Image processing device and image processing method
Patent Information
- Application Number
- JP2021181437
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-11-05
- Publication Date
- 2024-11-01
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing image restoration techniques effectively correct blur caused by aberration but also inadvertently remove intentional blurring intended for photographic expression.
An image processing apparatus and method that applies image recovery processing to correct blur due to aberration while selectively adding a blur effect to high-brightness areas, using image restoration filters and machine learning models to maintain both sharpness and artistic blur.
Achieves high-quality images by correcting aberration-induced blur while preserving intentional blur in high-brightness areas, enhancing overall image quality and artistic expression.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing apparatus and an image processing method, and particularly to an image correction technique.
Background Art
[0002] There is known an image restoration technique for correcting blurring of an image caused by aberration of an imaging optical system using the optical transfer function (OTF) of the imaging optical system (Patent Document 1).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] By correcting the blurring due to aberration, the sharpness of the image can be enhanced. On the other hand, there has been a problem that the blurring that is desired to be left as a photographic expression is also corrected.
[0005] In view of such problems of the prior art, in one aspect, the present invention provides an image processing apparatus and an image processing method that achieve both correction of blurring caused by aberration of an optical system and blurring as a photographic expression.
Means for Solving the Problems
[0006] The above object is achieved by an image processing apparatus including: a restoration unit that applies an image restoration process for correcting blurring caused by characteristics of an imaging optical system that captured an input image to the input image; and an application unit that applies a predetermined effect to a high-luminance portion that satisfies a predetermined condition among the images to which the image restoration process has been applied.
Effects of the Invention
[0007] According to the present invention, it is possible to provide an image processing apparatus and an image processing method that can achieve both correction of blur caused by optical system aberrations and blur as a photographic expression. [Brief explanation of the drawing]
[0008] [Figure 1] Block diagram showing an example of the functional configuration of a digital camera according to the first embodiment. [Figure 2] Flowchart relating to the operation of the image processing unit 104 in the first embodiment [Figure 3] A diagram showing an example of a settings screen provided in the embodiment. [Figure 4] Figure illustrating the operation and effects of the embodiment. [Figure 5] A diagram showing an example of a template image available in the embodiment. [Figure 6] Block diagram showing an example of the functional configuration of the image processing system according to the second embodiment. [Figure 7] This figure shows an example of the external appearance of the image processing system according to the second embodiment. [Figure 8] Flowchart relating to the operation of the learning phase in the second embodiment [Figure 9] Flowchart relating to the operation of the estimated face in the second embodiment [Modes for carrying out the invention]
[0009] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.
[0010] The following description will focus on an embodiment of the present invention using a digital camera. However, imaging functions such as an image sensor and imaging optical system are not essential to the present invention. Therefore, the present invention can be implemented using any electronic device capable of handling image data. Such electronic devices include, in addition to imaging devices, computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and dashcams. These are examples, and the present invention can be implemented using other electronic devices as well.
[0011] ●(First Embodiment) Figure 1 is a block diagram showing an example of the functional configuration of a digital camera 100, which is an example of an image processing apparatus according to the present invention. The imaging optical system 101 is, for example, a detachable lens unit and includes multiple lenses, including a focus lens 101b, and an aperture 101a. The imaging optical system 101 also includes motors, actuators, etc., that drive movable members such as the aperture 101a and the focus lens 101b. The imaging optical system 101 generates an optical image of the subject on the image-forming plane of the image sensor 102.
[0012] The image sensor 102 may be a known CCD image sensor or CMOS image sensor. The image sensor 102 has a pixel array in which a plurality of pixels, each having a photoelectric conversion unit, are arranged in two dimensions. Each pixel accumulates a charge corresponding to the amount of incident light, thereby converting the optical image of the subject into a group of pixel signals (analog image signals).
[0013] The analog image signal is converted into a digital image signal by the A / D converter 103 and input to the image processing unit 104. If the image sensor 102 has an A / D converter function, the digital image signal is input from the image sensor 102 to the image processing unit 104.
[0014] The image processing unit 104 applies various processes to the digital image signal to generate signals and image data, and to obtain and / or generate various types of information. The image processing unit 104 may be a dedicated hardware circuit such as an ASIC designed to implement a specific function, or may be configured such that a programmable processor such as a DSP realizes a specific function by executing software.
[0015] Here, the image processing applied by the image processing unit 104 includes color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Color interpolation processing is processing for interpolating values of color components that cannot be obtained during shooting, and is also called demosaicing processing. Correction processing includes processing such as white balance adjustment, gradation correction, correction of image degradation (image restoration) caused by optical aberration of the imaging optical system 101, correction of the effect of peripheral light reduction of the imaging optical system 101, and color correction. Detection processing includes detection of feature regions (for example, face regions and human body regions) and their movements, and person recognition processing. Data processing includes processing such as synthesis, scaling, encoding and decoding, and header information generation (data file generation). Evaluation value calculation processing includes processing such as generation of signals and evaluation values used for autofocus detection (AF), and generation of evaluation values used for automatic exposure control (AE). Special effect processing includes processing such as addition of a blur effect, change of color tone, and relighting. Note that the above-described processing is an example of processing applicable to the image processing unit 104, and does not limit the processing applied by the image processing unit 104.
[0016] [[ID=I22]]The image processing unit 104 can generate image data according to the application. Among them, the image data for recording is stored in a data file in a format according to the setting and recorded on a recording medium 109 such as a memory card. Further, the image data for display is displayed on the display unit 105.
[0017] The display unit 105 is a display device having, for example, video memory, a display control circuit, and a display panel. The display unit 105 displays image data generated by the image processing unit 104 and images based on GUI (Graphical User Interface) data supplied from the system controller 110. The display panel of the display unit 105 may be a touch display panel.
[0018] The state detection unit 107 acquires the current shooting status of the digital camera 100 from the system controller 110. The shooting status may be, for example, the current focal length, aperture value, and focusing distance (subject distance) of the imaging optical system 101. The shooting status may also include other shooting conditions such as ISO sensitivity and other information such as specific information of the imaging optical system 101 (product name, model number, etc.). The state detection unit 107 may acquire information regarding the imaging optical system 101 from the optical system control unit 106. The state detection unit 107 supplies the acquired shooting status to the image processing unit 104.
[0019] The storage unit 108 is a storage device accessible by the image processing unit 104 and the system controller 110, and may be, for example, a memory or a magnetic recording device. The storage unit 108 has at least a non-volatile area. The non-volatile area of the storage unit 108 stores parameters of the image recovery filter of the imaging optical system 101, and / or data or information for generating the image recovery filter, associated with a plurality of different imaging conditions. The information for generating the image recovery filter may be, for example, information regarding the optical transfer function (OTF) of the imaging optical system 101.
[0020] The system controller 110 includes one or more processors (hereinafter referred to as CPUs) capable of executing programs, non-volatile memory, and working memory. The system controller 110 controls the operation of the components of the digital camera 100 and the imaging optical system 101 by loading programs stored in the non-volatile memory into the working memory and executing them with the CPU, thereby providing the functions of the digital camera 100. If the imaging optical system 101 is interchangeable, the system controller 110 communicates with the controller of the imaging optical system 101 and controls the operation of the imaging optical system 101.
[0021] The operation unit 111 is a general term for the input devices (buttons, switches, dials, etc.) provided on the digital camera 100 for the user to input instructions to the digital camera 100. The input devices that make up the operation unit 111 have names according to the function they are assigned to. For example, the operation unit 111 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a select key. The release switch is a switch for recording still images, and the system controller 110 recognizes the half-pressed state of the release switch as a shooting preparation instruction and the fully pressed state as a shooting start instruction. The function assigned to the same input device may be variable. In addition, the input devices may be software buttons or keys using a touch display.
[0022] (Image recovery process) Next, the image recovery process performed by the image processing unit 104 will be described. The image recovery process is a process that corrects image blur caused mainly by aberrations (spherical aberration, coma aberration, field curvature, astigmatism, etc.) of the imaging optical system 101. More specifically, the image recovery process corrects the blur component of the image, which is represented by the point spread function (PSF). The PSF is a function that represents the intensity distribution of the point light source image generated on the imaging surface by the imaging optical system 101. Due to the effects of aberrations and diffraction of the imaging optical system 101, the image of the point light source becomes a spread image.
[0023] Furthermore, image blurring due to aberrations in the imaging optical system 101 differs from blurring caused by the imaging optical system 101 not being in focus on the subject; it occurs even when the imaging optical system 101 is in focus on the subject.
[0024] The Optical Transfer Function (OTF) of the imaging optical system 101 is obtained by performing a Fourier transform on the PSF of the imaging optical system 101. The OTF represents the frequency components of the aberrations of the imaging optical system 101 using complex numbers. Similarly, the PSF of the imaging optical system 101 is obtained by performing an inverse Fourier transform on the OTF of the imaging optical system 101.
[0025] The amplitude component of the OTF is called the Modulation Transfer Function (MTF), and the phase component is called the Phase Transfer Function (PTF). MTF and PTF are the frequency characteristics of the amplitude and phase components of image degradation due to aberrations, respectively. If we denote the real and imaginary parts of the OTF as Re(OTF) and Im(OTF), respectively, and express the PTF in terms of phase angle, the following relationship holds. PTF=tan-1(Im(OTF) / Re(OTF)) (Formula 1) Thus, because the OTF of the imaging optical system 101 degrades the amplitude and phase components of the image, the image obtained by shooting has asymmetrical blurring of each point of the subject.
[0026] Here, if we let g(x,y) be the image generated by the imaging optical system 101, f(x,y) be the original image, and h(x,y) be the PSF of the imaging optical system 101, then the following relationship holds. g(x,y)=h(x,y)*f(x,y) (Formula 2) Here, (x,y) are the coordinates on the image when the image coordinate system is a 2-dimensional Cartesian coordinate system, and * represents the convolution operation.
[0027] When Equation 2 is Fourier transformed and converted to a frequency-plane representation, it takes the form of a frequency-by-frequency product, as shown in the following equation. G(u,v)=H(u,v)·F(u,v)···(Formula 3) H(u,v) is the Fourier transform of h(x,y), and therefore it is an OTF (Out-of-Tone Frequency). Also, the coordinate (u,v) is a coordinate on the 2D frequency plane, and therefore it is a frequency.
[0028] To eliminate the influence of OTF(H(u,v)) of the imaging optical system 101, both sides of equation 3 can be divided by H(u,v) as follows. G(u,v) / H(u,v)=F(u,v) (Equation 4)
[0029] Since F(u,v) is in the form of a frequency, an inverse Fourier transform is performed to return it to the image coordinate system, thereby obtaining an image f(x,y) from which the effect of the OTF of the imaging optical system 101 has been removed.
[0030] Here, if we take the inverse Fourier transform of 1 / H(u,v) as R(x,y), we can obtain an image f(x,y) with the effect of the OTF of the imaging optical system 101 removed by inverse convolution in the image coordinate system, without performing the Fourier transform and inverse Fourier transform. g(x,y)*R(x,y)=f(x,y) (Formula 5)
[0031] This R(x,y) is called the image recovery filter. Since actual images contain noise components, using an image recovery filter created from the reciprocal of OTF, 1 / H(u,v), amplifies the noise components, and generally does not yield a good image. Therefore, a method is known in which a Wiener filter is used as the image recovery filter, and the recovery of high-frequency components of the image is suppressed according to the intensity ratio of the image signal to the noise signal.
[0032] Image recovery processing ensures that the amount of blurring for each color component of the image becomes uniform, thereby correcting chromatic aberration of the imaging optical system 101 and color shifts caused by the spectral characteristics of each pixel.
[0033] Furthermore, since the OTF of the imaging optical system 101 fluctuates depending on the imaging conditions such as the focal length, aperture value, and distance to the subject, the image recovery filter also needs to be changed for each imaging condition. In addition, if an optical low-pass filter or infrared light cut filter is provided in the imaging optical system 101 or image sensor 102, an image recovery filter that takes into account the effect of these on the OTF can be used.
[0034] Next, the image recovery process performed by the image processing unit 104 will be further explained using the flowchart shown in Figure 2. Here, it is assumed that the image sensor 102 is equipped with a primary color Bayer array color filter. Therefore, the digital image signal input to the image processing unit 104 has one color component per pixel: R, G, or B. Before applying the image recovery process, the image processing unit 104 applies color interpolation processing to ensure that each pixel has all three color components: R, G, and B.
[0035] In S201, the image processing unit 104 determines whether or not to apply image recovery processing. For example, the image processing unit 104 can make this determination based on whether or not the user setting stored in the non-volatile area of the storage unit 108 is a setting to apply image recovery processing.
[0036] Figure 3 shows an example of a settings screen 300, which is a user interface that allows users to configure settings related to image recovery processing. This settings screen can be accessed through operation of the operation unit 111 from a menu screen displayed on the display unit 105 by the system controller 110, which detects operation of the menu button on the operation unit 111.
[0037] Figure 3(a) shows an example of the settings screen 300 when image recovery is disabled. The settings screen 300 is a GUI for setting image recovery and includes a checkbox 310 for setting whether or not to perform image recovery, and a slider 320 for setting the intensity of image recovery. The slider 320 has a knob 321, and the position of the knob 321 corresponds to the intensity. For example, when the knob 321 is at the right end of the slider, it corresponds to the maximum intensity, and when the knob 321 is at the left end of the slider, it corresponds to the minimum intensity. In the example in Figure 3(a), the checkbox 310 is not checked, so the intensity setting by the slider 320 is disabled.
[0038] When the checkbox 310 is checked in the state shown in Figure 3(a), the settings screen 300 changes to the state shown in Figure 3(b). The check of checkbox 310 can be toggled by touching the checkbox or by pressing a predetermined button on the operation unit 111.
[0039] When checkbox 310 is checked, the settings screen 300 displays a GUI for setting whether or not to apply a blur effect to the high-brightness areas of the image. Specifically, a checkbox 330 and a slider 340 for setting the intensity of the blur effect applied to the high-brightness areas are added. The slider 340 has a knob 341, and the position of the knob 341 corresponds to the intensity. However, when checkbox 330 is not checked, operation of the slider 340 and intensity setting using the slider 340 are disabled.
[0040] Note that the GUI for setting the blur effect on high-brightness areas may be displayed regardless of the checked state of checkbox 310. For example, when displaying checkbox 330 and slider 340 in the state shown in Figure 3(a), they can be grayed out and / or their operation can be disabled.
[0041] Figure 3(c) shows the state in Figure 3(b) with checkbox 330 checked. This enables operation of slider 340 and intensity setting using slider 340.
[0042] The system controller 110, for example, based on the state of the settings screen 300 at the time it detects an operation to close the settings screen 300, saves the settings related to image recovery to the non-volatile area of the storage unit 108. That is, the settings related to image recovery are, • Whether or not to perform image recovery (status of checkbox 310) • Image recovery intensity (position of slider 320 knob 321) • Whether or not to apply a bokeh effect to high-brightness areas (state of checkbox 330) • The intensity of the bokeh effect applied to high-brightness areas (position of knob 341 on slider 340) This includes [the specified parameters]. Note that the intensity may be calculated by converting the knob position to intensity, rather than using the knob position itself.
[0043] The image processing unit 104 executes S202 if it determines that image recovery is possible, and terminates the process shown in the flowchart in Figure 2 if it does not determine that recovery is possible.
[0044] In S202, the image processing unit 104 acquires the imaging state through the state detection unit 107. Then, by referring to the non-volatile area of the storage unit 108 based on the imaging state, it acquires information regarding the aberrations of the imaging optical system 101 corresponding to the imaging state. Here, the information regarding aberrations may be, for example, information regarding the OTF of the imaging optical system 101 (OTF information). The OTF information may be OTF or PSF.
[0045] Furthermore, if the memory unit 108 stores OTF information corresponding to discrete combinations of parameters constituting the imaging conditions, there may be no OTF information stored that matches the acquired imaging state. In this case, OTF information for the acquired imaging state can be obtained by correcting or interpolating one or more OTF pieces of information corresponding to a similar combination.
[0046] In S203, the image processing unit 104 acquires an image recovery filter (parameters of the image recovery filter) according to the imaging state and the intensity setting of the image recovery. For example, the image processing unit 104 selects an image recovery filter from the image recovery filters stored in the storage unit 108 according to the imaging state and the intensity setting of the image recovery. Alternatively, the image recovery filter may be corrected according to the intensity setting after selecting an image recovery filter according to the imaging state.
[0047] Furthermore, image recovery filters corresponding to discrete combinations of parameters constituting the imaging conditions are stored, and there may be cases where an image recovery filter that matches the acquired imaging state is not stored. In this case, an image recovery filter for the acquired imaging state may be generated by correcting or interpolating one or more image recovery filters corresponding to combinations close to the acquired imaging state. Alternatively, instead of selecting an image recovery filter, an image recovery filter corresponding to the imaging state may be generated from information about the OTF, etc. There are no particular restrictions on the specific configuration of the image recovery filter, and any known filter configuration capable of realizing the R(x,y) effect described above can be used.
[0048] In S204, the image processing unit 104 applies the image recovery filter R(x,y) acquired in S203 to each pixel of each color component of the target image g(x,y) as shown in Equation 5, thereby obtaining a recovered image f(x,y). By applying the image recovery filter, an image can be obtained in which asymmetric blur components caused by aberrations of the imaging optical system 101 are removed or reduced.
[0049] Figure 4 is a schematic diagram illustrating the effect of the image restoration process. Figure 4(a) shows the entire input image with a starry sky at the top. Figure 4(b) shows a magnified view of the star images within region R in Figure 4(a). The star images, which are point light sources, are formed as images with asymmetric blur components due to the effects of aberrations and diffraction in the imaging optical system 101. By applying the image restoration filter, the asymmetric blur components are removed or reduced, and the image is corrected to its original shape, as shown in Figure 4(c).
[0050] In S205, the image processing unit 104 refers to the image recovery settings stored in the memory unit 108 and determines whether or not to perform adjustments (applying a blur effect) to the high-brightness areas. If the checkbox 330 in the settings screen 300 explained using Figure 3 is checked, it is set to apply a blur effect to the high-brightness areas. If the image processing unit 104 determines that it should perform adjustments to the high-brightness areas, it executes S206; otherwise, it terminates the process shown in the flowchart of Figure 2. If the process is terminated at this point, an image with only the recovery process applied is obtained.
[0051] In S206, the image processing unit 104 detects high-brightness regions from the input image. The image processing unit 104 can detect pixels with a value greater than or equal to a threshold predetermined based on the characteristics of the image sensor 102 as high-brightness areas. The image processing unit 104 may also create a high-brightness map showing the pixel coordinates of the detected high-brightness areas.
[0052] In S207, the image processing unit 104 generates a filter B(x,y) that applies a blurring effect to the high-brightness areas. In this embodiment, a configuration using a Gaussian filter is shown as an example. B(x,y) = 1 / (2πσ) 2 )exp{-(x 2 -y 2 ) / 2σ 2} ...(Formula 6)
[0053] In Equation 6, x and y indicate the positions of the filter taps, and σ is a parameter that controls the magnitude of the blur. By determining σ based on the OTF information of the imaging optical system 101 acquired in S202, a blur effect based on the characteristics of the imaging optical system 101 can be applied.
[0054] For example, the smaller the MTF value of the imaging optical system 101, the greater the aberration of the imaging optical system 101. Therefore, the image processing unit 104 sets σ to a large value so that the blurring effect in the region with a small MTF value becomes stronger. As described above, the MTF can be obtained as the amplitude component of the OTF.
[0055] Furthermore, if the image processing unit 104 acquires the PSF as OTF information in S202, it may increase the σ value as the PSF spreads larger. Alternatively, the image processing unit 104 may acquire the correction intensity from the characteristics of the image recovery filter acquired in S203, and set the σ value to be larger as the intensity increases.
[0056] In this way, the image processing unit 104 can generate a filter (bokeh effect filter) that produces a greater blur effect the greater the aberration of the imaging optical system 101.
[0057] Since OTF information generally exhibits different characteristics depending on the distance from the optical axis, the image processing unit 104 may generate multiple blur effect filters according to the distance from the optical axis. In this case, a blur effect filter corresponding to the distance from the optical axis of the target pixel is applied. Alternatively, the image processing unit 104 may generate a blur effect filter common to the entire image based on typical characteristics (e.g., average characteristics).
[0058] If the user has set the intensity of the blur effect, in addition to the magnitude of the aberration of the imaging optical system 101 described above, the image processing unit 104 reflects the intensity according to the user setting in σ of equation 6.
[0059] In addition, If the aberration of the imaging optical system 101 is determined to be sufficiently small, If the image recovery setting strength is determined to be weak, • If the setting intensity of the bokeh effect is determined to be weak. If one or more of the following conditions are met, the image processing unit 104 may skip the processing from S206 onward and not perform the processing to apply a filter that imparts a blurring effect. These conditions can be determined using predetermined thresholds. For example, if the MTF at a predetermined image height is greater than or equal to the threshold, it may be determined that the magnitude of the aberration is sufficiently small. Alternatively, the image processing unit 104 may apply a filter that outputs the input image as is (with coefficients of 1 only at the center of the filter and 0 everywhere else) as a blurring effect filter.
[0060] In S208, the image processing unit 104 applies the blur effect filter generated in S207 to the pixels in the high-brightness areas, thereby adding a blur effect to the high-brightness areas of the input image. The pixels in the high-brightness areas can be identified based on the high-brightness map of the input image.
[0061] Specifically, the image processing unit 104 uses a high-brightness map to determine whether high-brightness pixels are included in the image region (the x,y range in Equation 6) used for the convolution operation to apply the blur effect filter. If high-brightness pixels are included, the blur effect filter is applied to the target pixels.
[0062] Alternatively, the image processing unit 104 may generate an image by applying a blur effect filter to all pixels of the image, and then synthesize the blurred image with the original image based on the high-luminance map. In this case, the blurred image is synthesized in the areas surrounding the high-luminance pixels of the original image.
[0063] Furthermore, for saturated pixels among the high-luminance pixels, it is desirable to increase the signal value before applying the blur effect filter. This is because if the brightness of saturated pixels decreases due to smoothing by the blur effect filter, the resulting image will differ from its original brightness.
[0064] Figure 4(d) shows an example of applying a bokeh effect filter to the star image (high-brightness area) shown in Figure 4(c). The asymmetrical depiction due to aberrations, as seen in Figure 4(b) before image restoration, is reduced by image restoration, and the addition of uniform blur makes it possible to generate a high-quality image while maintaining the impression of the size of the light source. Furthermore, since the addition of blur is limited to the high-brightness area and its surroundings in the image, in a scene like the one shown in Figure 4(a), the landscape portion at the bottom of the image can be output in a high-resolution state due to the image restoration process.
[0065] If the aberrations of the imaging optical system are sufficiently small, the image changes due to image restoration processing are minimal. Furthermore, unless the user has set a large amount of blur to be added, the filter used outputs the input image as is, as described above, so no blur is added. In this way, even when the aberrations of the imaging optical system are small, it is possible to output an image without significantly changing the impression of the photograph taken at the time of shooting.
[0066] While an example of applying a Gaussian filter to create a bokeh effect has been described, the bokeh effect can also be created using other filters or methods. If no filter is used, for example, pixel data indicating the bokeh shape can be stored as a template, and the template can be selected or processed based on the aberration information of the imaging optical system 101 and added to the image to create the bokeh effect.
[0067] For example, the image data of the light rays shown in Figure 5 is stored in the memory unit 108 as a template. Then, this image data is scaled according to the magnitude of the aberration of the imaging optical system 101 and added to the image of the high-brightness area. Even with this type of processing, it is possible to add a blur effect with a shape corresponding to the aberration of the imaging optical system 101.
[0068] Image data that has undergone image restoration (and the application of a blur effect to high-brightness areas according to the settings) is used in the image processing unit 104 to generate recording image data and display image data. The original image and the image after image restoration processing may also be displayed on the display unit 105 for comparison.
[0069] According to this embodiment, when applying image recovery processing to correct blur caused by aberrations, it is possible to apply a blur effect to the high-brightness areas. Therefore, it is possible to effectively achieve both high image quality by correcting blur caused by aberrations in the imaging optical system and expression through blur in high-brightness areas.
[0070] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. In this embodiment, image restoration and the application of a blur effect to high-brightness areas in the first embodiment are performed using a machine learning model.
[0071] Machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), and RNNs (Recurrent Neural Networks).
[0072] In the following explanation, the stage of learning the weights of the machine learning model will be referred to as the learning phase. The stage of performing image restoration and applying a blur effect to high-brightness areas using the machine learning model with the learned weights will be referred to as the estimation phase.
[0073] Figure 6 is a block diagram showing an example configuration of the image processing system 300 in this embodiment. Figure 7 is a diagram showing an example of the external appearance of the image processing system 300. The image processing system 300 includes a learning device (image processing device) 301, a lens unit 302, an imaging device 303, an image estimation device 304, a display device 305, a recording medium 306, an output device 307, and a network 308.
[0074] The learning device 301 may be, for example, a server device on a network, or a general-purpose computer device. By running a learning application, the computer device functions as the learning device 301. The learning device 301 has a storage unit 301a, an acquisition unit 301b, an arithmetic unit 301c, and an update unit 301d, and learns the weights of a machine learning model used for image recovery and applying a blur effect to high-brightness areas.
[0075] The memory unit 301a is a storage device such as non-volatile memory or a hard disk. The other blocks are described as individual blocks for convenience, but in reality, their functions are realized by the processor executing programs. Details regarding weight learning and processing using weights will be described later.
[0076] The lens unit 302 is a detachable interchangeable lens for the imaging device 303. The focal length, aperture, and focusing distance range of the lens unit 302 vary depending on the model. In addition, the PSF (or OTF) of the lens unit 302 differs depending on the model, so the shape of the bokeh caused by the aberrations and diffraction of the lens unit 302 also differs.
[0077] The imaging device 303 has an image sensor 303a. Depending on the type, one or more of the following may differ in the image sensor 303a: the presence or absence of an optical low-pass filter, the type of optical low-pass filter (such as the separation method and cutoff frequency), the pixel pitch (including the pixel aperture), and the configuration of the color filter array.
[0078] The image estimation device 304 is a general-purpose computer device, similar to the server device, and is, for example, a personal computer used by the user of the image processing system 300. By running a specific application on the computer device, the computer device functions as the image estimation device described later.
[0079] The image estimation device 304 includes a storage unit 304a, an acquisition unit 304b, an image recovery unit 304c, a blur application unit 304d, and a synthesis unit 304e. Of these, the storage unit 304a is a storage device such as non-volatile memory or a hard disk. The other blocks are described as individual blocks for convenience, but in reality, their functions are realized by the processor executing a program.
[0080] The image estimation device 304 corrects blurring caused by aberrations and diffraction of the lens unit 302 in the image captured by the imaging device 303, and generates an image with a blur effect applied to the high-brightness areas. Different combinations of the lens unit 302 and imaging device 303 can be connected to the image estimation device 304. The image estimation device 304 uses a machine learning model with weights trained by the learning device 301 to generate an image (hereinafter referred to as a recovered image) from the captured image, in which image recovery and the application of a blur effect to the high-brightness areas have been performed.
[0081] The learning device 301 and the image estimation device 304 are connected by a network 308. The network 308 may include multiple networks. It may also be either a wired network or a wireless network.
[0082] The image estimation device 304 acquires the learned weight information from the learning device 303 at any time before applying the machine learning model to the original image. The image estimation device 304 outputs the recovered image to at least one of the display device 305, recording medium 306, and output device 307. The display device 305 may be, for example, a liquid crystal display or a projector.
[0083] The user can perform editing tasks while checking the image being processed, which is displayed on the display device 305. The recording medium 306 is, for example, a semiconductor memory, a hard disk, or a server on a network. The output device 307 is, for example, a printer.
[0084] Next, the weight learning (learning phase) performed by the learning device 301 will be explained using the flowchart shown in Figure 8. In this embodiment, a machine learning model using a CNN will be explained, but weights can be learned in the same way for other machine learning models.
[0085] In S401, the acquisition unit 301b acquires training data from the storage unit 301a, which includes one or more sets of correct images 1 and training images. The storage unit 301a has pre-stored training images for each combination of lens unit 302 and imaging device 303 models.
[0086] In this embodiment, weight learning is performed for each model of lens unit 302. Therefore, the acquisition unit 301b first determines the model of lens unit 302 for which weight learning will be performed. Then, it acquires training images corresponding to the determined model of lens unit 302.
[0087] The set of training images corresponding to one model of lens unit 302 is: • Lens unit condition (focal length, aperture value, focusing distance (subject distance)) • The condition of the image sensor in the corresponding imaging device (image height and azimuth, presence and type of optical low-pass filter, pixel pitch, color filter pattern) This is a collection of images with different combinations of elements (i.e., images where blurring originates from the optical system, such as aberrations in the lens unit).
[0088] These training images can be acquired by actually taking photographs, but they can also be generated from the original images using characteristic information such as the OTF of the lens unit 302. On the other hand, the ground truth image 1 is an image in which no blurring originating from the optical system has occurred. The training image corresponds to an image with blurring, such as Figure 4(b), and the corresponding ground truth image 1 corresponds to an image with the blur corrected, such as Figure 4(c).
[0089] In S402, the arithmetic unit 301c inputs the training image to the CNN and generates the output image. In S403, the update unit 301d updates the CNN weights using a known method based on the error between the output image and the ground truth image 1.
[0090] In S404, the update unit 301d determines, for example, whether the first learning is complete based on the magnitude of the error. If the update unit 301d determines that the first learning is complete, it stores the CNN weights at that point in the memory unit 301a as weight information for machine learning model 1 and proceeds to S405. If the update unit 301d does not determine that the first learning is complete, it returns to S401.
[0091] In S405, the acquisition unit 301b acquires one or more sets of ground truth images 2 and training images from the storage unit 301a. The training images may be the same as those acquired in S401. On the other hand, ground truth image 2 is an image in which a blurring effect has been applied to the high-brightness areas of ground truth image 1, and corresponds to, for example, the image shown in Figure 4(d).
[0092] The applied bokeh effect takes into account the characteristics of the lens unit 302, such as OTF, as described in the first embodiment. If the aberration of the lens unit 302 is small, or if the applied bokeh effect is weak, the same image as the training image may be used as the ground truth image 2.
[0093] In S406, the arithmetic unit 301c inputs the training image to the CNN and generates the output image. In S407, the update unit 301d updates the CNN weights using a known method based on the error between the output image and the ground truth image 2.
[0094] In S408, the update unit 301d determines, for example, whether the second learning is complete based on the magnitude of the error. If the update unit 301d determines that the second learning is complete, it stores the CNN weights at that point in the memory unit 301a as weight information for the machine learning model 2 and terminates the process shown in the flowchart. If the update unit 301d does not determine that the second learning is complete, it returns to S405.
[0095] Next, the recovery image generation process (estimation phase) performed by the image estimation device 304 will be explained using the flowchart shown in Figure 9. In S501, the acquisition unit 304b determines whether or not to apply image recovery processing. The acquisition unit 304b can make this determination based, for example, on whether or not the user settings stored in the storage unit 304a are settings to apply image recovery processing. In this embodiment as well, as in the first embodiment, it is assumed that the settings related to image recovery processing are stored in the storage unit 304b. If the acquisition unit 304b determines that image recovery should be performed, it executes S502; otherwise, it terminates the processing shown in the flowchart of Figure 9.
[0096] In S502, the acquisition unit 304b acquires the weights of the machine learning model 1 corresponding to the model of the lens unit 302 that captured the input image from the learning device 301 and stores them in the storage unit 304a. If there are weights of the machine learning model 1 corresponding to a combination of the lens unit 302 model and the imaging device 303 model, those may be acquired instead. The model names of the lens unit 302 and the imaging device 303 can be identified from the image metadata.
[0097] In S503, the image recovery unit 304c sets the weights of the machine learning model 1 stored in the memory unit 304a to the CNN. Then, it inputs the input image into the CNN and generates the recovered image 1.
[0098] In S504, the acquisition unit 304b refers to the image recovery settings stored in the storage unit 304a and determines whether or not to perform adjustments (applying a blur effect) to the high-brightness areas. If the acquisition unit 304b determines that adjustments to the high-brightness areas should be performed, it proceeds to S505; otherwise, it terminates the process shown in the flowchart of Figure 9. If the process is terminated at this point, an image with only the recovery process applied is obtained.
[0099] In S505, the acquisition unit 304b detects high-brightness regions from the input image. The acquisition unit 304b can detect pixels having a value greater than or equal to a threshold predetermined based on the characteristics of the image sensor 102 as high-brightness regions. The acquisition unit 304b may also create a high-brightness map showing the pixel coordinates of the detected high-brightness regions.
[0100] In S506, the acquisition unit 304b acquires the weights of machine learning model 2 corresponding to the model of the lens unit 302 that captured the input image from the learning device 301 and stores them in the storage unit 304a. If there are weights of machine learning model 2 corresponding to the combination of the lens unit 302 model and the imaging device 303 model, those may be acquired instead. The weights of machine learning model 1 and machine learning model 2 may be acquired at the same time.
[0101] In S507, the image recovery unit 304c sets the weights of the machine learning model 2 stored in the memory unit 304a to the CNN. Then, it inputs the input image into the CNN to generate the recovered image 2. The input image is the same as the one used in S503.
[0102] In S508, the synthesis unit 304e synthesizes recovery image 1 and recovery image 2. The synthesis unit 304e synthesizes the recovery image 1, which has undergone image recovery processing, by replacing the high-luminance pixels and their surrounding areas with recovery image 2, which has a blur effect applied. Other synthesis methods may also be used. The high-luminance pixels and their surrounding areas can be identified using a high-luminance map.
[0103] In this embodiment, instead of applying an image restoration filter or a blurring filter, a machine learning model is used to generate an image with image restoration processing and an image with a blurring effect applied to the high-brightness areas from the input image. This embodiment can achieve the same effects as the first embodiment.
[0104] On the other hand, by using machine learning models, it is possible to estimate the spatial frequency spectrum of the subject with high accuracy, even when the MTF drops to near zero, thereby improving the accuracy of blur correction through image restoration.
[0105] Since the blurring effect is limited to the high-brightness areas of the image and their surroundings, processing an input image like the one shown in Figure 4(a) allows the resolution of the landscape portion at the bottom of the image to be improved through image restoration processing. On the other hand, for the starry sky portion at the top of the image, a symmetrical blurring effect (the so-called circular blurring effect) is applied to the star images, allowing for expressions using blurring.
[0106] As explained in the first embodiment, when the aberration of the lens unit 302 is sufficiently small, the image change due to the image restoration process is small, and the bokeh effect is not substantially applied. Therefore, even if settings are made to perform image restoration processing or apply a bokeh effect, if the aberration of the lens unit 302 is small, it will not significantly change the impression of the original image.
[0107] (Other embodiments) The above-described embodiment describes a configuration that applies a symmetrical blur effect to the high-brightness areas of the image after image restoration. However, other effects may be applied to the high-brightness areas of the image after image restoration. For example, a light ray pattern produced by aperture blades or the effect of bright lines obtained with a cross filter can be applied. In this case as well, a method of applying filters or a method of combining them can be used.
[0108] Furthermore, although the second embodiment described a configuration in which image recovery processing and the application of effects to high-brightness areas are performed by an external device of the imaging device, the same processing as that performed by the image estimation device 304 in the first embodiment may also be performed by the image processing unit 104.
[0109] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.
[0110] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]
[0111] 101...Imaging optical system, 102...Image sensor, 104...Image processing unit, 105...Display unit, 107...Status detection unit, 108...Storage unit, 110...System controller
Claims
1. a restoration means for applying an image restoration process to the input image to correct blur caused by the characteristics of an imaging optical system that captured the input image; an applying means for applying a predetermined effect to a high-brightness portion of the image to which the image restoration processing has been applied that satisfies a predetermined condition; 1. An image processing device comprising:
2. 2. The image processing apparatus according to claim 1, wherein the applying means applies a symmetrical blur effect to the high-brightness portion.
3. 2. The image processing apparatus according to claim 1, wherein the applying means applies a light beam pattern or a bright line effect to the high-brightness portion.
4. 4. The image processing device according to claim 1, wherein the applying unit applies the effect in consideration of characteristics of the imaging optical system.
5. 5. The image processing apparatus according to claim 4, wherein the effect applying means increases the strength of the effect applied as the aberration of the imaging optical system increases.
6. 5. The image processing apparatus according to claim 4, wherein said applying means does not apply said effect when the magnitude of the aberration of said imaging optical system satisfies a predetermined condition.
7. 7. The image processing device according to claim 1, wherein the restoration means applies the image restoration process to the input image by applying a filter based on the characteristics of the imaging optical system to the input image.
8. The image processing device according to any one of claims 1 to 6, wherein the restoration means applies the image restoration process to the input image by inputting the input image into a trained machine learning model.
9. 9. The image processing apparatus according to claim 1, further comprising a user interface that enables a user to set whether or not to apply the image restoration process and whether or not to apply the effect.
10. An imaging element; an image processing device according to claim 1 , wherein an image captured by the image sensor is used as an input image; An imaging device comprising:
11. An image processing method executed by an image processing device, applying an image restoration process to the input image to correct blur caused by the characteristics of the imaging optical system that captured the input image; imparting a predetermined effect to a high-brightness portion of the image to which the image restoration processing has been applied that satisfies a predetermined condition; An image processing method comprising:
12. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 9.
