Image processing method, image processing system, and program

Abstract

To suppress negative effect in a sharpness processing.SOLUTION: An image processing method includes: a step S101 of acquiring information on an imaging image obtained by imaging it with an optical system 101 and an object distance; a step S104 of acquiring an objective defocus amount of a partial region in the imaging image on the basis of the information on the objective distance; a step S106 of obtaining a correction strength of a sharpness processing adopted to the partial region on the basis of an objective defocus amount; and a step S107 of adopting the sharpness processing on the basis of the correction strength. The correction strength becomes small in the case where the objective defocus amount is 0 than the case where a difference between an imaging position and an imaging surface in each image height of the optical system 101 is equal to the objective defocus amount.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device that performs image sharpening processing. [Background technology]

[0002] In imaging using an optical system, light generated from a single point on a subject reaches the image plane with a small spread due to the effects of diffraction and aberration that occur in the optical system. Therefore, the higher the image height in the captured image, the more likely it is that blur will occur. Such blur in the captured image is corrected by a sharpening process based on the optical characteristics of the optical system, such as the point spread function (PSF) and optical transfer function (OTF).

[0003] Patent Document 1 discloses a method of generating an OTF for each image height based on the OTF when the optical system is focused on the optical axis and coefficient data, and sharpening an image based on the OTF for each image height. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-038563 A Summary of the Invention [Problem to be solved by the invention]

[0005] When the subject captured by an imaging device is a three-dimensional object with depth, the optical characteristics change depending on the subject's depth (subject distance). In addition, in the case of an optical system that has aberrations (field curvature and astigmatism) that cause the image point to shift vertically (in the optical axis direction) relative to the image plane, the blur of a subject that exists at a depth different from the focal plane may be reduced.

[0006] The method of Patent Document 1 does not take into account the depth of the subject for each image height of the subject. Therefore, if the blur at a certain image height due to the depth of the subject is smaller than the blur assumed based on the coefficient data, over-correction is performed at that image height. Over-correction in the sharpening process may cause problems such as blackening and ringing in the processed image.

[0007] Therefore, an object of the present invention is to suppress the adverse effects of sharpening processing. [Means for solving the problem]

[0008] An image processing method according to one aspect of the present invention includes a step of acquiring a captured image obtained by imaging using an optical system and information related to a subject distance, and a step of acquiring a subject defocus amount for a partial region in the captured image based on the information related to the subject distance. The image processing method further includes a step of acquiring a correction strength of a sharpening process to be applied to the partial region based on the subject defocus amount, and a step of applying the sharpening process based on the correction strength. The correction strength is smaller when the subject defocus amount is 0 than when the difference between the imaging position and the imaging plane at each image height of the optical system is equal to the subject defocus amount. Effect of the Invention

[0009] According to the present invention, it is possible to suppress adverse effects in sharpening processing. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram showing an imaging relationship in an optical system when focused on an axial object point. [Diagram 2] FIG. 2 is a diagram showing an imaging relationship by an optical system when focusing on an off-axis object point. [Diagram 3] 1 is a block diagram of an imaging device according to a first embodiment. [Figure 4] 4 is a flowchart showing image processing in the first embodiment. [Diagram 5]6 is a diagram showing a change in MTF with respect to the depth of a subject in Example 1. FIG. [Figure 6] FIG. 11 is a diagram illustrating a correction strength. [Figure 7] 13 is a diagram showing a change in MTF with respect to the depth of a subject in Example 2. FIG. [Figure 8] FIG. 1 is a diagram illustrating a two-viewpoint shooting method. [Figure 9] 2 is a diagram illustrating the relationship between a light receiving portion of an image sensor and a pupil of an image capturing optical system. FIG. [Figure 10] 2 is a diagram illustrating the relationship between a light receiving portion of an image sensor and a subject. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same reference numerals are used to refer to the same components, and duplicated explanations will be omitted.

[0012] The image processing method in this embodiment includes a step of acquiring a captured image and a subject distance obtained by imaging using an optical system, and a step of acquiring a subject defocus amount of a partial region in the captured image based on information on the subject distance.Furthermore, the image processing method includes a step of acquiring a correction strength of a sharpening process to be applied to the partial region based on the subject defocus amount, and a step of applying the sharpening process based on the correction strength.Furthermore, the correction strength is characterized in that the correction strength is smaller when the subject defocus amount is 0 than when the difference between the imaging position at each image height of the optical system and the imaging surface is equal to the subject defocus amount.By adopting such a configuration, it is possible to suppress adverse effects in the sharpening process.

[0013] In this embodiment, a sharpening process is performed on an input image generated by imaging using an optical system, using a sharpening filter based on the optical characteristics of the optical system. The optical characteristics in this embodiment indicate the aberration of the optical system and the blur of the input image due to the aberration, and vary depending on the imaging conditions, spatial frequency, image height of the optical system, and subject distance relative to the focal position. The optical characteristics include, for example, a point spread function (PSF), an optical transfer function (OTF), a modulation transfer function (MTF) which is the amplitude component of the OTF, and a phase transfer function (PTF) which is the phase component of the OTF.

[0014] Next, the sharpening process of this embodiment will be described. In this embodiment, the sharpening process is an aberration correction process based on optical characteristics, and is also called image restoration process or point image restoration process. In addition, the optical characteristics may be characteristics including optical elements such as a low-pass filter and an infrared cut filter as well as an imaging lens, and may take into account the influence of the structure of the pixel array of the image sensor. The sharpening process described here will be used as appropriate in each of the embodiments described later.

[0015] The following is an overview of the sharpening process. When the captured image (degraded image) is g(x,y), the original image is f(x,y), and the point spread function PSF, which is a Fourier pair of the optical transfer function OTF, is h(x,y), the following formula (1) is established. g(x,y)=h(x,y)*f(x,y) (1)

[0016] Here, * denotes convolution (convolution integral, product sum), and (x, y) are coordinates on the captured image.

[0017] Furthermore, if equation (1) is Fourier transformed and converted into a frequency domain display format, equation (2), expressed as a product for each frequency, is obtained. G(u,v)=H(u,v)·F(u,v) (2)

[0018] Here, H is the optical transfer function OTF obtained by Fourier transforming the point spread function PSF(h), and G and F are functions obtained by Fourier transforming the degraded image g and the original image f, respectively. (u,v) are the coordinates in the two-dimensional frequency plane, i.e., the frequency.

[0019] To obtain the original image f from the captured degraded image g, both sides of the following equation (3) can be divided by the optical transfer function H. G(u,v) / H(u,v)=F(u,v) (3)

[0020] Then, F(u,v), i.e., G(u,v) / H(u,v), is transformed back to the real plane by inverse Fourier transform, and the original image f(x,y) is obtained as a restored image.

[0021] If R is the inverse Fourier transform of 1 / H, the original image f(x, y) can be obtained by performing convolution processing on the image on the real plane as shown in the following equation (4). g(x,y)*R(x,y)=f(x,y) (4)

[0022] Here, R(x,y) is called a sharpening filter. When the image is a two-dimensional image, the sharpening filter R is generally a two-dimensional filter having taps (cells) corresponding to each pixel of the image. In addition, the greater the number of taps (cells) of the sharpening filter R, the higher the restoration accuracy. For this reason, a feasible number of taps is set according to the required image quality, image processing capabilities, aberration characteristics, etc. The sharpening filter R must at least reflect the characteristics of aberration, and therefore is different from conventional edge enhancement filters with about three taps each horizontally and vertically. The sharpening filter R is set based on the optical transfer function OTF, so it can correct both the deterioration of the amplitude component and the phase component with high accuracy.

[0023] Furthermore, because actual images contain noise components, if a sharpening filter R created by taking the reciprocal of the optical transfer function OTF as described above is used, the noise components will be significantly amplified while restoring the degraded image. This is because the MTF (amplitude component) of the optical system is raised to return it to 1 across all frequencies in a state where the amplitude of noise is added to the amplitude component of the image. The MTF, which is the amplitude degradation caused by the optical system, returns to 1, but at the same time the power spectrum of the noise is also raised, and as a result, the noise is amplified according to the degree to which the MTF is raised (restoration gain).

[0024] Therefore, when noise is included, an image of good quality for viewing cannot be obtained. This is expressed by the following equations (5-1) and (5-2). G(u,v)=H(u,v)·F(u,v)+N(u,v) (5-1) G(u,v) / H(u,v)=F(u,v)+N(u,v) / H(u,v) (5-2)

[0025] Here, N is the noise component. For images containing noise components, there is a method of controlling the degree of restoration according to the intensity ratio SNR of the image signal to the noise signal, such as the Wiener filter expressed by the following equation (6). M(u,v)=1 / H(u,v)·(|H(u,v)|^2) / (|H(u,v)|^2+SNR^2) (6)

[0026] Here, M(u,v) is the frequency characteristic of the Wiener filter, and |H(u,v)| is the absolute value of the optical transfer function OTF (MTF). In this method, for each frequency, the smaller the MTF is, the smaller the restoration gain (degree of restoration) is, and the larger the MTF is, the larger the restoration gain is. Generally, the MTF of an optical system is high on the low frequency side and low on the high frequency side, so this method essentially reduces the restoration gain on the high frequency side of the image.

[0027] In this way, the sharpening filter can be obtained by inverse Fourier transforming a function designed based on the inverse function of the optical transfer function OTF of the optical system. The sharpening filter that can be used for the sharpening process of this embodiment can be changed as appropriate, and for example, the above-mentioned Wiener filter can be used. When using a Wiener filter, it is possible to create a real-space sharpening filter that is actually convolved with an image by performing an inverse Fourier transform on Equation (6). Furthermore, the optical transfer function OTF changes depending on the image height (image position) of the optical system even in one imaging state. For this reason, the sharpening filter is changed and used depending on the image height.

[0028] The sharpening filter used in the sharpening process is not limited to one based on the inverse function of the optical transfer function, and may be any process based on the optical characteristics for each imaging condition. For example, the sharpening process may use a filter based on the PSF instead of the blurring filter in the unsharp mask process.

[0029] In addition, the sharpening process is not limited to convolution of a single sharpening filter, and may be performed using machine learning, for example. In order to perform the sharpening process using machine learning, an original image corresponding to a subject is used as a correct image, and a training image is generated by adding blur based on an optical transfer function to the original image. For example, a neural network can be used as the machine learning model. In this case, a sharpened image is estimated based on the training image using the neural network, and the machine learning model can be optimized so that the difference with the correct image approaches. Note that the correct image and the training image may have a relationship in which blur due to optical characteristics can be corrected, and conditions such as the presence or absence of noise, the presence or absence of development processing, and resolution may be different.

[0030] Next, the principle of calculating the subject distance using parallax images is explained. Figure 8 is a diagram explaining a model of the two-viewpoint shooting method. The coordinate system has the center of the left and right cameras as the origin, the x axis in the horizontal direction, and the y axis in the depth direction. The height direction is omitted for simplification. The principal points of the imaging optical systems of the left and right cameras are respectively located at (-Wc, 0) and (Wc, 0). Here, the focal lengths of the imaging optical systems of the left and right cameras are f. In this state, it is assumed that subject A, which is located at (0, y1) on the y axis, is shot by each camera. As the parallax in shooting, the amount of deviation of the image of subject A from the sensor center of the left and right cameras is Plc and Prc, respectively, and can be expressed by the following formula.

[0031]

number

[0032] By photographing the same subject from different viewpoints using the above principle, it is possible to obtain left and right parallax images with the shift amounts shown in formulas (7) and (8) in the shift (baseline) direction (parallax direction) of the viewpoint positions. In this case, the distance y1 to subject A can be calculated using the following formula (9).

[0033]

number

[0034] In this embodiment, the parallax shift amount is considered based on the focal plane. Therefore, the parallax shift amount for a given subject is obtained by subtracting the parallax shift amount for a focused subject from the difference between the corresponding Prc and Plc. The distance (defocus amount) from the image sensor to the focal point (imaging point) of the imaging optical system is obtained by multiplying the parallax shift amount by a given conversion coefficient, and the conversion coefficient is determined according to the position shift amount of the viewpoint. Note that the parallax shift amount and the defocus amount are signed amounts that can take positive or negative values. In this embodiment, when the focal point of the imaging optical system is located on the image sensor, the amount is 0, and the direction toward the object side from the focal point is negative.

[0035] Therefore, by identifying corresponding object regions between the parallax images, the object distance can be calculated using the parallax images. Various methods can be used to identify the same object region between images. For example, a block matching method using one of the parallax images as a reference image can be used. This makes it possible to obtain the parallax shift amount and the defocus amount. The method of obtaining the object distance is not limited to the above, and the defocus amount may be obtained directly from the parallax images by machine learning, for example.

[0036] Parallax images can also be obtained using an imaging unit that guides multiple light beams that have passed through different regions of the pupil of a single imaging optical system to different light receiving sections (pixels) in a single imaging element for photoelectric conversion. In other words, parallax images required for distance calculation can be obtained with a single imaging unit (consisting of one optical system and one imaging element).

[0037] FIG. 9 shows the relationship between the light receiving part of the image sensor and the pupil of the image sensor in such an imaging section. ML is a microlens, and CF is a color filter. EXP indicates the exit pupil of the image sensor. G1 and G2 are light receiving parts (hereinafter referred to as G1 pixel and G2 pixel, respectively), and one G1 pixel and one G2 pixel are paired with each other. A plurality of pairs (pixel pairs) of G1 and G2 pixels are arranged in the image sensor. The G1 pixel and the G2 pixel included in the pixel pair have an approximately conjugate relationship with the exit pupil EXP via a common microlens ML (i.e., one for each pixel pair). A plurality of G1 pixels arranged in the image sensor are collectively referred to as a G1 pixel group, and a plurality of G2 pixels arranged in the image sensor are collectively referred to as a G2 pixel group.

[0038] Figure 10 shows a schematic of the imaging section when it is assumed that a thin lens is located at the position of the exit pupil EXP in Figure 9. The G1 pixel receives the light beam that has passed through the P1 region of the exit pupil EXP, and the G2 pixel receives the light beam that has passed through the P2 region of the exit pupil EXP. OSP is the object point being imaged. An object does not necessarily need to exist at the object point OSP, and the light beam that has passed through this point is incident on the G1 pixel or the G2 pixel depending on the region (position) in the pupil that it passes through.

[0039] The passage of light beams through different regions in the pupil corresponds to the separation of incident light from the object point OSP by angle (parallax). That is, among a plurality of G1 and G2 pixels provided for each microlens ML, an image generated using an output signal from the G1 pixel and an image generated using an output signal from the G2 pixel become a plurality of parallax images (here, a pair) having parallax with each other.

[0040] In the following description, receiving light beams that have passed through different regions in the pupil by different light receiving sections (pixels) is also referred to as pupil division. Here, an example in which the pupil is divided into two in one pupil division direction has been described, but the pupil division direction and the number of divisions are arbitrary, and the pupil may be divided into two directions, for example, the horizontal direction and the vertical direction. Furthermore, the pupil division direction and the number of divisions may differ depending on the pixel position of the image sensor. In the pupil division method, the parallax direction is the direction of positional deviation when the divided pupils are regarded as the respective viewpoints. The viewpoint position is defined based on the pupil region through which each light beam passes, and may be, for example, the center of gravity position of the light beam. Furthermore, the pupil division direction is not limited to the horizontal direction and the vertical direction, and may be an oblique direction.

[0041] Furthermore, in Figures 9 and 10, even if the position of the exit pupil EXP is shifted, the above-mentioned conjugate relationship is no longer complete, or the P1 area and the P2 area partially overlap, the multiple images obtained can be treated as parallax images.

[0042] In the split-pupil method, the amount of parallax shift and the amount of defocus can be obtained in the same way as in the two-viewpoint shooting method. The distance from the image sensor to the focal point of the imaging optical system (the amount of defocus) can be obtained by multiplying the amount of parallax shift by a predetermined conversion coefficient, and the conversion coefficient is determined according to the amount of positional shift of the viewpoint.

[0043] Next, over-correction in the sharpening process (blur correction) of a captured image will be described with reference to Figures 1 and 2. Figure 1 is a diagram showing the image-forming relationship of an optical system when focusing on an on-axis object point OB. Figure 2 is a diagram showing the image-forming relationship of an optical system when focusing on an off-axis object point OC. Note that Figures 1 and 2 only show marginal rays, and other rays are omitted.

[0044] In FIG. 1, object point OB is located at the intersection of object plane OP and the optical axis of optical system 101, and is imaged by optical system 101 to image point IB on a conjugate image plane IP. Since optical system 101 is focused on object point OB, image point IB is located on image plane IP. In addition, image plane IP is at the same position as the image sensor. Object point OC exists on object plane OP, but not on the optical axis. Object point OC is imaged by optical system 101 to a conjugate image point IC. Note that object point OC has a first object height, and image height IC has a first image height.

[0045] Since the optical system 101 in this embodiment has a curvature of field, an image point IC indicating an imaging position at an arbitrary image height of the optical system 101 is shifted (defocused) in the propagation direction of light rays due to the curvature of field, and is not located on an image plane IP (imaging plane). In addition, a curved surface 11 on which the image point IC is located is an imaging position at each image height corresponding to each object height of the object plane OP due to the curvature of field of the optical system 101. The distance between the image plane IP and the image point IC (first image point) is defined as a first defocus amount. The first defocus amount is not limited to the curvature of field, and may be a value based on the defocus amount due to the aberration of the optical system 101. The aberration in this embodiment includes, for example, curvature of field and astigmatism, and is preferably an aberration that causes a shift of the image point in the optical axis direction.

[0046] Here, the object point OC' is at the same object height as the object point OC, and is located at a different depth from the object point OC (has an object distance). The object point OC' is imaged at a conjugate image point IC' by the optical system 101. In FIG. 1, the object point OC' is located on an image plane IP. That is, the optical system 101, which is focused on an on-axis object point OB, is focused on a position having the depth of the object point OC' at a first image height. At this time, the distance between the image plane IP and the image point IC' (second image point) is defined as a second defocus amount.

[0047] Although the second defocus amount is ideally zero, in this embodiment, "imaging" and "focusing" do not have to be imaging and focusing based on optical paraxial theory. For example, it may be that the value of the optical imaging spot spread (PSF) due to aberration is the smallest. It may also be that the value of the MTF at a specific spatial frequency is the largest. Furthermore, the "conjugate" in this embodiment does not have to be a relationship determined by optical paraxial theory. For example, it may be a relationship in which the optical transfer function (OTF) is the best based on a predetermined criterion.

[0048] In this case, in the conventional technology (Patent Document 1), even if the subject is located at object point OC', sharpening processing is performed based on the optical characteristics (corresponding to the first optical characteristics described later) for object point OC on object plane OP. At this time, overcorrection occurs in a partial region including image point IC' where no blur occurs, resulting in image quality problems such as undershoot and ringing. Here, object point OC' where the second defocus amount is 0 is taken as an example, but if the subject has an absolute value of the defocus amount from image plane IP that is small for object point OC corresponding to the optical characteristics used for sharpening processing, overcorrection may occur in a similar manner.

[0049] Next, Fig. 2 will be described. Fig. 2 is a diagram showing the image formation relationship by the optical system when focusing on an off-axis object point OC. Descriptions of the same contents as Fig. 1 will be omitted.

[0050] In Fig. 2, optical system 101 having distortion aberration is focused on object point OC located off-axis. Therefore, image point IC is located at the intersection of image plane IP and curved surface 11. The first defocus amount is the amount of field curvature at each image height, and corresponds to the distance between curved surface 11 and image plane IP at each image height in the figure. Thus, when focused on object point OC, the defocus of optical system 101 at image point IC is smaller than when focused on object point OB.

[0051] In this case, in the conventional technology (Patent Document 1), the strength of the sharpening process is set higher at higher image heights, so that image point IC is processed at a higher strength even though it is less blurred than image point IB. In this case, overcorrection occurs in the partial area including image point IC where no blur occurs, resulting in image quality problems such as undershooting and ringing.

[0052] In this embodiment, the purpose is to suppress excessive correction by the sharpening process performed based on the optical characteristics (first optical characteristics) when defocused by the first defocus amount at an arbitrary image height. Note that in this embodiment, the defocus amount is a signed amount that can have a positive or negative value. In Figures 1 and 2, when the focal point of the optical system 101 is located on the object plane OP, the defocus amount is 0, and the defocus amount in the direction toward the object side is negative.

[0053] Although the optical system 101 in this embodiment is rotationally symmetric, the present invention is not limited to this, and the optical system 101 may be rotationally asymmetric.

[0054] The image processing method described above is merely an example, and the present invention is not limited to this. Details of other image processing methods will be described in the following examples.

[0055] [Example 1] Next, the configuration of the imaging device of the first embodiment will be described. The imaging device 100 of this embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the imaging device 100. An image processing program for performing the sharpening process of this embodiment is installed in the imaging device 100. The sharpening process of this embodiment is executed by an image processing unit (image processing device) 104 inside the imaging device 100 having imaging means including an optical system 101.

[0056] The imaging device 100 is configured to include an optical system (imaging optical system) 101 and an imaging device body (camera body). The optical system 101 has an aperture 101a and a focus lens 101b, and is configured integrally with the camera body. However, the present invention is not limited to this, and can also be applied to an imaging device in which the optical system 101 is detachably attached to the camera body. Furthermore, the optical system 101 may be configured to include an optical element having a diffractive surface, an optical element having a reflective surface, etc., in addition to an optical element having a refractive surface such as a lens.

[0057] The image sensor 102 has a CCD sensor or a CMOS sensor, and generates (outputs) a captured image (image data) by photoelectrically converting an object image (optical image formed by the optical system 101) formed via the optical system 101. That is, the object image is converted into an analog signal (electrical signal) by photoelectric conversion by the image sensor 102. The A / D converter 103 converts the analog signal input from the image sensor 102 into a digital signal, and outputs it to the image processing unit 104.

[0058] The image processing unit 104 performs a predetermined process on the digital signal and also performs the sharpening process of this embodiment. The image processing unit 104 has an image acquisition unit 104a, an image capture condition acquisition unit 104b, an information acquisition unit 104c, and a processing unit 104d. The image acquisition unit 104a acquires a captured image from the A / D converter 103. The image capture condition acquisition unit 104b acquires the image capture conditions of the image capture device 100 from the state detection unit 107. The image capture conditions are an aperture value, an image capture distance (focus position), or a focal length of a zoom lens. The state detection unit 107 may acquire the image capture conditions directly from the system controller 110 or from the optical system control unit 106. The process by the image processing unit 104 can be implemented by one or more processors (processing means) such as a CPU.

[0059] Also, the information acquisition unit 104c acquires information on the subject distance. In this embodiment, the defocus amount of the subject (subject defocus amount) based on the subject distance is acquired based on the above-mentioned pupil division method. However, the present invention is not limited to this. The subject distance may be acquired based on the acquired parallax image by configuring the optical system 101 as a stereo optical system. Also, it is not limited to the parallax image, and a method of acquiring the subject distance using a distance measuring device such as LiDAR, or a method of acquiring the subject distance from a monocular image by image processing may be used. The subject defocus amount can be calculated based on the optical specifications from the subject distance or calculated using a correspondence table.

[0060] In FIG. 1, when a subject exists at an object point OC' having a first image height, the subject distance corresponds to the distance from the image plane IP to the object point OC'. The subject distance may be the distance from the optical system 101 or the imaging device to the object point OC'. The subject defocus amount is the defocus amount at the object plane OP (focus plane) and the image plane corresponding to the object point OC', so in FIG. 1, the subject defocus amount is the distance between the image plane IP and the image point IC'. In addition, when the aberration of the optical system is taken into consideration, the subject defocus amount may be the distance from the curved surface 11 to the image point IC' at the image height corresponding to the object height of the subject. In this case, the subject defocus amount is the distance between the intersection points of the straight lines that pass through the curved surface 11 and the image point IC' and are parallel to the optical axis.

[0061] Furthermore, unlike the above-mentioned parallax shift amount, when the subject distance is acquired by a distance measuring device or the like without using the optical system 101, or when the subject distance is acquired based on the subject scene or composition, the defocus amount due to field curvature and astigmatism is not taken into consideration. Therefore, by storing information on field curvature and astigmatism in advance in the image processing unit 104, for example, it is possible to convert to a subject defocus amount that takes these into consideration.

[0062] The optical transfer function (OTF) or data required for generating the OTF is held in a storage unit (storage means) 108. The storage unit 108 is composed of, for example, a ROM. The output image processed by the image processing unit 104 is stored in a predetermined format in an image recording medium 109. An image obtained by performing a predetermined display process on an image that has been sharpened is displayed on a display unit 105 composed of a liquid crystal monitor or an organic EL display. However, the image displayed on the display unit 105 is not limited to this, and an image that has been subjected to simple processing for high-speed display may be displayed on the display unit 105.

[0063] The system controller 110 controls the imaging device 100. The optical system 101 is mechanically driven by the optical system control unit 106 based on instructions from the system controller 110. The optical system control unit 106 controls the aperture diameter of the diaphragm 101a to obtain a predetermined F-number. The optical system control unit 106 also controls the position of the focus lens 101b by an autofocus (AF) mechanism or a manual focus mechanism (not shown) to adjust the focus according to the subject distance. Note that functions such as the aperture diameter control of the diaphragm 101a and manual focus may not be executed depending on the specifications of the imaging device 100.

[0064] Note that optical elements such as a low-pass filter and an infrared cut filter may be disposed between the optical system 101 and the image sensor 102, but when an element that affects optical characteristics such as a low-pass filter is used, consideration may be required when creating a sharpening filter. The infrared cut filter also affects each OTF of the RGB channels, which is the integral value of the optical transfer function (OTF) of the spectral wavelength, particularly the OTF of the R channel, so consideration may be required when creating a sharpening filter. Therefore, the sharpening filter may be changed depending on the presence or absence of a low-pass filter or an infrared cut filter.

[0065] The image processing unit 104 is configured with an ASIC, and the optical system control unit 106, the state detection unit 107, and the system controller 110 are each configured with a CPU or an MPU. One or more of the image processing unit 104, the optical system control unit 106, the state detection unit 107, and the system controller 110 may be configured with the same CPU or MPU.

[0066] The imaging device 100 is not limited to this. The image processing method according to this embodiment may be implemented by an image processing system including, for example, an imaging device and an image processing device (corresponding to the image processing unit 104) provided separately from the imaging device. In this case, the imaging device outputs an input image to the image processing device, and the image processing device performs image processing described later. In addition, it is preferable that the imaging device also accompanies the input image with information necessary for correction, including imaging condition information and optical information. The imaging condition information and correction information may be directly or indirectly transferred from the imaging device to the image processing device by communication, other than being outputted accompanying the input image.

[0067] The sharpening process performed in the image processing unit 104 will be specifically described in each embodiment below. Fig. 4 is a flowchart of the sharpening process in the embodiment 1. The sharpening process in the embodiment 1 is executed based on an instruction from the image processing unit 104. Each step in Fig. 4 is executed mainly by the image acquisition unit 104a, the imaging condition acquisition unit 104b, the information acquisition unit 104c, and the processing unit 104d.

[0068] In step S101, the image acquisition unit 104a acquires an image captured by the imaging device 100 as an input image. In this embodiment, the input image is a parallax image corresponding to each of a plurality of viewpoints. The input image is stored in the storage unit 108. Note that an image stored in the image recording medium 109 may be acquired as the input image.

[0069] In step S102, the imaging condition acquisition unit 104b acquires imaging conditions when capturing an input image as necessary. The imaging conditions include the focal length of the optical system 101, the aperture value, and the imaging distance determined for a focused subject. In the case of an imaging device in which a lens is replaceably attached to a camera body, the imaging conditions may further include a lens ID and a camera ID. The imaging conditions may be acquired directly from the imaging device, or may be acquired from information attached to the input image. The information attached to the input image is, for example, EXIF ​​(Exchangeable image file format) information.

[0070] In step S103, the information acquisition unit 104c acquires the optical characteristic (first optical characteristic) of the optical system 101 from the storage unit 108. The first optical characteristic is the optical characteristic of the optical system 101 when the optical system 101 is defocused by the first defocus. The first defocus amount is a value based on the aberration of the optical system 101, and differs for each image height. Note that this embodiment is not limited to this, and any object point and an image point corresponding to the object point may be defined. For example, the object surface OP may be a curved surface such as a spherical surface. Also, the position where the optical system is focused does not have to be on the axis. Also, as the optical characteristic, for example, OTF, PSF, or coefficient data of a function approximately expressing PSF or OTF, etc. can be acquired. Furthermore, in this embodiment, the aberration is field curvature. Note that the aberration is not limited to this, and may be astigmatism or the like. The storage unit 108 may store optical characteristics discretely for imaging conditions and screen positions (image height, orientation, etc.), and the optical characteristics in each partial region may be acquired using interpolation for the imaging conditions and screen positions.

[0071] In S104, the information acquisition unit 104c acquires information about the subject distance by the above-mentioned method based on the captured image acquired in S101. The information about the subject distance is acquired, for example, by the above-mentioned pupil division method. In this embodiment, the map (defocus map) is a map in which the subject defocus amount based on the subject distance corresponding to the subject in each partial area of ​​the image is arranged two-dimensionally based on the above-mentioned pupil division method. In other words, the defocus map is a distribution of defocus values ​​indicating the degree of focus when the captured image is acquired for each position in the image. It is preferable that the defocus map includes values ​​corresponding to the entire image and has the same number of pixels as the captured image. Note that, in this embodiment, the information about the subject distance is a defocus map, but is not limited to this. Furthermore, step S104 may be executed regardless of the order of step S102, step S103, and step S105.

[0072] In step S105, the processing unit 104d acquires a sharpening filter. In this embodiment, the sharpening filter is acquired based on the first optical characteristic at a position corresponding to each partial region using formula (6). Since the first optical characteristic at each position of the input image is different, the sharpening filter applied to each position of the input image is different.

[0073] In step S106, the information acquisition unit 104c acquires the correction intensity. In this embodiment, the correction intensity is acquired based on the object defocus amount corresponding to each partial region. The correction intensity will be described in detail later.

[0074] In step S107, the processing unit 104d executes the sharpening process based on the sharpening filter and the correction strength. The sharpening process first convolves the input image with the sharpening filter to obtain an intermediate output image by sharpening the input image. Then, the input image and the intermediate output image are weighted-added based on the correction strength to adjust the correction effect.

[0075] The sharpening process in the processing unit 104d is a process for sharpening the blur caused by the first optical characteristic, and is executed based on the correction strength. For example, a machine learning model may be used for the sharpening process. The machine learning model is trained using learning data including a ground truth image and a training image that differ in the presence or absence of blur caused by the optical characteristic. Even in the sharpening process using the machine learning model, by using a correction strength based on the subject defocus amount, it is possible to suppress overcorrection of the sharpening process caused by the first optical characteristic and the depth of the subject.

[0076] When performing the sharpening process using a machine learning model, the information acquisition unit 104c acquires parameters (weight information) of the machine learning model. At this time, instead of convolving the sharpening filter in step S106, an intermediate output image obtained by sharpening the input image using the machine learning model is acquired. Note that the machine learning model may use the correction strength as input data.

[0077] The correction strength will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing the change in MTF with respect to the depth of a subject in a certain region. Fig. 6 is a diagram explaining the correction strength. In this embodiment, it is assumed that the blur due to astigmatism is sufficiently small, and only defocus due to field curvature is taken into consideration.

[0078] The vertical axis of Fig. 5 is the MTF value at a predetermined spatial frequency, and the horizontal axis is the depth of the subject from the optical system. Also, the on-axis focus position s and the off-axis focus position t of the subject depth at an arbitrary object height are illustrated. In this embodiment, MTF is used as the optical characteristic. However, this is not limited to this.

[0079] A in FIG. 5 shows the change in MTF with respect to the subject depth in a partial region including on the axis. In A, the subject depth showing the highest MTF is set to the on-axis focusing position s. In other words, the optical system focuses on a subject that is s away on the axis. B shows the change in sagittal MTF with respect to the subject depth in an off-axis partial region having an arbitrary image height. In B, the subject depth showing the highest MTF is set to the off-axis focusing position t. C shows the change in meridional MTF with respect to the subject depth in an off-axis partial region having the same image height as B. The on-axis focusing position s and the off-axis focusing position t differ depending on the image height and the field curvature.

[0080] In this embodiment, astigmatism is not taken into consideration, the meridional direction can be considered in the same way as the sagittal direction. Therefore, in addition to the MTF difference in the sagittal direction, the MTF difference in the meridional direction may be obtained. In that case, the correction strength may be set based on the value of the azimuth direction with the larger MTF difference, or the correction strength may be obtained for each azimuth direction.

[0081] 5, when a subject with an image height corresponding to B exists at a different depth from an on-axis subject corresponding to A, or when the optical system 101 is focused on a subject other than the on-axis subject, the optical characteristics may be improved at the position corresponding to B despite the image height being higher than that of A. In such a case, there is a risk of over-correction in the partial region corresponding to B.

[0082] Although the horizontal axis in Fig. 5 is the depth of the subject, it may be the image plane position corresponding to the optical characteristics. The optical characteristics do not strictly match when the subject depth is changed and when the image plane position is changed, but they can be considered equivalent if the change in depth is sufficiently small.

[0083] Next, the correction strength will be described with reference to Fig. 6. Fig. 6(A) to (C) each show an example of the relationship between the correction strength and the object defocus amount. In each diagram of Fig. 6, the vertical axis represents the correction strength for an arbitrary partial region, and the horizontal axis represents the object defocus amount.

[0084] In FIG. 6, the subject defocus amount of the subject having a depth corresponding to the off-axis focus position t is set to 0, and the subject defocus amount of the subject having a depth corresponding to the on-axis focus position s is set to S. At this time, the image point corresponding to the object point where the subject is located is defocused by the first defocus amount. That is, the subject defocus amount when it is equal to the difference (first defocus amount) between the image formation position at any image height of the optical system and the imaging surface is set to S. Also, in FIG. 6, when the subject defocus amount is S, the correction strength is set to be maximum. However, it is not limited to this. It is preferable that the correction strength is set to be smaller when the subject defocus amount is 0 than when the first defocus amount is equal to the subject defocus amount.

[0085] In the process of sharpening the blur caused by the first optical characteristic, if the optical characteristic at a high image height is higher than the optical characteristic at the in-focus position within the angle of view due to the depth of the subject, overcorrection occurs. Therefore, by acquiring the subject defocus amount for each image height based on the subject distance and setting the correction strength of the sharpening process based on the subject defocus amount, it is possible to suppress the adverse effects of the sharpening process.

[0086] As shown in FIG. 6A, the correction strength is set to be small in an area where the absolute value is smaller than the defocus amount S corresponding to the on-axis focusing position s. That is, the correction strength is maximum when the object defocus amount is equal to the first defocus amount. This is because the closer the absolute value of the object defocus amount is to zero, the higher the optical performance at the corresponding image height is, and the more likely overcorrection is to occur. Note that the minimum value of the correction strength does not have to be zero, as long as the adverse effects of overcorrection can be reduced. As described above, overcorrection can be suppressed by setting the correction strength according to the object defocus amount at the position where the optical system has the highest performance. Note that in this embodiment, the optical characteristics being high performance refers to the optical characteristics being evaluated based on a predetermined criterion and having good performance. For example, this corresponds to a large MTF value at a predetermined spatial frequency, a small spread of the point spread function (PSF), etc.

[0087] As shown in FIG. 6B, the correction intensity may be reduced when the absolute value of the subject defocus amount is too large for the defocus amount S. In other words, when the difference between the optical characteristics acquired in S103 and the optical characteristics of the actual subject is large, the correction intensity is reduced even under conditions where overcorrection does not occur. With this configuration, the effect of the sharpening process can be focused on the case corresponding to the desired defocus amount. In this way, the correction intensity may be different when the defocus amount changes in the positive direction and the negative direction with respect to the defocus amount S. The correction intensity may be reduced when the absolute value of the subject defocus amount is greater than the absolute value of the defocus amount S multiplied by an arbitrary coefficient or the value obtained by adding an arbitrary coefficient. The range in which the correction intensity is reduced may be determined based on the focal length of the optical system used for imaging, the imaging distance, and the like.

[0088] As shown in FIG. 6C, the correction strength may be asymmetric in the positive and negative directions (asymmetric with respect to the y-axis) with the defocus amount being zero as a reference. When the defocus amount S is negative, overcorrection does not occur even if the defocus amount changes further in the negative direction. On the other hand, by making the correction strength smaller when the defocus amount S changes in the positive direction, overcorrection can be suppressed. The same applies when the defocus amount S is positive.

[0089] With this configuration, even if the accuracy of acquiring information about the subject defocus amount or the subject distance is low, it is possible to suppress overcorrection by determining the correction strength based on the magnitude relationship with respect to the defocus amount S. Note that the information about the correction strength with respect to the subject defocus amount described above is held in table format.

[0090] As shown in FIG. 6, the positive and negative correction intensities may be asymmetric with respect to a line that passes through (S, 0) and is perpendicular to the x-axis.

[0091] In addition, the amount (degree) of overcorrection for the subject defocus amount varies depending on the difference between the MTF value at the on-axis focus position s and the MTF value at the off-axis focus position t, the image height, the imaging distance, or the imaging conditions. The correction strength may be acquired based on, in addition to the subject defocus amount, for example, the difference between the MTF value and the MTF value at the off-axis focus position t, the image height, the imaging distance, or the imaging conditions. For example, the minimum value of the correction strength may be made smaller as the MTF value at the off-axis focus position t is larger than the MTF value at the on-axis focus position s. In addition, the above-mentioned processing may be applied only to the image height or the imaging conditions in which the difference between the MTF value at the on-axis focus position s and the MTF value at the off-axis focus position t is large. If necessary, the storage unit 108 holds the above-mentioned information such as the imaging conditions.

[0092] Since the amount of overcorrection occurring in the image differs for each partial region (image height), it is preferable that the correction intensity in this embodiment differs depending on the position of the partial region. In the case of a rotationally symmetric optical system, the field curvature and astigmatism change depending on the image height and are uniform in the rotation direction about the optical axis. In this case, the degree of overcorrection may change depending on the image height of the captured image and be uniform in the rotation direction about the optical axis. In this case, by storing only the change according to the image height as the change in correction intensity for the partial region, the data capacity for storing the data of the correction intensity can be reduced.

[0093] In this embodiment, the information acquisition unit 104c acquires the first optical characteristic and the correction intensity from the storage unit 108 of the imaging device 100, but is not limited to this. For example, in the case of an imaging device in which the optical system 101 is detachably attached to a camera body, the information acquisition unit 104c may acquire the optical characteristic and the correction intensity stored in a storage unit in a lens device including the optical system 101 by communication with the imaging device. In this case, the lens device (optical system 101) has a storage unit (not shown) that stores the optical characteristic and the correction intensity, and a communication unit (not shown) that transmits the optical characteristic and the correction intensity to the camera body.

[0094] The first optical characteristic and the correction intensity may be stored in a storage unit in the imaging device or the lens device, or may be held in advance on a server. In this case, the optical characteristic and the correction intensity can be downloaded by communicating with the imaging device or the lens device as necessary.

[0095] In this embodiment, the correction intensity is obtained from the storage unit 108, but a value (index value) such as MTF may be obtained from the storage unit 108, and the correction intensity may be calculated from the index value. With this configuration, it is possible to set a different correction intensity for each imaging device or optical system.

[0096] [Example 2] Next, the configuration of an imaging device of the second embodiment will be described. In the first embodiment, the astigmatism is sufficiently small, but in the present embodiment, the case where the astigmatism is large enough to not be ignored will be described. Note that the flowchart related to the sharpening process of the present embodiment is the same as that of the first embodiment.

[0097] The correction strength in an optical system having astigmatism will be described in detail with reference to FIG. 7. FIG. 7 is a diagram showing the change in MTF with respect to the depth of the object. The vertical axis of FIG. 7 is the value of MTF at a predetermined spatial frequency, and the horizontal axis is the depth of the object from the optical system. Also, among the depth of the object at an arbitrary object height, the in-focus position s on the axis, the in-focus position t1 in the sagittal direction off-axis, and the in-focus position t2 in the meridional direction off-axis are shown. In this embodiment, MTF is used as the optical characteristic. Also, the azimuth direction is not limited to the sagittal direction and the meridional direction, but may be a first direction and a second direction different from each other among a plurality of azimuth directions.

[0098] A in the figure shows the change in MTF with respect to the subject depth in an area including on the axis. In A, the subject depth showing the highest MTF is set to the on-axis in-focus position s. In other words, the optical system focuses on a subject that is s away on the axis. B shows the change in sagittal MTF with respect to the subject depth in an off-axis partial area having an arbitrary image height. In B, the subject depth showing the highest MTF is set to the off-axis sagittal in-focus position t1. C shows the change in meridional MTF with respect to the subject depth in an off-axis partial area having the same image height as B. In C, the subject depth showing the highest MTF is set to the off-axis meridional in-focus position t2.

[0099] This embodiment takes astigmatism into consideration, and therefore differs from the first embodiment in that the in-focus positions t1 and t2 are different for each azimuth direction even in the same off-axis region. Note that the correction strength in this embodiment is obtained based on the difference between the in-focus position and the subject distance for each azimuth direction (subject defocus amount). For example, the correction strength in a partial region including a certain image height is obtained based on the smaller value (high optical performance) of the subject defocus amount at t1 and the subject defocus amount at t2.

[0100] With this configuration, it is possible to suppress excessive correction in the sharpening process performed on an image captured using an optical system having astigmatism.

[0101] (Other Examples) The present invention can also be realized by supplying a program for realizing one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for realizing one or more functions. The image processing device in the present invention may be any device having the image processing function of the present invention, and may be realized in the form of an imaging device or a PC.

[0102] According to each embodiment, it is possible to provide an image processing method, an image processing device, and a program capable of performing sharpening processing that suppresses over-correction due to the depth of the subject while reducing data volume and calculation load.

[0103] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0104] The embodiments of the present invention include the following methods and configurations.

[0105] [Method 1] acquiring a captured image obtained by imaging using an optical system and information regarding a subject distance; acquiring a subject defocus amount for a partial region in the captured image based on information about the subject distance; obtaining a correction strength of a sharpening process to be applied to the partial region based on the object defocus amount; applying the sharpening process based on the correction strength; an image processing method, characterized in that the correction strength is smaller when the subject defocus amount is 0 than when the difference between the image formation position and the imaging plane at each image height of the optical system is equal to the subject defocus amount.

[0106] [Method 2] The image processing method described in Method 1, characterized in that the sharpening process is performed based on the optical characteristics of the optical system based on the difference between the image formation position at each image height of the optical system and the imaging surface.

[0107] [Method 3] The image processing method according to Method 1 or 2, characterized in that the difference between the image forming position and the imaging plane at each image height of the optical system is the defocus amount at each image height caused by aberration of the optical system.

[0108] [Method 4] The image processing method according to Method 3, wherein the aberration includes at least one of field curvature and astigmatism.

[0109] [Method 5] The image processing method according to Method 2, wherein the sharpening process is a process for correcting blur corresponding to the optical characteristics.

[0110] [Method 6] 6. The image processing method according to claim 5, wherein the optical characteristic is a point spread function or an optical transfer function of the optical system.

[0111] [Method 7] the captured image is a plurality of parallax images respectively corresponding to a plurality of parallaxes, 7. The image processing method according to any one of Methods 1 to 6, characterized in that the information on the subject distance is obtained based on the plurality of parallax images.

[0112] [Method 8] The image processing method described in any one of Methods 1 to 7, characterized in that the information regarding the subject distance is a defocus map, which is a distribution of defocus values ​​indicating the degree of focus at each position within the image when the captured image was acquired.

[0113] [Method 9] The image processing method according to any one of Methods 1 to 8, characterized in that, when the correction intensity is represented on the y-axis and the defocus amount of the optical system is represented on the x-axis, and the change in the correction intensity relative to the defocus amount is represented, the change is asymmetric with respect to the y-axis.

[0114] [Method 10] The image processing method according to any one of Methods 1 to 9, wherein the sharpening process is a process using a machine learning model.

[0115] [Method 11] 11. The image processing method according to any one of Methods 1 to 10, wherein the correction intensity is uniform in a rotational direction about the optical axis of the optical system.

[0116] [Method 12] An image processing method according to any one of Methods 1 to 11, characterized in that the correction strength is based on the amount of defocus of the subject in a first direction among a plurality of azimuth directions in the partial region and the amount of defocus of the subject in a second direction different from the first direction.

[0117] [Method 13] 13. The image processing method according to claim 12, wherein the plurality of azimuth directions includes a sagittal direction and a meridional direction.

[0118] [Configuration 1] 14. An image processing apparatus comprising a processing means capable of executing the image processing method according to any one of Methods 1 to 13.

[0119] [Configuration 2] The image processing device according to configuration 1, an imaging device including the optical system and acquiring the captured image,

[0120] [Configuration 3] A processing means capable of executing the image processing method according to any one of methods 1 to 13; an imaging device including the optical system and configured to capture an image of an object via the optical system.

[0121] [Configuration 4] The imaging element includes pixels each having a plurality of light receiving portions, 4. The imaging device according to configuration 3, wherein the plurality of light receiving sections receive light beams that have passed through different pupils.

[0122] [Configuration 5] A program for causing a computer to execute the image processing method according to any one of Methods 1 to 13.

[0123] [Configuration 6] A storage medium storing the program according to configuration 5. [Explanation of symbols]

[0124] 101 Optical system S101: A step of acquiring information regarding a captured image and a subject distance S104: A step of acquiring a defocus amount of a subject S106 obtaining a corrected intensity; S107: Step of applying sharpening process

Claims

1. acquiring a captured image obtained by imaging using an optical system; acquiring a subject defocus amount for a partial region in the captured image; and performing a sharpening process on the partial region based on the first optical characteristic and the subject defocus amount, the first optical characteristic is an optical characteristic of the optical system when a defocus amount of the optical system is a first defocus amount, 10. An image processing method, comprising: a correction strength when the subject defocus amount is 0 being smaller than a correction strength when the subject defocus amount is equal to the first defocus amount.

2. 2. The image processing method according to claim 1, wherein the sharpening process is performed based on optical characteristics of the optical system based on a difference between an image-forming position at each image height of the optical system and an imaging surface.

3. 3. The image processing method according to claim 2, wherein the difference between the image forming position and the image pickup surface at each image height of the optical system is a defocus amount at each image height caused by aberration of the optical system.

4. 4. The image processing method according to claim 3, wherein the aberration includes at least one of field curvature and astigmatism.

5. 3. The image processing method according to claim 2, wherein the sharpening process corrects blurring corresponding to the optical characteristics.

6. 6. The image processing method according to claim 5, wherein the optical characteristic is a point spread function or an optical transfer function of the optical system.

7. the captured image is a plurality of parallax images corresponding to a plurality of parallaxes, 7. The image processing method according to claim 1, wherein the first defocus amount is acquired based on the plurality of parallax images.

8. 8. The image processing method according to claim 7, wherein the first defocus amount is a defocus map, which is a distribution of defocus values ​​indicating the degree of focus when the captured image was acquired for each position within the image.

9. 7. The image processing method according to claim 1, wherein the correction intensity is represented on the y-axis and the defocus amount of the optical system is represented on the x-axis, and when the change in the correction intensity relative to the defocus amount is represented, the change is asymmetric with respect to the y-axis.

10. 7. The image processing method according to claim 1, wherein the sharpening process is a process using a machine learning model.

11. 7. The image processing method according to claim 1, wherein the correction intensity is uniform in a rotation direction about the optical axis of the optical system.

12. 7. The image processing method according to claim 1, wherein the correction strength is based on the amount of defocus of the subject in a first direction among a plurality of azimuth directions in the partial region and the amount of defocus of the subject in a second direction different from the first direction.

13. 13. The image processing method according to claim 12, wherein the plurality of azimuth directions include a sagittal direction and a meridional direction.

14. 7. An image processing apparatus comprising processing means capable of executing the image processing method according to claim 1.

15. The image processing device according to claim 14; an imaging device that includes the optical system and acquires the captured image;

16. A processing means capable of executing the image processing method according to any one of claims 1 to 6; an imaging device including the optical system and configured to capture an image of an object via the optical system;

17. the imaging element includes pixels each having a plurality of light receiving portions, 17. The imaging device according to claim 16, wherein the plurality of light receiving sections receive light beams that have passed through different pupils.

18. A program causing a computer to execute the image processing method according to any one of claims 1 to 6.

19. A storage medium storing the program according to claim 18.

Images