Imaging device, imaging method, program, and recording medium

Abstract

To solve a problem that when a plurality of images captured while continuously changing a focal position during light exposure is composed, luminance unevenness develops in a composed image.SOLUTION: An imaging apparatus according to the present invention comprises: imaging means which captures a plurality of images with different focal positions and continuously changes the focal positions even while capturing each of the images; and correction means which corrects an exposure difference caused by the change in the focal position while imaging a sheet of image by the imaging means. The exposure difference between the images after the correction by the correction means is further reduced than that before the correction.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an imaging device that captures a plurality of images with different focus positions. [Background technology]

[0002] When capturing images of multiple subjects at significantly different distances from a digital camera or a subject that is long in the depth direction, the depth of field may be insufficient to focus on only a portion of the subject. To solve this problem, Patent Document 1 discloses a so-called depth stacking technique that captures multiple images with different focus positions, extracts only the in-focus areas from each image, and combines them into a single image to generate a composite image in which the entire captured area is in focus.

[0003] Furthermore, in order to reduce the processing time of focus stacking as much as possible, a technique for continuously changing the focus position during exposure, as described in Patent Document 2, has been disclosed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-216532 A [Patent Document 2] International Publication No. 12 / 117733 Summary of the Invention [Problem to be solved by the invention]

[0005] However, as described above, when capturing an image by moving the focus position during exposure, the effective aperture value changes when the upper and lower parts of the image sensor are exposed, causing brightness unevenness within the image plane. If only the in-focus areas of multiple captured images are extracted and synthesized into a single image, the uneven brightness makes the synthesis boundary stand out, which can result in roughness in the synthesized image.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has an object to reduce roughness in a composite image when the focus position is moved during exposure in imaging for depth stacking. [Means for solving the problem]

[0007] In order to solve the above problem, the present invention provides a method for adjusting the focus position. Capture images while changing An imaging means, One The focus position while capturing the image Changes in and a correction means for correcting an exposure difference caused by the focus position. Changes in If the exposure difference caused by is equal to or smaller than a predetermined threshold, the correction means does not perform correction. do not have The present invention provides an imaging device characterized by the above-mentioned. Effect of the Invention

[0008] According to the configuration of the present invention, it is possible to obtain a depth stacking image in which the image capturing time required to capture multiple images with different focuses is shortened, while reducing the roughness of the composite boundary caused by differences in brightness unevenness and noise. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a hardware configuration of a digital camera as an image capture apparatus according to an embodiment of the present invention. [Diagram 2] 5 is a graph for explaining the relationship between the displayed aperture value, the effective aperture value, and the focus position in an embodiment of the present invention. [Diagram 3] 11 is a graph for explaining the relationship between the effective aperture value change due to the change in focus position and the exposure timing when the exposure time is not corrected in an embodiment of the present invention. [Figure 4] 11 is a graph for explaining the relationship between the effective aperture value change due to the change in focus position of a plurality of images and the exposure timing when the exposure time is not corrected in an embodiment of the present invention. [Diagram 5] 1 is a flowchart illustrating a process of focus stacking according to an embodiment of the present invention. [Figure 6] 5 is a graph for explaining the relationship between pixel reset timing for each row and effective aperture value in an embodiment of the present invention. [Figure 7] 11 is a graph for explaining the relationship between pixel reset timing for each row and effective aperture value after exposure time correction in an embodiment of the present invention. [Figure 8] 11 is a graph illustrating the relationship between pixel reset timing for each row of a plurality of images and effective aperture value after exposure time correction in an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0011] 1 is an example of a block diagram showing the structure of a digital camera as an imaging device according to this embodiment. Digital camera 100 can capture still images, record information on the focus position, calculate contrast values, and synthesize images. Furthermore, digital camera 100 can perform enlargement or reduction processing on captured and saved images or images input from the outside.

[0012] The control unit 101 is, for example, a signal processor such as a CPU or MPU, and controls each part of the digital camera 100 while reading a program previously stored in a ROM 105 described later. For example, as described later, the control unit 101 issues commands to an imaging unit 104 described later to start and end imaging. Alternatively, the control unit 101 issues commands for image processing to an image processing unit 107 described later based on a program stored in the ROM 105. Commands from a user are input to the digital camera 100 by an operation unit 110 described later, and reach each part of the digital camera 100 through the control unit 101.

[0013] The driving unit 102 is constituted by a motor and the like, and mechanically operates the optical system 103 described below under the command of the control unit 101. For example, based on the command of the control unit 101, the driving unit 102 moves the position of a focus lens included in the optical system 103 to adjust the focal length of the optical system 103.

[0014] The optical system 103 is composed of a zoom lens, a focus lens, an aperture, etc. The aperture is a mechanism for adjusting the amount of light that passes through. The focal position can be changed by changing the position of the lens.

[0015] The imaging unit 104 is an imaging element that performs photoelectric conversion to convert an incident optical signal into an electrical signal. For example, a CCD sensor or a CMOS sensor can be applied to the imaging unit 104. The imaging unit 104 is provided with a video imaging mode, and can capture a plurality of images that are continuous in time as individual frames of a video. The imaging unit 104 can measure the luminance of a subject through the optical system 103. Instead of photometry in the imaging unit 104, an AE sensor or the like can be used.

[0016] The ROM 105 is a read-only non-volatile memory serving as a recording medium, and stores operation programs for each block of the digital camera 100 as well as parameters and the like required for the operation of each block.

[0017] The RAM 106 is a rewritable volatile memory, and is used as a temporary storage area for data output during the operation of each block included in the digital camera 100 .

[0018] The image processing unit 107 performs various image processing such as white balance adjustment, color interpolation, filtering, and synthesis processing on the image output from the imaging unit 104 or on image signal data recorded in the internal memory 109 described later. In addition, the image processing unit 107 performs compression processing on the image signal data captured by the imaging unit 104 in accordance with a standard such as JPEG.

[0019] Image processing unit 107 is configured as an application specific integrated circuit (ASIC) that is a collection of circuits that perform specific processing. Alternatively, control unit 101 may perform part or all of the functions of image processing unit 107 by processing according to a program read from ROM 105 by control unit 101. When control unit 101 performs all of the functions of image processing unit 107, there is no need to have image processing unit 107 as hardware.

[0020] The display unit 108 is a liquid crystal display or an organic EL display for displaying images temporarily stored in the RAM 106, images stored in the internal memory 109 described below, or a setting screen of the digital camera 100.

[0021] The built-in memory 109 is a place for recording images captured by the imaging unit 104, images processed by the image processing unit 107, and information on the focal position when capturing an image, etc. A memory card or the like may be used instead of the built-in memory.

[0022] The operation unit 110 is, for example, a button, switch, key, or mode dial attached to the digital camera 100, or a touch panel that also serves as the display unit 108. Commands from a user reach the control unit 101 via the operation unit 110.

[0023] Next, with reference to the drawings, the reason why luminance unevenness occurs within the plane when imaging is performed by moving the focus position during exposure using a rolling shutter will be described.

[0024] When the digital camera 100 is in manual exposure mode or aperture priority exposure mode, the user can set the aperture value for the digital camera 100 using the operation unit 110. Also, when the digital camera 100 is in auto mode or shutter speed priority, the digital camera 100 automatically determines the aperture value. The aperture value that the user sets in the digital camera 100 is called the "display aperture value" or "nominal aperture value".

[0025] Digital camera 100 instructs drive unit 102 via control unit 101 to set the aperture of optical system 103 to the aperture set by the user or the aperture value automatically determined by the digital camera. However, even if the aperture of optical system 103 becomes the displayed aperture value, the amount of light that imaging unit 104 actually receives through optical system 103 depends on the positional relationship between optical system 103 and imaging unit 104, so imaging unit 104 may not be able to receive an amount of light equivalent to the displayed aperture value. The amount of light that imaging unit 104 actually receives, expressed as an aperture value, is called the "effective aperture value." The actual aperture value is called the "effective aperture value."

[0026] However, a common feature of lenses is that the effective aperture value changes when the focus position is moved, and the actual aperture value of a lens may differ from the displayed aperture value depending on the state of the lens. The difference between the displayed aperture value and the effective aperture value depends on the position of the focus lens, that is, the focus position. Also, in depth stacking imaging, a rolling shutter is often used to capture a large number of images. Imaging with a rolling shutter has the feature that the exposure timing of each row is slightly shifted because pixel reset and pixel readout are performed sequentially for each row. The exposure time of the pixels in each row is basically the same.

[0027] Fig. 2 is a graph for explaining the relationship between the displayed aperture value, the effective aperture value, and the focal position in this embodiment. As shown in the graph in Fig. 2, in the optical system in this embodiment, when the focal position is at the closest point, the effective aperture value is larger than the displayed aperture value, and as the focal position approaches infinity, the effective aperture value approaches the displayed aperture value. However, whether the change in the effective aperture value depending on the focal position is linear or nonlinear, whether it increases or decreases monotonically, and the absolute amount of the difference from the displayed aperture value, etc., differ depending on the type of lens. The relationship between the effective aperture value and the focal position as shown in Fig. 2 is merely one example.

[0028] FIG. 3 is a graph for explaining the relationship between the effective aperture value change due to the change in the focus position and the exposure timing when the exposure time is not corrected in this embodiment. FIG. 4 is a graph for explaining the relationship between the effective aperture value change due to the change in the focus position of multiple images and the exposure timing when the exposure time is not corrected in this embodiment. When the imaging unit 104 performs imaging by continuously moving the focus position toward the infinity side using the driving unit 102 while the imaging unit 104 is exposing by the rolling shutter, the effective aperture value becomes smaller. Therefore, in FIG. 3, the Nth row (last row) of the pixel array becomes brighter than the first row, and luminance unevenness occurs within the plane due to the change in the focus position. Also, in FIG. 4, when multiple images for depth stacking are captured, the state of occurrence of luminance unevenness within the plane as shown in FIG. 3 is shown for all of the multiple images. If the focus area of ​​multiple images with luminance unevenness is extracted and synthesized into one image, the synthesis boundary becomes noticeable due to the luminance unevenness, and the synthesized image becomes rough.

[0029] Next, the focus stacking process in this embodiment when the exposure time is not corrected will be described below. Fig. 5 is a flowchart for explaining the focus stacking process in this embodiment.

[0030] In step S501, the user operates the operation unit 110 to set each parameter for imaging, such as exposure setting and focus bracketing setting. The control unit 101 calculates the exposure time from the shutter speed in response to the user's operation, and calculates the amount of movement of the focus position in the optical axis method based on the focus bracketing setting. Alternatively, the control unit 101 may set each parameter for imaging based on a predetermined setting, such as a default setting, instead of a user's operation immediately before imaging.

[0031] In step S502, the control unit 101 calculates an exposure time correction amount that reduces luminance unevenness. A method for calculating the exposure time correction amount will be described below with reference to the drawings. The control unit 101 calculates the amount of change in effective aperture value in capturing one image from the amount of movement of the focus position set in step S501 and the relationship between the focus position and effective aperture value as shown in FIG. 2. Note that information regarding the relationship between the focus position and effective aperture value as shown in FIG. 2 may be stored in advance in the ROM 105 or the like. Alternatively, in the case of a lens-mounted camera, the control unit 101 may read out information stored in the lens at the start of capturing an image.

[0032] FIG. 6 is a graph for explaining the relationship between pixel reset timing for each row and effective aperture value in this embodiment. The control unit 101 calculates the difference amount of the effective aperture value between the first row and the Nth row from the exposure time set in step S501 and the change amount of the effective aperture value of the optical system 103 in capturing the first image calculated in step S502, and converts the difference amount of the effective aperture value into an exposure time. The result obtained by the conversion by the control unit 101 here becomes the exposure time correction amount for reducing the brightness unevenness in the plane of the first image. The control unit 101 calculates the exposure time correction amount for reducing the exposure difference between the second row to the Nth row and the first row using the exposure difference between the first row and each of the second row to the Nth row (described as the "last row" in FIG. 6). Then, the control unit 101 applies the calculated exposure time correction amount to each of the second row to the Nth row, and captures the image with the corrected exposure time, which is expected to reduce the brightness unevenness in one image.

[0033] However, if the difference in the effective aperture value is extremely small, the effect of the correction is considered to be negligible. Therefore, the control unit 101 may compare the difference in the effective aperture value with a predetermined threshold value, and if the difference in the effective aperture value is equal to or less than the threshold value, the exposure time may not be corrected.

[0034] FIG. 7 is a graph for explaining the relationship between the pixel reset timing for each row and the effective aperture value after the exposure time is corrected in this embodiment. As shown in FIG. 7, the control unit 101 corrects the exposure time to reduce the difference in exposure between the second to Nth rows and the first row, so that the exposure time of each row is no longer the same. In the situation shown in FIG. 7 of this embodiment, the control unit 101 sequentially reduces the exposure time from the first row to the Nth row, but this is not limited to this. In addition, the control unit 101 may change the pixel reset timing as shown in FIG. 7 or may change the pixel readout timing to correct the exposure time for each row. Furthermore, in FIG. 7, the control unit 101 uses the exposure time of the first row as a reference and corrects the other rows to match the effective aperture value of the first row, but this is not limited to this and the exposure time of any row may be used as a reference.

[0035] Fig. 8 is a graph for explaining the relationship between the pixel reset timing for each row of a plurality of images and the effective aperture value after the exposure time is corrected in this embodiment. In the case shown in Fig. 8, the control unit 101 corrects the exposure time so that the effective aperture value of all rows of all images is matched with that of the first row of the first image. It is expected that such a correction reduces luminance unevenness in the composite image.

[0036] In step S503, the control unit 101 detects whether or not there is an image capture instruction from the user. If an image capture instruction is given by the user through operation of the operation unit 110, the process proceeds to step S504, and if there is no image capture instruction from the user, the process returns to step S502.

[0037] In step S504, the control unit 101 drives the driving unit 102 to perform focus driving based on the imaging conditions determined in step S501, to change the focus position to a focus position at which imaging will be performed in the next step S505.

[0038] In step S505, the imaging unit 104 captures an image at the focus position in the optical axis direction set in step S504, using the exposure time corrected by the correction amount determined in step S502. As described above, in this embodiment, the focus position is moved even during imaging (focus drive is not stopped). As shown in Fig. 8, in this embodiment, the focus position is moved while correcting the exposure time, and imaging is performed.

[0039] In step S506, the control unit 101 determines whether or not imaging has ended. Here, the control unit 101 uses, as a criterion for determining whether imaging has ended, for example, whether a predetermined number of captured images has been reached. Also, for example, the criterion is whether a predetermined image storage capacity has been reached. Also, the criterion is whether a predetermined focus range has been reached.

[0040] After the image capturing is completed, in step S507, the driving unit 102 stops the focus driving.

[0041] In step S508, the image processing unit 107 performs a depth stacking process on the captured image to generate a composite image. An example of a depth stacking method will be described. First, the control unit 101 calculates the amount of positional shift between two images to be combined. An example of the calculation method will be described below. First, the control unit 101 sets a plurality of blocks in one image. It is preferable that the control unit 101 sets each block to have the same size. Next, the control unit 101 sets a search range in the other image at the same position as each of the set blocks, with a range wider than the block. Finally, the control unit 101 calculates a corresponding point in each search range of the other image that has the smallest absolute difference (Sum of Absolute Difference, hereinafter referred to as SAD) between the first set block and the corresponding point. The system control unit 210 calculates the positional shift as a vector from the center of the first set block and the above-mentioned corresponding point. In calculating the corresponding points described above, the control unit 210 may use the sum of squared difference (hereinafter referred to as SSD) or normalized cross correlation (hereinafter referred to as NCC) in addition to SAD. Next, the control unit 101 calculates a conversion coefficient from the amount of positional deviation. The control unit 101 uses, for example, a projection transformation coefficient as the conversion coefficient. However, the conversion coefficient is not limited to the projection transformation coefficient, and an affine transformation coefficient or a simplified conversion coefficient of only horizontal and vertical shift may be used. Next, the image processing unit 107 calculates a contrast value for each image after the alignment. As an example of a method of calculating the contrast value, for example, the image processing unit 107 first calculates a luminance Y from the color signals Sr, Sg, and Sb of each pixel using the following (Equation 7). Y=0.299Sr+0.587Sg+0.114Sb...Formula (1)

[0042] Next, a contrast value I is calculated for a 3×3 pixel luminance Y matrix L using a Sobel filter as shown in the following equations (2), (3), and (4).

number

number

number

[0043] The above-mentioned method of calculating the contrast value is merely an example, and it is also possible to use an edge detection filter such as a Laplacian filter or a bandpass filter that passes a predetermined band as the filter to be used. Next, the image processing unit 107 generates a composite map. As a method of generating the composite map, the image processing unit 107 compares the contrast values ​​of pixels at the same position in each image, sets the composite ratio of the pixel with the highest contrast value to 100%, and sets the composite ratio of other pixels at the same position to 0%. The image processing unit 107 sets such composite ratios for all positions in the image. Finally, the image processing unit 107 replaces pixels according to the composite map to generate a composite image. Note that, when the composite ratio between adjacent pixels changes from 0% to 100% (or from 100% to 0%) with respect to the composite ratio calculated in this way, unnaturalness at the composite boundary becomes noticeable. Therefore, a filter having a predetermined number of pixels (number of taps) is applied to the composite map to prevent the composite ratio from changing suddenly between adjacent pixels.

[0044] According to this embodiment, by capturing images while moving the focus position during exposure, it is possible to shorten the capturing time and obtain a depth stacking image in which the roughness of the composite boundary caused by differences in luminance unevenness is reduced.

[0045] (Other embodiments) The above-described embodiment has been described on the assumption that the digital camera 100 performs focus stacking. In many cases, the digital camera 100 is also equipped with a function other than focus stacking, which is to capture only one image and record the captured image. As another embodiment for implementing the present invention, it is considered that whether to correct the exposure time is determined depending on whether to record only one image or to perform focus stacking.

[0046] When digital camera 100 performs imaging by moving the focus position during exposure, even if only one image is captured without performing depth stacking, brightness unevenness occurs within the plane of the single image. However, even if brightness unevenness occurs within the plane of a single image, it is less than the brightness unevenness of a composite image obtained by depth stacking. Therefore, when digital camera 100 performs imaging by moving the focus position during exposure, the exposure time may not be corrected as described above when recording only one image, and the exposure time may be corrected as described above when performing depth stacking.

[0047] In the above embodiment, the description has been given based on a personal digital camera, but as long as the camera is equipped with a depth stacking function, the present invention can also be applied to a mobile device, a smartphone, or a network camera connected to a server. Alternatively, part of the above-described processing may be performed by a mobile device, a smartphone, or a network camera connected to a server.

[0048] The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and run the program. Also, the present invention can be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0049] 100 Digital Camera 101 Control section 102 Drive unit 103 Optical system 104 Imaging unit 105 ROM 106 RAM 107 Image Processing Unit 108 Display section 109 Internal Memory 110 Operation section

Claims

1. An imaging means for capturing an image while changing a focus position; a correction unit that corrects an exposure difference in the image caused by a change in the focus position while capturing one of the images, The imaging apparatus according to claim 1, wherein the correction means does not perform correction if an exposure difference in the image caused by a change in the focus position is equal to or smaller than a predetermined threshold value.

2. Further comprising a synthesis means for synthesizing the multiple images captured by the imaging means to generate a synthetic image, The imaging device according to claim 1 , wherein the depth of field of the composite image is deeper than the depth of field of the plurality of images.

3. 3. The imaging apparatus according to claim 2, wherein the synthesizing means extracts in-focus areas of the plurality of images and performs the synthesis.

4. 4. The imaging apparatus according to claim 1, wherein the correction means performs the correction by changing an exposure time between images taken by the imaging means.

5. 5. The image pickup apparatus according to claim 4, wherein the correction means changes at least one of a reset timing and a read timing of each pixel of the image pickup means to change the exposure time.

6. 6. The imaging device according to claim 1, wherein the correction means corrects the effective aperture value so that the effective aperture values ​​when the imaging means captures each of the multiple images are the same, and corrects the effective aperture values ​​of the multiple images so that the effective aperture values ​​are the same.

7. When the imaging means captures an image, image signals are sequentially read out one by one from the imaging element; 7. The imaging apparatus according to claim 6, wherein the correction means corrects the effective aperture value when the imaging means reads out the image signals of the plurality of images so as to be uniform when the imaging means reads out the image signals of the portions of the images.

8. 8. The imaging device according to claim 7, wherein the portion of the imaging element is a row of a pixel array of the imaging element.

9. 9. The imaging apparatus according to claim 8, wherein the correction means adjusts effective aperture values ​​of the plurality of images based on the effective aperture value of any row of any image among the plurality of images.

10. 10. The imaging apparatus according to claim 9, wherein the correction means uses the effective aperture value of a first row of a first image of the plurality of images as the reference.

11. The optical system includes a rolling shutter.

11. The imaging apparatus according to claim 1, wherein the imaging means captures an image by receiving light through the optical system.

12. An imaging means for capturing an image while changing a focus position; a synthesis means for synthesizing the multiple images captured by the imaging means to generate a synthesized image; a correction unit that corrects an exposure difference in the image caused by a change in the focus position while capturing one of the images, If the exposure difference in the image caused by the change in the focus position is equal to or smaller than a predetermined threshold, the correction means does not perform correction; a depth of field of the composite image is greater than a depth of field of the plurality of images; 11. An imaging apparatus comprising: a first image capturing unit configured to capture an image of an image captured by an imaging device, the first image capturing unit configured to capture an image of an image captured by an imaging device;

13. An imaging step of capturing an image while continuously changing a focus position; a correction step of correcting an exposure difference caused by a change in the focus position while capturing one of the images in the imaging step, an exposure difference in the image caused by the change in the focus position in the correction step being equal to or smaller than a predetermined threshold value, no correction is performed.

14. An imaging step of capturing an image while changing a focus position; a synthesis step of synthesizing the plurality of images captured in the imaging step to generate a synthetic image; a correction step of correcting an exposure difference caused by a change in the focus position while one image is captured in the imaging step, If the exposure difference caused by the change in the focus position in the correction step is equal to or smaller than a predetermined threshold value, no correction is performed; a depth of field of the composite image is greater than a depth of field of the plurality of images; An imaging method, comprising: if the combining is performed in the combining step, performing the correction in the correcting step; and if the combining is not performed, not performing the correction in the correcting step.

15. A computer program for causing a computer to operate, A program causing a computer to execute each step of the imaging method according to claim 13 or 14.

16. A computer-readable recording medium having the program according to claim 15 recorded thereon.

Images