Camera device, camera method, and storage medium

Abstract

The present invention discloses an imaging device, an imaging method, and a storage medium. The imaging device includes: an imaging unit configured to capture an image while continuously changing a focus position during image capture; and a correction unit configured to correct exposure differences caused by changes in the focus position while the imaging unit captures an image.

Description

Technical Field

[0001] The present invention relates to an image pickup device that reduces exposure differences between images. Background Art

[0002] When photographing multiple subjects at widely varying distances, or when photographing subjects with a great depth of field, a lack of depth of field can result in only some or a portion of the subjects being in focus. To address this issue, Japanese Patent Application Laid-Open No. 2015-216532 discusses a focus stacking technique that captures multiple images at different focus positions, extracts only the focused area from the captured images, and synthesizes the focused areas into a single image, thereby generating a composite image in which the entire captured area is in focus.

[0003] Meanwhile, in order to reduce the processing time of depth synthesis as much as possible, International Publication No. 2012 / 117733 discusses a technique for continuously changing the focus position during exposure.

[0004] However, if imaging is performed while shifting the focus position during exposure as described above, the effective aperture value changes between the exposure timings of the upper and lower portions of the imaging element, resulting in uneven brightness within the image plane. If only the focused area is extracted from multiple images captured in this manner and synthesized into a single image, the boundaries between the synthesized areas may become noticeable due to the uneven brightness, potentially causing artifacts in the synthesized image. Summary of the Invention

[0005] The present invention is directed to reducing defects in a synthesized image when the focus position is moved during exposure in focus synthesis imaging.

[0006] According to one aspect of the present invention, an imaging apparatus includes an imaging unit configured to capture an image while continuously changing a focus position during image capture, and a correction unit configured to correct an exposure difference caused by a change in the focus position while the imaging unit captures one image.

[0007] In the configuration of the exemplary embodiment of the present invention, a depth synthesis image can be obtained in which defects at the boundaries between synthesis areas caused by brightness unevenness and noise level differences are reduced while reducing the imaging time for capturing multiple images with different focus positions.

[0008] Further features of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1is a block diagram illustrating a hardware configuration of a digital camera as an image pickup apparatus according to an exemplary embodiment of the present invention.

[0010] Figure 2 is a graph illustrating a relationship among a displayed aperture value, an effective aperture value, and a focus position according to an exemplary embodiment of the present invention.

[0011] Figure 3 is a graph illustrating a relationship between a change in an effective aperture value and an exposure timing due to a change in a focus position in a state where the exposure time is not corrected according to an exemplary embodiment of the present invention.

[0012] Figure 4 is a graph illustrating a relationship between a change in an effective aperture value and an exposure timing due to a change in a focus position in a plurality of images in a state where the exposure time is not corrected according to an exemplary embodiment of the present invention.

[0013] Figure 5 is a flowchart illustrating depth synthesis processing according to an exemplary embodiment of the present invention.

[0014] Figure 6 is a graph illustrating a relationship between a pixel reset timing and an effective aperture value for each row according to an exemplary embodiment of the present invention.

[0015] Figure 7 is a graph illustrating a relationship between a pixel reset timing and an effective aperture value for each row after exposure time correction according to an exemplary embodiment of the present invention.

[0016] Figure 8 is a graph illustrating a relationship between pixel reset timing and effective aperture values ​​for respective rows in a plurality of images after exposure time correction according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0017] Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings. The following exemplary embodiments are not intended to limit the present invention described in the claims, and all combinations of features described in the exemplary embodiments are not essential to the solutions of the present invention.

[0018] Figure 1 1 is an example of a block diagram illustrating the configuration of a digital camera as an imaging apparatus according to this exemplary embodiment. The digital camera 100 is capable of capturing still images, recording information regarding focus positions, calculating contrast values, and synthesizing images. The digital camera 100 is also capable of zooming in or out on captured and saved images or externally input images.

[0019] The control unit 101 is a signal processor such as a central processing unit (CPU) and a microprocessor unit (MPU). The control unit 101 reads a program stored in a read-only memory (ROM) 105, described below, to control each unit of the digital camera 100. For example, as described below, the control unit 101 issues commands to start and end image capture to the image capture unit 104, described below. Alternatively, the control unit 101 issues commands for image processing to the image processing unit 107, described below, based on the program stored in the ROM 105. Commands issued by the user are input into the digital camera 100 via the operation unit 110, described below, and reach the corresponding units of the digital camera 100 via the control unit 101.

[0020] The drive unit 102 includes a motor and the like, and mechanically operates the optical system 103 described below under the command of the control unit 101. For example, the drive unit 102 moves the position of a focus lens included in the optical system 103 based on the command of the control unit 101, thereby adjusting the focal length of the optical system 103.

[0021] The optical system 103 includes a zoom lens, a focus lens, and an aperture. The aperture is a mechanism for adjusting the amount of transmitted light. The focus position can be changed by changing the position of the lens.

[0022] The imaging unit 104 is an imaging element that photoelectrically converts an incident light signal into an electrical signal. For example, the imaging unit 104 may be a charge-coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, or the like. The imaging unit 104 may be provided with a motion image capture mode in which multiple temporally consecutive images are captured as frames of a motion image. The imaging unit 104 may measure the brightness of a subject via the optical system 103. An automatic exposure (AE) sensor or the like may be used instead of the imaging unit 104 for light metering.

[0023] The ROM 105 is a read-only nonvolatile memory as a recording medium, and stores an operation program of each block included in the digital camera 100 and parameters necessary for the operation of each block.

[0024] The RAM 106 is a writable volatile memory used as a temporary storage area for data output by operations of the respective blocks included in the digital camera 100 .

[0025] The image processing unit 107 performs various types of image processing, such as white balance adjustment, color interpolation, filtering, and synthesis, on the image data output from the imaging unit 104 or the image signal data recorded in the built-in memory 109 described below. The image processing unit 107 compresses the image signal data captured by the imaging unit 104 using a standard such as JPEG.

[0026] The image processing unit 107 includes an application-specific integrated circuit (ASIC) in which a circuit for performing specific processing is integrated. Alternatively, the control unit 101 may perform processing based on a program read from the ROM 105, whereby the control unit 101 performs some or all of the functions of the image processing unit 107. If the control unit 101 performs all of the functions of the image processing unit 107, the image processing unit 107 does not need to be provided as hardware.

[0027] The display unit 108 is a liquid crystal display or an organic electroluminescence (EL) display, and displays an image temporarily stored in the RAM 106 , an image stored in a built-in memory 109 described below, or a setting screen for the digital camera 100 .

[0028] The built-in memory 109 is an area for recording images captured by the imaging unit 104, images processed by the image processing unit 107, and information on a focus position in imaging. A memory card or the like may be used instead of the built-in memory 109.

[0029] The operation unit 110 includes, for example, buttons, switches, keys, or a mode dial attached to the digital camera 100, or a touch panel also serving as the display unit 108. Commands from the user reach the control unit 101 via the operation unit 110.

[0030] Next, the cause of occurrence of brightness unevenness in the plane of an image in the case where an image is captured while the focus position is moved during exposure using a rolling shutter will be described with reference to the drawings.

[0031] When the digital camera 100 is in manual exposure mode or aperture priority exposure mode, the user can set the aperture value for the digital camera 100 using the operation unit 110. When the digital camera 100 is in automatic mode or shutter speed priority mode, the digital camera 100 automatically determines the aperture value. The aperture value set by the user for the digital camera 100 is referred to as a displayed aperture value or a nominal aperture value.

[0032] The digital camera 100 instructs the drive unit 102, via the control unit 101, to change the aperture of the optical system 103 to an aperture value set by the user or automatically determined by the digital camera 100. However, even when the aperture of the optical system 103 is changed to the displayed aperture value, the actual amount of light received by the imaging unit 104 through the optical system 103 depends on the positional relationship between the optical system 103 and the imaging unit 104. Therefore, the imaging unit 104 may not receive the amount of light corresponding to the displayed aperture value. The actual amount of light received by the imaging unit 104, expressed in aperture values, is the effective aperture value. The actual aperture value is referred to as the effective aperture value.

[0033] However, lenses generally have the characteristic of changing their effective aperture value as the focus position shifts, and the actual aperture value of a lens may differ from the displayed aperture value depending on the lens's state. The difference between the displayed aperture value and the effective aperture value depends on the position of the focus lens, that is, the focus position. In focus synthesis photography, a rolling shutter is typically used to capture a large number of images. In photography using a rolling shutter, pixel resetting and pixel reading are performed sequentially for each row, thereby gradually shifting the exposure timing row by row. The exposure time for each row of pixels is essentially the same.

[0034] Figure 2 : is a graph showing the relationship between the aperture value, the effective aperture value, and the focus position according to the display of the present exemplary embodiment. Figure 2 As shown in the graph, in the optical system 103 according to this exemplary embodiment, when the focus position is at the minimum distance end, the effective aperture value is greater than the displayed aperture value. As the focus position approaches the infinity end, the effective aperture value approaches the displayed aperture value. However, whether the effective aperture value changes linearly or nonlinearly with the focus position, whether it increases or decreases monotonically, and the absolute difference from the displayed aperture value vary depending on the lens type. Figure 2 The shown relationship between the effective aperture value and the focus position is merely an example.

[0035] Figure 3 is a graph illustrating the relationship between a change in the effective aperture value due to a change in the focus position and the exposure timing in a state where the exposure time is not corrected according to the present exemplary embodiment. Figure 4 Graph showing the relationship between the change in the effective aperture value and the exposure timing due to the change in the focus position in a plurality of images without correcting the exposure time according to the present exemplary embodiment. If the imaging unit 104 performs imaging while continuously moving the focus position toward the infinite end using the drive unit 102 during exposure by the rolling shutter, the effective aperture value becomes smaller. Therefore, Figure 3 In the pixel array, the Nth row (the last row) is brighter than the first row, resulting in uneven brightness in the image plane due to the change in focus position. Figure 4 When taking multiple depth-of-field composite images, all images appear as follows in the plane. Figure 3 The brightness shown is uneven. If you extract focused areas from multiple images with uneven brightness and combine them into a single image, the boundaries between the combined areas become noticeable due to the uneven brightness, causing defects in the combined image.

[0036] Next, the depth of field synthesis processing in a state where the exposure time is not corrected according to the present exemplary embodiment will be described. Figure 5 is a flowchart illustrating depth-of-field synthesis processing according to the present exemplary embodiment.

[0037] In step S501, the user operates the operation unit 110 to set imaging parameters, such as exposure and focus bracketing. In response to the user operations, the control unit 101 calculates the exposure time based on the shutter speed and the amount of focus position shift along the optical axis based on the focus bracketing settings. Alternatively, the control unit 101 may set imaging parameters based on predetermined settings, such as default settings, rather than in response to user operations immediately before imaging.

[0038] In step S502, the control unit 101 calculates an exposure time correction amount for reducing uneven brightness. The following describes a method for calculating the exposure time correction amount with reference to the accompanying drawings. The control unit 101 calculates the exposure time correction amount based on the movement amount of the focus position set in step S501 and the Figure 2 The relationship between the focus position and the effective aperture value is shown, and the change in the effective aperture value during shooting one image is calculated. Figure 2 Information on the relationship between the focus position and the effective aperture value shown may be stored in advance in the ROM 105, etc. Alternatively, in the case of a camera with a lens mounted thereon, the control unit 101 may read information stored in a storage unit of the lens.

[0039] Figure 6 is a graph showing the relationship between the pixel reset timing and the effective aperture value of each row according to the present exemplary embodiment. The control unit 101 calculates the difference in effective aperture value between the first row and the Nth row based on the exposure time set in step S501 and the amount of change in the effective aperture value in the optical system 103 during the shooting of the first image calculated in step S502, and converts the difference in effective aperture value into exposure time. The result obtained by the conversion by the control unit 101 constitutes an exposure time correction amount for reducing uneven brightness in the first image plane. The control unit 101 uses the first row and the second to Nth rows ( Figure 6 The control unit 101 then applies the calculated exposure time correction amount to each of the second to Nth rows and captures images using the corrected exposure time, thereby reducing uneven brightness in one image.

[0040] However, if the difference in effective aperture values ​​is extremely small, the effect of the correction is considered to be slight. Therefore, the control unit 101 may compare the difference in effective aperture values ​​with a predetermined threshold. If the difference in effective aperture values ​​is less than or equal to the threshold, the control unit 101 does not need to perform exposure time correction.

[0041] Figure 7Graph showing the relationship between the pixel reset timing and the effective aperture value of each row after exposure time correction according to the present exemplary embodiment. Figure 7 As shown, the control unit 101 corrects the exposure time to reduce the exposure difference between the second to Nth rows and the first row, so that the exposure time of each row is different. Figure 7 In the case of the exemplary embodiment shown in FIG. 1 , the control unit 101 sequentially reduces the exposure time from the first row to the Nth row, but the present invention is not limited thereto. In order to correct the exposure time of each row, the control unit 101 may change the exposure time as follows: Figure 7 The pixel reset timing shown or the timing for reading the pixels can be changed. Figure 7 In the embodiment, the control unit 101 uses the exposure time of the first row as a reference and corrects the exposure time of other rows with reference to the effective aperture value of the first row. However, the present invention is not limited thereto, and the control unit 101 may use the exposure time of any row as a reference.

[0042] Figure 8 Graph showing the relationship between the pixel reset timing and the effective aperture value for each row in a plurality of images after exposure time correction according to the present exemplary embodiment. Figure 8 In the case shown, the control unit 101 corrects the exposure time so that the effective aperture value of all rows in all images matches the effective aperture value of the first row in the first image. By such correction, it is expected that the brightness unevenness in the composite image will be reduced.

[0043] In step S503, the control unit 101 detects whether a shooting instruction has been issued by the user. If the shooting instruction has been issued by the user by operating the operation unit 110 ("YES" in step S503), the process proceeds to step S504. If the shooting instruction has not been issued by the user ("NO" in step S503), the process returns to step S502.

[0044] In step S504 , the control unit 101 drives the drive unit 102 to perform focus driving for moving the focus position to a focus position to be used for imaging next in step S505 , based on the imaging conditions set in step S501 .

[0045] In step S505, the imaging unit 104 captures an image at the focus position in the optical axis direction set in step S504 using the exposure time corrected by the correction amount determined in step S502. As described above, the focus position is moved during imaging according to the present exemplary embodiment (without stopping the focus drive). Figure 8 As shown, in the present exemplary embodiment, imaging is performed in a state where the focus position is moved while the exposure time is corrected.

[0046] In step S506, the control unit 101 determines whether image capture is complete. Here, as a criterion for determining whether image capture is complete, the control unit 101 uses, for example, the following condition: a predetermined number of captured images has been reached. Otherwise, the control unit 101 uses, for example, the following condition: a predetermined image storage capacity has been reached. Otherwise, the control unit 101 uses, for example, the following condition: a predetermined focus range has been reached.

[0047] After image capture ends (YES in step S506 ), in step S507 , the drive unit 102 stops focus driving.

[0048] In step S508, the image processing unit 107 performs depth synthesis processing on the captured image to generate a composite image. An example of a method for depth synthesis will be described below. First, the control unit 101 calculates the amount of positional gap between the two images to be synthesized. An example of the calculation method will be described below. First, the control unit 101 assigns multiple blocks to one image. Preferably, the control unit 101 assigns blocks of the same size. Next, the control unit 101 assigns a search range larger than the size of these blocks to the other image at positions corresponding to these assigned blocks. Finally, the control unit 101 calculates corresponding points within the search range in the other image so that the sum of absolute differences (hereinafter referred to as SAD) with respect to the brightness of the assigned blocks is minimized. The system control unit 210 calculates the positional gap as a vector based on the centers of the assigned blocks and the corresponding points. When calculating these corresponding points, the system control unit 210 may use the sum of squared differences (hereinafter referred to as SSD) or normalized cross correlation (hereinafter referred to as NCC) instead of the SAD. Next, the control unit 101 calculates transform coefficients based on the amount of positional gap. The control unit 101 uses, for example, projective transform coefficients as the transform coefficients. However, the transformation coefficients are not limited to the projective transformation coefficients and may be affine transformation coefficients or simplified transformation coefficients for horizontal and vertical shifting. The image processing unit 107 then calculates the contrast value of the aligned images. As an example of a method for calculating the contrast value, the image processing unit 107 first calculates the brightness Y from the color signals Sr, Sg, and Sb of the pixel by using the following (Formula 1).

[0049] Y=0.299Sr+0.587Sg+0.114Sb Formula (1)

[0050] Next, the image processing unit 107 calculates the contrast value I by using a Sobel filter for the matrix L of luminance Y of 3×3 pixels as described in the following formulas (2), (3), and (4):

[0051] [Mathematical Operation 1]

[0052]

[0053] [Mathematical Operation 2]

[0054]

[0055] [Mathematical Operation 3]

[0056]

[0057] The above-described method for calculating contrast values ​​is merely an example. For example, an edge detection filter such as a Laplacian filter or a bandpass filter that passes a predetermined frequency band may be used instead. The image processing unit 107 then generates a composite image. As a method for generating the composite image, the image processing unit 107 compares the contrast values ​​of pixels at the same location in each image, sets the composite ratio of the pixel with the highest contrast value to 100%, and sets the composite ratio of the other pixels at the same location to 0%. The image processing unit 107 sets this composite ratio at all locations in the image. Finally, the image processing unit 107 replaces pixels based on the composite image to generate a composite image. If the composite ratio between adjacent pixels calculated in this way changes from 0% to 100% (or vice versa), the boundaries between composite areas become noticeably unnatural. Therefore, a filter with a predetermined number of pixels (taps) is applied to the composite image so that the composite ratio does not change dramatically between adjacent pixels.

[0058] In the present exemplary embodiment, by performing imaging while moving the focus position during exposure, a depth synthesis image can be obtained in which defects in the boundaries between synthesis areas caused by brightness unevenness are reduced in a state in which the imaging time is shortened.

[0059] In the exemplary embodiment described above, the digital camera 100 performs focus synthesis as a prerequisite. In many cases, the digital camera 100 has the function of capturing and recording a single image in addition to the focus synthesis function. In another exemplary embodiment for implementing the present invention, whether to correct the exposure time depends on whether to record a single image or perform focus synthesis.

[0060] If the digital camera 100 captures a single image without performing focus stacking while shifting the focus position during exposure, brightness unevenness will appear within the plane of the single image. However, the brightness unevenness within the plane of the single image is less than that of the composite image after focus stacking. Therefore, if the digital camera 100 captures images while shifting the focus position during exposure, the digital camera 100 may not correct the exposure time as described above when recording only a single image, but may correct the exposure time as described above when performing focus stacking.

[0061] The above exemplary embodiments are described based on a personal digital camera. However, the exemplary embodiments are also applicable to mobile devices, smartphones, or webcams connected to a server, as long as they have a depth synthesis function. Alternatively, some of the above-described processing can be performed by the mobile device, smartphone, or webcam connected to the server.

[0062] The embodiments of the present invention may also be implemented by providing a program for implementing one or more functions of the exemplary embodiments described above to a system or device via a network or storage medium, and having one or more processors in a computer of the system or device read and execute the program. The present invention may also be implemented by a circuit (e.g., an application-specific integrated circuit (e.g., ASIC)) that implements one or more functions.

[0063] Other embodiments

[0064] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0065] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A camera device, comprising: an image capturing unit configured to capture an image while continuously changing a focus position during image capturing; as well as a correction unit configured to correct exposure differences caused by changes in focus position during exposure while the image pickup unit captures one image using a rolling shutter, and If the exposure difference caused by the change in the focus position is less than or equal to a predetermined threshold, the correction unit does not perform the correction, The correction unit corrects the exposure time converted from the exposure difference between the second to Nth rows and the first row in the captured image.

2. The imaging device according to claim 1, wherein The correction unit performs the correction by changing an exposure time while the image pickup unit captures the one image.

3. The imaging device according to claim 2, wherein: The correction unit changes the exposure time by changing at least one of a timing for resetting pixels and a timing for reading pixels of the image pickup unit.

4. The imaging device according to claim 1, in, The imaging unit sequentially reads image signals from various parts of the image sensor, and The correction unit performs correction to make the brightness of the image signal from each portion uniform. The imaging device according to claim 4 , wherein: The exposure time of pixels included in each portion of the image sensor is the same. The imaging device according to claim 4 , wherein: The sections of the image sensor are rows of the pixel array of the image pickup element.

7. The imaging device according to claim 1, in, The camera unit captures a plurality of images with different focus positions, and The exposure difference between the multiple images corrected by the correction unit is smaller than the difference before correction.

8. The imaging device according to claim 7, further comprising a synthesis unit configured to synthesize the plurality of images to generate a synthesized image. in, The depth of field of the composite image is deeper than the depth of field of each of the multiple images.

9. The imaging device according to claim 8, wherein: The synthesis unit performs synthesis by extracting a focus area from each of the plurality of images.

10. The imaging device according to claim 7, wherein: The correction unit performs correction based on an effective aperture value when each of the plurality of images is captured.

11. The imaging device according to claim 7, wherein: The correction unit corrects the exposure time to match the effective aperture value of the multiple images with the effective aperture value of any row in any one of the multiple images as a reference.

12. The imaging device according to claim 11, wherein: The correction unit uses the effective aperture value of the first row in the first image of the plurality of images as a reference.

13. The imaging device according to any one of claims 1 to 12, further comprising an optical system including a rolling shutter. in, The image capturing unit performs image capturing by receiving light from the optical system using a rolling shutter, wherein pixel resetting and pixel reading of an image sensor are sequentially performed for each row of pixels.

14. A camera device, comprising: an image pickup unit configured to capture a plurality of images having different focus positions and to continuously change the focus position during the capturing of each of the plurality of images; a synthesis unit configured to synthesize a plurality of images to generate a synthesized image; as well as a correction unit configured to correct an exposure time of a captured image based on an exposure difference between a reference pixel row and other pixel rows in a pixel array caused by a change in a focus position while the image capturing unit captures an image; Wherein, when the synthesis unit performs the synthesis, the correction unit performs the correction, and Wherein, when the synthesis unit does not perform the synthesis, the correction unit does not perform the correction.

15. The imaging device according to claim 14, wherein: The depth of field of the composite image is deeper than the depth of field of each of the plurality of images.

16. The imaging device according to claim 14, wherein: The exposure differences among the plurality of images after correction by the correction unit are smaller than the differences before correction.

17. A method for photographing, comprising: During image capture, images are captured while continuously changing the focus position; as well as Correcting exposure differences caused by changes in focus position during exposure while capturing one image using a rolling shutter, and If the exposure difference caused by the change in focus position is less than or equal to a predetermined threshold, no correction is performed, The exposure time converted from the exposure difference between the second to Nth rows and the first row in the captured image is corrected.

18. A method for photographing, comprising: capturing a plurality of images having different focus positions, and continuously changing the focus position during capturing each of the plurality of images; synthesizing the plurality of images to generate a composite image; as well as Correcting the exposure time of a captured image based on an exposure difference between a reference pixel row and other pixel rows in a pixel array caused by a focus position change when capturing an image during shooting. wherein, in the case where synthesis is performed in the synthesizing step, correction is performed in the correcting step, and However, when synthesis is not performed in the synthesis step, correction is not performed in the correction step. 19 . A non-transitory computer-readable storage medium storing a program for causing a computer of an apparatus to execute the imaging method according to claim 17 .

Images