Imaging device, its control method, and program

Abstract

To provide a suitable composite image with reduced noise and reduced image blur.SOLUTION: An imaging apparatus that includes an imaging unit and synthesizes a plurality of images obtained by continuous imaging by the imaging unit to generate a composite image, a correction unit that detects the amount of vibration of the imaging apparatus and corrects the image blur by changing the position of a correction member on the basis of the amount of vibration, an acquisition unit that acquires a correctable amount representing the degree to which correction by the correction unit is possible from the current position of the correction member and the amount of vibration, a determination unit that determines an exposure time for the next image on the basis of the correctable amount acquired by the acquisition unit when shooting for generating the composite image is being performed, and a synthesizing unit that synthesizes the plurality of images obtained by imaging on the basis of the exposure time determined by the determining unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technique for synthesizing a plurality of captured images.

Background Art

[0002] Conventionally, there is known a technique for generating a synthesized image with less noise while preventing image blur by capturing a plurality of images with an exposure time that does not cause image blur and performing image synthesis while correcting the positional deviation of the obtained plurality of images.

[0003] For example, Patent Document 1 discloses a technique for detecting shake applied to a lens and setting the exposure time based on the detection result of the shake in a camera system that divides the set exposure time and synthesizes the obtained plurality of images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] % However, in the method disclosed in Patent Document 1 described above, since the divided exposure time is set uniformly, if there is a change in the shake state during the capture of a plurality of images, the exposure time may become inappropriate. For example, the technical ability to hold the camera steadily varies among individuals, and therefore, some photographers may not be able to maintain a certain degree of shake during imaging.

[0006] The present invention has been made in view of the above problems, and aims to provide a technique capable of obtaining a suitable synthesized image with less noise and suppressed image blur.

Means for Solving the Problems

[0007] To solve this problem, for example, the imaging apparatus of the present invention has the following configuration. That is, An imaging device having an imaging means, which performs continuous shooting with the imaging means to obtain a plurality of images used to generate a composite image, A correction means that detects the amount of shake of the imaging device and corrects image shake by changing the position of a correction member based on the amount of shake, An acquisition means for acquiring a correctable amount representing the degree to which correction by the correction means is possible, based on the current position of the correction member and the amount of runout, When continuous shooting is performed to obtain the plurality of images used to generate the composite image, the correctionable amount obtained by each shooting is used based on the correctionable amount acquired by the acquisition means before each shooting of the plurality of images obtained by each shooting. each A determination means for determining the composite ratio when generating the aforementioned composite image, to have death, The determination means increases the composite ratio for images with lower ISO sensitivity, which is determined based on the brightness information and the correctable amount among the multiple images. do. [Effects of the Invention]

[0008] According to the present invention, a suitable composite image with suppressed image shake and low noise can be obtained. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram of the imaging device in the embodiment. [Figure 2] A flowchart illustrating the procedure for generating a composite image in the embodiment. [Figure 3] This figure illustrates the calculation of the amount of shake correction possible and the setting of imaging conditions in the embodiment. [Figure 4] A flowchart illustrating the procedure for the alignment process in the embodiment. [Figure 5] A diagram illustrating the method for setting the synthesis ratio in the embodiment. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] Figure 1 is a block diagram showing the configuration of the imaging device 100 in this embodiment. Typical imaging devices include digital cameras and digital video cameras, but any device with imaging capabilities, such as a smartphone or tablet PC, may also be used. In addition, Figure 1 shows an interchangeable lens imaging device, but an integrated lens imaging device may also be used.

[0012] In Figure 1, the shutter unit 105 is a mechanical focal-plane shutter and is equipped with a front curtain blade and a rear curtain blade. When not shooting and during video recording, the shutter unit 105 moves its front and rear curtain blades to a position retracted from the optical axis of the image, allowing the light beam to pass to the image sensor 106. When shooting, the shutter unit 105 allows the light beam to pass through by performing an exposure run, where the front curtain blade moves from the light-shielding position to the exposure position. After the set exposure time (shutter seconds) has elapsed, the rear curtain blade moves from the exposure position to the light-shielding position, completing the capture of one image data.

[0013] The above describes exposure control that adjusts the amount of light using the mechanical front curtain and mechanical rear curtain of the shutter unit 105. Alternatively, it is also possible to employ an electronic front curtain and mechanical rear curtain exposure control method that combines an electrical reset with the rear curtain of the shutter unit 105.

[0014] The subject image formed after passing through an interchangeable-lens or integrated-lens imaging optical system is imaged onto the light-receiving surface of an image sensor 106, such as a CCD (charge-coupled device) or a CMOS sensor (complementary metal-oxide-semiconductor). The image sensor 106 converts the amount of light imaged onto the light-receiving surface into an electrical signal. In this embodiment, the image sensor 106 has a mechanism for optically correcting image shake by moving it in a rotational direction around the optical axis and in a direction perpendicular to the optical axis. Alternatively, image shake may be optically corrected by moving the lens included in the imaging optical system in a direction perpendicular to the optical axis, or by combining the lens and the image sensor 106. The target amount of movement of the lens when correcting image shake using a lens, and the target amount of movement when correcting image shake using a combination of the lens and the image sensor 106, can be determined by known methods. In addition, other known methods may be used as methods for optically correcting image shake. In either case, the calculation of the amount of shake correction possible and the determination of the exposure time according to the amount of shake correction possible can be performed based on the same approach as in steps S204 and S205 described later.

[0015] The AD converter 107 performs noise reduction, gain adjustment, and AD conversion processing on the electrical signal output from the image sensor 106. The timing generator 108 generates and outputs signals related to the drive timing of the image sensor 106 and the output timing of the AD converter 107, and controls them according to the commands of the camera control unit 115.

[0016] The image processing circuit 109 performs pixel interpolation, color conversion, and other processing on the image data output from the AD converter 107, and then stores the processed image data in the internal memory 110. The display unit 111 displays the image data held in the internal memory 110 along with shooting information and other data. The compression / decompression processing unit 131 performs compression or decompression processing on the data stored in the internal memory 110 according to the image format. The storage memory 113 also stores various data such as parameters. The operation unit 114 is a user interface for the user to perform various menu operations and mode switching operations, and is composed of various switches, buttons, touch panels, etc.

[0017] The camera control unit 115 is composed of an arithmetic unit such as a CPU (Central Processing Unit). Then, the camera control unit 115 executes various control programs stored in the internal memory 110 according to the user's operation by the operation unit 114. The control programs are programs for performing, for example, shake correction control, automatic exposure control, and automatic focus adjustment control. In the case of an interchangeable-lens photographing apparatus, the camera control unit 115 performs information transmission between the photographing apparatus and the lens via the communication unit 116.

[0018] The shutter drive unit 128 drives the shutter unit 105. The luminance signal detection unit 130 detects the signal read from the imaging element 106 and passed through the AD converter 107 as the luminance of the subject and the scene.

[0019] The exposure control unit 129 calculates the exposure time, which is one of the exposure values, based on the luminance information obtained by the luminance signal detection unit 130, and communicates the calculation result to the shutter drive unit 128. Also, the exposure control unit 129 simultaneously performs control to amplify the imaging signal read from the imaging element 106. The amplification factor at this time corresponds to the ISO sensitivity, and automatic exposure control (AE control) is performed by adjusting the aperture value of the aperture of the photographing optical system, the exposure time, and the ISO sensitivity. Although details will be described later, the exposure control unit 129 also controls the shutter drive unit 128 based on information from the imaging element position detection unit 120 and the shake detection unit 112.

[0020] The shake detection unit 112 detects shake and vibration applied to the imaging apparatus 100. Generally, a gyro sensor (hereinafter simply referred to as a gyro) is used as a sensor for detecting vibrations such as shake and vibration, and detects the angular velocity of shake and vibration.

[0021] The image sensor drive unit 121 drives the image sensor 106. The image sensor position detection unit 120 detects the position of the image sensor 106, which is driven in a direction perpendicular to the optical axis. The image sensor drive unit 121 drives the image sensor 106. The image sensor PID control unit 122 performs PID control (ratio control, integral control, derivative control) on the deviation between the target movement amount of the image sensor 106 and the current position detected by the image sensor position detection unit 120. Since PID control is a common technique, a detailed explanation is omitted.

[0022] Next, we will explain how to calculate the target movement amount of the image sensor 106.

[0023] The position of the image sensor 106 can be controlled by converting the angular velocity detected by the vibration detection unit 112 into an angle, which has one more dimension. As a way to increase the dimension of the angular velocity by one, there are methods such as integration processing by the integration unit 125 or low-pass filter processing (LPF processing), but in this embodiment, integration processing is used.

[0024] The shake correction amount calculation unit 124 multiplies the output value of the integration unit 125 by the lens sensitivity information acquired by the communication unit 116 to calculate the target movement amount of the image sensor 106 to correct image shake. The lens sensitivity information is a value that changes depending on the focal length of the lens and the distance to the subject, and the optimal shake correction amount is weighted according to each lens and each lens condition.

[0025] Next, the method for generating the composite image in this embodiment will be described.

[0026] Figure 2 is a flowchart illustrating the composite image generation process in this embodiment. The composite image generation process by continuous shooting in this embodiment is executed when the user operates the operation unit 114 to select a composite mode. The camera control unit 115 executes a program stored in the internal memory 110 to control the image processing circuit 109, etc., thereby realizing this composite image generation process. The processing program may be recorded on a computer-readable recording medium, for example.

[0027] First, in S201, the camera control unit 115 sets imaging conditions such as the exposure time for the subject to be imaged. The imaging conditions set here are generally those that are determined according to the focal length of the lens, such as an exposure time that does not cause camera shake.

[0028] In S202, the camera control unit 115 starts capturing multiple images. The number of images to be captured to create a composite image can be set in advance by the user via the operation unit 114. For the sake of demonstration, this explanation will continue assuming that four images are captured.

[0029] In S203, the camera control unit 115 determines whether or not the acquisition of all images has been completed. If the camera control unit 115 determines that the acquisition of the target number of images (four in this embodiment) has been completed, it proceeds to S207; otherwise, it proceeds to S204.

[0030] In S204, the camera control unit 115 controls the shake correction amount calculation unit 124 to calculate the amount of shake correction possible. The calculation of the amount of shake correction possible will be described later.

[0031] In S205, the camera control unit 115 sets the imaging conditions for the subsequent imaging according to the amount of shake correction that can be calculated in S204. The setting of the imaging conditions will be described later.

[0032] In S206, the camera control unit 115 controls the image sensor 106 and the exposure control unit 129 to perform imaging according to the imaging conditions set in S205. The image obtained from imaging is temporarily stored in the internal memory 110. Then, the camera control unit 115 returns to processing in S203.

[0033] After repeating the above process and storing the desired number of images in the internal memory 110, the camera control unit 115 performs alignment of each image stored in the internal memory 110 in S207. Then, in S208, the camera control unit 115 generates a composite image using the aligned images. The generated composite image is stored as an image file in a storage medium (not shown), for example.

[0034] Next, with reference to Figures 3(a) to (d), the calculation of the shake correction amount in step S204, the setting of imaging conditions in step S205, and the imaging in step S206 in this embodiment will be explained in more detail.

[0035] In Figures 3(a) to (d), the rightward direction of the horizontal axis represents the passage of time, starting from the time the first image was captured. The vertical axis represents the time change in the amount (position) of drive of the image sensor 106 for correcting image shake. The line in the center of the vertical axis represents the position of the image sensor 106 when it is not driven (at the drive center position), while the lines at the top and bottom represent the position of the image sensor 106 when it is driven to its limit.

[0036] Figure 3(a) shows the state before the exposure of the first image begins, with the position of the image sensor 106 at this time being on the center line. The camera control unit 115 calculates the amount of shake correction that can be performed at this time based on the distance from the current position of the image sensor 106 to the nearest boundary of the range in which the image sensor 106 can move (shortest distance 31 to the upper or lower end) and the amount of shake detected by the shake detection unit 112. Once the amount of shake correction that can be performed is calculated, the camera control unit 115 modifies the conditions for the first imaging so that the exposure time becomes longer as the amount of shake correction that can be performed increases.

[0037] Here, when d is the distance between the current position of the image sensor 106 and the closer end of the position driveable range (either the upper or lower end), and R is the amount of shake detected by the shake detection unit 112, the camera control unit 115 calculates the amount of shake correction C using a function f(d,R) with d and R as arguments. An example of the function f(d,R) is as follows. C = f(d, R) = α × d - β × R Here, α and β are positive coefficients. As shown in the above equation, the amount of motion compensation C increases as the distance d increases and the amount of motion R decreases. Conversely, the amount of motion compensation C decreases as the distance d decreases and the amount of motion R increases.

[0038] Alternatively, the amount of shake compensation C can be determined by referring to the table corresponding to the function f(d, R) described above.

[0039] Figure 3(b) shows the state before exposure of the second image begins, with the image sensor 106 positioned near the top edge. In this case, the distance 32 from the image sensor 106 to the top edge is small, leaving little margin for shake correction, and thus the calculated amount of shake correction possible becomes small. As a result, the camera control unit 115 changes the second imaging conditions to shorten the exposure time.

[0040] Figures 3(c) and 3(d) show the state before exposure for the third and fourth images begins, respectively, and the imaging conditions for each image are determined in the same way as described above.

[0041] A concrete example of the above is shown below. Let Ts be the exposure time set in S201. Also, the maximum value of the possible shake correction amount C is defined as Cmax, and the minimum value as Cmin.

[0042] In this case, the camera control unit 115 determines the exposure time T in S205 based on the exposure time Ts set in S201, for example, according to the following formula. T = Ts + {C - Cmax} According to the above formula, the larger the amount of shake correction C (the closer it is to Cmax), the closer the exposure time T approaches the exposure time Ts set in S205. On the other hand, the smaller the amount of shake correction C (the closer it is to Cmin), the smaller the exposure time T becomes (provided that T > 0).

[0043] Furthermore, if the subject being imaged is a moving object, extending the exposure time unnecessarily tends to increase the degree of image shake of the moving object. Therefore, if the presence of a moving object is detected, the upper limit of the exposure time set by the camera control unit 115 may be reduced compared to when no moving object is present. Alternatively, the exposure time T may be selected from a plurality of preset exposure times according to the amount of shake correction C. For example, the exposure time Ts may be set as the first exposure time, and the exposure time that is a predetermined time shorter than the exposure time Ts may be set as the second exposure time. If the amount of shake correction C is greater than or equal to a threshold, the first exposure time may be selected; if the amount of shake correction C is less than the threshold, the second exposure time may be selected. In addition, upper and lower limits may be set so that the exposure times do not differ drastically from one image to another, keeping the exposure times within a predetermined range. After determining the exposure time in this manner, at least the ISO sensitivity is set based on the brightness information. The aperture value is set along with the ISO sensitivity unless manually set by the user. A known method may be used to set the ISO sensitivity and aperture value based on the brightness information.

[0044] Next, we will explain the alignment process in S207. Please note that when performing this process, the desired number of images must already be acquired. The following explanation will refer to the flowchart in Figure 4.

[0045] In S401, the image processing circuit 109 acquires a reference image for alignment from each of the images stored in the internal memory 110. The reference image for alignment is, for example, the one captured earliest. The images other than the reference image are then used as images to be corrected.

[0046] Next, in S402, the image processing circuit 109 acquires one of the images to be corrected for the alignment process. The image to be corrected is an image other than the reference image acquired in S401 that has not yet undergone alignment processing. If the image processing circuit 109 uses the image acquired earliest as the reference image, then it should acquire the images to be corrected sequentially in the order they were acquired.

[0047] In S403, the image processing circuit 109 calculates the amount of positional shift between the reference image and the image to be corrected. An example of the calculation method is described below.

[0048] First, the image processing circuit 109 sets up multiple blocks for the reference image. It is preferable that each block be set to the same size. Next, the image processing circuit 109 sets a search range for the image to be corrected that is wider than the reference image block, at the same position as each block in the reference image. Finally, the image processing circuit 109 calculates the corresponding point in each search range of the image to be corrected that minimizes the sum of absolute differences in brightness (SAD) between the image and the reference image block. The image processing circuit 109 calculates the positional shift between images as a vector from the center of the reference image block and the aforementioned corresponding point. In calculating the corresponding point described above, the image processing circuit 109 may use methods other than SAD, such as the sum of squared differences (SSD) or normalized cross-correlation (NCC).

[0049] In S404, the image processing circuit 109 calculates a transformation coefficient from the positional shift between the reference image and the target image. The image processing circuit 109 uses, for example, a projection transformation coefficient as the transformation coefficient. However, the transformation coefficient is not limited to only projection transformation coefficients; affine transformation coefficients or simplified transformation coefficients consisting only of horizontal and vertical shifts may also be used.

[0050] In S405, the image processing circuit 109 uses the conversion coefficient calculated in S404 to perform a transformation on the image to be corrected and generates a corrected image with the amount of displacement corrected.

[0051] For example, the image processing circuit 109 can deform the target image according to the following equation (1) and generate a corrected image I'.

[0052]

number

[0053] In equation (1), (x', y') represents the coordinates after deformation, and (x, y) represents the coordinates before deformation. Matrix A represents the deformation coefficients calculated by the image processing circuit 109 in S404.

[0054] In S406, the image processing circuit 109 determines whether alignment has been performed on all images except the reference image. If it determines that alignment has been performed on all images except the reference image, the image processing circuit 109 terminates the process shown in this flowchart. If it determines that there are unprocessed images, the image processing circuit 109 returns to processing in S402.

[0055] Once the alignment process is completed for all the images captured as described above, the image synthesis process in S208 will be performed.

[0056] In the image compositing process, a reference image is combined with multiple corrected images that have undergone alignment processing other than the reference image. Here, the image compositing process is performed by averaging the signals of the corresponding coordinates of each image, and random noise in the image is reduced by averaging.

[0057] The blending ratio of each image is determined according to the exposure time of each image. In this embodiment, as shown in Figure 5, the blending ratio is set to be higher for images taken with longer exposure times. This makes it possible to reduce the noise in the blended image because the blending ratio of images with less random noise will be relatively higher. Also, if the exposure times are the same, the blending ratio of each image is set uniformly. Note that this is not the only way to determine the blending ratio; other methods may be used as long as they minimize the random noise in the blended image.

[0058] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. In the above embodiments, the ISO sensitivity is determined based on the exposure time and brightness information, and the image blending ratio is set according to the exposure time, assuming that the ISO sensitivity decreases as the exposure time increases for the same brightness. However, since the effect of random noise becomes greater for images with higher ISO sensitivity, the image blending ratio may be set according to the ISO sensitivity (the blending ratio should be increased for images with lower ISO sensitivity).

[0059] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0060] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0061] 100...Imaging device, 106...Image sensor, 112...Shake detection unit, 115...Camera control unit, 120...Image sensor position detection unit, 124...Shake correction amount calculation unit, 121...Image sensor drive unit

Claims

1. An imaging device having an imaging means, which performs continuous shooting with the imaging means to obtain a plurality of images used to generate a composite image, A correction means that detects the amount of shake of the imaging device and corrects image shake by changing the position of a correction member based on the amount of shake, An acquisition means for acquiring a correctable amount representing the degree to which correction by the correction means is possible, based on the current position of the correction member and the amount of runout, When continuous shooting is performed to obtain the plurality of images used to generate the composite image, a determination means determines the composite ratio when generating the composite image for each of the plurality of images obtained by each shooting, based on the correctable amount acquired by the acquisition means before each shooting, It has, The determination means is an imaging device characterized in that it increases the composite ratio for images with lower ISO sensitivity, determined based on brightness information and the correctable amount, among the plurality of images.

2. The acquisition means calculates the correctable amount based on the distance from the nearest boundary of the movable range of the correction member and the amount of deviation. The imaging apparatus according to feature 1.

3. The determination means determines that the exposure time should be reduced as the amount of correctable amount decreases. The imaging apparatus according to claim 1 or 2.

4. The imaging apparatus according to claim 3, characterized in that the determination means determines the ISO sensitivity based on luminance information and the exposure time.

5. The imaging apparatus according to any one of claims 1 to 4, characterized in that the correction member is an image sensor of the imaging means.

6. The imaging apparatus according to any one of claims 1 to 4, characterized in that the corrective member is a lens included in the imaging optical system.

7. A control method for an imaging device having an imaging means, which performs continuous shooting using the imaging means to obtain a plurality of images used to generate a composite image, A correction process that detects the amount of vibration of the imaging device and corrects image vibration by changing the position of the correction member based on the amount of vibration, An acquisition step to obtain a correctable amount representing the degree to which correction by the correction step is possible, from the current position of the correction member and the amount of runout, When continuous shooting is performed to obtain the plurality of images used to generate the composite image, a determination step is performed before each shooting to determine the composite ratio when generating the composite image for each of the plurality of images obtained by each shooting, based on the correctable amount obtained in the acquisition step, It has, In the determination step, the composite ratio is increased for images with lower ISO sensitivity, which is determined based on the brightness information and the correctable amount among the multiple images. A control method for an imaging device, characterized by the following:

8. A program that, when read and executed by a computer, causes the computer to perform each of the steps of the method according to claim 7.

Images