Control device, imaging device, control method, and program

Abstract

To provide a control device that can accurately correct image blur even during long exposures in still image photography. [Solution] The control device is used in an imaging device that includes an imaging means for acquiring multiple images in succession and a combining means for combining multiple images based on motion vectors between the multiple images, and comprises an estimation means for estimating the offset component of the output of a detection means for detecting vibrations of the imaging device, and a changing means for changing the settings used by the estimation means when estimating the offset component during preliminary shooting, in which multiple images used in live view are acquired in succession, and during main shooting, in which multiple images used by the combining means are acquired in succession.

Description

[Technical Field]

[0001] The present invention relates to a control device used in an imaging device. [Background technology]

[0002] In image stabilization, which suppresses the impact of camera shake on images, the output signal of a detection means that detects vibrations of the imaging device is used. However, this output signal contains DC components such as variations in the reference voltage due to individual differences and drift due to temperature changes (hereinafter collectively referred to as offset components). Therefore, a configuration has been proposed in which the offset component included in the output signal is estimated using a motion vector obtained from the difference between multiple images acquired by the image sensor, and the estimated offset component is removed from the output signal. Patent Document 1 discloses a configuration in which the update speed of the offset estimate is changed using the reliability of the motion vector. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2019-124871 [Overview of the project] [Problems that the invention aims to solve]

[0004] The configuration described in Patent Document 1 is effective when continuously acquiring images at short intervals, such as during live view. However, in still image shooting, the next image cannot be obtained until the exposure is complete, and motion vectors cannot be acquired either. Therefore, it is difficult to obtain effectiveness, especially during long exposures. Furthermore, since motion vectors cannot be acquired during exposure, it is not possible to estimate and remove the offset component included in the output signal. As the exposure time increases, the influence of the offset component becomes greater, making accurate blur correction impossible. Consequently, even if the offset component is removed until just before the start of exposure, the accuracy of blur correction may decrease as the exposure time increases.

[0005] The present invention aims to provide a control device that can accurately correct image blur even during long exposures in still image photography. [Means for solving the problem]

[0006] An imaging device as one aspect of the present invention is a control device used in an imaging device comprising: an imaging means for successively acquiring a plurality of images; and a combining means for combining a plurality of images based on motion vectors between the plurality of images, characterized in that it includes an estimation means for estimating the offset component of the output of a detection means for detecting vibrations of the imaging device; and a changing means for changing the settings used by the estimation means when estimating the offset component during preliminary shooting, in which a plurality of images used in live view are successively acquired, and during main shooting, in which a plurality of images used in the combining means are successively acquired. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a control device that can perform blur correction with high accuracy even during long exposures when taking still images. [Brief explanation of the drawing]

[0008] [Figure 1] This is an explanatory diagram of a camera system according to an embodiment of the present invention. [Figure 2] This is an explanatory diagram for estimating the offset component. [Figure 3] This diagram illustrates the relationship between exposure time and image blur in image synthesis image stabilization. [Figure 4] This is a block diagram of the image synthesis mechanism. [Figure 5] This is a flowchart showing the image synthesis and image stabilization process. [Figure 6] This flowchart shows the still image capture process when performing image synthesis image stabilization. [Figure 7] This is a flowchart showing the processing steps taken during this shoot. [Figure 8] This figure simply shows the change in the amount of offset component removed due to differences in image acquisition intervals.

Best Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant explanations are omitted.

[0010] FIG. 1 is an explanatory diagram of a camera system according to an embodiment of the present invention. FIG. 1(a) is a block diagram of the camera system. FIG. 1(b) is a central cross-sectional view of the camera system. In this embodiment, an interchangeable-lens camera will be described as an example, but the present invention is also applicable to video cameras, digital still cameras, and electronic devices having an imaging unit.

[0011] The camera system includes a camera (imaging device) 1 and a lens 2. The camera 1 and the lens 2 can exchange electrical signals via a lens contact 20 that is electrically connected.

[0012] Camera 1 comprises an image sensor (imaging means) 11, an image processing unit 12, a memory unit 13, a focal-plane shutter 14 (hereinafter referred to as the shutter), an operation unit 15, a display unit 16, and a viewfinder optical system 19. The image sensor 11 receives light rays that have passed through the lens 2. The image processing unit 12 generates an image from the information photoelectrically converted by the image sensor 11. The image processing unit 12 also includes motion vector detection means 12a and image synthesis means 12b. The memory unit 13 stores information such as image information. The shutter 14 controls the blocking and passage of light rays to the image sensor 11. The operation unit 15 recognizes user operations. The display unit 16 displays images and the like. As shown in Figure 1(b), the display unit 16 has a rear LCD unit 16a located on the back of the camera 1 and a viewfinder display unit 16b located in the viewfinder optical system 19 and viewable through the eyepiece lens 19a. The display unit 16 is controlled by a display control means (not shown) that controls the displayed image, and the user can arbitrarily switch between displaying on the rear LCD unit 16a and the viewfinder display unit 16b. In addition, the display unit 16 can display a live view before shooting via the camera system control unit 10, which will be described later.

[0013] Camera 1 includes a camera system control unit (control device) 10, a camera-side vibration damping means 17, and a blur detection means (detection means) 18. The camera-side vibration damping means 17 shifts the image sensor 11 in a direction approximately perpendicular to the optical axis. The blur detection means 18 detects vibrations of Camera 1 (movements occurring in Camera 1) and outputs a detection signal of camera blur information (amount of blur) to the camera system control unit 10. The camera system control unit 10 includes an offset estimation means (estimation means) 103, an estimation control means (modification means) 104, an exposure condition setting means (setting means) 107, and an image synthesis determination means (determination means) 108. The exposure condition setting means 107 can automatically or manually set the total exposure time and image acquisition interval, which will be described later. When setting manually, the setting is done via the operation unit 15. The exposure condition setting means 107 can also determine whether the total exposure time and image acquisition interval have been changed since the previous shooting. The image synthesis determination means 108 determines whether or not to perform the actual shooting described later based on the total exposure time and image acquisition interval set by the exposure condition setting means 107.

[0014] Lens 2 comprises a lens system control unit 21, an imaging optical system 22, a focus driving means 23, and a lens-side vibration damping means 24. The lens system control unit 21 oversees the control of lens 2. The imaging optical system 22 allows light rays to pass through. The focus driving means 23 moves the focus lens included in the imaging optical system 22. The lens-side vibration damping means 24 shifts a correction lens, such as a shift lens, included in the imaging optical system 22 in a direction approximately perpendicular to the optical axis.

[0015] The shutter 14 has shutter curtains consisting of a front curtain and a rear curtain, and controls the blocking and passage of light rays from the imaging optical system 22 to the image sensor 11 by moving each shutter curtain within the aperture. The shutter 14 is driven and controlled by the camera system control unit 10.

[0016] Light rays that pass through the aperture of the imaging optical system 22 and shutter 14 and are received by the image sensor 11 are photoelectrically converted, and the photoelectrically converted output is quantized by an A / D converter (not shown). The image processing unit 12 is equipped with a white balance circuit, a gamma correction circuit, and an interpolation calculation circuit, and receives commands from the camera system control unit 10 to generate image data from the signal acquired from the image sensor 11. The motion vector detection means 12a detects motion vectors based on a comparison between multiple images obtained from the image sensor 11. The image synthesis means 12b aligns multiple images acquired consecutively by the image sensor 11 and outputs image data that has been image-synthesized. The data output from the image synthesis means 12b is stored in the memory unit 13.

[0017] The camera system control unit 10 is equipped with a CPU (Central Processing Unit) and other components, and oversees the control of the camera 1, including communication with the lens 2. The camera system control unit 10 generates timing signals for image capture and outputs them to each unit. When the release button included in the operation unit 15 is pressed and an operation instruction is received, the camera system control unit 10 transmits command signals to the image sensor 11 and the lens system control unit 21 according to the instruction. The release button can detect a so-called half-press operation (hereinafter referred to as S1 operation), where the amount of pressing is the first stage, and a so-called full-press operation (hereinafter referred to as S2 operation), where the amount of pressing is further than that, where the amount of pressing is the second stage. When an S1 operation is detected, a command for shooting preparation operations such as autofocus (hereinafter referred to as AF) operation is issued. If an S2 operation is detected from that state, the shutter 14 is driven and the exposure operation for still image capture is started. Depending on the settings, it is also possible to perform multiple consecutive exposure operations for still image capture with a single press of the release button.

[0018] Figure 2 is an explanatory diagram of the offset component estimation (offset estimation).

[0019] The blur detection means 18 outputs the detection signal of the camera 1's blur information to the subtractors 102 and 105.

[0020] The motion vector detection means 12a outputs the detected motion vector to the adder 101.

[0021] The vibration-damping member position detection means 171 detects the position of the image sensor 11, which is a vibration-damping member of the camera-side vibration-damping means 17. The output signal from the vibration-damping member position detection means 171 is output to the differentiator 172.

[0022] The differentiator 172 performs differentiation on the output signal of the vibration isolation member position detection means 171. The output signal of the differentiator 172 is output to the adder 101.

[0023] The adder 101 adds the motion vector detected by the motion vector detection means 12a and the output signal of the differentiator 172. The output signal of the adder 101 is output to the subtractor 102.

[0024] The subtractor 102 subtracts the output signal of the adder 101 from the output signal of the deviation detection means 18. The output signal of the subtractor 102 is output to the offset estimation means 103.

[0025] The offset estimation means 103 estimates the offset component of the output signal of the blur detection means 18 based on the output signal of the subtractor 102. The offset component estimated by the offset estimation means 103 is output to the subtractor 105.

[0026] The estimation control means 104 changes the update characteristics (settings) when the offset estimation means 103 estimates the offset component. The method for changing the update characteristics of the offset estimation by the estimation control means 104 will be described later. The output data of the estimation control means 104 is output to the offset estimation means 103.

[0027] The subtractor 105 subtracts the offset component estimated by the offset estimation means 103 from the output signal of the shake detection means 18. The output signal of the subtractor 105 is output to the integrator 106.

[0028] The integrator 106 performs integration on the output signal of the subtractor 105. The output signal of the integrator 106 is output to the camera-side vibration damping means 17.

[0029] The camera-side image stabilization means 17 converts the output value of the integrator 106 into a correction target value and controls the image sensor 11, which is an image stabilization member, to cancel out movements such as camera shake. In addition, as shown in Figure 1(a), if the lens 2 has a lens-side image stabilization means 34, the output value of the integrator 106 may also be output to the lens-side image stabilization means 34 via the lens contact 20, converted into a correction target value, and used to control a correction lens such as a shift lens, which is an image stabilization member.

[0030] The offset estimation method using the offset estimation means 103 will be described below. When the offset estimation means 103 is configured as a linear Kalman filter, the linear Kalman filter can be expressed by the following equations (1) to (7).

[0031]

number

[0032] JPEG2026049800000003.jpg11100

[0033] Here, equation (1) represents the operation model in state-space representation, and equation (2) represents the observation model. A represents the system matrix in the operation model, and B represents the input matrix. Also, C represents the output matrix in the observation model, and each is expressed as a determinant. Also, ε t This is process noise, δ t represents observation noise, and t represents discrete time.

[0034]

number

[0035] JPEG2026049800000005.jpg11134

[0036] Here, equation (3) represents the prior estimate in the prediction step, and equation (4) represents the prior error covariance. Also, Σ x This represents the noise variance of the operating model.

[0037]

number

[0038] JPEG2026049800000007.jpg12114

[0039] JPEG2026049800000008.jpg12112

[0040] Here, equation (5) represents the formula for calculating the Kalman gain in the filtering step, and the subscript T represents the transpose matrix. Equation (6) represents the posterior estimate by the Kalman filter, and equation (7) represents the posterior error covariance. Also, Σz represents the variance of the noise of the observation model.

[0041] In this embodiment, in order to estimate the offset component of the output signal of the shake detection means 18, the offset component of the output signal of the shake detection means 18 is set as x t , the amount of shake observed (detected) by the shake detection means 18 (the output of the shake detection means 18) is set as z t and. Also, ε t is the process noise, and δ t is the observation noise. Then, the model representing the offset component can be expressed by the following first-order linear model in which the input term u in Equation (1) is absent and A = C = 1 in Equations (1) and (2).

[0042] [[ID=JPEG2026049800000010.jpg12111

[0044] Here, the variance Σ x of the noise of the operation model in Equation (4) is represented by the variance

[0045]

Number

[0047]

Number

[0048] is represented by. The prior estimate at time t is

[0049]

Number

[0050] , posterior error variance

[0051]

number

[0052] , Kalman gain is kt, observation noise variance is

[0053]

number

[0054] The amount of shake observed by the shake detection means 18 is z t Therefore, the Kalman filter can be constructed using the following equation.

[0055]

number

[0056] JPEG2026049800000017.jpg12110

[0057] JPEG2026049800000018.jpg21108

[0058] JPEG2026049800000019.jpg11111

[0059] JPEG2026049800000020.jpg13109

[0060] The offset estimation means 103 is composed of equations (10) to (14), and the offset estimate at time t-1 of the estimation calculation update period

[0061]

number

[0062] and dispersion of system noise output by the shake detection means 18

[0063]

number

[0064] , posterior error variance at time t-1

[0065]

number

[0066] Therefore, prior estimates

[0067]

number

[0068] and prior error variance

[0069]

number

[0070] The following is calculated: and the prior error variance.

[0071]

number

[0072] and observation noise dispersion output by the blur detection means 18

[0073]

number

[0074] Based on the Kalman gain k t The following is calculated. Then, according to equation (13), the amount of shake z observed by the shake detection means 18 is calculated. t and prior estimates

[0075]

number

[0076] The error with respect to the Kalman gain k t The prior estimate is obtained by multiplying by the value obtained by

[0077]

number

[0078] The offset estimate has been corrected.

[0079]

number

[0080] The following is calculated. Also, the prior error variance is calculated using equation (14).

[0081]

number

[0082] The posterior error variance has been corrected.

[0083]

number

[0084] The following is calculated. By repeating the prior estimation update and correction at each calculation cycle using these calculations, the offset estimate is calculated.

[0085] In the above explanation, the output of the vibration detection means 18 is used as the observed value of the offset component. However, in this embodiment, offset estimation is performed using the difference between the signal obtained by adding the vibration isolation member's moving speed and motion vector, which is obtained by differentiating the output of the vibration isolation member position detection means 171, and the output of the vibration detection means 18 as the observed value of the offset component.

[0086] The following describes the image synthesis vibration isolation performed by the image synthesis means 12b.

[0087] First, we will explain a specific example and effect of image composite image stabilization with reference to Figure 3. Figure 3 is a diagram showing the relationship between exposure period and image blur in image composite image stabilization. In Figure 3, the vertical axis is the blur amount, and the horizontal axis is the exposure period. The dashed lines 31, 33, and 33a to 33c show the blur amount when image composite image stabilization is not performed, and the dashed lines 33a to 33c are the dashed line 33 slid in the direction of exposure period. The solid lines 32a to 32d, 34, and 34a to 34c show the blur amount per image before stabilization used in image composite image stabilization. B1 shows the blur amount when image composite image stabilization is not performed when the total exposure period is 1 second, and B2 and B3 show the blur amount when image composite image stabilization is performed when the total exposure period is 1 second.

[0088] Image synthesis image stabilization is a technique that divides the exposure period required to acquire one composite image (hereinafter referred to as the total exposure time) into multiple short exposure periods (hereinafter referred to as the image acquisition interval), and combines the multiple images obtained at each image acquisition interval while aligning them. The sum of the time of the image acquisition intervals is the total exposure time. By performing image synthesis image stabilization, it is possible to reduce blur in the image while obtaining the same exposure as when shooting with the total exposure time, and it is particularly effective when the total exposure time is long. For example, as shown in Figure 3(a), when shooting with a total exposure time of 1 second and an image acquisition interval of 1 / 4 second, four images are taken consecutively within the total exposure time, and these images are aligned and combined.

[0089] Without image composite image stabilization, offset estimation cannot be performed during the exposure period, making accurate blur correction impossible. As shown by the dashed line 31, the amount of blur continues to increase from the start to the end of exposure. On the other hand, with image composite image stabilization, exposure is performed in multiple stages at image acquisition intervals. As shown by the solid lines 32a to 32d, the amount of blur from the previous exposure period can be reset to zero by image alignment at each exposure start. These aligned images are then combined to form a single composite image. If the alignment is done properly, the amount of blur in the image will be B2, which is the amount of blur shown by the solid line 32d. This is lower than the amount of blur B1 when image composite image stabilization is not performed. However, since offset estimation is not performed during the exposure period in this method, the offset component generated during the exposure period is not removed. Therefore, even if the total exposure time is divided by image acquisition intervals, the longer the total exposure time, the less blur can be suppressed compared to when image composite image stabilization is not performed. However, because accurate blur correction is not performed, the increase in the amount of blur itself cannot be suppressed. However, during image synthesis vibration isolation, multiple images are acquired at image acquisition intervals within the total exposure time. Therefore, motion vectors can be obtained even during the exposure period, making offset estimation possible.

[0090] Figure 3(b) shows the relationship between exposure period and blur when offset estimation is performed during the exposure period. In this case, when exposure starts at the image acquisition interval, not only is the blur of the previous exposure period reduced to zero by image alignment, but the offset component that occurred during the previous exposure period is also removed by offset estimation. Therefore, the increase in blur at each exposure start timing is represented by the dashed line 33, which is slid to lines 33a to 33c for each exposure start timing, and the blur per image is also represented by solid lines 34a to 34c, just like the initial solid line 34, and all are B3. Furthermore, if alignment is performed appropriately during image synthesis, the blur will also be B3, and even if the total exposure time is long, the blur can be reduced to a level equivalent to the image acquisition interval.

[0091] The configuration of the image synthesis means 12b will be described below. Figure 4 is a block diagram of the image synthesis means 12b.

[0092] The motion vector detection means 12a detects motion vectors between multiple images obtained from the image sensor 11 and outputs them to the alignment amount calculation means 121.

[0093] The image synthesis means 12b includes a positioning amount calculation means 121 and a positioning synthesis means 122.

[0094] The alignment amount calculation means 121 calculates the alignment amount of multiple images based on the motion vectors obtained from the motion vector detection means 12a and outputs it to the alignment synthesis means 122.

[0095] The alignment and synthesis means 122 aligns and synthesizes multiple input images based on the alignment amounts of multiple images obtained from the alignment amount calculation means 121.

[0096] The following describes the process flow for image synthesis image stabilization. Figure 5 is a flowchart showing the image synthesis image stabilization process, which begins when an image is input from the image sensor 11 to the image processing unit 12.

[0097] In step S501, the motion vector detection means 12a detects motion vectors between multiple images obtained from the image sensor 11. Motion vectors can be detected by extracting feature points from the images and determining the correspondence between the feature points. Feature point extraction can be performed using known corner detection methods. The correspondence between feature points can be determined using known template matching or feature matching methods. If the reliability of the matching is low, it is highly likely that the correspondence between feature points has not been correctly determined, so it is preferable to exclude such feature points before detecting the motion vectors.

[0098] In step S502, the alignment amount calculation means 121 calculates the alignment amount of multiple images based on the motion vector obtained in step S501. The method of expressing the alignment amount differs depending on the blur component to be corrected. When correcting only the translational blur component, the alignment amount is expressed as the amount of horizontal and vertical movement. In this case, histograms can be generated for the horizontal and vertical movement amounts of the motion vector, and the mode of each histogram can be calculated. Since this mode is a representative value of the blur occurring between frames, the alignment amount that cancels out the blur can be obtained by taking the inverse sign of this mode. On the other hand, when correcting both the translational and rotational blur components, the alignment amount is expressed as a projection transformation matrix (or affine transformation matrix) that shows the correspondence between images. In this case, the projection transformation matrix can be calculated from the correspondence between feature points between frames obtained by the motion vector using a known method such as the least squares method. Since the calculated projection transformation matrix represents the blur occurring between frames, the alignment amount that cancels out the blur can be obtained by calculating the inverse matrix of this matrix.

[0099] In step S503, the alignment and compositing means 122 aligns and combines the input images based on the alignment amounts of the multiple images obtained in step S502. Image alignment is performed by geometrically deforming the multiple images using the acquired alignment amounts. Then, by adding and compositing the aligned multiple images, image compositing image stabilization is achieved. Note that if each of the multiple images is photographed with the correct exposure, it is necessary to take the average after adding the images. Once the alignment and compositing of the multiple images have been performed, the composite image is output to the memory unit 13 and stored.

[0100] The following describes the challenges of offset estimation in image synthesis image stabilization. In image synthesis image stabilization, multiple images are taken at the image acquisition interval within the total exposure time, making it possible to acquire motion vectors. However, the opportunity to acquire them is limited to the intervals between exposures in the image acquisition interval. Therefore, the offset estimate can only be updated at that time. To remove errors caused by the offset component that occurred during the previous exposure during the interval between exposures in the image acquisition interval, it is necessary to increase the update speed of the offset estimate (increase the degree to which the estimate is corrected as the offset estimate is updated). In other words, the time required to estimate the offset component should be shortened. To achieve this, when using a Kalman filter, the Kalman gain k, expressed by equation (12), is used. t It needs to be enlarged.

[0101] The following describes the flow of the imaging operation with reference to Figure 6. Figure 6 is a flowchart of the still image shooting operation when performing image synthesis stabilization. This flow starts when the power of camera 1 is turned ON, or when the mode for still image shooting is switched from another mode while the power is ON.

[0102] In step S601, the camera system control unit 10 initiates a preliminary shooting operation. Preliminary shooting is the operation of successively acquiring multiple images using the image sensor 11 that will be used to display on the display unit 16 in live view.

[0103] In step S602, the camera system control unit 10 determines whether or not the S1 operation, which is a shooting preparation operation instruction, has been performed. If the camera system control unit 10 determines that the S1 operation has been performed, it executes the process in step S603; otherwise, it executes the process in this step again.

[0104] In step S603, the exposure condition setting means 107 sets the exposure conditions. In this step, the total exposure time and image acquisition interval for shooting using image synthesis image stabilization are set. At this time, the exposure conditions may be set automatically from the information if the F-number, ISO, etc. have already been set on the camera 1, or they may be set manually by the user of the camera 1 via the operation unit 15. In this embodiment, the exposure condition setting means 107 acquires the image acquisition interval, but it may also set the number of images to be taken.

[0105] In step S604, the image synthesis determination means 108 determines whether or not to perform image synthesis image stabilization based on the exposure conditions set in step S603. If the total exposure time and the image acquisition interval are different, the image synthesis determination means 108 determines to perform image synthesis image stabilization, and the camera system control unit 10 executes the process in step S605. If the total exposure time and the image acquisition interval are the same, the image synthesis determination means 108 determines not to perform image synthesis image stabilization, and the camera system control unit 10 executes the process in step S606.

[0106] In step S605, the camera system control unit 10 performs the main shooting operation. Main shooting is the operation in which the image sensor 11 successively acquires multiple images to be used by the alignment and synthesis means 122 with a single shooting instruction (S2 operation).

[0107] Figure 7 is a flowchart showing the processing during actual shooting. The flowchart starts when it is determined in step S604 that image synthesis stabilization should be performed.

[0108] In step S701, the exposure condition setting means 107 determines whether the total exposure time set in step S603 has changed since the previous shooting. If the exposure condition setting means 107 determines that the total exposure time has changed since the previous shooting, it executes the process in step S703; otherwise, it executes the process in step S702.

[0109] In step S702, the exposure condition setting means 107 determines whether the image acquisition interval set in step S603 (predetermined shooting timing) has changed from the previous shooting time (the shooting timing immediately preceding the predetermined shooting timing). If the exposure condition setting means 107 determines that the image acquisition interval has changed from the previous shooting time, it executes the process in step S704; otherwise, it executes the process in step S703.

[0110] In step S703, the estimation control means 104 modifies the update characteristics so that the update speed of the offset estimation is faster than during the preliminary imaging in step S601. The modification of the update characteristics here refers to the Kalman gain k during the main imaging. t The goal is to make it larger than the image taken during the preliminary shooting.

[0111] During preliminary shooting while in live view mode, the display unit 16 constantly displays an image, allowing for continuous image acquisition at short intervals and enabling constant offset estimation. However, during this time, the photographer performs framing actions such as determining the subject and composition while viewing the image displayed on the display unit 16. Therefore, even when the camera 1 is fixed, low-frequency blur caused by the photographer's movements is superimposed as noise on the observed offset component. Consequently, during preliminary shooting, the Kalman gain k is not affected by the noise superimposed on the observed offset component during offset estimation. t This is set. On the other hand, during the actual shooting, the photographer holds and fixes camera 1 in order to capture the subject, so low-frequency blur caused by the photographer's movement can be suppressed. Therefore, since no extraneous noise is added to the observed value of the offset component, the observed value of the offset component during the actual shooting can be trusted more than during the preliminary shooting, and the Kalman gain k during the actual shooting is lower than during the preliminary shooting. t It becomes possible to set the Kalman gain k to a larger value. t To increase the value, the prior error variance in equation (12)

[0112]

number

[0113] Observed noise dispersion

[0114]

number

[0115] Make it larger than, or the observation noise dispersion

[0116]

number

[0117] Prior error variance

[0118]

number

[0119] It will become even smaller.

[0120] In this embodiment, when transitioning from preliminary imaging to main imaging, the noise superimposed on the observed value of the offset component is reduced, so the latter method is used to gain the Kalman gain k t Set the value to a large value. This allows the update speed of the offset estimation during the main shooting to be faster than during the preliminary shooting, so that the offset component that occurred during exposure at the image acquisition interval can be removed each time during image composite image stabilization.

[0121] In step S704, the estimation control means 104 modifies the update characteristics so that the update speed of the offset estimation is the same as that of the previous image capture. The modification of the update characteristics here refers to the modification of the Kalman gain k so that the update speed of the offset estimation is the same as that of the previous image capture. t This involves changing the size. Furthermore, "same" includes not only cases where they are exactly the same, but also cases where they are substantially the same (almost identical).

[0122] Now, refer to Figure 8 for the Kalman gain k tThis section explains the relationship between the offset estimation update speed and the total exposure time. Figure 8 is a simplified diagram showing the change in the amount of offset component removed due to differences in image acquisition intervals, assuming a total exposure time of 8 seconds. In Figure 8, the vertical axis represents the offset component, and the horizontal axis represents the exposure period. The solid line 81 shows the change in the amount of offset component removed when the image acquisition interval is set to 1 second, and the dashed line 82 shows the change in the amount of offset removal when the image acquisition interval is set to 2 seconds. The symbols "●" and "▲" indicate the offset component at each image acquisition interval.

[0123] Kalman Gain k t Even if the Kalman gain is the same, if the image acquisition interval is different, the offset component corrected by one update of the offset estimate remains the same. Therefore, a difference arises in the update speed of the offset estimate, resulting in a difference in the amount of offset component removed at the end of the total exposure time, as shown by solid line 81 and dashed line 82. The slopes of solid line 81 and dashed line 82 represent the update speed of the offset estimate. In order to match the amount of offset component removed at the end of the total exposure time even if the image acquisition interval is different, the Kalman gain k t The update speed of the offset estimation needs to be adjusted by changing the size of the Kalman gain k. For example, if the image acquisition interval becomes shorter than the previous capture, as in the case of dashed line 82 to solid line 81, the Kalman gain k t If the magnitude remains unchanged, the update speed of the offset estimation will increase. Therefore, the Kalman gain k t The size is set to be smaller than in the previous capture, and the update speed of the offset estimation is slowed down so that the solid line 81 overlaps with the dashed line 82. Also, if the image acquisition interval becomes longer than in the previous capture, as in the case of the transition from solid line 81 to dashed line 82, the Kalman gain k t If the magnitude does not change, the update speed of the offset estimation will slow down. Therefore, the Kalman gain k t The size of the offset is set to be larger than in the previous shot, and the update speed of the offset estimation is increased so that the dashed line 82 overlaps with the solid line 81. This makes it possible to maintain a constant update speed for the offset estimation even when the total exposure time is the same as in the previous shot but the image acquisition interval is different.

[0124] In step S705, the camera system control unit 10 determines whether or not the S2 operation, which is an instruction to start shooting, has been performed. If the camera system control unit 10 determines that the S2 operation has been performed, it executes the process in step S706; otherwise, it executes the process in this step again.

[0125] In step S706, the image synthesis means 12b performs image synthesis vibration isolation based on the exposure conditions set in step S603 and acquires a composite image.

[0126] Now let's return to the explanation of the flowchart in Figure 6.

[0127] In step S606, the camera system control unit 10 performs normal still image shooting with the total exposure time set in step S603 and acquires an image. Here, normal still image shooting refers to shooting in which the total exposure time is used as the exposure period without image composite image stabilization, and a single still image is acquired.

[0128] In step S607, the camera system control unit 10 determines whether or not to continue shooting. If the camera system control unit 10 determines to continue shooting, it executes the process in step S601. At this time, the offset estimation update characteristics of the offset estimation means 103 revert to those corresponding to the preliminary shooting. If the camera system control unit 10 turns off the power to camera 1 or switches to a mode other than still image shooting, it determines not to continue shooting and terminates this flow.

[0129] In this embodiment, a determination is made as to whether to perform image synthesis image stabilization. However, if the selected mode for still image capture presupposes the execution of image synthesis image stabilization, this determination may be omitted.

[0130] Through the above series of operations, accurate image stabilization can be performed even during long exposures when taking still images. [Other examples] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0131] This embodiment includes the following configurations and methods. (Composition 1) A control device used in an imaging device comprising: an imaging means for successively acquiring multiple images; and a synthesis means for synthesizing the multiple images based on motion vectors between the multiple images, Estimation means for estimating the offset component of the output of the detection means for detecting vibrations of the imaging device, A control device characterized by having a changing means for changing the settings used by the estimation means when estimating the offset component, during preliminary shooting, in which multiple images used in live view are acquired in succession, and during main shooting, in which multiple images used in the synthesis means are acquired in succession. (Configuration 2) The control device according to configuration 1, characterized in that the modification means modifies the settings so that the speed at which the estimation means estimates the offset component differs between the preliminary shooting and the main shooting. (Composition 3) The control device according to configuration 2, characterized in that the modification means modifies the settings so that the speed at which the estimation means estimates the offset component during the main shooting is faster than the speed at which the estimation means estimates the offset component during the preliminary shooting. (Composition 4) The control device according to any one of configurations 1 to 3, characterized in that the changing means changes the setting according to the image acquisition interval, which is the exposure time per image when acquiring multiple images in the actual shooting. (Composition 5) The control device according to configuration 4, characterized in that the modification means changes the settings so that the estimation means estimates the offset component at the same speed when the image acquisition interval changes between a predetermined shooting timing and the shooting timing immediately preceding the predetermined shooting timing during the actual shooting. (Composition 6) A control device according to any one of configurations 1 to 5, further comprising a determination means for determining whether or not to perform the aforementioned photography. (Composition 7) The system further includes a setting means for setting the exposure period required to acquire a single image, The control device according to configuration 6, characterized in that the determination means determines whether or not to perform the actual shooting based on the exposure period. (Composition 8) The control device according to any one of configurations 1 to 7, further comprising a setting means for setting the exposure time required to acquire a single composite image in the aforementioned shooting. (Composition 9) The control device according to configuration 8, characterized in that the setting means sets at least one of the image acquisition interval, which is the exposure time per image when acquiring multiple images in the actual shooting, and the number of multiple images. (Composition 10) A control device according to any one of claims 1 to 9, The imaging means, An imaging apparatus characterized by having the aforementioned synthesis means. (Method 1) A control method used in an imaging device comprising: an imaging means for successively acquiring multiple images; and a synthesis means for synthesizing the multiple images based on motion vectors between the multiple images, A step of estimating the offset component of the output of the detection means for detecting vibrations of the imaging device, A control method characterized by having the step of changing the settings used when estimating the offset component during preliminary shooting, in which multiple images used in live view are acquired in succession, and during main shooting, in which multiple images used in the synthesis means are acquired in succession. (Composition 11) A program characterized by causing a computer to execute the control method described in Method 1.

[0132] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of Symbols]

[0133] 1. Camera (imaging device) 10 Camera system control unit (control device) 11 Image sensor (imaging means) 12b Image synthesis means (synthesis means) 18. Blur detection means (detection means) 103 Offset estimation means (estimation means) 104 Estimation control means (modification means)

Claims

1. A control device used in an imaging device comprising: an imaging means for successively acquiring multiple images; and a synthesis means for synthesizing the multiple images based on motion vectors between the multiple images, Estimation means for estimating the offset component of the output of the detection means for detecting vibrations of the imaging device, A control device characterized by having a changing means for changing the settings used by the estimation means when estimating the offset component, during preliminary shooting when acquiring multiple images used in live view in succession, and during main shooting when acquiring multiple images used in the synthesis means in succession.

2. The control device according to claim 1, characterized in that the modification means modifies the settings so that the speed at which the estimation means estimates the offset component differs between the preliminary shooting and the main shooting.

3. The control device according to claim 2, characterized in that the modification means modifies the settings such that the speed at which the estimation means estimates the offset component during the main shooting is faster than the speed at which the estimation means estimates the offset component during the preliminary shooting.

4. The control device according to any one of claims 1 to 3, characterized in that the changing means changes the setting according to the image acquisition interval, which is the exposure time per image when acquiring multiple images in the actual shooting.

5. The control device according to claim 4, characterized in that the modification means changes the settings so that the estimation means estimates the offset component at the same speed when the image acquisition interval changes between a predetermined shooting timing and the shooting timing immediately preceding the predetermined shooting timing during the actual shooting.

6. The control device according to any one of claims 1 to 3, further comprising a determination means for determining whether or not to perform the aforementioned photograph.

7. The system further includes a setting means for setting the exposure period required to acquire a single image, The control device according to claim 6, characterized in that the determination means determines whether or not to perform the actual shooting based on the exposure period.

8. The control device according to any one of claims 1 to 3, further comprising setting means for setting the exposure period required when acquiring a single composite image in the aforementioned main shooting.

9. The control device according to claim 8, wherein the setting means sets at least one of the image acquisition interval, which is the exposure time per image when acquiring multiple images in the actual shooting, and the number of multiple images.

10. A control device according to any one of claims 1 to 3, The imaging means, An imaging apparatus characterized by having the aforementioned synthesis means.

11. A control method used in an imaging device comprising: an imaging means for successively acquiring multiple images; and a synthesis means for synthesizing the multiple images based on motion vectors between the multiple images, A step of estimating the offset component of the output of the detection means for detecting vibrations of the imaging device, A control method characterized by having the step of changing the settings used when estimating the offset component during preliminary shooting, in which multiple images used in live view are acquired in succession, and during main shooting, in which multiple images used in the synthesis means are acquired in succession.

12. A program characterized by causing a computer to execute the control method described in claim 11.

Images