Image Blur Correction Device and Its Control Method, Imaging Device, and Storage Medium

Abstract

The present invention relates to an image blur correction device and a control method thereof, a photographing device, and a storage medium. The image blur correction device includes: an acquisition component for acquiring shake information; a plurality of image blur correction amount calculation components, including a first image blur correction amount calculation component for calculating image blur correction amounts respectively corresponding to partial images included in a plurality of consecutive images, and a second image blur correction amount calculation component for calculating image blur correction amounts respectively corresponding to partial images included in a plurality of consecutive images by using a method different from the method used by the first correction amount calculation component; and a control component for controlling the plurality of image blur correction amount calculation components.

Description

Technical Field

[0001] The present invention relates to a technique for performing image blur correction in cooperation with each other via communication by image blur correction components installed on multiple devices. Background Art

[0002] In recent years, the video industry has used a method of obtaining a smoother slow video by shooting at a high frame rate and using a reproduction speed equal to or lower than the shooting frame rate. However, this method shoots at a high frame rate, and thus increases the processing load of the camera. As a result, in the case where image blur correction components installed on multiple devices perform image blur correction control in cooperation with each other via communication, communication errors and communication delays occur. In view of this, a method has been proposed in which, even in the case where communication errors and communication delays have occurred, a reduction in the image blur correction effect is suppressed.

[0003] Japanese Unexamined Patent Application Publication No. 2017-219635 discloses a method in which, in the case where shake information has been obtained via communication, a blur correction amount obtained based on the shake information is selected, and in the case where shake information has not been obtained via communication, a blur correction amount predicted by a prediction component is selected to correct image blur.

[0004] However, Japanese Unexamined Patent Application Publication No. 2017-219635 has the following problem. According to Japanese Unexamined Patent Application Publication No. 2017-219635, since a method of predicting based on the previous correction amount (such as Kalman filtering and the least squares method, etc.) is used to calculate the blur correction amount, in the case where an incorrect correction amount has been calculated, the image blur correction effect is significantly reduced.

[0005] In addition, although communication errors and communication delays are improved by setting a low-speed communication cycle, setting a low-speed communication cycle also reduces the speed of the cycle for obtaining a signal for performing image blur correction, thereby reducing the image blur correction effect. Summary of the Invention

[0006] The present invention has been made in consideration of the above problems, and suppresses a reduction in the image blur correction effect when a low-speed communication cycle is set in the case of performing image blur correction control while communicating between multiple devices.

[0007] According to a first aspect of the present invention, there is provided an image blur correction device, comprising: an acquisition component configured to acquire shake information from a plurality of shake detection components for detecting shake; a plurality of image blur correction amount calculation components, including: a first image blur correction amount calculation component configured to calculate image blur correction amounts respectively corresponding to partial images included in a plurality of consecutive images based on the shake information from a first shake detection component acquired by the acquisition component, and a second image blur correction amount calculation component configured to calculate image blur correction amounts respectively corresponding to the partial images included in the plurality of consecutive images by using a method different from the method used by the first image blur correction amount calculation component based on the shake information from a second shake detection component acquired by the acquisition component; and a control component configured to control the plurality of image blur correction amount calculation components such that at least one of the plurality of image blur correction amount calculation components acquires image blur correction amounts respectively corresponding to a plurality of consecutive captured images, and such that images for which the image blur correction amounts are calculated by the first image blur correction amount calculation component among the plurality of image blur correction amount calculation components and images for which the image blur correction amounts are calculated by the second image blur correction amount calculation component among the plurality of image blur correction amount calculation components are periodically generated.

[0008] According to a second aspect of the present invention, there is provided an imaging device, comprising: an imaging component configured to capture a plurality of consecutive images; the image blur correction device according to the above; a second shake detection component; and an image blur correction component configured to perform image blur correction on the plurality of consecutive images based on the image blur correction amounts acquired by the plurality of image blur correction amount calculation components, wherein the acquisition component acquires the detection result of the first shake detection component included in the mounted lens device by communication with a second period that is longer than a first period of capturing the plurality of consecutive images.

[0009] According to a third aspect of the present invention, there is provided an imaging device including: an imaging sensor configured to capture a plurality of consecutive images in a first period; a shake detection component configured to detect shake based on the plurality of consecutive images; a plurality of image blur correction amount obtaining components including: a first image blur correction amount obtaining component configured to obtain image blur correction amounts respectively corresponding to partial images included in the plurality of consecutive images based on shake information obtained from a mounted lens device through communication, and a second image blur correction amount obtaining component configured to obtain image blur correction amounts respectively corresponding to the partial images included in the plurality of consecutive images based on a detection result of the shake detection component and the shake information obtained through communication by using a method different from the method used by the first image blur correction amount obtaining component; and an image blur correction component configured to perform image blur correction on the plurality of consecutive images based on the image blur correction amounts obtained by the plurality of image blur correction amount obtaining components, wherein at least one of the plurality of image blur correction amount obtaining components obtains an image blur correction amount corresponding to each of the plurality of consecutive images respectively.

[0010] According to a fourth aspect of the present invention, there is provided a control method for an image blur correction device, the control method including: obtaining shake information from a plurality of shake detection components for detecting shake; calculating an image blur correction amount of a captured image based on the obtained shake information from the plurality of shake detection components; and controlling the obtaining such that shake information from at least one of the plurality of shake detection components is obtained corresponding to each of a plurality of consecutively captured images, and such that images in which corresponding shake information from a first shake detection component among the plurality of shake detection components cannot be obtained and images in which corresponding shake information from a second shake detection component among the plurality of shake detection components cannot be obtained are periodically generated.

[0011] According to a fifth aspect of the present invention, there is provided a control method for a imaging device, the imaging device having an imaging sensor for capturing a plurality of consecutive images at a first period, the control method comprising: detecting shake based on the plurality of consecutive images; obtaining a plurality of image blur correction amounts, including: a first obtaining for obtaining image blur correction amounts respectively corresponding to partial images included in the plurality of consecutive images based on shake information obtained through communication from a mounted lens device, and a second obtaining for obtaining image blur correction amounts respectively corresponding to partial images included in the plurality of consecutive images by using a method different from the method used in the first obtaining based on a shake detection result and shake information obtained through communication; and performing image blur correction on the plurality of consecutive images based on the image blur correction amounts obtained through the obtaining, wherein, in at least one of the obtaining of the plurality of image blur correction amounts, image blur correction amounts are obtained corresponding to the plurality of consecutive images respectively.

[0012] According to a sixth aspect of the present invention, there is provided a computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to the above.

[0013] Further features of the present invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram schematically showing the configuration of an interchangeable-lens digital camera system as an embodiment of an image blur correction device of the present invention.

[0015] Figure 2 is a block diagram for describing image blur correction control in the first embodiment.

[0016] Figure 3 is a block diagram for describing an image blur correction amount calculation unit in the first embodiment.

[0017] Figure 4A and 4B is a block diagram for describing a correction amount division unit in the first embodiment.

[0018] Figure 5A and 5B is a timing chart for describing cooperative communication in which a low-speed processing period is set compared to the imaging period.

[0019] Figure 6A and 6B is a timing chart for describing motion vector detection in which a low-speed detection period is set compared to the imaging period.

[0020] Figure 7A and7B It is a timing chart for describing problems in the case of setting a low speed for cooperative communication and setting a low speed period for motion vector detection as compared with the imaging period.

[0021] Figure 8 It is a timing chart for describing the operation of electronic correction amount conversion in the first embodiment.

[0022] Figure 9 It is a flowchart for describing the operation of electronic correction amount conversion in the first embodiment.

[0023] Figure 10 It is a timing chart for describing the operation of electronic correction amount conversion in the second embodiment.

[0024] Figure 11 It is a flowchart for describing the operation of electronic correction amount conversion in the second embodiment. Detailed Description of the Embodiment

[0025] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. In the embodiments, multiple features are described, but it is not limited that the invention necessarily requires all these features, and multiple such features can be appropriately combined. Further, in the drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions thereof are omitted.

[0026] (First Embodiment)

[0027] Hereinafter, a first embodiment of the present invention will be described in detail with reference to the accompanying drawings; first, matters common to all embodiments will be described.

[0028] Figure 1 It is a block diagram schematically showing the configuration of an interchangeable-lens digital camera system for taking still images and moving images, which is an embodiment of the image blur correction device of the present invention. Note that the application scope of the present invention is not limited to digital cameras, and the present invention is applicable to various types of imaging devices.

[0029] Note that in the following description of the embodiments, the vibration of the imaging device is expressed as "shake", and the influence of the shake of the imaging device on the captured image is expressed as "image blur".

[0030] In Figure 1 , the digital camera system 100 is composed of an interchangeable lens 150 and a camera body 180, and the interchangeable lens 150 is used when attached to the camera body 180.

[0031] The zoom unit 101 of the interchangeable lens 150 includes a zoom lens that changes the magnification. The zoom drive control unit 102 controls the drive of the zoom unit 101. The aperture unit 103 has the function of an aperture. The aperture drive control unit 104 controls the drive of the aperture unit 103. The image blur correction unit 105 includes an image blur correction lens such as a shift lens (hereinafter also referred to as a correction lens or OIS). The image blur correction unit 105 is a first image blur correction component, and its drive is controlled by the optical image blur correction control unit 106. The focus unit 107 includes a focus lens that forms a subject image by performing focus adjustment. The focus drive control unit 108 controls the drive of the focus unit 107.

[0032] The lens operation unit 109 is an operation unit used by the user when operating the interchangeable lens. The lens shake detection unit 110 detects the amount of shake of the interchangeable lens 150 and outputs the detected signal to the lens system control unit 111. The lens system control unit (hereinafter referred to as the lens control unit) 111 for controlling the entire interchangeable lens 150 includes a CPU (central computing processing device) and integrally controls each drive control unit and correction control unit for the interchangeable lens. The lens control unit 111 communicates with the camera system control unit 124 of the camera body 180 via the lens communication control unit 112.

[0033] Next, the camera body 180 will be described. The camera body 180 includes a shutter unit 113. The shutter drive control unit 114 controls the drive of the shutter unit 113. The imaging unit 115 includes an image sensor and outputs an electrical signal by performing photoelectric conversion on the optical image formed after passing through the lens group. The imaging signal processing unit 116 performs processing for converting the electrical signal output from the imaging unit 115 into a video signal. The video signal processing unit 117 processes the video signal output from the imaging signal processing unit 116 according to the desired use. For example, the video signal processing unit 117 changes the clipping position of the video signal according to the correction amount of the electronic image blur correction control unit 123. The electronic image blur correction control unit 123 is a second image blur correction component and controls image blur correction by clipping the image.

[0034] The display unit 118 displays an image as required based on the signal output from the video signal processing unit 117. The storage unit 119 stores various types of data such as video information. The power supply unit 120 supplies power to the entire system according to the desired use. The camera operation unit 121 is an operation unit used by the user when operating the camera system, and outputs an operation signal to the camera system control unit 124. The camera shake detection unit 122 detects the amount of shake of the camera, and outputs the detected signal to the camera system control unit 124. The camera system control unit (hereinafter referred to as the camera control unit) 124 includes a CPU, and controls the entire camera system integrally. The camera control unit 124 communicates with the lens communication control unit 112 of the interchangeable lens 150 via the camera communication control unit 125. That is, in a state where the interchangeable lens 150 is attached to and electrically connected to the camera body 180, mutual communication is performed via the lens communication control unit 112 and the camera communication control unit 125.

[0035] Next, the general operation of the digital camera system 100 configured as described above is described. The lens operation unit 109 and the camera operation unit 121 include an image blur correction switch, and the on / off of image blur correction can be selected via the image blur correction switch. Once the user selects on for image blur correction by operating the image blur correction switch, the lens control unit 111 or the camera control unit 124 instructs the optical image blur correction control unit 106 or the electronic image blur correction control unit 123 to perform an image blur correction operation. Each image blur control unit controls the image blur correction until an instruction to turn off the image blur correction is issued.

[0036] In combination with the image blur correction, the camera operation unit 121 also includes an image blur correction mode switch, and the first mode and the second mode can be selected via the image blur correction mode switch. The first mode is a mode in which image blur correction is performed by using both optical image blur correction (first image blur correction) and electronic image blur correction (second image blur correction). In the case where the first mode is selected, correction can be achieved by the cooperation between optical (optically based) image blur correction and electronic (electrically based) image blur correction, and a wider correction angle can be realized. In addition, by reducing the clipping range of the video signal and changing the clipping position according to the amount of image blur correction in the video signal processing unit 117, larger shake can be resolved.

[0037] The camera operation unit 121 includes a release button configured such that a first switch (SW1) and a second switch (SW2) are turned on in sequence according to the amount of pressing. When the user presses the release button about halfway, the first switch SW1 is turned on, and when the user has pressed the release button all the way, the second switch SW2 is turned on. Once the first switch SW1 has been turned on, the focus drive control unit 108 performs focusing by driving the focus unit 107, and the aperture drive control unit 104 sets an appropriate exposure amount by driving the aperture unit 103. Once the second switch SW2 has been turned on, the image data obtained from the optical image that has been exposed by the imaging unit 115 is stored in the storage unit 119.

[0038] The camera operation unit 121 further includes a moving image recording switch. After the moving image recording switch is pressed, the camera starts shooting a moving image and ends the recording when the user presses the moving image recording switch again during the recording. When the user operates the release button to turn on the first switch SW1 and the second switch SW2 during the shooting of the moving image, a process for obtaining and recording a still image during the recording of the moving image is executed. The camera operation unit 121 further includes a reproduction mode selection switch, and the reproduction mode can be selected via the reproduction mode selection switch. When the reproduction mode is selected by operating the reproduction mode selection switch, the camera stops the image blur correction operation.

[0039] The camera operation unit 121 further includes a component for changing the imaging frame rate of the camera. The camera can change the imaging frame rate according to the user operation and select low and high frame rates.

[0040] Next, the image blur correction control executed by the lens control unit 111 and the camera control unit 124 is described.

[0041] Figure 2 is a block diagram for describing the control of image blur correction by driving an optical image blur correction unit (first image blur correction unit 211) and an electronic image blur correction unit (second image blur correction unit 214) based on the shake information of the digital camera system 100. In this embodiment, the correction amount for image blur correction is calculated by using an angular velocity sensor (first shake detection component) provided in the interchangeable lens as a shake detection component, and the first image blur correction unit 211 is driven. At the same time, the correction amount for image blur correction in the second image blur correction unit 214 is sent from the interchangeable lens 150 to the camera body 180 via communication to drive the second image blur correction unit 214. That is, the interchangeable lens 150 and the camera body 180 work as a master device and a slave device, respectively, to form an image blur correction system. Note that the first image blur correction unit 211 includesFigure 1 The image blur correction unit 105 and the optical image blur correction control unit 106. In addition, the second image blur correction unit 214 includes Figure 1 the video signal processing unit 117 and the electronic image blur correction control unit 123 in

[0042] Figure 2 The angular velocity sensor 201 of Figure 1 is included in the lens shake detection unit 110 of

[0043] to detect the angular velocity of the shake of the digital camera system 100 and output a voltage corresponding to the angular velocity. The voltage output from the angular velocity sensor is converted into digital angular velocity data by the A / D converter 202 and provided to the image blur correction amount calculation unit 203.

[0044] Figure 3 is a block diagram for describing the detailed configuration of the image blur correction amount calculation unit 203. A high-pass filter (hereinafter referred to as HPF) 301 is used to remove the DC component or the low-frequency component of the angular velocity data detected by the angular velocity sensor 201. By performing first-order integration in the integrator 303, the angular velocity data that has passed through the HPF 301 is converted into angular displacement data. Incomplete integration is applied as the integration calculation performed here to prevent saturation, and a known primary low-pass filter (hereinafter referred to as primary LPF) is used for this integration calculation.

[0045] The angular displacement data calculated by the integrator 303 is supplied to the viewfinder control unit 305 and the limiter 304. The limiter 304 limits the angular displacement data so that the first image blur correction unit 211 and the second image blur correction unit 214 do not reach the edges of the movable range. The angular displacement data limited by the limiter 304 is output as the output from the image blur correction amount calculation unit 203, that is, the image blur correction amount of the captured image. Note that the image blur correction amount (angular displacement data) calculated by the image blur correction amount calculation unit 203 is the sum value of the correction amounts of the first image blur correction unit 211 and the second image blur correction unit 214. Therefore, the displacement amount obtained by adding the control ranges of the first image blur correction unit 211 and the second image blur correction unit 214 is set as the limit value of the limiter 304.

[0046] The viewfinder control unit 305 determines whether an operation such as panning and tilting, which is the user's intention, has been performed, and controls to return the angular displacement data to the center. In other words, the jitter component caused by the user's intended camera viewfinder is removed from the angular velocity data or the angular displacement data detected by the angular velocity sensor 201, and control is performed to correct the image blur caused by the camera shake when performing the intended viewfinder operation.

[0047] Specifically, a predetermined threshold is set within the control edges of the angular displacement data set in the limiter 304, and when the angular displacement data output from the integrator 303 exceeds these thresholds, it is determined that panning has been performed. In the case where it is determined that panning has been performed, the angular velocity data is limited by removing more low-frequency components by increasing the cut-off frequency of the HPF 301. Alternatively, the deviation is subtracted from the angular velocity data input to the integrator 303 to return the output from the integrator 303 to the center. Alternatively, control is performed to return the output from the integrator 303 to the center by increasing the cut-off frequency of the LPF calculation performed in the integrator 303. By controlling in the above-described manner, even when jitter such as panning and tilting, which is the user's intention, has occurred, the image blur correction operation can be controlled so that it falls within the movable ranges of the first image blur correction unit 211 and the second image blur correction unit 214.

[0048] Figure 2 The correction amount splitting unit 204 splits the entire system's image blur correction amount calculated by the image blur correction amount calculation unit 203 into a first correction amount intended for correction in the first image blur correction unit 211 and a second correction amount intended for correction in the second image blur correction unit 214.

[0049] Figure 4A and 4BThis is a diagram showing an exemplary configuration of the correction amount division unit 204. In Figure 4A the multiplier 401 outputs a first correction amount by multiplying the image blur correction amount calculated in the image blur correction amount calculation unit 203 by a predetermined scale factor K1. Here, the scale factor that satisfies the following formula is set as K1.

[0050] 0 ≤ K1 ≤ 1...(Expression 1)

[0051] After multiplying by the predetermined scale factor K1 in the multiplier 401, the image blur correction amount is used as the correction amount used for image blur correction in the first image blur correction unit 211. In addition, the subtractor 402 subtracts the first correction amount from the image blur correction amount calculated in the image blur correction amount calculation unit 203, thereby calculating the second correction amount used for image blur correction in the second image blur correction unit 214. As a result of performing the above calculation, the correction amount can be divided in such a way that the first correction amount and the second correction amount are added to generate the image blur correction amount of the entire system.

[0052] Note that although Figure 4A an example of dividing the image blur correction amount by a predetermined percentage is shown, the image blur correction amount can be divided by frequency band. Figure 4B This shows an exemplary configuration of the correction amount division unit 204 for the case of dividing the image blur correction amount by frequency band. A high-pass filter (hereinafter referred to as HPF) 403 allows only the high-frequency band of the image blur correction amount calculated in the image blur correction amount calculation unit 203 to pass through, and outputs this high-frequency band as the first correction amount. The second correction amount is obtained by extracting the low-frequency band by subtracting the first correction amount (high-frequency band) from the image blur correction amount calculated in the image blur correction amount calculation unit 203.

[0053] Return Figure 2, the driving amount conversion unit 207 converts the first correction amount into a movement amount for appropriately performing image blur correction in the first image blur correction unit 211, and outputs the movement amount as a driving target position. The position sensor 212 detects the position information of the first image blur correction unit 211, and the subtractor 208 obtains a deviation by subtracting the position information of the first image blur correction unit 211 from the driving target position. This deviation is input to the control filter 209, undergoes various types of signal processing such as gain amplification and phase compensation, and is supplied to the OIS driving unit 210. The first image blur correction unit 211 is driven by the OIS driving unit 210; as a result, the correction optical system moves in a direction perpendicular to the optical axis. This results in the formation of a feedback loop, whereby the position sensor 212 again detects the position information of the moved first image blur correction unit 211, calculates the next deviation data, and performs control to reduce the difference between the driving target position and the position information. As a result, the correction optical system is driven to follow the driving target position.

[0054] The second correction amount calculated in the correction division unit 204 is sent to the camera body 180 via the lens communication control unit 112 and the camera communication control unit 125. The electronic correction amount conversion unit 213 adds the motion vector correction amount obtained from the motion vector conversion unit 216 to the correction amount received from the camera communication control unit 125, and outputs the added result as a shearing target position. Based on the image included in the video signal from the imaging signal processing unit 116, the motion vector detection unit (second shake detection component) 215 detects motion vectors in two directions (i.e., the horizontal direction and the vertical direction perpendicular to each other on a plane perpendicular to the optical axis).

[0055] Specifically, examples of the motion vector detection method include the correlation method and the block matching method. Here, it is assumed that the motion vector detection unit 215 adopts the block matching method as an example. In this block matching method, first, the input image signal is divided into a plurality of blocks of an appropriate size (e.g., 16×16 pixels), and the difference from the pixels within a specific range in the previous field or frame is calculated based on each block. Then, the method searches for the block with the minimum sum of the absolute values of these differences in the previous field or frame, and detects the relative offset of this block as the motion vector of this block (differential shake detection method). As a result, the movement amounts in the vertical direction and the horizontal direction (i.e., the motion vectors) are obtained based on each pixel. These motion vectors indicate the movement amount of the images continuously captured per unit time, i.e., the movement amount of the imaging device. In addition, in the case where the motion vector cannot be detected well, it is determined that a motion vector error has occurred. A possible example of the method for determining a motion vector error uses conditions such as a low-luminance signal and a detection value as a peak.

[0056] The motion vector conversion unit 216 sets a rotation axis Y (yaw axis) in the vertical direction and a rotation axis X (pitch axis) in the horizontal direction such that the two axes represent axes perpendicular to each other in a plane perpendicular to the optical axis. Then, using the motion vector output from the motion vector detection unit 215 and the focal length, the yaw angle and the pitch angle, which are rotation angles around the respective axes, are converted. Note that the motion vector conversion unit 216 adds together the difference between the motion vector obtained from the motion vector detection unit 215 and the previous electronic correction amount, and outputs the added result to the electronic correction amount conversion unit 213. The second image blur correction unit 214 shears the image in a direction perpendicular to the optical axis according to the shear position specified by the electronic correction amount conversion unit 213.

[0057] In the above-described manner, the first image blur correction unit 211 and the second image blur correction unit 214 cooperate with each other to share the shake correction of the entire system; thus, the correction range of image blur correction can be expanded.

[0058] Now, the influence of an increase in processing load or the like, which is a problem to be solved by the present invention, on the communication between the interchangeable lens 150 and the camera body 180 will be described.

[0059] The increase in the above-described processing load occurs, for example, due to performing imaging and image display processing at a high frame rate. In the case of imaging at a high frame rate, the communication frequency between the interchangeable lens 150 and the camera body 180 increases. Examples include autofocus control, aperture control, and zoom control, etc., and the communication during the cooperative operation in the above-described image blur correction (hereinafter referred to as cooperative communication) is no exception.

[0060] There are limitations in the processing capabilities of the CPUs of the interchangeable lens 150 and the camera body 180, and there are cases where it is necessary to reduce the cycle of cooperative communication to satisfy, for example, imaging and image display processing at a high frame rate. In this case, a continuous correction amount cannot be obtained, which results in a reduction in the image blur correction effect. Figure 5A shows the relationship between the imaging timing and the cooperative communication timing, and Figure 5B shows the relationship between the imaging timing and the cooperative communication timing in the case where the cooperative communication has a low-speed cycle. As Figure 5B shown, if there is a frame (frame 2) captured between frame 1 and frame 3, for example, for which the shake amount cannot be obtained periodically, a continuous correction amount cannot be calculated.

[0061] In addition to reducing the processing load by reducing the speed of the cooperation communication cycle as described above, there is also a case where it is necessary to set a low-speed detection cycle for the motion vector to meet, for example, the imaging and display processing at a high frame rate. In this case as well, the continuous correction amount cannot be calculated, which similarly results in a reduction in the image blur correction effect. Figure 6A shows the relationship between the imaging timing and the motion vector detection timing, and Figure 6B shows the relationship between the imaging timing and the motion vector detection timing in the case where the motion vector detection has a low-speed cycle. As Figure 6B shown, there is a frame (frame 2) such as the one captured between frame 1 and frame 3 for which the jitter amount cannot be obtained, and the continuous correction amount cannot be calculated.

[0062] In addition, there is a difference between the jitter information detection interval (time period) obtained through the cooperation communication performed in Figure 5B and the jitter information detection interval (time period) obtained in the case where the cycle speed of the motion vector detection performed in Figure 6B is reduced, and the motion vector obtained in Figure 6B represents the jitter information corresponding to two frames. That is, the jitter amount (motion vector) corresponding to frame 3 is the difference between frame 1 and frame 3.

[0063] As a method of performing image blur correction using the above two pieces of jitter information, Figure 7A shows the relationship between imaging, cooperation communication, and motion vector detection in an ideal case, and Figure 7B shows the relationship between imaging, cooperation communication, and motion vector detection in the case where the image blur correction effect is reduced. When the processing load on the CPUs of the interchangeable lens 150 and the camera body 180 is large, control is performed as Figure 7B shown, and the continuous image blur correction amount cannot be calculated.

[0064] Taking this into account, in the present embodiment, a continuous correction amount is generated by appropriately combining the case of the above-mentioned low-speed cooperation communication and the case of the above-mentioned low-speed detection cycle of the motion vector, thereby preventing a reduction in the image blur correction effect.

[0065] Figure 8 is a diagram showing the relationship between the imaging timing, the acquisition of the image blur correction amount through cooperation communication with a low-speed cycle, and the acquisition of the image blur correction amount based on the motion vector with a low-speed detection cycle in the case of performing image blur correction while suppressing a reduction in the image blur correction effect in the electronic correction amount conversion unit 213. In Figure 8 , in Figure 7BThe jitter of the frames that cannot be obtained in [reference document] can be calculated by offsetting the overlapping period of the jitter obtained through cooperative communication and the jitter detected using the motion vector. The offset between the timing of obtaining the jitter through cooperative communication and the timing of obtaining the jitter detected using the motion vector is constant. That is, for the previous frame, the image blur correction amount is calculated based on the jitter amount obtained through cooperative communication, and for the next frame, the image blur correction amount is calculated by subtracting the image blur correction amount calculated for the previous frame from the motion vectors corresponding to the two detected frames. In this way, the continuous correction amounts can be calculated. That is, for one frame, image blur correction is performed using the output from the angular velocity sensor 201 obtained through cooperative communication, and for the next frame (i.e., the next image after the image for which jitter information is obtained through cooperative communication or an image after that), image blur correction is performed using the image blur correction amount calculated based on the motion vector. These operations are alternately performed on a per-frame basis. In other words, when frames (frames 2, 4) that do not obtain jitter information from the angular velocity sensor and frames (frames 1, 3, 5) that do not obtain jitter information based on the motion vector are alternately generated, the control timing is such that one of these types of jitter information can be obtained for each frame.

[0066] Next, with reference to Figure 8 and Figure 9 the flowchart of [reference document] describes an example of the method for calculating the image blur correction amount. Note that Figure 9 the processing shown in [reference document] is repeatedly executed at an arbitrary predetermined period such as the imaging period; in the present embodiment, it is assumed that the imaging period is used.

[0067] First, in step S101, the camera control unit 124 determines whether to reduce the speed of the cooperative communication period. In the present embodiment, it is determined whether to perform cooperative communication based on the processing load on a per-frame basis. In the case of a frame for which it is determined that cooperative communication is not to be performed, the processing proceeds to step S102; in the case of a frame for which it is determined that cooperative communication is to be performed, the processing proceeds to step S104. Note that in the present embodiment, it is assumed that frames for which cooperative communication is performed and frames for which cooperative communication is not performed are alternately arranged.

[0068] Now, the case where it is determined in step S101 that cooperative communication is not to be performed is described. In step S102, the motion vector is obtained, and the processing proceeds to step S103. Here, since the previous execution of step S102 is two frames earlier than the current step, the motion vectors B1 corresponding to two frames can be obtained. In step S103, as shown in Expression 2, the image blur correction amount C1 is calculated by calculating the difference between the difference between the correction amount A1 calculated in step S109 one frame before and the previous image blur correction amount C1_old and the motion vector B1 obtained in step S102, and integrating the calculated difference.

[0069] C1 = ∫(B1 - (A1 - C1_old))...(Expression 2)

[0070] The calculated amount of image blur correction C1 is restricted to fall within the range that can be corrected by the second image blur correction unit 214, and the correction amount that has been applied with panning processing or the like is calculated; then, the process proceeds to step S110.

[0071] Next, the case where it is determined in step S101 that cooperative communication is to be performed is described. In step S104, the camera control unit 124 transmits imaging information such as limiter information, shutter speed, and exposure period in which image blur correction can be applied in the second image blur correction unit 214 from the camera body 180 to the interchangeable lens 150, and the process proceeds to step S105. In step S105, the lens control unit 111 receives the data transmitted from the camera body 180, and the process proceeds to step S106.

[0072] In step S106, the lens control unit 111 uses Figure 2 the angular velocity sensor 201 therein to detect the shake of the camera with respect to the yaw axis and the pitch axis. Note that since the shake information from the angular velocity sensor 201 is angular velocity, an HPF is applied thereto to remove low-frequency components, and the angular velocity is converted into angular displacement data in the integrator, and the process proceeds to step S107.

[0073] In step S107, the lens control unit 111 generates the amount of image blur correction used in the first image blur correction unit 211 and the second image blur correction unit 214, and divides the amount of image blur correction. In the present embodiment, as a method for calculating the amount of image blur correction, the correction amount is calculated within the range representing the sum of the correction ranges in which the first image blur correction unit 211 and the second image blur correction unit 214 can perform correction; in addition, as an example of the division method, the correction amount is divided according to the percentage of each correction range. In the case where image blur correction is performed using the calculated and divided correction amount used in the first image blur correction unit 211, the process proceeds to step S108. In addition, in the case where image blur correction is performed using the calculated and divided correction amount used in the second image blur correction unit 214, the correction amount is transmitted from the interchangeable lens 150 to the camera body 180, and the process proceeds to step S109.

[0074] In step S108, the lens control unit 111 performs image blur correction by controlling the first image blur correction unit 211 using the correction amount calculated in step S107.

[0075] In step S109, the camera control unit 124 receives the data that has been transmitted from the interchangeable lens 150 to the camera body 180, and the process proceeds to step S110. In step S110, the camera control unit 124 performs image blur correction by controlling the second image blur correction unit 214 by using the correction amount calculated and obtained in step S103 or step S109.

[0076] Note that the above description of the embodiment has been provided for the case of performing image blur correction using both the first image blur correction unit 211 and the second image blur correction unit 214. However, as long as the camera body 180 can receive the shake amount detected on the interchangeable lens 150 side in Figure 9 steps S107 and S109, image blur correction may not be performed by the first image blur correction unit 211 on the interchangeable lens 150, and image blur correction may be performed only by the second image blur correction unit 214.

[0077] As described above, according to the present embodiment, even when the detection period of cooperative communication and shake information (such as motion vectors, etc.) is reduced for the purpose of reducing the processing load, a continuous image blur correction amount can be calculated. In this way, a decrease in the image blur correction effect can be suppressed, and appropriate image blur correction can be performed. That is, even when performing high-load processing using, for example, a high frame rate, appropriate image blur correction control can be achieved through cooperation between the camera and the lens.

[0078] (Second Embodiment)

[0079] Figure 10 is a diagram showing the processing of the electronic correction amount conversion unit 213 different from that of the first embodiment; this diagram is a diagram showing the relationship between the imaging timing, the acquisition of the image blur correction amount of cooperative communication reduced by the communication period, and the acquisition of the image blur correction amount of motion vector detection reduced by the detection period.

[0080] In the first embodiment, processing is performed in units of two frames (i.e., the previous frame and the next frame); that is, processing for cooperative communication and motion vector detection is performed every two frames. In the present embodiment, processing is performed in units of three frames; that is, the frequency of cooperative communication is two times every three frames, and the frequency of motion vector detection is one time every three frames. In this case, the detection period of shake by cooperative communication becomes longer than the imaging period of the frame (the detection frequency of shake by cooperative communication becomes lower than the imaging frequency of the frame image). In this way, while reducing the processing load associated with cooperative communication, the processing load associated with motion vector detection can be reduced compared to the first embodiment.

[0081] The following refers to Figure 10 and Figure 11 to describe an example of the method for calculating the amount of image blur correction using the flowchart. Note that components identical to those in Figure 9 are also given the same reference numerals and their descriptions are omitted. The processing shown in Figure 11 is repeatedly executed at an arbitrary predetermined period such as the imaging period; in the present embodiment, it is assumed that the imaging period is used.

[0082] First, in step S201, the camera control unit 124 determines whether cooperative communication has been executed for two consecutive frames. If it is determined that cooperative communication has been executed for less than two consecutive frames, the processing proceeds to step S104; if it is determined that cooperative communication has been executed for more than two consecutive frames, the processing proceeds to step S202.

[0083] In step S202, the camera control unit 124 obtains a motion vector, and the processing proceeds to step S203. Here, since the previous execution of step S202 is three frames earlier than the current step, the motion vector C2 corresponding to three frames can be obtained. In step S203, as shown in Expression 3, by calculating the differences between the correction amounts A2 and B2 calculated in step S109 one frame before and two frames before, the difference from the previous image blur correction amount D2_old, and the difference from the motion vector C2 obtained in step S202, and by performing integration, the image blur correction amount D2 is calculated.

[0084] D2 = ∫(C2 - ((B2 - A2) + (A2 - D2_old)))...(Expression 3)

[0085] The calculated image blur correction amount D2 is restricted to fall within the range that can be corrected by the second image blur correction unit 214, and the correction amount applied with panning processing or the like is calculated; then, the processing proceeds to step S110.

[0086] As described above, according to the present embodiment, even when the detection periods of cooperative communication and jitter information (such as motion vectors) are unevenly reduced for the purpose of reducing the processing load, continuous image blur correction amounts can be calculated. In this way, appropriate image blur correction can be performed while suppressing a decrease in the image blur correction effect. That is, even when performing high-load processing using, for example, a high frame rate, appropriate image blur correction control can be achieved through cooperation between the camera and the lens.

[0087] In the first embodiment, the camera control unit 124 controls the acquisition of shake information in such a way that frames for acquiring shake information through cooperative communication and frames for acquiring shake information based on motion vectors are alternately generated. Further, in the present embodiment, the acquisition of shake information is controlled in such a way that after generating frames for acquiring shake information through cooperative communication twice in succession, a frame for acquiring shake information based on motion vectors is generated once. The offset method for the two timings is not limited to the above method, as long as it is controlled in such a way that the offset between the timing for acquiring shake information through cooperative communication and the timing for acquiring shake information based on motion vectors is constant, so that a continuous correction amount can be obtained. In order to make the offset between the two timings constant, it is sufficient to make the acquisition period of shake information based on cooperative communication coincide with the acquisition period of shake information based on motion vectors. Note that the acquisition period of shake information represents the period of frames related to the acquisition of shake information and frames not related to the acquisition of shake information. For example, in the case where, as in the present embodiment, after generating frames for acquiring shake information through cooperative communication twice in succession, a frame not for acquiring shake information through cooperative communication is generated once and this is repeated a number of times, the acquisition period of shake information based on cooperative communication is three frames. Similarly, in the case where after generating frames not for acquiring shake information based on motion vectors twice in succession, a frame for acquiring shake information based on motion vectors is generated once and this is repeated a number of times, the acquisition period of shake information based on motion vectors is three frames. Further, although in the first embodiment and the present embodiment, only one of shake information based on cooperative communication and shake information based on motion vectors is acquired corresponding to each frame, there may be frames for acquiring both types of shake information. In terms of processing load, the mode of frames for acquiring both types of shake information is more disadvantageous; however, for example, since there may be a timing when shake information based on cooperative communication and shake information based on motion vectors become necessary in operations other than shake correction, there may be a timing for acquiring both types of shake information.

[0088] Note that although the above description of each embodiment has been provided for the case of an example where an angular velocity sensor is used as a shake detection component, other shake detection components may also be used. For example, an acceleration sensor may be used to calculate the shake amount based on acceleration, and the shake amount of the camera system may be calculated by detecting shake using a combination of multiple sensors.

[0089] Further, the lens shift method of moving some lenses constituting an interchangeable lens in a direction perpendicular to the optical axis and the image shear of moving a captured image in a direction perpendicular to the optical axis have been described above as examples of an image blur correction component. However, a method of correcting image blur by using, for example, image sensor shift of moving an image sensor in a direction perpendicular to the optical axis is allowed.

[0090] Other embodiments

[0091] Embodiments of the present invention can also be implemented by the following method, that is, software (program) that executes the functions of the above-described embodiments is provided to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0092] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image blur correction device, comprising: An acquisition component for acquiring shake information from a plurality of shake detection components for detecting shake; And A plurality of image blur correction amount calculation components, including: A first image blur correction amount calculation component for calculating image blur correction amounts corresponding to partial images included in a plurality of consecutive images based on the shake information from a first shake detection component acquired by the acquisition component, and A second image blur correction amount calculation component for calculating image blur correction amounts corresponding to the partial images included in the plurality of consecutive images by using a method different from the method used by the first image blur correction amount calculation component based on the shake information from a second shake detection component acquired by the acquisition component, Characterized in that the image blur correction device further comprises: A control component for controlling the plurality of image blur correction amount calculation components such that at least one of the plurality of image blur correction amount calculation components acquires an image blur correction amount corresponding to a plurality of consecutive captured images respectively, and such that images in which the first image blur correction amount calculation component among the plurality of image blur correction amount calculation components calculates an image blur correction amount without using the shake information from the second shake detection component and images in which the second image blur correction amount calculation component among the plurality of image blur correction amount calculation components calculates an image blur correction amount are generated periodically.

2. The image blur correction device according to claim 1, wherein The acquisition component acquires the shake information from the first shake detection component through communication.

3. The image blur correction device according to claim 1, further comprising: An image blur correction control component for controlling image blur correction of a captured image based on the image blur correction amounts calculated by the plurality of image blur correction amount calculation components.

4. The image blur correction device according to claim 3, wherein The image blur correction control component controls electronic image blur correction for electronically correcting image blur of the plurality of consecutive captured images.

5. The image blur correction device according to claim 4, wherein The image blur correction control component further controls a correction component for optically correcting image blur of the captured image.

6. The image blur correction device according to claim 1, wherein The second shake detection component is a differential shake detection component for detecting shake based on the difference between the plurality of consecutive captured images.

7. The image blur correction device according to claim 6, wherein The second shake detection component obtains the difference based on an image for which the corresponding image blur correction amount cannot be obtained by at least the first image blur correction amount calculation component.

8. The image blur correction device according to claim 1, wherein The acquisition component acquires the shake information from the first shake detection component at a second period longer than a first period for capturing the plurality of consecutive images.

9. The image blur correction device according to claim 1, wherein The acquisition component acquires jitter information from the second jitter detection component in a third period that is longer than the first period in which the multiple consecutive images are captured in a ratio.

10. The image blur correction device according to claim 1, wherein the control component controls the multiple image blur correction amount calculation components such that two periods are made consistent, that is, the period of the image in which the first image blur correction amount calculation component calculates the corresponding image blur correction amount and the image in which the first image blur correction amount calculation component does not calculate the corresponding image blur correction amount, and the period of the image in which the second image blur correction amount calculation component calculates the corresponding image blur correction amount and the image in which the second image blur correction amount calculation component does not calculate the corresponding image blur correction amount.

11. The image blur correction device according to claim 10, wherein the multiple image blur correction amount calculation components include two image blur correction amount calculation components represented by the first image blur correction amount calculation component and the second image blur correction amount calculation component, and the control component controls the multiple image blur correction amount calculation components such that images in which the first image blur correction amount calculation component calculates the image blur correction amount and images in which the second image blur correction amount calculation component calculates the image blur correction amount are alternately generated.

12. The image blur correction device according to claim 1, wherein the second jitter detection component is a differential jitter detection component for detecting jitter based on the difference between the multiple consecutively captured images, and the second image blur correction amount calculation component calculates the image blur correction amount corresponding to the first image based on the jitter information from the second jitter detection component based on the difference between the first image and the second image and the jitter information from the first jitter detection component corresponding to a third image captured between the first image and the second image, where for the first image, the corresponding jitter information from the first jitter detection component cannot be obtained.

13. The image blur correction device according to claim 12, wherein the third image is an image captured before the first image.

14. The image blur correction device according to claim 1, wherein when the first period of capturing the multiple consecutive images is shorter than a predetermined value, the control component controls the multiple image blur correction amount calculation components such that at least one of the multiple image blur correction amount calculation components calculates the image blur correction amount corresponding to each of the multiple consecutively captured images.

15. An imaging device, comprising: an imaging component for capturing multiple consecutive images, characterized in that the imaging device further comprises: the image blur correction device according to claim 1; a second jitter detection component; and an image blur correction component for performing image blur correction on the multiple consecutive images based on the image blur correction amounts obtained by the multiple image blur correction amount calculation components. Among them, the obtaining component obtains the detection result of the first shake detection component included in the installed lens device through communication with a second period longer than the first period for photographing the plurality of consecutive images.

16. The imaging device according to claim 15, wherein the first image blur correction amount calculation component calculates the image blur correction amount at the second period.

17. An imaging device, comprising: an imaging sensor for photographing a plurality of consecutive images at a first period; a shake detection component for detecting shake based on the plurality of consecutive images; a plurality of image blur correction amount obtaining components, including: a first image blur correction amount obtaining component for obtaining image blur correction amounts respectively corresponding to partial images included in the plurality of consecutive images based on shake information obtained from the installed lens device through communication, and a second image blur correction amount obtaining component for obtaining image blur correction amounts respectively corresponding to partial images included in the plurality of consecutive images by using a method different from the method used by the first image blur correction amount obtaining component based on the detection result of the shake detection component and the shake information obtained through communication; and an image blur correction component for performing image blur correction on the plurality of consecutive images based on the image blur correction amounts obtained by the plurality of image blur correction amount obtaining components, wherein at least one of the plurality of image blur correction amount obtaining components obtains an image blur correction amount corresponding to each of the plurality of consecutive images respectively, characterized in that the first image blur correction amount obtaining component obtains the image blur correction amount without using the detection result of the shake detection component.

18. A control method for an image blur correction device, the control method comprising: obtaining shake information from a plurality of shake detection components for detecting shake; and calculating a plurality of image blur correction amounts, including: a first calculation for calculating image blur correction amounts respectively corresponding to partial images included in a plurality of consecutive images based on the shake information from a first shake detection component obtained in the obtaining, and a second calculation for calculating image blur correction amounts respectively corresponding to partial images included in the plurality of consecutive images by using a method different from the method used in the first calculation based on the shake information from a second shake detection component obtained in the obtaining, characterized in that the control method further comprises: controlling the calculation such that at least one of the image blur correction amounts is obtained corresponding to each of the plurality of consecutively photographed images, and such that images for calculating the image blur correction amount without using the shake information from the second shake detection component in the first calculation and images for calculating the image blur correction amount in the second calculation are periodically generated.

19. A control method for an imaging device, the imaging device having an imaging sensor for photographing a plurality of consecutive images at a first period, the control method comprising: detecting shake based on the plurality of consecutive images, The control method is characterized in that it further includes: Obtaining a plurality of image blur correction amounts, including: A first obtaining, for obtaining image blur correction amounts respectively corresponding to partial images included in the plurality of consecutive images without based on a shake detection result but based on shake information obtained through communication from an installed lens device, and A second obtaining, for obtaining image blur correction amounts respectively corresponding to partial images included in the plurality of consecutive images by using a method different from the method used in the first obtaining, based on the shake detection result and the shake information obtained through communication; and Performing image blur correction on the plurality of consecutive images based on the image blur correction amounts obtained through the obtaining; wherein, in at least one of the first obtaining and the second obtaining, image blur correction amounts are obtained corresponding to the plurality of consecutive images respectively.

20. A computer-readable storage medium storing a program, the program being used to cause a computer to execute each step of the control method according to claim 18 or 19.

21. A computer program product, which includes a program, the program being used to cause a computer to execute each step of the control method according to claim 18 or 19.

Images