Image blur correction device, imaging device, control method thereof, program, and storage medium
A dual-image blur correction system using optical and electronic methods shares the correction load across devices, addressing communication delays and high processing loads to maintain effective stabilization during high frame rates.
Patent Information
- Application Number
- JP2020147675
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-02
- Publication Date
- 2024-11-07
- Estimated Expiration
- 2040-09-02
AI Technical Summary
Existing image stabilization methods face a decline in effectiveness due to communication errors and delays, especially when image blur correction devices communicate at low speeds, leading to reduced processing efficiency and stabilization performance.
A dual-image blur correction system is implemented, combining optical and electronic correction methods, where one device acts as a master and the other as a slave, sharing the blur correction load and using angular velocity sensors and motion vector detection to calculate and divide correction amounts across multiple devices, ensuring continuous correction even during high processing loads.
This approach maintains effective image stabilization by alternating blur correction methods, reducing the impact of communication delays and high processing loads, thereby ensuring consistent image quality during high frame rate operations.
Smart Images

Figure 0007579088000001 
Figure 0007579088000002 
Figure 0007579088000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for performing image blur correction in which image blur correction units mounted on a plurality of devices cooperate with each other via communication. [Background technology]
[0002] In recent years, in the video industry, a method has been used in which smooth slow-motion images are obtained by shooting at a high frame rate and setting the playback speed to be equal to or lower than the shooting frame rate. However, this method increases the processing load on the camera because the shooting is performed at a high frame rate. As a result, when image stabilization means installed in multiple devices cooperate with each other via communication to perform image stabilization control, communication errors and delays may occur. For this reason, a method has been proposed for suppressing the deterioration of the image stabilization effect even when a communication error or delay occurs.
[0003] Patent Document 1 discloses a method for correcting image blur by selecting an amount of blur correction obtained from shake information when shake information can be acquired via communication, and by selecting an amount of blur correction predicted by a prediction means when shake information cannot be acquired via communication. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2017-219635 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, Patent Document 1 has the following problem: In Patent Document 1, the image blur correction amount is calculated by a method such as a Kalman filter or a least squares method that predicts from the previous correction amount, so if an incorrect correction amount is calculated, the effect of the image blur correction is significantly reduced.
[0006] In addition, communication errors and delays can be improved by slowing down the communication cycle, but slowing down the communication cycle also slows down the cycle for acquiring signals for image shake compensation, reducing the effectiveness of image shake compensation.
[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to suppress a decrease in the effect of image stabilization when the communication cycle is set to a slow speed when image stabilization control is performed while communicating between a plurality of devices. [Means for solving the problem]
[0008] An image blur correction device according to the present invention includes a plurality of image blur correction amount calculation means including an acquisition means for acquiring shake information from a plurality of shake detection means that detect shake, a first image blur correction amount calculation means for calculating an image blur correction amount corresponding to each of a portion of a plurality of consecutive images based on the shake information from a first shake detection means acquired by the acquisition means, and a second image blur correction amount calculation means for calculating an image blur correction amount corresponding to each of the portion of the plurality of consecutive images by a method different from that of the first image blur correction amount calculation means based on the shake information from a second shake detection means acquired by the acquisition means, and a control means for controlling the plurality of image blur correction amount calculation means so that the image blur correction amount is obtained by at least any of the plurality of image blur correction amount calculation means corresponding to each of a plurality of consecutive images captured, and an image for which the image blur correction amount is calculated by the first image blur correction amount calculation means and an image for which the image blur correction amount is calculated by the second image blur correction amount calculation means are periodically generated among the plurality of image blur correction amount calculation means. When a first period during which the consecutive images are captured is shorter than a predetermined value, the control means controls the plurality of image blur correction amount calculation means so that the image blur correction amount is calculated corresponding to each of the consecutive images captured by at least one of the plurality of image blur correction amount calculation means. It is characterized by: Effect of the Invention
[0009] According to the present invention, when image stabilization control is performed while communicating between a plurality of devices, it is possible to suppress a decrease in the image stabilization effect that occurs when the communication cycle is set to a low speed. [Brief description of the drawings]
[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of a lens-interchangeable digital camera system that is an embodiment of an image stabilization device of the present invention; [Diagram 2] FIG. 2 is a block diagram illustrating image blur correction control according to the first embodiment. [Diagram 3] FIG. 4 is a block diagram illustrating an image blur correction amount calculation unit according to the first embodiment. [Figure 4] FIG. 4 is a block diagram illustrating a correction amount dividing unit according to the first embodiment. [Diagram 5] FIG. 11 is a timing diagram illustrating cooperative communication in which the processing cycle is slower than the imaging cycle. [Figure 6] 11 is a timing chart illustrating motion vector detection in which the detection cycle is slowed down relative to the imaging cycle. [Figure 7] 11 is a timing diagram for explaining a problem that arises when cooperative communication and a motion vector detection cycle are slowed down relative to an imaging cycle. [Figure 8] FIG. 4 is a timing chart for explaining an electronic correction amount conversion operation in the first embodiment. [Figure 9] 5 is a flowchart illustrating an electronic correction amount conversion operation in the first embodiment. [Figure 10] FIG. 11 is a timing chart for explaining an electronic correction amount conversion operation in the second embodiment. [Figure 11] 10 is a flowchart illustrating an electronic correction amount conversion operation in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.
[0012] (First embodiment) A first embodiment of the present invention will be described in detail below with reference to the accompanying drawings, but first, matters common to each embodiment will be described.
[0013] 1 is a block diagram showing a typical configuration of a lens-interchangeable digital camera system for taking still images and moving images, which is an embodiment of an image stabilization device of the present invention. Note that the scope of application of the present invention is not limited to digital cameras, and can be applied to various types of imaging devices.
[0014] In the following description of the embodiments, vibration of the imaging device is referred to as "shake", and the effect on a captured image caused by the shake of the imaging device is referred to as "image blur".
[0015] In FIG. 1, a digital camera system 100 is made up of an interchangeable lens 150 and a camera body 180, and the interchangeable lens 150 is attached to the camera body 180 when in use.
[0016] The zoom unit 101 of the interchangeable lens 150 includes a zoom lens that changes magnification. The zoom drive control unit 102 drives and controls the zoom unit 101. The aperture unit 103 has an aperture function. The aperture drive control unit 104 drives and controls the aperture unit 103. The image blur correction unit 105 includes an image blur correction lens (hereinafter also referred to as a correction lens or OIS) such as a shift lens. The image blur correction unit 105 is a first image blur correction means, and is driven and controlled by the optical image blur correction control unit 106. The focus unit 107 includes a focus lens that performs focus adjustment to form a subject image. The focus drive control unit 108 drives and controls the focus unit 107.
[0017] The lens operation unit 109 is an operation unit that the user uses to operate the interchangeable lens. The lens shake detection unit 110 detects the amount of shake of the interchangeable lens 150 and outputs a detection signal to a lens system control unit 111. The lens system control unit (hereinafter referred to as the lens control unit) 111, which controls the entire interchangeable lens 150, has a CPU (Central Processing Unit) and generally controls each drive control unit and correction control unit of the interchangeable lens. The lens control unit 111 communicates with the camera system control unit 124 of the camera body 180 via a lens communication control unit 112.
[0018] Next, the camera body 180 will be described. The camera body 180 includes a shutter unit 113. The shutter drive control unit 114 drives and controls the shutter unit 113. The imaging unit 115 includes an imaging element, and performs photoelectric conversion on the light image formed by passing through each lens group to output an electric signal. The imaging signal processing unit 116 converts the electric signal output from the imaging unit 115 into a video signal. The video signal processing unit 117 processes the video signal output from the imaging signal processing unit 116 according to the application. For example, the video signal processing unit 117 changes the cut-out position of the video signal according to the correction amount of the electronic image stabilization control unit 123. The electronic image stabilization control unit 123 is a second image stabilization means, and controls the image stabilization by cutting out the image.
[0019] The display unit 118 displays an image as necessary based on a signal output from the video signal processing unit 117. The storage unit 119 stores various data such as video information. The power supply unit 120 supplies power to the entire system depending on the purpose. The camera operation unit 121 is an operation unit used by the user to operate the camera system, and outputs an operation signal to the camera system control unit 124. The camera shake detection unit 122 detects the amount of shake of the camera, and outputs a detection signal to the camera system control unit 124. The camera system control unit (hereinafter referred to as the camera control unit) 124 includes a CPU, and controls the entire camera system. The camera control unit 124 communicates with the lens communication control unit 112 of the interchangeable lens 150 via the camera communication control unit 125. In other words, when the interchangeable lens 150 is attached to the camera body 180 and electrically connected, mutual communication is performed by the lens communication control unit 112 and the camera communication control unit 125.
[0020] Next, an outline of the operation of the digital camera system 100 having the above configuration will be described. The lens operation unit 109 and the camera operation unit 121 each include an image stabilization switch that can select ON / OFF of image stabilization. When the user operates the image stabilization switch to select ON for image stabilization, the lens control unit 111 or the camera control unit 124 instructs the optical image stabilization control unit 106 or the electronic image stabilization control unit 123 to perform an image stabilization operation. Each image stabilization control unit continues to control the image stabilization until an instruction to turn OFF image stabilization is given.
[0021] Furthermore, the camera operation unit 121 includes an image blur correction mode switch that can select a first mode or a second mode for image blur correction. The first mode is a mode in which image blur correction is performed by using both optical image blur correction (first image blur correction) and electronic image blur correction (second image blur correction). When the first mode is selected, a wider correction angle is possible by performing correction in cooperation with optical image blur correction and electronic image blur correction. Furthermore, the image signal processing unit 117 narrows the cut-out range of the image signal and changes the cut-out position according to the amount of image blur correction, so that it is possible to handle larger shakes.
[0022] The camera operation unit 121 includes a release button configured to turn on a first switch (SW1) and a second switch (SW2) in sequence according to the amount of depression. When the user presses the release button about halfway, the first switch SW1 turns on, and when the user presses the release button all the way, the second switch SW2 turns on. When the first switch SW1 is turned on, the focus drive control unit 108 drives the focus unit 107 to adjust the focus, and the aperture drive control unit 104 drives the aperture unit 103 to set an appropriate exposure amount. When the second switch SW2 is turned on, image data obtained from the light image exposed on the imaging unit 115 is stored in the memory unit 119.
[0023] Camera operation unit 121 also includes a movie recording switch. The camera starts shooting a movie after the movie recording switch is pressed, and stops recording when the user presses the movie recording switch again during recording. When the user operates the release button to turn on first switch SW1 and second switch SW2 during movie shooting, a process is executed to acquire and record a still image during movie recording. Camera operation unit 121 also includes a playback mode selection switch that can select a playback mode. When playback mode is selected by operating the playback mode selection switch, the camera stops the image stabilization operation.
[0024] Furthermore, the camera operation unit 121 includes a means for changing the image capture frame rate of the camera. The camera can change the image capture frame rate in response to a user operation, and can select a low-speed or high-speed frame rate.
[0025] Next, the image blur correction control executed by the lens control unit 111 and the camera control unit 124 will be described.
[0026] FIG. 2 is a block diagram for explaining control for performing image blur correction by driving an optical image blur correction section (first image blur correction section 211) and an electronic image blur correction section (second image blur correction section 214) based on shake information of the digital camera system 100. In this embodiment, an angular velocity sensor (first shake detection section) provided in the interchangeable lens as shake detection section is used to calculate a correction amount for image blur correction and drive the first image blur correction section 211. At the same time, a correction amount for performing image blur correction by the second image blur correction section 214 is transmitted from the interchangeable lens 150 to the camera body 180 via communication, and the second image blur correction section 214 is driven. That is, this is an image blur correction system in which the interchangeable lens 150 operates as a master and the camera body 180 operates as a slave. The first image blur correction section 211 includes the image blur correction unit 105 and the optical image blur correction control section 106 in FIG. 1. 1. The second image blur correction unit 214 includes the video signal processing unit 117 and the electronic image blur correction control unit 123 shown in FIG.
[0027] 2 is included in the lens shake detection unit 110 in Fig. 1, detects the angular velocity of the shake of the digital camera system 100, and outputs a voltage according to the angular velocity. The output voltage of the angular velocity sensor is converted into digital angular velocity data by an A / D converter 202 and supplied to an image shake correction amount calculation unit 203.
[0028] The image blur correction amount calculation unit 203 calculates a correction amount for correcting image blur caused by shaking of the digital camera system 100. The digital camera system 100 includes two image blur correction units, a first image blur correction unit 211 and a second image blur correction unit 214. However, the image blur correction amount calculated by the image blur correction amount calculation unit 203 does not calculate the correction amount for each of the two image blur correction units, but calculates the image blur correction amount for the entire digital camera system 100.
[0029] 3 is a block diagram illustrating a detailed configuration of the image blur correction amount calculation unit 203. A high-pass filter (hereinafter, HPF) 301 is used to remove DC components or low-frequency components from the angular velocity data detected by the angular velocity sensor 201. The angular velocity data passed through the HPF 301 is converted into angular displacement data by first-order integration in an integrator 303. In the integration calculation performed here, incomplete integration is applied to prevent saturation, and is performed by a generally known first-order low-pass filter (hereinafter, first-order LPF).
[0030] The angular displacement data calculated by the integrator 303 is supplied to a framing control unit 305 and a limiter 304. The limiter 304 limits the angular displacement data so that the first image blur correction unit 211 and the second image blur correction unit 214 do not hit the ends of their movable ranges. The angular displacement data limited by the limiter 304 is output from the image blur correction amount calculation unit 203, that is, output as the image blur correction amount of the captured image. Note that the image blur correction amount (angular displacement data) calculated by the image blur correction amount calculation unit 203 is the sum of the correction amounts of the first image blur correction unit 211 and the second image blur correction unit 214. Therefore, the displacement amount obtained by summing the control range of the first image blur correction unit 211 and the control range of the second image blur correction unit 214 is set as the limit value of the limiter 304.
[0031] The framing control unit 305 determines whether or not the user has performed an intended operation such as panning or tilting, and performs control to return the angular displacement data to the center. In other words, it removes the shake component due to the user's intended camera framing from the angular velocity data or angular displacement data detected by the angular velocity sensor 201, and performs control to correct image blur caused by camera shake while performing the intended framing.
[0032] Specifically, a predetermined threshold is set further inside the control end of the angular displacement data provided in the limiter 304, and if the angular displacement data output from the integrator 303 exceeds the threshold, it is determined that panning has occurred. If panning is determined, the cutoff frequency of the HPF 301 is increased to remove more low-frequency components, thereby limiting the angular velocity data. Alternatively, an offset is subtracted from the angular velocity data input to the integrator 303 so that the output of the integrator 303 returns to the center. Alternatively, the cutoff frequency of the LPF calculation performed by the integrator 303 is increased to control the output of the integrator 303 to return to the center. By controlling in this way, even if a shake intended by the user such as panning or tilting occurs, it is possible to control the image stabilization operation so that it falls within the movable range of the first image stabilization unit 211 and the second image stabilization unit 214.
[0033] The correction amount division unit 204 in FIG. 2 divides the image blur correction amount of the entire system calculated by the image blur correction amount calculation unit 203 into a first correction amount for correction by the first image blur correction unit 211 and a second correction amount for correction by the second image blur correction unit 214.
[0034] 4 is a diagram showing an example of the configuration of the correction amount division unit 204. In FIG. 4(a), a multiplier 401 multiplies the image blur correction amount calculated by the image blur correction amount calculation unit 203 by a predetermined magnification K1 to output a first correction amount. Here, K1 is expressed as follows: 0 ≦ K1 ≦ 1 … (Formula 1) A magnification is set so that: The image blur correction amount multiplied by the predetermined magnification K1 by the multiplier 401 becomes the correction amount when the first image blur correction unit 211 performs image blur correction. Moreover, a subtractor 402 calculates a second correction amount when the second image blur correction unit 214 performs image blur correction by subtracting the first correction amount from the image blur correction amount calculated by the image blur correction amount calculation unit 203. By performing calculations in this manner, the correction amount can be divided so that the sum of the first correction amount and the second correction amount becomes the image blur correction amount for the entire system.
[0035] Although Fig. 4(a) shows an example in which the image blur correction amount is divided at a predetermined ratio, it may be divided by frequency band. Fig. 4(b) shows an example of the configuration of the correction amount dividing unit 204 when dividing by frequency band. A high-pass filter (hereinafter, HPF) 403 passes only the high frequency band of the image blur correction amount calculated by the image blur correction amount calculation unit 203, and outputs it as a first correction amount. The second correction amount is obtained by extracting a low frequency band by subtracting the first correction amount (high frequency band) from the image blur correction amount calculated by the image blur correction amount calculation unit 203.
[0036] Returning to FIG. 2, the drive amount conversion unit 207 converts the first correction amount into a movement amount for the first image blur correction unit 211 to appropriately perform image blur correction, and outputs it as a drive target position. The position sensor 212 detects position information of the first image blur correction unit 211, and the subtractor 208 subtracts the position information of the first image blur correction unit 211 from the drive target position to obtain a deviation. The deviation is input to the control filter 209, and various signal processes such as gain amplification and phase compensation are performed, and the deviation is supplied to the OIS drive unit 210. The first image blur correction unit 211 is driven by the OIS drive unit 210, so that the correction optical system moves in a direction perpendicular to the optical axis. Then, the position information of the moved first image blur correction unit 211 is detected again by the position sensor 212, and the next deviation data is calculated, forming a feedback loop, and the difference between the drive target position and the position information is controlled to be small. As a result, the correction optical system is driven to follow the drive target position.
[0037] The second correction amount calculated by the correction division unit 204 is transmitted to the camera body 180 via the lens communication control unit 112 and the camera communication control unit 125. The electronic correction amount conversion unit 213 adds the motion vector correction amount acquired from the motion vector conversion unit 216 to the correction amount received by the camera communication control unit 125, and outputs the result as a cut-out target position. The motion vector detection unit (second shake detection means) 215 detects motion vectors in two directions, the horizontal direction and the vertical direction, which are mutually orthogonal on a plane orthogonal to the optical axis, based on an image included in the video signal from the imaging signal processing unit 116.
[0038] Specifically, the motion vector detection method includes a correlation method, a block matching method, and the like. Here, as an example, the block matching method is adopted in the motion vector detection unit 215. In the block matching method, the input image signal is first divided into a plurality of blocks of an appropriate size (for example, 16×16 pixels), and the difference between each block and a certain range of pixels in the previous field or frame is calculated. Then, the block in the previous field or frame that has the smallest sum of the absolute values of the differences is searched for, and the relative deviation of the block is detected as the motion vector of the block (differential shake detection method). As a result, the vertical and horizontal movement amounts (i.e., the motion vectors) are obtained in pixel units. The motion vectors indicate the movement amount per unit time of successive captured images, that is, the movement amount of the imaging device. If the motion vector cannot be detected successfully, a motion vector error judgment is performed. As an example of a method for judging an error in a motion vector, a condition such as a small luminance signal or a peak value detected can be considered.
[0039] The motion vector conversion unit 216 sets a vertical rotation axis Y (Yaw axis) and a horizontal rotation axis X (Pitch axis) so that they form mutually orthogonal axes on a plane orthogonal to the optical axis. Then, the motion vector conversion unit 216 converts the Yaw angle and Pitch angle, which are the rotation angles around each axis, using the motion vector and focal length output from the motion vector detection unit 215. The motion vector conversion unit 216 integrates the difference between the motion vector acquired from the motion vector detection unit 215 and the previous electronic correction amount, and outputs the result to the electronic correction amount conversion unit 213. The second image blur correction unit 214 cuts out the image in a direction perpendicular to the optical axis according to the cut-out position specified by the electronic correction amount conversion unit 213.
[0040] In this way, the first image blur correction unit 211 and the second image blur correction unit 214 operate in cooperation to share and correct the shake of the entire system, making it possible to expand the correction range of image blur correction.
[0041] Here, the effect that an increase in processing load, which is a problem of the present invention, has on communication between the interchangeable lens 150 and the camera body 180 will be described.
[0042] One of the causes of the increase in the processing load described above is imaging and development processing at a high frame rate. When imaging at a high frame rate, the frequency of communication between the interchangeable lens 150 and the camera body 180 increases. Examples of this include autofocus control, aperture control, and zoom control, and communication when performing cooperative operations in the above-mentioned image stabilization (hereinafter referred to as cooperative communication) is no exception.
[0043] The processing power of the CPU of the interchangeable lens 150 and the camera body 180 is limited, and it may be necessary to slow down the cooperative communication cycle to accommodate imaging and development processes at a high frame rate. In such cases, a continuous correction amount cannot be obtained, which leads to a decrease in the image blur correction effect. FIG. 5(a) shows the relationship between the imaging timing and the cooperative communication timing, and FIG. 5(b) shows the relationship between the imaging timing and the cooperative communication timing when the cooperative communication cycle is slow. As shown in FIG. 5(b), when a frame occurs for which the shake amount cannot be obtained periodically, such as a frame (Frame 2) captured between Frame 1 and Frame 3, a continuous correction amount cannot be calculated.
[0044] In addition to reducing the processing load by slowing down the cooperative communication cycle as described above, there are cases where it is necessary to slow down the motion vector detection cycle to accommodate imaging and development processes at a high frame rate. In such cases, a continuous correction amount cannot be calculated, which similarly leads to a decrease in the image stabilization effect. Figure 6(a) shows the relationship between the imaging timing and the motion vector detection timing, and Figure 6(b) shows the relationship between the imaging timing and the motion vector detection timing when the motion vector detection cycle is slow. As shown in Figure 6(b), there are frames in which the shake amount cannot be obtained, such as the frame (Frame 2) captured between Frame 1 and Frame 3, and a continuous correction amount cannot be calculated.
[0045] In addition, there is a difference between the blur information detection interval (time) obtained by the cooperative communication implemented in Fig. 5(b) and the blur information detection interval (time) obtained when the motion vector detection cycle is slowed down implemented in Fig. 6(b), and the motion vector obtained in Fig. 6(b) is blur information for two frames. In other words, the shake amount (motion vector) corresponding to Frame 3 is the difference between Frame 1 and Frame 3.
[0046] As a method of performing image blur correction using the above-mentioned two pieces of blur information, Fig. 7(a) shows the relationship between imaging, cooperative communication, and motion vector detection in an ideal case, and Fig. 7(b) shows the relationship between imaging, cooperative communication, and motion vector detection in a case where the image blur correction effect is reduced. When the processing load of the CPU of the interchangeable lens 150 and the camera body 180 is large, the control shown in Fig. 7(b) is performed, and a continuous image blur correction amount cannot be calculated.
[0047] Therefore, in this embodiment, the above-mentioned case where the cooperative communication is slow and the case where the motion vector detection period is slow are appropriately combined to create a continuous correction amount, thereby preventing a decrease in the image blur correction effect.
[0048] FIG. 8 is a diagram showing the relationship between the imaging timing, the acquisition of the image blur correction amount by the cooperative communication with a slow cycle, and the acquisition of the image blur correction amount by the motion vector with a slow detection cycle, when the electronic correction amount conversion unit 213 performs image blur correction while suppressing the deterioration of the image blur correction effect. In FIG. 8, the overlap period of the shake acquired by the cooperative communication and the shake detected by the motion vector is shifted, so that the shake of the frame that could not be acquired in FIG. 7(b) can be calculated. The shift between the timing of acquiring the shake by the cooperative communication and the timing of acquiring the shake detected by the motion vector is constant. That is, in the previous frame, the image blur correction amount is calculated according to the shake amount acquired by the cooperative communication, and in the next frame, the image blur correction amount is calculated by subtracting the image blur correction amount calculated in the previous frame from the detected motion vector for two frames. This makes it possible to calculate a continuous correction amount. That is, in one frame, image blur correction is performed using the output from the angular velocity sensor 201 acquired by the cooperative communication, and in the next frame (the image after the image after which the shake information is acquired by the cooperative communication), image blur correction is performed using the image blur correction amount calculated from the motion vector. This is done alternately for each frame. In other words, frames (Frames 2 and 4) in which no shake information is obtained from the angular velocity sensor and frames (Frames 1, 3 and 5) in which no shake information is obtained from the motion vector occur alternately, but the timing is controlled so that some shake information can be obtained in all frames.
[0049] An example of a method for calculating the image blur correction amount will be described below with reference to the flowcharts of Fig. 8 and Fig. 9. Note that the process shown in Fig. 9 is repeatedly executed at an arbitrary predetermined cycle such as an image capture cycle, which is set as the image capture cycle in this embodiment. First, in step S101, the camera control unit 124 determines whether or not to slow down the period of cooperative communication. In this embodiment, it is determined whether or not to perform cooperative communication on a frame-by-frame basis depending on the processing load. For a frame for which it is determined that cooperative communication is not to be performed, the process proceeds to step S102, and for a frame for which it is determined that cooperative communication is to be performed, the process proceeds to step S104. Note that in this embodiment, frames for which cooperative communication is performed and frames for which cooperative communication is not performed are arranged alternately.
[0050] A case where it is determined in step S101 that cooperative communication will not be performed will be described. In step S102, a motion vector is acquired, and the process proceeds to step S103. Here, since the previously executed step S102 was two frames ago, it is possible to acquire motion vectors B1 for two frames. In step S103, as shown in (Equation 2), the difference between the correction amount A1 calculated in step S109 one frame ago and the previous image blur correction amount C1_old, and the difference between the motion vector B1 acquired in step S102 are calculated and integrated to calculate the image blur correction amount C1.
[0051] C1=∫(B1-(A1-C1_old)) …(Formula 2) The calculated image blur correction amount C1 is limited so as to fall within a range that can be corrected by the second image blur correction unit 214, and a correction amount that has undergone panning processing or the like is calculated, and the process proceeds to step S110.
[0052] Next, a case where it is determined in step S101 that cooperative communication is to be performed will be described. In step S104, the camera control unit 124 transmits information on a limiter that can correct image blur by the second image blur correction unit 214, and imaging information such as the shutter speed and exposure time from the camera body 180 to the interchangeable lens 150, and the process proceeds to step S105. In step S105, the lens control unit 111 receives the data transmitted from the camera body 180, and the process proceeds to step S106.
[0053] In step S106, the lens control unit 111 detects camera shake with respect to the Yaw and Pitch axes using the angular velocity sensor 201 in Fig. 2. Note that since the shake information from the angular velocity sensor 201 is angular velocity, it is passed through an HPF to remove low-frequency components, and converted from angular velocity to angular displacement data by an integrator, and the process proceeds to step S107.
[0054] In step S107, the lens control unit 111 creates and divides an image blur correction amount to be used by the first image blur correction unit 211 and the second image blur correction unit 214. In this embodiment, as a method for calculating the image blur correction amount, the correction amount is calculated within a range that is the sum of the ranges that can be corrected by the first image blur correction unit 211 and the second image blur correction unit 214, and as an example of a division method, the correction amount is divided according to the ratio of each correction range. If image blur correction is to be performed using the calculated and divided correction amount to be used by the first image blur correction unit 211, the process proceeds to step S108. On the other hand, if image blur correction is to be performed using the calculated and divided correction amount to be used by the second image blur correction unit 214, the correction amount is transmitted from the interchangeable lens 150 to the camera body 180, and the process proceeds to step S109.
[0055] In step S108, the lens control unit 111 controls the first image blur correction unit 211 using the correction amount calculated in step S107 to perform image blur correction.
[0056] In step S109, the camera control unit 124 receives the data transmitted from the interchangeable lens 150 to the camera body 180, and proceeds to step S110. In step S110, the camera control unit 111 controls the second image blur correction unit 214 using the correction amount calculated and acquired in step S103 or step S109, and performs image blur correction.
[0057] In the above embodiment, the case has been described where image blur correction is performed using both the first image blur correction unit 211 and the second image blur correction unit 214. However, if the camera body 180 can receive the amount of shake detected on the interchangeable lens 150 side in steps S107 and S109 in Fig. 9, image blur correction may be performed only by the second image blur correction unit 210, without the first image blur correction unit 211 on the interchangeable lens 150 side performing image blur correction.
[0058] As described above, according to this embodiment, even if the detection period of cooperative communication and shake information such as motion vectors is reduced in order to reduce the processing load, it is possible to calculate a continuous image blur correction amount. This makes it possible to suppress a decrease in the image blur correction effect and perform appropriate image blur correction. In other words, it is possible to realize appropriate image blur correction control by cooperating with the camera and lens even during high-load processing such as high frame rates.
[0059] Second Embodiment FIG. 10 is a diagram showing processing of the electronic correction amount converter 213 which is different from that in the first embodiment, and shows the relationship between the imaging timing, acquisition of the image blur correction amount by cooperative communication with a reduced communication cycle, and acquisition of the image blur correction amount by motion vector detection with a reduced detection cycle.
[0060] In the first embodiment, processing is performed in units of two frames, that is, the previous frame and the next frame, meaning that cooperative communication and motion vector detection are performed once every two frames. In this embodiment, processing is performed in units of three frames, meaning that cooperative communication is performed twice every three frames and motion vector detection is performed once every three frames. In this case, the shake detection period by cooperative communication is longer than the frame image capture period (the shake detection frequency by cooperative communication is lower than the frame image capture frequency). This makes it possible to reduce the processing load of cooperative communication while reducing the processing load of motion vector detection more than in the first embodiment.
[0061] An example of an image blur correction amount calculation method will be described with reference to the flowcharts of Fig. 10 and Fig. 11. Note that the same components as those in Fig. 9 are given the same reference numerals and will not be described. The process shown in Fig. 11 is repeatedly executed at any predetermined period such as the imaging period, which is set to the imaging period in this embodiment.
[0062] First, in step S201, the camera control unit 124 determines whether or not the cooperative communication has been performed for two consecutive frames. If it is determined that the cooperative communication has been performed for two consecutive frames or less, the process proceeds to step S104, and if it is determined that the cooperative communication has been performed for two consecutive frames or more, the process proceeds to step S202.
[0063] In step S202, the camera control unit 214 acquires a motion vector and proceeds to step S203. Here, since the previously executed step S202 was three frames ago, the motion vector C2 for three frames can be acquired. In step S203, as shown in (Equation 3), the difference between the correction amounts A2 and B2 calculated in step S109 one frame ago and two frames ago, the difference between the previous image blur correction amount D2_old, and the difference between the motion vector C2 acquired in step S202 are calculated and integrated to calculate the image blur correction amount D2.
[0064] D2=∫(C2-((B2-A2)+(A2-D2_old))) …(Formula 3) The calculated image blur correction amount D2 is limited so as to be within a range that can be corrected by the second image blur correction unit 214, and a correction amount that has undergone panning processing or the like is calculated, and the process proceeds to step S110.
[0065] As described above, according to this embodiment, even if the detection cycles of cooperative communication and shake information such as motion vectors are unevenly reduced in order to reduce the processing load, it is possible to calculate a continuous image blur correction amount. This makes it possible to perform appropriate image blur correction while suppressing a decrease in the image blur correction effect. In other words, it is possible to realize appropriate image blur correction control by cooperating with the camera and lens even during high-load processing such as high frame rates.
[0066] In the first embodiment, the camera control unit 214 controlled the acquisition of the shake information so that a frame in which shake information is acquired by cooperative communication and a frame in which shake information is acquired by a motion vector occur alternately. In the present embodiment, the acquisition of the shake information was controlled so that after two consecutive frames in which shake information is acquired by cooperative communication occur, one frame in which shake information is acquired by a motion vector occurs. In this way, as long as the shift between the timing of acquiring shake information by cooperative communication and the timing of acquiring shake information by a motion vector is controlled so that a continuous correction amount can be acquired, the shift between the two timings is not limited to the above. In order to make the shift between the two timings constant, the acquisition period of shake information by cooperative communication and the acquisition period of shake information by a motion vector should match. However, the acquisition period of shake information is the period between a frame in which shake information is acquired and a frame in which shake information is not acquired. For example, as in the present embodiment, when two consecutive frames in which shake information is acquired by cooperative communication occur, one frame in which shake information is not acquired occurs, and this is repeated multiple times, the acquisition period of shake information by cooperative communication is set to three frames. Similarly, if two consecutive frames occur in which no shake information is obtained by the motion vector, and then one frame is obtained, and this is repeated several times, the acquisition period of the shake information by the motion vector is three frames. In addition, in the first and the present embodiment, only one of the shake information by the cooperative communication and the shake information by the motion vector is obtained for each frame, but there may be a frame in which both are obtained. In terms of processing load, however, a form in which there is a frame in which both are obtained is disadvantageous, but for example, there may be a timing when the shake information by the cooperative communication and the shake information by the motion vector are required in addition to the shake correction, so there may be a timing when both shake information are obtained accordingly.
[0067] In the above embodiments, an angular velocity sensor is used as an example of a shake detection unit, but other shake detection units may be used. For example, an acceleration sensor may be used to calculate the amount of shake from acceleration, or a combination of multiple sensors may be used to detect shake and calculate the amount of shake of the camera system.
[0068] As examples of image blur correction means, the lens shift method in which a part of a lens constituting an interchangeable lens is moved in a direction perpendicular to the optical axis and the image cropping method in which a captured image is moved in a direction perpendicular to the optical axis have been described. However, a method of correcting image blur using an image sensor shift in which an image sensor is moved in a direction perpendicular to the optical axis or the like may also be used.
[0069] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0070] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]
[0071] 100: digital camera system, 105: image stabilization unit, 106: optical image stabilization control unit, 110: lens shake detection unit, 111: lens system control unit, 112: lens communication control unit, 115: imaging unit, 116: imaging signal processing unit, 117: video signal processing unit, 122: camera shake detection unit, 123: electronic image stabilization control unit, 124: camera system control unit, 125: camera communication control unit
Claims
1. An acquisition means for acquiring shake information from a plurality of shake detection means for detecting shakes; a first image blur compensation amount calculation means for calculating an image blur compensation amount corresponding to each of a portion of a plurality of consecutive images based on shake information from a first shake detection means acquired by the acquisition means, and a second image blur compensation amount calculation means for calculating an image blur compensation amount corresponding to each of the portion of a plurality of consecutive images by a method different from that of the first image blur compensation amount calculation means based on shake information from a second shake detection means acquired by the acquisition means; a control means for controlling the plurality of image blur compensation amount calculation means so that at least one of the plurality of image blur compensation amount calculation means obtains the image blur compensation amount corresponding to each of a plurality of consecutive images that have been captured, and so that an image for which an image blur compensation amount is calculated by a first image blur compensation amount calculation means and an image for which an image blur compensation amount is calculated by a second image blur compensation amount calculation means are periodically generated among the plurality of image blur compensation amount calculation means; Equipped with the control means controls the plurality of image blur compensation amount calculation means so that, when a first period during which the plurality of consecutive images are captured is shorter than a predetermined value, the image blur compensation amount is calculated corresponding to each of the plurality of consecutive images captured by at least any of the plurality of image blur compensation amount calculation means.
2. 2. The image stabilization apparatus according to claim 1, wherein the acquiring means acquires the shake information from the first shake detecting means through communication.
3. 3. The image blur correction device according to claim 1, further comprising an image blur correction control unit that controls correction of an image blur of a captured image based on the image blur correction amounts calculated by the plurality of image blur correction amount calculation units.
4. 4. The image blur correction device according to claim 3, wherein the image blur correction control means controls electronic image blur correction for electronically correcting image blur in the plurality of consecutive captured images.
5. 5. The image blur correction device according to claim 4, wherein the image blur correction control means further controls a correction means for optically correcting image blur in the captured image.
6. 6. The image stabilization device according to claim 1, wherein the second shake detection means is a difference shake detection means that detects shake from a difference between the plurality of consecutive captured images.
7. 7. The image blur correction device according to claim 6, wherein the second shake detection means obtains the difference based on at least an image for which an image blur correction amount cannot be obtained by the first image blur correction amount calculation means.
8. 8. The image stabilization device according to claim 1, wherein the acquisition means acquires the shake information from the first shake detection means at a second period longer than a first period during which the plurality of consecutive images are captured.
9. 9. The image stabilization device according to claim 1, wherein the acquisition means acquires the shake information from the second shake detection means at a third period longer than a first period in which the plurality of consecutive images are captured.
10. The control means a period between an image for which the first image blur correction amount calculation means calculates a corresponding image blur correction amount and an image for which the first image blur correction amount calculation means does not calculate a corresponding image blur correction amount; 10. The image blur correction device according to claim 1, wherein the plurality of image blur correction amount calculation means are controlled so that a period between an image for which an image blur correction amount is calculated by the second image blur correction amount calculation means and an image for which an image blur correction amount is not calculated coincides with a period between the image for which an image blur correction amount is calculated by the second image blur correction amount calculation means.
11. 11. The image blur correction device according to claim 10, wherein the plurality of image blur correction amount calculation means include two image blur correction amount calculation means, that is, the first image blur correction amount calculation means and the second image blur correction amount calculation means, and the control means controls the plurality of image blur correction amount calculation means so that an image for which an image blur correction amount is calculated by the first image blur correction amount calculation means and an image for which an image blur correction amount is calculated by the second image blur correction amount calculation means are generated alternately.
12. the second shake detection means is a difference shake detection means that detects shake from a difference between the plurality of consecutive captured images, 12. The image blur correction device according to claim 1, wherein the second image blur correction amount calculation means calculates an image blur correction amount corresponding to a first image for which no corresponding shake information is obtained from the first shake detection means, based on shake information based on a difference between the first image and the second image, which is obtained from the second shake detection means, and on shake information corresponding to a third image taken between the first image and the second image, which is obtained from the first shake detection means.
13. 13. The image stabilization apparatus according to claim 12, wherein the third image is an image taken before the first image.
14. An imaging means for capturing the plurality of successive images; An image stabilization device according to any one of claims 1 to 13, The second shake detection means; an image blur correction unit that performs image blur correction on the plurality of consecutive images based on the image blur correction amounts calculated by the plurality of image blur correction amount calculation units, an acquisition means for acquiring a detection result of the first shake detection means provided in an attached lens device via communication at a second period longer than a first period during which the plurality of consecutive images are captured, said acquisition means comprising:
15. 15. The image pickup apparatus according to claim 14, wherein the first image blur correction amount calculation means calculates the image blur correction amount in the second period.
16. an imaging sensor that captures a plurality of successive images in a first period; a vibration detection means for detecting a vibration based on the plurality of images; a first image blur compensation amount acquisition means for acquiring an image blur compensation amount corresponding to each of the partial images of the consecutive images based on shake information acquired via communication from a mounted lens device, and a second image blur compensation amount acquisition means for acquiring an image blur compensation amount corresponding to each of the partial images of the consecutive images by a method different from that used by the first image blur compensation amount acquisition means based on a detection result by the shake detection means and the shake information acquired via the communication; an image blur correction unit that performs image blur correction on the plurality of consecutive images based on the image blur correction amounts acquired by the plurality of image blur correction amount acquisition units; and a control means for controlling the plurality of image blur compensation amount acquisition means so that, when the first period is shorter than a predetermined value, the image blur compensation amount corresponding to each of the plurality of consecutive images is obtained by at least one of the plurality of image blur compensation amount acquisition means.
17. An acquisition step of acquiring shake information from a plurality of shake detection means for detecting shake; a first image blur compensation amount calculation step of calculating an image blur compensation amount corresponding to each of a portion of a plurality of consecutive images based on shake information from a first shake detection means acquired in the acquisition step, and a second image blur compensation amount calculation step of calculating an image blur compensation amount corresponding to each of the portion of a plurality of consecutive images by a method different from that of the first image blur compensation amount calculation step based on shake information from a second shake detection means acquired in the acquisition step; a control step of controlling the plurality of image blur compensation amount calculation steps so that the image blur compensation amount is obtained in at least one of the plurality of image blur compensation amount calculation steps corresponding to each of a plurality of consecutive images that have been captured, and an image for which an image blur compensation amount is calculated in a first image blur compensation amount calculation step and an image for which an image blur compensation amount is calculated in a second image blur compensation amount calculation step are periodically generated among the plurality of image blur compensation amount calculation steps, the control step includes controlling the image blur compensation amount calculation steps so that the image blur compensation amount is calculated corresponding to each of the consecutive images captured in at least one of the image blur compensation amount calculation steps when a first period during which the consecutive images are captured is shorter than a predetermined value.
18. 1. A method for controlling an imaging device having an imaging sensor that captures a plurality of successive images in a first cycle, the method comprising: a vibration detection step of detecting vibration based on the plurality of images; a first image blur compensation amount acquisition step of acquiring an image blur compensation amount corresponding to each of the partial images of the consecutive images based on shake information acquired via communication from a mounted lens device, and a second image blur compensation amount acquisition step of acquiring an image blur compensation amount corresponding to each of the partial images of the consecutive images by a method different from that of the first image blur compensation amount acquisition step based on a detection result in the shake detection step and the shake information acquired via communication; an image blur correction step of performing image blur correction for the plurality of consecutive images based on the image blur correction amounts acquired in the image blur correction amount acquisition step; and a control step of controlling the plurality of image blur compensation amount acquisition steps so that, when the first period is shorter than a predetermined value, the image blur compensation amount corresponding to each of the plurality of consecutive images is obtained by at least one of the plurality of image blur compensation amount acquisition steps.
19. A program for causing a computer to execute each step of the control method according to claim 17 or 18.
20. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 17 or 18.
Citation Information
Patent Citations
Image stabilizing device, image stabilizing method and electronic equipment
JP2017169074A
Image processing device and method, imaging device, and imaging system
JP2017219635A
Imaging device
JP2019152888A
