Image stabilization apparatus, control method thereof, optical apparatus, storage medium, and computer program product

Abstract

The invention relates to an image stabilization apparatus, a control method thereof, an optical system, a storage medium, and a computer program product. The image stabilization apparatus includes: a first acquisition unit configured to acquire a first dither signal using a first sensor; a second acquisition unit configured to acquire, using a second sensor, a second dither signal having an amount of noise in a high-frequency band larger than an amount of noise in a high-frequency band of the first dither signal and an amount of noise in a low-frequency band smaller than an amount of noise in a low-frequency band of the first dither signal; a generator configured to generate a combined image stabilization signal based on a first signal of the first dither signal having a frequency higher than the cutoff frequency and a second signal of the second dither signal having a frequency lower than the cutoff frequency; and a setting unit configured to set the cutoff frequency. The setting unit changes the cutoff frequency in accordance with the imaging condition information.

Description

Technical Field

[0001] This disclosure relates to an image stabilization device and its control method, optical system, and storage medium. Background Technology

[0002] Image-capturing devices (optical devices), such as digital cameras, use known techniques where an image sensor, such as a CMOS sensor, or a portion of an optical element in a camera optical system moves in a direction orthogonal to the optical axis to correct image blur caused by camera shake. Such image-stabilized camera devices typically use a gyroscope sensor to detect shake applied to the camera.

[0003] Japanese Patent Application Publication No. 2018-116134 discloses a method that calculates the offset component of the gyroscope sensor's detection signal based on a combined signal obtained by passing a complementary filter through a detection signal from an accelerometer or geomagnetic sensor and a gyroscope sensor, and subtracts the offset component from the gyroscope sensor's detection signal. Japanese Patent Application Publication No. 2018-205551 discloses a method that, when using a complementary filter or similar method to combine a motion vector and a gyroscope sensor, changes the weighting of the combination based on the reliability of the motion vector.

[0004] When calculating a combined signal using complementary filters or similar methods based on the low-frequency components of an accelerometer or magnetometer and the high-frequency components of a gyroscope, the noise characteristics of the combined signal change depending on the cutoff frequency between the low-frequency and high-frequency components. Therefore, setting the cutoff frequency appropriately is important.

[0005] However, Japanese Patent Application Publication No. 2018-116134 does not disclose a method for setting the cutoff frequency. Japanese Patent Application Publication No. 2018-205551 discloses a method for setting the cutoff frequency, but does not mention a method for setting the cutoff frequency when combined with a sensor other than a motion vector. Summary of the Invention

[0006] An image stabilization device according to one aspect of this disclosure includes: a first acquisition unit configured to acquire a first jitter signal; a second acquisition unit configured to acquire a second jitter signal, the second jitter signal having a high-frequency noise level greater than the noise level in the high-frequency band of the first jitter signal and a low-frequency noise level less than the noise level in the low-frequency band of the first jitter signal; a generator configured to generate a combined image stabilization signal based on a first signal in the first jitter signal and a second signal in the second jitter signal, wherein the first signal has a frequency higher than a cutoff frequency and the second signal has a frequency lower than the cutoff frequency; and a setting unit configured to set the cutoff frequency. The setting unit changes the cutoff frequency according to imaging condition information. An optical device having the above-described image stabilization device, a control method for the above-described image stabilization device, and a storage medium storing a program for causing a computer to execute the above-described control method also constitute another aspect of this disclosure.

[0007] Further features of various embodiments of this disclosure will become apparent from the following description of the embodiments with reference to the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a block diagram of an image stabilization device according to the first embodiment.

[0009] Figure 2 The relationship between the camera device and the axial direction in each embodiment is shown.

[0010] Figure 3 This is a flowchart illustrating the processing of the image stabilization device according to the first embodiment.

[0011] Figure 4 The relationship between the target cutoff frequency and the exposure time in the first embodiment is shown.

[0012] Figure 5 The time evolution of the cutoff frequency in the first embodiment is shown.

[0013] Figure 6 This is a block diagram of an image stabilization device according to the second embodiment.

[0014] Figure 7 This is a flowchart illustrating the processing of the image stabilization device according to the second embodiment.

[0015] Figure 8 The time evolution of the cutoff frequency in the second embodiment is shown.

[0016] Figure 9 This is a schematic diagram illustrating the frequency characteristics of the complementary filters in various embodiments. Detailed Implementation

[0017] In the following text, the term "unit" may refer to a software context, a hardware context, or a combination of both. In a software context, the term "unit" refers to a function, application, software module, feature, routine, instruction set, or program that can be executed by a programmable processor, such as a microprocessor, central processing unit (CPU), or specially designed programmable device or controller. Memory contains instructions or programs that, when executed by the CPU, cause the CPU to perform operations corresponding to the unit or function. In a hardware context, the term "unit" refers to a hardware element, circuit, component, physical structure, system, module, or subsystem. According to a specific embodiment, the term "unit" may include mechanical, optical, or electrical components or any combination thereof. The term "unit" may include active components (e.g., transistors) or passive components (e.g., capacitors). The term "unit" may include a semiconductor device having a substrate and other material layers having various conductivity concentrations. The unit may include a CPU or programmable processor capable of executing a program stored in memory to perform a specific function. The term "unit" may include logic elements (e.g., AND, OR) implemented by transistor circuitry or any other switching circuitry. In a combination of software and hardware contexts, the term "unit" or "circuit" refers to any combination of software and hardware contexts as described above. Additionally, the terms "element," "component," "part," or "device" may also refer to a "circuit" integrated with or not integrated with packaging material.

[0018] Various embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. The following embodiments do not limit the scope of this disclosure in relation to the appended claims. Although multiple features are described in the embodiments, not all of these features are essential to this disclosure, and multiple features can be combined in any manner. Corresponding elements in the corresponding drawings will be indicated by the same reference numerals, and repeated descriptions thereof will be omitted.

[0019] First Embodiment

[0020] Figure 1 The structure of an image stabilization device 100 according to a first embodiment of the present disclosure is shown. The image stabilization device 100 is provided in an optical device such as a camera device or a lens device.

[0021] The first jitter detector (first acquisition unit) 101 is a gyroscope sensor configured to detect (acquire) jitter in the rotational axis directions (pitch, yaw, and roll directions) about three mutually orthogonal axes (i.e., the X-axis, Y-axis, and Z-axis). Figure 2The relationship between the camera device and the axial direction is illustrated. The horizontal direction of the camera device is defined as the X-axis, the vertical direction as the Y-axis, the optical axis as the Z-axis, the rotation axis around the X-axis as the pitch axis, the rotation axis around the Y-axis as the yaw axis, and the rotation axis around the Z-axis as the tilt axis. The first jitter detector 101 detects the first jitter signal (angular velocity signal) applied to the camera device and outputs the first jitter signal to the first image stabilization signal calculator (first calculator) 107 and the combined image stabilization signal calculator (generator) 111.

[0022] The second jitter detector (second acquisition unit) 102 is configured to detect (acquire) at least one of an accelerometer and a geomagnetic sensor for jitter in three mutually orthogonal axial directions (X-axis, Y-axis, and Z-axis directions). The second jitter detector 102 detects a second jitter signal (at least one of an acceleration signal and a geomagnetic signal) applied to the camera device and outputs the second jitter signal to the angle attitude calculator 103, the low-pass filter (LPF) 105, and the camera condition information acquisition unit 108. In this embodiment, the second jitter signal has the following characteristics: the noise level in the high-frequency band is greater than the noise level in the high-frequency band of the first jitter signal, and the noise level in the low-frequency band is less than the noise level in the low-frequency band of the first jitter signal.

[0023] The gyroscope sensor, serving as the first jitter detector 101, is configured to detect rotational jitter in angular velocity states applied to the image stabilization device 10, and improving the detection accuracy of the gyroscope sensor is crucial. One important issue in improving the detection accuracy of the gyroscope sensor is how to remove the low-frequency noise (offset component) present in the gyroscope sensor. When using the gyroscope sensor signal without removing the offset component, by integrating the output signal of the gyroscope sensor and treating it as an angle, a so-called drift, which accumulates as an integration error, occurs, making accurate image stabilization difficult.

[0024] Therefore, the second jitter detector 102 can detect jitter by combining detection information from other sensors (such as accelerometers or magnetometers) with detection information from a gyroscope sensor. Typically, accelerometers or magnetometers tend to have less low-frequency noise than gyroscopes and more high-frequency noise than gyroscopes. Therefore, jitter detection accuracy can be improved by using a complementary filter or similar method to combine the low-frequency components of the accelerometer or magnetometer with the high-frequency components of the gyroscope.

[0025] When using complementary filters or similar methods to calculate a combined signal based on the low-frequency components of an accelerometer or magnetometer and the high-frequency components of a gyroscope, the noise characteristics of the combined signal change depending on the cutoff frequencies of the low-frequency and high-frequency components. When the cutoff frequency is set low, high-frequency noise from the accelerometer or magnetometer is less likely to be superimposed, but low-frequency noise from the gyroscope cannot be sufficiently removed, and the combined signal may have less high-frequency noise and more low-frequency noise. On the other hand, when the cutoff frequency is set high, high-frequency noise from the accelerometer or magnetometer may be superimposed, but low-frequency noise from the gyroscope can be sufficiently removed, and the combined signal may have more high-frequency noise and less low-frequency noise. In other words, the lower the cutoff frequency, the less high-frequency noise the combined signal has, and the higher the cutoff frequency, the more high-frequency noise it has; conversely, the lower the cutoff frequency, the more low-frequency noise it has, and the higher the cutoff frequency, the less low-frequency noise it has. Therefore, there is a trade-off between reducing low-frequency noise and superimposing high-frequency noise based on the cutoff frequency, and thus, setting the cutoff frequency appropriately is important.

[0026] The angle attitude calculator 103 calculates the angle attitude (orientation) of the camera device (angle attitude information) by using a second jitter signal to find Euler angles using a known method. The angle attitude calculator 103 can be used as an attitude detector configured to acquire information related to whether the camera device is in a horizontal (normal) or vertical position. The angle attitude can be calculated using the output signal of the LPF 105, described later. The calculated angle attitude information is output to the angle calculation axis determination unit (sensor selector) 104.

[0027] The angle calculation axis determination unit 104 determines, based on angle attitude information, which of the detection signals from the accelerometer and the geomagnetic sensor should be used to calculate the angles in each rotation axis direction for pitch, yaw, and roll. For example, when the camera device is in a roughly normal (horizontal) position (where the Y-axis is parallel to the direction of gravity, and the X and Z axes are orthogonal to the direction of gravity), the angles in the pitch and roll axes are calculated based on the detection signals from the accelerometer, and the angle in the yaw axis direction is calculated based on the detection signals from the geomagnetic sensor. This is because rotation in the yaw axis direction does not involve changes in gravitational acceleration relative to the X, Y, and Z axes, and the angle in the yaw axis direction cannot be calculated based on the detection signals from the accelerometer. Alternatively, when the camera device is in a roughly vertical position (where the X-axis is parallel to the direction of gravity, and the Y and Z axes are orthogonal to the direction of gravity), the angles in the yaw and roll axes are calculated based on the detection signals from the accelerometer, and the angle in the pitch axis direction is calculated based on the detection signals from the geomagnetic sensor. This is for similar reasons.

[0028] LPF 105 outputs the second jitter signal (the second signal) from which the high-frequency components have been removed to the angle calculator 106. The cutoff frequency of LPF 105 can be approximately 1 to 10 Hz, but is not limited to this example.

[0029] Angle calculator 106 calculates the angles (angle signals) in the pitch, yaw, and roll axes based on the combination of sensor detection signals determined by angle calculation axis determination unit 104 and the directions of each rotation axis. The calculated angle signals are output to combined image stabilization signal calculator 111.

[0030] The first image stabilization signal calculator 107 converts the angular velocity signal (which is the first jitter signal) into an angle through integration processing to calculate the first image stabilization signal, and outputs the first image stabilization signal to the image stabilization signal selector 112.

[0031] The camera condition information acquisition unit 108 acquires the camera condition information of the camera device and outputs it to the cutoff frequency setting unit 110. The camera condition information includes at least one of the following: the norm of the second jitter signal in three mutually orthogonal axis directions, the exposure time, and the shutter type of the camera device. The shutter type of the camera device typically includes three types: mechanical shutter type (front-curtain and rear-curtain mechanical shutter), electronic front-curtain shutter type (electronic front-curtain shutter and mechanical rear-curtain shutter), and electronic shutter (front-curtain and rear-curtain electronic shutter). However, this embodiment is not limited to this example.

[0032] Counter 109 counts at least one of the elapsed time calculated from the start of the LPF 105 and the elapsed time after the mechanical shutter is actuated, and notifies the cutoff frequency setting unit 110 of the count value. Cutoff frequency setting unit 110 sets the cutoff frequency based on the imaging condition information and the count value, and notifies the combined image stabilization signal calculator 111 of the cutoff frequency. In other words, cutoff frequency setting unit 110 changes the cutoff frequency according to the imaging condition information.

[0033] The combined image stabilization calculator 111 uses a complementary filter to combine the high-frequency component signal (first signal) obtained from the first jitter detector 101 and the low-frequency component signal (second signal) obtained from the second jitter detector 102 to calculate the combined image stabilization signal. In this embodiment, the combined image stabilization calculator 111 combines the first signal and the second signal based on the cutoff frequency notified by the cutoff frequency setting unit 110.

[0034] The combined image stabilization signal calculator 111 constitutes at least a portion of a complementary filter for combining the first and second signals. The cutoff frequency corresponds to the cutoff frequency of the complementary filter. The complementary filter has an HPF that allows the first signal in the first jitter signal detected by the first jitter detector 101 to pass through, and an LPF that allows the second signal in the second jitter signal detected by the second jitter detector 102 to pass through.

[0035] Figure 9 This is a schematic diagram illustrating the frequency characteristics of the complementary filter in this embodiment. Figure 9 In the diagram, the vertical axis represents the gain (dB) of the complementary filter, and the horizontal axis represents the frequency (Hz). The complementary filter has the characteristic of imparting appropriate gains to both the first and second jitter signals according to the frequency (band), and the sum of the gains of the HPF and LPF is 1 in any band. Figure 9 In this context, the frequency at which the gains of the first and second signals are equal (the frequency at which the first and second signals cross) is the cutoff frequency, i.e., the cutoff frequency Fc.

[0036] The first signal is a signal in the first jitter signal whose frequency is higher than the cutoff frequency, and the second signal is a signal in the second jitter signal whose frequency is lower than the cutoff frequency. However, this embodiment is not limited to this example, and the first signal may be a signal in the first jitter signal that includes frequencies higher than the cutoff frequency, and the second signal may be a signal in the second jitter signal that includes frequencies lower than the cutoff frequency.

[0037] The image stabilization signal selector 112 selects one of the first image stabilization signal and the combined image stabilization signal based on the cutoff frequency, and outputs the selected signal (final image stabilization signal) to the image stabilization drive signal calculator (second calculator) 113.

[0038] The image stabilization drive signal calculator 113 performs known calculations, such as multiplying the image stabilization signal output from the image stabilization signal selector 112 by the focal length or the drive resolution of the image stabilization member (correction member) 114. The image stabilization drive signal calculator 113 calculates the image stabilization drive signal used to drive the image stabilization member 114 and outputs the image stabilization drive signal to the image stabilization member 114. The image stabilization member 114 performs image stabilization by being driven according to the image stabilization drive signal.

[0039] Some of the aforementioned units are implemented by loading and executing programs using a processor, such as one or more CPUs. For example, the angle attitude calculator 103, the angle calculation axis determination unit 104, the angle calculator 106, and the first image stabilization signal calculator 107 can be implemented by a processor. The camera condition information acquisition unit 108, the cutoff frequency setting unit 110, the combined image stabilization signal calculator 111, the image stabilization signal selector 112, and the image stabilization drive signal calculator 113 can also be implemented by a processor.

[0040] Now refer to Figure 3 This describes the processing of the image stabilization device 100 (image stabilization operation and control method of the image stabilization device 100). Figure 3 This is a flowchart illustrating the processing of the image stabilization device 100.

[0041] Figure 3 The "Start" indicator in the code begins due to a turning-on state (such as powering on the camera or activating image stabilization mode). In the turning-on state, the process returns to the "Start" box without proceeding to the final "End" box, and repeats. Figure 3 The process follows the detection cycle of the first jitter detector 101 and the second jitter detector 102, as well as the calculation cycle of the complementary filter. Figure 3 The "End" box indicates that the recording has ended due to a power outage of the camera or the image stabilization mode being turned off.

[0042] First, in step S301, the first jitter detector 101 detects a first jitter signal applied to the camera device. Next, in step S302, the first image stabilization signal calculator 107 converts the first jitter signal into an angle through integration processing to calculate a first image stabilization signal. Next, in step S303, the second jitter detector 102 detects a second jitter signal applied to the camera device. Next, in step S304, the angle attitude calculator 103 calculates the angle attitude of the camera device based on the second jitter signal. Next, in step S305, the angle calculation axis determination unit 104 determines the combination of rotation axis directions and sensor types based on the angle attitude of the camera device. Next, in step S306, the LPF 105 removes high-frequency components from the second jitter signal.

[0043] Next, in step S307, the angle calculator 106 calculates the angle based on the low-frequency component of the second jitter signal. As an example, the angle calculation in the angle orientation of the camera device in a roughly normal posture will be described. The low-frequency component (a) of the triaxial accelerometer signal (which is the second jitter signal) is used. x ,a y ,a zThe roll axis angle θ is calculated according to the following equations (1) and (2), respectively. a_Roll and the angle θ of the pitch axis a_Pitch .

[0044]

[0045] As shown in equation (3) below, the angle θ of the roll axis is used. a_Roll The pitch axis angle θ a_Pitch And the low-frequency component (m) of the triaxial geomagnetic sensor signal (which is the second jitter signal). x ,m y ,m z To calculate the angle θ of the yaw axis. m Yaw .

[0046]

[0047] The angle obtained by integrating the gyroscope signal is a relative angle from the start time of the integration calculation, while the angle (θ) calculated above... a_Roll ,θ a_Pitch ,θ m_Yaw These are absolute angles relative to the direction and orientation of gravity, and therefore need to be converted into relative angles. These absolute angles can be converted into relative angles by subtracting a fixed absolute angle value at any given time from the absolute angles calculated over time. Any time can be, for example, the time the camera or image stabilization device is powered on, or the time during the processing of step S310 or step S320.

[0048] Next, in step S308, the camera condition information acquisition unit 108 calculates the norm of the second jitter signal based on the second jitter signal. The norm of the second jitter signal is calculated using the second jitter signal (X, Y, Z) in the three-axis directions by the following formula (4).

[0049]

[0050] The second jitter signal has a large amount of noise in the high-frequency band. Therefore, the norm of the second jitter signal can be calculated using the low-frequency component of the second jitter signal output from the LPF 105.

[0051] Next, in step S309, the cutoff frequency setting unit 110 determines whether the elapsed time counted by the counter 109 since the start of calculation using the LPF 105 and the elapsed time after the mechanical shutter is actuated have each exceeded a predetermined time. The cutoff frequency setting unit 110 also determines whether the norm of the second jitter signal has exceeded the threshold TH. Based on these determination results, in step S310 and thereafter, the cutoff frequency setting unit 110 sets the cutoff frequency.

[0052] If either the elapsed time calculated from LPF 105 or the elapsed time after the mechanical shutter is actuated is less than a predetermined time, or the norm of the second jitter signal exceeds the threshold TH, the process proceeds to step S310. Otherwise, the process proceeds to step S311.

[0053] The predetermined time corresponding to the elapsed time since the start of the calculation of LPF 105 can be approximately the time constant of LPF 105. This is because the output signal from LPF 105 contains transient responses and does not provide an accurate output until approximately the time constant has elapsed. The predetermined time corresponding to the elapsed time after the mechanical shutter is actuated can be set to the time until the vibration of the camera device caused by the mechanical shutter actuation is sufficiently attenuated.

[0054] Regarding the threshold for the norm of the second jitter signal, the norm of the acceleration signal is determined based on whether the absolute value of the difference between the acceleration signal and the 1G gravitational acceleration is equal to or greater than the threshold. For example, with the threshold set to 0.2G, it is determined that the norm of the acceleration signal does not exceed the threshold when it is between 0.8G and 1.2G. For example, when the user significantly pans the camera or the camera is on a moving object such as a car, the norm of the acceleration signal is significantly different from 1G. In this case, the angle cannot be correctly calculated based on the acceleration signal. Similarly, the norm of the geomagnetic sensor can be determined based on whether the absolute value of the difference between the norm of the geomagnetic sensor and the norm after calibration using known methods is equal to or greater than the threshold. Typically, when a magnet such as iron approaches the geomagnetic sensor, the norm may exceed the threshold. Since the causes of changes in the norm of the acceleration signal and the norm of the geomagnetic signal are different, independent determinations can be made for each sensor.

[0055] In step S310, the cutoff frequency setting unit 110 sets the cutoff frequency (Hz) of the complementary filter to a high frequency. In step S311, the cutoff frequency setting unit 110 sets (determines) the target cutoff frequency based on the exposure time obtained from the imaging condition information acquisition unit 108.

[0056] For example, such as Figure 4 As shown, the exposure time can be correlated with the target cutoff frequency. Figure 4 This illustrates the relationship between the target cutoff frequency and exposure time. Figure 4In this case, the horizontal axis (abscissa) represents the exposure time, and the vertical axis (ordinate) represents the target cut-off frequency. This example provides a lower limit Fc1 and an upper limit Fc2 for the target cut-off frequency. Even when the exposure time is short, the target cut-off frequency is set to be not lower than the lower limit Fc1. On the other hand, even when the exposure time is long, the target cut-off frequency is set to be not higher than the upper limit Fc2. The exposure times Tv1 and Tv2 satisfy the relationship Tv1 ≤ Tv2, and the lower limit Fc1 and the upper limit Fc2 satisfy the relationship Fc1 < Fc2. Alternatively, Fc1 = 0 (Hz) can be used. Fc2 can be determined according to the noise characteristics of the first jitter detector 101 and the second jitter detector 102, and for example, it is approximately Fc2 = 0.005 to 0.5 (Hz).

[0057] In this embodiment, the cut-off frequency setting unit 110 sets the cut-off frequency higher as the exposure time is longer, and sets the cut-off frequency lower as the exposure time is shorter. That is, when the exposure time is the first exposure time (Tv1), the cut-off frequency setting unit 110 sets the cut-off frequency to the first frequency (Fc1), and when the exposure time is the second exposure time (Tv2) which is longer than the first exposure time, the cut-off frequency setting unit 110 sets the cut-off frequency to the second frequency (Fc2) which is higher than the first frequency. The change of the target cut-off frequency when the exposure time is between the first exposure time and the second exposure time can be Figure 4 linear as shown, or non-linear.

[0058] Next, in step S312, the cut-off frequency setting unit 110 compares the current cut-off frequency actually used in the calculation of the complementary filter with the target cut-off frequency, and if the current cut-off frequency is greater than the target cut-off frequency, the process proceeds to step S313. On the other hand, if the current cut-off frequency is less than the target cut-off frequency, the process proceeds to step S315.

[0059] In step S313, the cut-off frequency setting unit 110 sets the cut-off frequency of the complementary filter to be lower than the cut-off frequency of the previous sample (the previous step). In steps S314 and S315, the combined image stabilization signal calculator 111 uses the complementary filter to combine the high-frequency component of the first jitter signal and the low-frequency component of the second jitter signal according to the cut-off frequency by using the following equations (5) and (6) to calculate the combined image stabilization signal.

[0060]

[0061] θ(n) = K(θ(n - 1)+T s ω g (n))+(1 - K)θ am(n) (6)

[0062] Here, θ is the combined image stabilization signal, F C It is the cutoff frequency, K is the complementary filter coefficient, and ω is the cutoff frequency. g It is the first jitter signal, θ am It is the second jitter signal, T S is the sampling time, n is the current sample, and (n-1) is the sample immediately preceding the current sample (the previous sample).

[0063] In step S316, the image stabilization signal selector 112 selects a first image stabilization signal, and the image stabilization drive signal calculator 113 uses the first image stabilization signal to calculate the image stabilization drive signal. In step S317, the image stabilization signal selector 112 selects a combined image stabilization signal, and the image stabilization drive signal calculator 113 uses the combined image stabilization signal to calculate the image stabilization drive signal.

[0064] Then, in step S318, the image stabilization device 100 drives the image stabilization member 114 according to the image stabilization drive signal. Then, in step S319, the image stabilization device 100 determines whether the mechanical shutter has been driven by the camera. If the mechanical shutter has been driven, the process proceeds to step S320. On the other hand, if the mechanical shutter has not been driven, the process returns to the start frame or moves to the end frame to end the process.

[0065] In step S320, the image stabilization device 100 initializes the LPF 105 and resets the counter 109. The count value of the counter 109 is used in the conditional branch in step S309. Then, the process returns to the start box or moves to the end box to terminate the process.

[0066] Now refer to Figure 5 The document describes a method for setting the cutoff frequency using the cutoff frequency setting unit 110 and a method for selecting an image stabilization signal using the image stabilization signal selector 112. Figure 5 The time evolution of the cutoff frequency is shown. Figure 5 In the diagram, the vertical axis represents the cutoff frequency, and the horizontal axis represents time. Here, it is assumed that the norm of the second jitter signal does not exceed a threshold, except immediately following the mechanical rear curtain shutter driven by the electronic front curtain shutter type and at time T510.

[0067] The T500 is in image stabilization mode and starts. Figure 3The flowchart shows a series of calculations taking place. Simultaneously, counter 109 starts counting. In the conditional branch of step S309, the elapsed time since the calculation started using LPF 105 is less than a predetermined time. Therefore, the process proceeds to step S310, where the cutoff frequency Fc is set to a high cutoff frequency Fc_high. In step S314, the combined image-stabilized signal is calculated by the complementary filter based on the cutoff frequency Fc_high.

[0068] The period from time T500 to time T501 is a stabilization period for the transient response of LPF 105, during which the cutoff frequency is set to Fc_high. However, the cutoff frequency does not need to be fixed and can, for example, be gradually decreased from this period.

[0069] At time T501, the elapsed time since the start of calculation using LPF 105 exceeds the predetermined time, so the process moves from the conditional branch in step S309 to step S311, and the target cutoff frequency is determined to be low Fc_low based on the set exposure time Tv_long (not shown).

[0070] During the time interval from time T501 to time T502, according to the conditional branch of step S312, the current cutoff frequency Fc is greater than the target cutoff frequency Fc_low, therefore the process proceeds to step S313, where, for each sample, the cutoff frequency Fc is reduced compared to the previous sample. The cutoff frequency can be as follows: Figure 5 The cutoff frequency decreases linearly, or it can decrease non-linearly. Then, in step S314, the complementary filter calculates the combined image stabilization signal based on the current cutoff frequency Fc. By providing a period of gradually decreasing cutoff frequency, the transient response period of the complementary filter can be reduced. That is, immediately after the combined image stabilization signal calculator 111 starts generating the combined image stabilization signal, the cutoff frequency setting unit 110 sets the cutoff frequency to a predetermined high cutoff frequency. Then, the cutoff frequency setting unit 110 causes the cutoff frequency to decrease over time to a predetermined low cutoff frequency.

[0071] At time T502, the cutoff frequency Fc becomes the target cutoff frequency Fc_low, and the process moves from the conditional branch of step S312 to step S315, where the complementary filter calculates the combined image stabilization signal based on the cutoff frequency Fc_low. At time T503, it is assumed that the electronic front-curtain shutter is driven to begin exposure. Since it is not a mechanical shutter, the cutoff frequency does not increase. If the mechanical front-curtain shutter has already been driven to begin exposure at time T503, the cutoff frequency Fc will be set to Fc_high from time T503 onwards.

[0072] At time T504, the cutoff frequency Fc is set to Fc_high because the mechanical rear curtain shutter is driven. Simultaneously, according to the conditional branch of step S319, the process proceeds to step S320, where LPF 105 is initialized and counter 109 is reset. If the electronic shutter type is set and exposure begins at time T504 by driving the electronic rear curtain shutter instead of the mechanical rear curtain shutter, the cutoff frequency does not increase but remains at Fc_low at time T504.

[0073] At time T505, the elapsed time since the start of calculation using LPF 105 exceeds the predetermined time, but the elapsed time since the mechanical shutter was activated is still less than the predetermined time, so the cutoff frequency Fc remains at Fc_high.

[0074] At time T506, the elapsed time since the mechanical shutter was actuated exceeds a predetermined time, therefore the process moves from the conditional branch of step S309 to step S311, and the target cutoff frequency is determined as Fc_Low based on the set exposure time Tv_long. This example sets the predetermined time used to determine the elapsed time after the mechanical shutter is actuated to be longer than the predetermined time used to determine the elapsed time calculated from the start using LPF 105, but this embodiment is not limited to this example. If the camera device has a mechanical shutter that requires time for vibration decay, or if the time constant of the LPF is set long, the relationship between the lengths of the predetermined times will be reversed.

[0075] In other words, when the shutter type is a mechanical shutter, the cutoff frequency setting unit 110 sets the cutoff frequency to a higher level than that in the case of an electronic shutter, until a predetermined time has elapsed after the mechanical shutter is activated. When the shutter type is an electronic front-curtain shutter, the cutoff frequency setting unit 110 sets the cutoff frequency to a higher level than that until a predetermined time has elapsed after the mechanical rear-curtain shutter is activated, after the predetermined time has elapsed, or when the shutter type is an electronic shutter. After the predetermined time has elapsed, the cutoff frequency setting unit 110 causes the cutoff frequency to decrease over time to a predetermined low cutoff frequency.

[0076] At time T507, the exposure time is set to Tv_short, which is shorter than Tv_long, and in step S311, for this sample, the target cutoff frequency is determined to be Fc_middle, which is higher than Fc_Low.

[0077] During the time interval from time T506 to time T508, similar to the time interval from time T501 to time T502, according to the conditional branch of step S312, the current cutoff frequency Fc is greater than the target cutoff frequency Fc_middle, so the process proceeds to step S313. Then, the cutoff frequency Fc of each sample becomes lower than the cutoff frequency of the sample immediately preceding it. Then, in step S314, the combined image stabilization signal is calculated by the complementary filter based on the current cutoff frequency Fc.

[0078] At time T508, the cutoff frequency Fc becomes the target cutoff frequency Fc_middle, and the process moves from the conditional branch of step S312 to step S315, where the complementary filter calculates the combined image stabilization signal based on the cutoff frequency Fc_middle.

[0079] At time T509, the exposure time is returned from Tv_short to Tv_long, and in step S311, the target cutoff frequency is determined to be Fc_low. The process then proceeds from the conditional branch of step S312 to step S313, where the cutoff frequency is reduced to Fc_low. For the next sample, the process proceeds from the conditional branch of step S312 to step S315, where the combined image-stabilized signal is calculated by the complementary filter based on the cutoff frequency Fc_low.

[0080] At time T510, the norm of the second jitter signal momentarily exceeds a threshold (predetermined value) due to an object impacting the camera device, etc. Therefore, the process moves from the conditional branch of step S309 to step S310, and the cutoff frequency is set to Fc_high. By setting the cutoff frequency high once, the transient response of the complementary filter is shortened. That is, when the norm of the second jitter signal is equal to or greater than the predetermined value, the cutoff frequency setting unit 110 sets the cutoff frequency to be higher than the cutoff frequency when the norm is less than the predetermined value. When the norm changes from a value equal to or greater than the predetermined value to a value less than the predetermined value, the cutoff frequency setting unit 110 causes the cutoff frequency to decrease over time to a predetermined low cutoff frequency.

[0081] The time period from time T510 to time T511 is similar to the time period from time T500 to time T501, and the time period from time T511 to time T512 is similar to the time period from time T501 to time T502, so their description will be omitted.

[0082] When the cutoff frequency Fc is the target cutoff frequency, that is, when the process moves from the conditional branch in step S312 to step S315, the image stabilization signal selector 112 selects the combined image stabilization signal. On the other hand, in other cases, the image stabilization signal selector 112 selects the first image stabilization signal. Therefore, in Figure 5 In the process, a first image stabilization signal is selected during the time periods from time T500 to time T502, from time T504 to time T508, and from time T510 to time T512. A combined image stabilization signal is selected during the time periods from time T502 to time T504, from time T508 to time T510, and from time T512 onwards.

[0083] This embodiment allows for setting an appropriate cutoff frequency based on exposure time, shutter type, and the norm of the accelerometer or magnetometer when combining an accelerometer or magnetometer with a gyroscope sensor using a complementary filter. Furthermore, this embodiment enables accurate image stabilization by appropriately selecting and switching between an image stabilization signal calculated solely by the gyroscope sensor and a combined image stabilization signal calculated by the complementary filter.

[0084] Second Embodiment

[0085] Next, a second embodiment of this disclosure will be described. In the first embodiment, the image stabilization signal selector 112 selects one of a first image stabilization signal and a combined image stabilization signal to calculate the image stabilization drive signal. However, when the combined image stabilization signal calculator 111 generates the combined image stabilization signal using a complementary filter with a cutoff frequency of 0Hz, it becomes equivalent to the first image stabilization signal. Therefore, as Figure 6 As shown, the image stabilization device 100a according to this embodiment has a [missing information - likely a function or feature] from [missing information - likely a function or feature] Figure 1 The image stabilization device 100 excludes the structure of the first image stabilization signal calculator 107 and the image stabilization signal selector 112. Figure 6 This is a block diagram of the image stabilization device 100a according to this embodiment. Figure 6 The components shown are the same as those in the first embodiment, so their description is omitted.

[0086] Now refer to Figure 7 This describes the processing of the image stabilization device 100a (image stabilization operation and control method of the image stabilization device 100a). Figure 7 This is a flowchart illustrating the processing of the image stabilization device 100a. Figure 7 In the figures, the same reference numerals are used with Figure 3 The common steps will be described separately, and their descriptions will be omitted.

[0087] In step S710, the cutoff frequency setting unit 110 sets the cutoff frequency of the complementary filter to a low frequency. At this time, the cutoff frequency can be set to approximately 0 (Hz). In step S711, the cutoff frequency setting unit 110 sets the target cutoff frequency based on the exposure time obtained from the imaging condition information acquisition unit 108.

[0088] Now refer to Figure 8 This describes a method for setting the cutoff frequency by the cutoff frequency setting unit 110. Figure 8 The time evolution of the cutoff frequency in this embodiment is shown. Figure 8 In the diagram, the vertical axis indicates the cutoff frequency, and the horizontal axis indicates the time.

[0089] The T800 is in image stabilization mode and starts. Figure 7 The flowchart shows a series of calculations taking place. Simultaneously, counter 109 starts counting. In the conditional branch of step S309, the elapsed time since the calculation started using LPF 105 is less than a predetermined time. Therefore, the process proceeds to step S710, where the cutoff frequency is set to an extremely low cutoff frequency Fc_zero. In step S314, the combined image-stabilized signal is calculated by the complementary filter based on the cutoff frequency Fc_zero.

[0090] The period from time T800 to time T801 is a period for waiting for the transient response of LPF 105 to stabilize, and during this period, the cutoff frequency is set to Fc_zero.

[0091] At time T801, the elapsed time since the start of calculation using LPF 105 exceeds the predetermined time, so the process moves from the conditional branch in step S309 to step S711, where the cutoff frequency is set to a low cutoff frequency Fc_low based on the set exposure time Tv_long (not shown).

[0092] At time T802, it is assumed that the electronic front-curtain shutter is driven to begin exposure. Since it is not a mechanical shutter, the cutoff frequency is not reduced. If the mechanical front-curtain shutter is driven to begin exposure at time T802, the cutoff frequency Fc is set to Fc_zero from time T802. That is, when the shutter type is a mechanical shutter, the cutoff frequency setting unit 110 sets the cutoff frequency to a lower value than the cutoff frequency in the case of an electronic shutter until a predetermined time has elapsed after the mechanical shutter is driven. When the shutter type is an electronic front-curtain shutter, the cutoff frequency setting unit 110 sets the cutoff frequency to a lower value than the cutoff frequency until a predetermined time has elapsed after the mechanical rear-curtain shutter is driven, after the predetermined time has elapsed, or when the shutter type is an electronic shutter.

[0093] At time T803, the cutoff frequency Fc is set to Fc_zero because the mechanical rear curtain shutter is driven. Simultaneously, the process moves from the conditional branch of step S319 to step S320, where LPF 105 is initialized and counter 109 is reset. If the electronic shutter type is set and exposure begins at time T803 by driving the electronic rear curtain shutter instead of the mechanical rear curtain shutter, the cutoff frequency at time T803 will not be as low as Fc_zero, but will remain at Fc_low.

[0094] At time T804, the elapsed time since the start of the LPF 105 operation exceeds the predetermined time, but the elapsed time since the start of the mechanical shutter is still less than the predetermined time, therefore the cutoff frequency Fc remains at Fc_zero. At time T805, the elapsed time since the start of the mechanical shutter exceeds the predetermined time, therefore the process moves from the conditional branch in step S309 to step S711, where the cutoff frequency is set to Fc_Low based on the set exposure time Tv_long.

[0095] At time T806, the exposure time is set to Tv_short, which is shorter than Tv_long, and in step S711, for this sample, the cutoff frequency is set to Fc_middle, which is higher than Fc_Low. At time T807, the set exposure time is returned from Tv_short to Tv_long, and in step S711, the cutoff frequency is set to Fc_low.

[0096] At time T808, the norm of the second jitter signal momentarily exceeds a threshold (predetermined value) due to an object impacting the camera device, and the process moves from the conditional branch of step S309 to step S310, and the cutoff frequency is set to Fc_zero. That is, when the norm of the second jitter signal is equal to or greater than the predetermined value, the cutoff frequency setting unit 110 sets the cutoff frequency to a lower frequency than when the norm is less than the predetermined value.

[0097] At time T809, the elapsed time since the start of calculation using LPF 105 exceeds the predetermined time, so the process moves from the conditional branch of step S309 to step S711, where the cutoff frequency is set to low Fc_low based on the set exposure time Tv_long.

[0098] This embodiment allows setting an appropriate cutoff frequency based on exposure time, shutter type, and the norm of the accelerometer or magnetometer when combining an accelerometer or magnetometer with a gyroscope sensor using a complementary filter.

[0099] Other embodiments

[0100] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs / instructions) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU), microprocessor unit (MPU) of the system or device) reads and executes the computer program / instructions.

[0101] Each embodiment can set an appropriate cutoff frequency based on imaging condition information when combining an accelerometer or geomagnetic sensor with a gyroscope sensor using a complementary filter, thereby removing the offset component of the gyroscope sensor. Therefore, each embodiment can provide its own image stabilization device and control method, optical device, and storage medium capable of high-precision image stabilization.

[0102] While exemplary embodiments have been described in this disclosure, it should be understood that this disclosure is not limited to the exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation to cover all such modifications and equivalent structures and functions. For example, in various embodiments, the camera condition information includes information relating to at least one of the exposure time, the shutter type of the camera device, and the norm of the second jitter signal, but the embodiments are not limited to this example. The camera condition information may be camera condition information that affects the detection of at least one of the first sensor and the second sensor.

Claims

1. An image stabilization apparatus comprising: a first acquisition unit configured to acquire a first blur signal using a first sensor; a second acquisition unit configured to acquire a second blur signal using a second sensor, the second blur signal having a noise amount in a high frequency band that is greater than a noise amount in the high frequency band of the first blur signal and a noise amount in a low frequency band that is less than a noise amount in the low frequency band of the first blur signal; a generator configured to generate a combined image stabilization signal based on a first signal in the first blur signal and a second signal in the second blur signal, the first signal having a frequency higher than a cutoff frequency, the second signal having a frequency lower than the cutoff frequency; and a setting unit configured to set the cutoff frequency, wherein the setting unit changes the cutoff frequency in accordance with imaging condition information. The imaging condition information is imaging condition information that affects detection by at least one of the first sensor and the second sensor.

2. The image stabilizing apparatus according to claim 1, characterized by The imaging condition information includes information about at least one of an exposure time, a shutter type of an imaging apparatus, and a norm of the second blur signal.

3. The image stabilizing apparatus according to claim 1, characterized by, The imaging condition information is information about an exposure time, 4. The image stabilizing apparatus according to claim 1, characterized by wherein the setting unit is configured to: set the cutoff frequency to a first frequency in a case where the exposure time is a first exposure time, and set the cutoff frequency to a second frequency that is higher than the first frequency in a case where the exposure time is a second exposure time that is longer than the first exposure time. The imaging condition information is information about a norm of the second blur signal, 5. The image stabilizing apparatus according to claim 1, characterized by wherein the setting unit is configured to set the cutoff frequency to be lower in a case where the norm is equal to or greater than a predetermined value than in a case where the norm is less than the predetermined value. The imaging condition information is information about a shutter type of an imaging apparatus, 6. The image stabilizing apparatus according to claim 1, characterized by wherein the setting unit is configured to set the cutoff frequency to be lower in a case where the shutter type is a mechanical shutter type than in a case where the shutter type is an electronic shutter type, until a predetermined time elapses after driving a mechanical shutter. The imaging condition information is information about a shutter type of an imaging apparatus, 7. The image stabilizing apparatus according to claim 1, wherein wherein the setting unit is configured to set the cutoff frequency to be lower in a case where the shutter type is an electronic front curtain shutter type than in a case where a predetermined time elapses after driving a mechanical rear curtain shutter, after elapsing of the predetermined time, or in a case where the shutter type is an electronic shutter type. 8.The image stabilization apparatus according to claim 1, further comprising: a first calculator configured to calculate a first image stabilization signal based on the first blur signal; a selector configured to select and output one of the first image stabilization signal and the combined image stabilization signal based on the cutoff frequency; and a second calculator configured to calculate a driving signal of a correction member that performs image stabilization based on the selected image stabilization signal. ​ ​ 9. The image stabilizing apparatus according to claim 8, characterized by The imaging condition information is information about a norm of the second blur signal, wherein the setting unit is configured to: in a case where the norm is equal to or greater than a predetermined value, set the cutoff frequency to be higher than a cutoff frequency in a case where the norm is less than the predetermined value, and in a case where the norm changes from a value equal to or greater than the predetermined value to a value less than the predetermined value, decrease the cutoff frequency over time, and set the cutoff frequency to a predetermined low cutoff frequency.

10. The image stabilizing apparatus according to claim 8, characterized by The imaging condition information is information about a shutter type of an imaging device, wherein the setting unit is configured to: in a case where the shutter type is a mechanical shutter type, set the cutoff frequency to be higher than a cutoff frequency in a case where the shutter type is an electronic shutter until a predetermined time elapses after driving a mechanical shutter, and after the predetermined time elapses, decrease the cutoff frequency over time, and set the cutoff frequency to a predetermined low cutoff frequency.

11. The image stabilizing apparatus according to claim 8, characterized by The imaging condition information is information about a shutter type of an imaging device, wherein the setting unit is configured to: in a case where the shutter type is an electronic front curtain shutter type, set the cutoff frequency to be higher than a cutoff frequency until a predetermined time elapses after driving a mechanical back curtain shutter, after the predetermined time elapses, or in a case where the shutter type is an electronic shutter type, and after the predetermined time elapses, decrease the cutoff frequency over time, and set the cutoff frequency to a predetermined low cutoff frequency.

12. The image stabilizing apparatus according to claim 8, characterized by The setting unit is configured to: immediately after the generator starts generating the combined image stabilization signal, set the cutoff frequency to a predetermined high cutoff frequency, and decrease the cutoff frequency over time to a predetermined low cutoff frequency.

13. The image stabilizing apparatus according to claim 1, characterized by The first sensor is a gyro sensor.

14. The image stabilizing apparatus according to claim 1, characterized by The second sensor is at least one of an acceleration sensor and a geomagnetic sensor.

15. The image stabilizing apparatus according to claim 1, characterized by The generator constitutes at least a portion of a complementary filter configured to combine the first signal and the second signal, wherein the cutoff frequency is a cutoff frequency of the complementary filter.

16. An optical device comprising: the image stabilization device according to any one of claims 1-15.

17. A control method of an image stabilization device, the control method comprising: obtaining a first blur signal; obtaining a second blur signal having a noise amount in a high frequency band that is greater than a noise amount in a high frequency band of the first blur signal and a noise amount in a low frequency band that is less than a noise amount in a low frequency band of the first blur signal; generating a combined image stabilization signal based on a first signal in the first blur signal and a second signal in the second blur signal, wherein the first signal has a frequency higher than a cutoff frequency and the second signal has a frequency lower than the cutoff frequency; and setting the cutoff frequency, wherein setting the cutoff frequency includes changing the cutoff frequency in accordance with imaging condition information.

18. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 17.

19. A computer program product comprising a program for causing a computer to execute the control method according to claim 17.

Images