Signal processing device, signal processing method, signal processing program, imaging device, and lens device

Abstract

Provided are a signal processing device, a signal processing method, a signal processing program, an imaging device, and a lens device which make it possible to accurately calculate a blur amount on the basis of a signal outputted from a blur detection sensor. This signal processing device processes a signal outputted from a blur detection sensor during exposure of an imaging element, and is provided with a processor. The processor performs processing for generating a second signal obtained by performing high-pass filter processing on a first signal outputted from the blur detection sensor, and processing for calculating a blur amount on the basis of the first signal or the second signal.

Description

[Technical field]

[0001] The present invention relates to a signal processing device, a signal processing method, a signal processing program, an imaging device, and a lens device, and more particularly to a signal processing device, a signal processing method, a signal processing program, an imaging device, and a lens device that process a signal output from a shake detection sensor. [Background technology]

[0002] The image stabilization technology for an imaging device consists of a technology for detecting camera shake and a technology for correcting the camera shake. As a technology for detecting camera shake, a technology for detecting camera shake using a shake detection sensor such as a gyro sensor is known (for example, Patent Documents 1 and 2, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-283443 A [Patent Document 2] JP 2017-107093 A Summary of the Invention

[0004] One embodiment of the technique disclosed herein provides a signal processing device, a signal processing method, a signal processing program, an imaging device, and a lens apparatus that are capable of calculating the amount of blur with high accuracy based on a signal output from a blur detection sensor. [Means for solving the problem]

[0005] (1) A signal processing device that processes a signal output from a shake detection sensor during exposure of an image sensor, the signal processing device comprising a processor, the processor performing a process of generating a second signal by performing high-pass filtering on a first signal output from the shake detection sensor, and a process of calculating an amount of shake based on the first signal or the second signal.

[0006] (2) A signal processing device according to (1), in which the processor switches the signal used to calculate the amount of blur based on the exposure time of the image sensor and the elapsed calculation time of the high-pass filter process when calculating the amount of blur.

[0007] (3) In the signal processing device of (2), in the case where the amount of blur is calculated based on the second signal, when the calculation of the high-pass filter processing is reset, the processor switches from the second signal to the first signal and calculates the amount of blur.

[0008] (4) A signal processing device according to (2) or (3), in which, when the exposure time exceeds a first threshold and the calculation elapsed time is equal to or less than a second threshold, the processor calculates the amount of blur based on the first signal during the period from the start of exposure to the third threshold, and calculates the amount of blur based on the second signal after the third threshold has elapsed from the start of exposure.

[0009] (5) Any one of the signal processing devices (2) to (4), wherein the processor calculates an amount of blur based on the first signal when the exposure time is equal to or less than a first threshold value.

[0010] (6) Any one of the signal processing devices (2) to (5), wherein the processor calculates the amount of blur based on the second signal when the exposure time exceeds a first threshold and the calculation elapsed time exceeds a second threshold.

[0011] (7) The signal processing device of (4), wherein the first threshold, the second threshold, and the third threshold are set based on a cutoff frequency in high-pass filter processing and an offset of the shake detection sensor.

[0012] (8) The signal processing device of (7), wherein the processor further performs a process of calculating an offset based on a signal output from the shake detection sensor.

[0013] (9) In the signal processing device of (8), in calculating the offset, the processor performs a process of calculating a first feature by obtaining a DC component of the signal output from the shake detection sensor, a process of calculating a second feature by obtaining a slope of the accumulated integral value of the signal output from the shake detection sensor, and a process of calculating the offset based on the first feature and the second feature.

[0014] (10) An imaging device comprising: a shake detection sensor provided in a device body; and a signal processing device according to any one of (1) to (9) for processing a signal output from the shake detection sensor.

[0015] (11) A lens device comprising: a shake detection sensor provided in a device body; and a signal processing device according to any one of (1) to (9) for processing a signal output from the shake detection sensor.

[0016] (12) A signal processing method for processing a signal output from a shake detection sensor during exposure of an image sensor, the signal processing method including: a process for generating a second signal by performing high-pass filtering on a first signal output from the shake detection sensor; and a process for calculating an amount of shake based on the first signal or the second signal.

[0017] (13) A signal processing program that causes a computer to execute the following processes: acquiring a signal output from a shake detection sensor during exposure of an image sensor; generating a second signal by performing high-pass filtering on a first signal output from the shake detection sensor; and calculating an amount of shake based on the first signal or the second signal. [Brief description of the drawings]

[0018] [Figure 1] Block diagram showing the schematic configuration of a digital camera [Diagram 2] Conceptual diagram of correction lens movement [Diagram 3] FIG. 1 is a diagram showing a schematic configuration of a correction lens driving unit; [Figure 4] FIG. 2 is a diagram showing a schematic configuration of a first detection unit and a second detection unit; [Diagram 5]Block diagram of functions realized by the system control unit regarding image stabilization control [Figure 6] FIG. 2 is a block diagram of functions of a first correction amount calculation unit and a second correction amount calculation processing unit; [Figure 7] FIG. 1 is a block diagram of functions of a filter processing unit of a first correction amount calculation processing unit; [Figure 8] Conceptual diagram of how the signal output from the filter processing section during the actual imaging exposure is determined [Figure 9] A flowchart showing a processing procedure when a high-pass signal is forcibly switched to a high-passless signal during exposure in actual imaging. [Figure 10] A flowchart showing the processing procedure of the filter processing unit during exposure of the actual image capture. [Figure 11] A flowchart showing the procedure for setting the HPF use flag during exposure of the actual image. [Figure 12] A graph comparing the image stabilization effect depending on the output form of the signal from the filter processing unit [Figure 13] A graph showing the distribution of blurring using the conventional method when the exposure time is 1 [s]. [Figure 14] A graph showing the distribution of blurring using this method when the exposure time is 1 [s]. [Figure 15] A graph comparing the variance of blurring between the conventional method and this method when the exposure time is 1 [s]. [Figure 16] A graph comparing the variance of the blur amount obtained by excluding outliers between the conventional method and this method when the exposure time is 1 [s]. [Figure 17] A graph showing the distribution of blurring using the conventional method when the exposure time is set to 2 [s]. [Figure 18] A graph showing the distribution of blurring using this method when the exposure time is 2 seconds. [Figure 19] A graph comparing the variance of blurring between the conventional method and this method when the exposure time is set to 2 [s]. [Figure 20] A graph comparing the variance of the blur amount obtained by excluding outliers between the conventional method and this method when the exposure time is set to 2 [s]. [Figure 21]A graph comparing the average amount of blur for different imaging timings when the exposure time is set to 1 s. [Figure 22] Histogram of blurring amount by imaging timing when the exposure time is 1 [s] [Diagram 23] A graph comparing the average amount of blur for different capture timings when the exposure time is set to 2 s. [Figure 24] Histogram of blurring amount by imaging timing when the exposure time is 2 [s] [Diagram 25] FIG. 13 is a conceptual diagram showing a modified example of a method for determining a signal to be output from a filter processing unit during exposure for actual imaging. [Figure 26] A flowchart showing a procedure for setting an HPF use flag during exposure for main imaging in a modified example. [Figure 27] Block diagram of the offset correction unit [Figure 28] Block diagram of functions of the second offset calculation unit [Figure 29] A conceptual diagram of a process for extracting second features performed by a second feature extraction unit. [Diagram 30] Block diagram of functions of a second offset calculation unit [Diagram 31] FIG. 1 is a conceptual diagram of a calculation process of a second offset performed by a second offset calculation unit; [Diagram 32] A flowchart showing a procedure for calculating a second offset. [Diagram 33] FIG. 13 is a diagram showing an example of an output signal of a gyro sensor in a stationary state; [Diagram 34] FIG. 11 is a diagram showing the extraction result of the first feature amount. [Diagram 35] A diagram showing the calculation results of integral values [Diagram 36] A diagram showing the changes over time of the first and second features DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0020] Here, an example will be described in which the present invention is applied to a digital camera having a function of detecting camera shake using a gyro sensor and optically correcting the camera shake. The digital camera is an example of an imaging device. The gyro sensor is an example of a shake detection sensor.

[0021] When camera shake is detected by a gyro sensor, the amount of shake (angular shake amount) is calculated by integrating the output signal (angular velocity signal) of the gyro sensor. However, the output signal of the gyro sensor contains an error called an offset. An offset is a reference value deviation (zero point deviation) that prevents the output signal from becoming 0 even when the angular velocity is 0 [deg / s]. If the amount of shake is calculated in a state that includes an offset, an error occurs and accurate correction cannot be performed. For this reason, in general imaging devices, the output signal of the gyro sensor is subjected to high-pass filtering to reduce the effects of the offset.

[0022] However, when the output signal of the gyro sensor is subjected to high-pass filtering to calculate the amount of shake, there is a problem that low-frequency shake remains. Therefore, in order to achieve more accurate image stabilization, it is necessary to perform the correction without using a high-pass filter.

[0023] [Digital camera configuration] FIG. 1 is a block diagram showing a schematic configuration of a digital camera.

[0024] Digital camera 1 of this embodiment is a digital camera equipped with a so-called lens-shift type image stabilization function. A lens-shift type image stabilization mechanism corrects camera shake by moving a correction lens provided in the imaging optical system.

[0025] As shown in FIG. 1, the digital camera 1 includes an imaging optical system 10, an angular velocity detection unit 20, an image sensor 30, a main memory unit 40, a digital signal processing unit 41, an auxiliary memory unit 42, a display unit 43, an operation unit 44, and a system control unit 50.

[0026] The imaging optical system 10 includes a zoom lens 11, a focus lens 12, a correction lens 13, and an aperture 14.

[0027] The zoom lens 11 is a lens for adjusting the focal length. The focal length of the imaging optical system 10 is adjusted by moving the zoom lens 11 back and forth along the optical axis. The zoom lens 11 is driven by a zoom lens driving unit 15 to operate.

[0028] The focus lens 12 is a lens for adjusting focus. The focus of the imaging optical system 10 is adjusted by moving the focus lens 12 back and forth along the optical axis. The focus lens 12 is driven by a focus lens driver 16 to operate.

[0029] The correction lens 13 is a lens for correcting camera shake. The correction lens 13 moves within a plane perpendicular to the optical axis to correct camera shake.

[0030] FIG. 2 is a conceptual diagram of the movement of the correction lens.

[0031] The optical axis is the z-axis, the axis perpendicular to the z-axis is the x-axis, and the axis perpendicular to the z-axis and x-axis is the y-axis. The x-axis is the horizontal (horizontal) axis. The y-axis is the vertical (vertical) axis. The correction lens 13 moves along a plane (xy plane) defined by the x-axis and y-axis. This plane (xy plane) is parallel to the light receiving surface of the image sensor 30.

[0032] When correcting camera shake, the correction lens 13 is moved in a direction to cancel out the shake. The correction lens 13 is driven by a correction lens drive unit 17 to operate.

[0033] FIG. 3 is a diagram showing a schematic configuration of the correction lens driving unit.

[0034] The correction lens driver 17 has a first driver 17x and a second driver 17y. The first driver 17x moves the correction lens 13 along the x-axis. The second driver 17y moves the correction lens 13 along the y-axis.

[0035] The diaphragm 14 is, for example, an iris diaphragm. The amount of light passing through the imaging optical system 10 is adjusted by the diaphragm 14. The diaphragm 14 is driven by a diaphragm driver 18 to operate.

[0036] 2, the angular velocity detection unit 20 includes a first detection unit 20A and a second detection unit 20B. The first detection unit 20A detects the angular velocity of the digital camera body (device body) in the yaw direction Yaw. The second detection unit 20B detects the angular velocity of the digital camera body in the pitch direction Pit.

[0037] Here, the yaw direction Yaw is the rotation direction around the y axis as shown in Fig. 2. The yaw direction Yaw is the horizontal (lateral) rotation direction in the digital camera body.

[0038] Moreover, the pitch direction Pit is the rotation direction around the x-axis as shown in Fig. 2. The pitch direction Pit is the rotation direction in the vertical direction (lengthwise direction) in the digital camera body.

[0039] FIG. 4 is a diagram showing a schematic configuration of the first detection unit and the second detection unit.

[0040] The first detection unit 20A and the second detection unit 20B each include a gyro sensor 21A, 21B and an ADC (Analog to Digital Converter) 22A, 22B. The signals (angular velocity signals) output from the gyro sensors 21A, 21B are converted into digital signals by the ADCs 22A, 22B and input to the system control unit 50.

[0041] The image sensor 30 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) having a predetermined color filter array (for example, a Bayer array, etc.). etThe image sensor 30 is made up of an Al-Oxide Semiconductor type image sensor. In the digital camera 1 of this embodiment, the image sensor 30 is made up of a drive section, an ADC (Analog to Digital Converter), a signal processor, and the like. In this case, the image sensor 30 is driven by the built-in drive section to operate. The signal of each pixel is converted into a digital signal by the built-in ADC. Furthermore, the signal of each pixel is subjected to processes such as correlated double sampling, gain processing, and correction processing by the built-in signal processor as necessary. The signal processing can be configured to be performed on the analog signal of each pixel, or can be configured to be performed on the digital signal of each pixel.

[0042] The imaging element 30 can be configured with an organic thin-film imaging element, an XY address type, a CCD (Charged Coupled Device) type image sensor, or the like, in addition to a CMOS type image sensor.

[0043] The main storage unit 40 is used as a temporary storage area for data. The signal (image signal) output from the imaging element 30 is stored in the main storage unit 40 for each frame. The main storage unit 40 is formed of, for example, a semiconductor memory.

[0044] The digital signal processing unit 41 performs signal processing such as offset processing, gamma correction processing, demosaic processing, RGB / YCrCb conversion processing, etc. on the image signal obtained by imaging to generate image data. The digital signal processing unit 41 is formed of, for example, a microprocessor.

[0045] The auxiliary storage unit 42 functions as a storage unit for various data. The auxiliary storage unit 42 is composed of a non-volatile semiconductor memory such as an EEPROM (Electrically Erasable Programmable Read Only Memory) or a flash memory. The data recorded in the auxiliary storage unit 42 includes image data obtained by imaging as well as data required for control.

[0046] The display unit 43 is used as a playback monitor when playing back captured images, and is also used as a live view monitor for displaying live view images during image capture. It is also used as a setting monitor when performing various settings. The display unit 43 is configured with a display such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode).

[0047] Operation unit 44 is configured to include various operation members for operating digital camera 1. The operation members include a power button, a shutter button, and various other operation buttons. The various operation buttons include a button for turning on and off the image stabilization function. Furthermore, if display unit 43 is configured with a display unit equipped with a touch panel, the operation members constituting operation unit 44 include the touch panel. Operation unit 44 outputs a signal according to the operation of each operation member to system control unit 50.

[0048] The system control unit 50 performs overall control of the digital camera 1. The system control unit 50 also calculates various physical quantities required for control. The system control unit 50 is configured, for example, by a microcomputer equipped with a processor, memory, and the like. The processor is configured, for example, by a CPU (Central Processing Unit), and the like. The memory is configured, for example, by a RAM (Ra n The system controller 50 is made up of a ROM (Read Only Memory) and a DOMAccess Memory (ROM). The memory stores programs executed by the processor (such as a signal processing program) and various data. The control performed by the system controller 50 includes control of camera shake correction.

[0049] [System configuration for image stabilization control] FIG. 5 is a block diagram of functions implemented by the system control unit regarding control of camera shake correction.

[0050] As shown in the figure, with regard to the control of camera shake correction, the system control unit 50 functions as offset correction units 60A and 60B, a first correction amount calculation unit 70A, a second correction amount calculation unit 70B, a first drive control unit 80A, and a second drive control unit 80B, etc. These functions are realized by a processor constituting the system control unit 50 executing a predetermined control program.

[0051] The offset correction unit 60A performs a process of removing an offset from the signal (angular velocity signal in the yaw direction) output from the first detection unit 20A based on the offset information stored in an offset storage unit (not shown). The offset correction unit 60B performs a process of removing an offset from the signal (angular velocity signal in the pitch direction) output from the second detection unit 20B based on the offset information stored in an offset storage unit (not shown). The offset storage unit may be configured, for example, by the auxiliary storage unit 42, or may be configured by incorporating a memory of a required capacity inside the offset correction unit.

[0052] The first correction amount calculation unit 70A calculates the correction amount in the x-axis direction of the correction lens 13 based on the yaw direction angular velocity signal after the offset has been removed by the offset correction unit 60A. In other words, it calculates the movement amount in the x-axis direction of the correction lens 13 for correcting shake in the yaw direction.

[0053] FIG. 6 is a block diagram of functions possessed by the first correction amount calculation unit and the second correction amount calculation processing unit.

[0054] As shown in the figure, the first correction amount calculation unit 70A has the functions of a filter processing unit 71A, a sensitivity correction unit 72A, an integration unit 73A, and a correction amount calculation unit 74A. In the present embodiment, the system control unit 50 functioning as the first correction amount calculation unit 70A and the second correction amount calculation unit 70B is an example of a signal processing device.

[0055] The filter processing unit 71A generates a signal by performing high-pass filtering on the angular velocity signal in the yaw direction after the offset has been removed. The filter processing unit 71A also selectively switches between a signal that has not been subjected to high-pass filtering (a first signal) and a signal that has been subjected to high-pass filtering (a second signal) according to a predetermined condition, and outputs the signal. The conditions for switching and the details of the switching process will be described later.

[0056] The sensitivity correction unit 72A multiplies the signal output from the filter processing unit 71A (angular velocity signal that has been subjected to high-pass filtering or angular velocity signal that has not been subjected to high-pass filtering) by the sensitivity. The sensitivity is a unitless coefficient that is uniquely determined using the focus lens position and focal length (zoom lens position) as variables. Therefore, when the focus lens position and focal length change from moment to moment, the sensitivity also changes from moment to moment.

[0057] The integrator 73A integrates the angular velocity signal whose sensitivity has been corrected by the sensitivity corrector 72A, thereby calculating the amount of shaking in the yaw direction.

[0058] Correction amount calculation section 74A calculates the correction amount in the x-axis direction of correction lens 13 based on the amount of shake in the yaw direction calculated by integration section 73A. In other words, it calculates the amount of movement in the x-axis direction of correction lens 13 that is necessary to offset the shake in the yaw direction.

[0059] The second correction amount calculation unit 70B calculates the correction amount in the y-axis direction of the correction lens 13 based on the angular velocity signal in the pitch direction after the offset has been removed by the offset correction unit 60B. That is, it calculates the amount of movement in the y-axis direction of the correction lens 13 required to offset the shake in the pitch direction. The second correction amount calculation unit 70B has the same functions as the first correction amount calculation unit 70A. As shown in FIG. 6, the second correction amount calculation unit 70B has the functions of a filter processing unit 71B, a sensitivity correction unit 72B, an integration unit 73B, and a correction amount calculation unit 74B. A detailed description of each function will be omitted.

[0060] The first drive control unit 80A controls the drive of the first drive unit 17x to control the movement in the x-axis direction of the correction lens 13. The first drive control unit 80A controls the drive of the first drive unit 17x based on the correction amount in the x-axis direction of the correction lens 13 calculated by the first correction amount calculation unit 70A.

[0061] The second drive control unit 80B controls the driving of the second drive unit 17y to control the movement in the y-axis direction of the correction lens 13. The second drive control unit 80B controls the driving of the second drive unit 17y based on the correction amount in the y-axis direction of the correction lens 13 calculated by the second correction amount calculation unit 70B.

[0062] The control of the camera shake correction is executed in the following procedure. First, the offset correction units 60A and 60B perform a process of removing the offset from the signals (angular velocity signals) output from the first detection unit 20A and the second detection unit 20B. Next, the first correction amount calculation unit 70A and the second correction amount calculation unit 70B calculate the correction amounts in the x-axis direction and the y-axis direction of the correction lens 13 based on the signals from which the offset has been removed. Next, the first drive unit 17x and the second drive unit 17y are driven by the first drive control unit 80A and the second drive control unit 80B based on the calculated correction amounts. As a result, the correction lens 13 is moved so as to cancel out the shake, and the camera shake is corrected.

[0063] [Filter processing part] [Filter processing section configuration] Next, the functions of the filter processing units 71A and 71B provided in the first correction amount calculation unit 70A and the second correction amount calculation unit 70B will be described. The functions of the filter processing unit 71A provided in the first correction amount calculation unit 70A and the functions of the filter processing unit 71B provided in the second correction amount calculation unit 70B are the same. Therefore, here, the functions of the filter processing unit 71A provided in the first correction amount calculation unit 70A will be described, and the description of the functions of the filter processing unit 71B provided in the second correction amount calculation unit 70B will be omitted.

[0064] FIG. 7 is a block diagram of functions included in the filter processor of the first correction amount calculation processor.

[0065] As shown in the figure, the filter processing unit 71A has the functions of a high pass filter (HPF) processing unit 71A1, a first elapsed time measurement unit 71A2, an imaging information acquisition unit 71A3, a second elapsed time measurement unit 71A4, an output signal determination unit 71A5, and an output signal switching unit 71A6. Each function is realized by a processor constituting the system control unit 50 executing a predetermined program (signal processing program).

[0066] The high-pass filter processing unit 71A1 , O A signal (second signal) is generated by performing high-pass filtering on the yaw direction angular velocity signal (first signal) after the offset has been removed.

[0067] The following equation is an example of the calculation of the high-pass filter processing performed by the high-pass filter processing unit 71A1. The signal value (angular velocity value) of the angular velocity signal input to the high-pass filter processing unit 71A1 is ω, and the signal value of the angular velocity signal output from the high-pass filter processing unit 71A1 when the cutoff frequency is n is HPFn. HPFn is calculated by the following equation.

[0068] HPFn[k] = ω[k] × HPFn_COEF_A - ω[k-1] × HPFn_COEF_A + HPFn[k-1] × HPFn_COEF_B Here, HPFn_COEF_A and HPFn_COEF_B are filter coefficients when the cutoff frequency is n.

[0069] If a signal that has not been subjected to high-pass filtering is designated as HPF_LESS, then HPF_LESS can be calculated using the following formula.

[0070] HPF_LESS[k] = ω[k]

[0071] The first elapsed time measurement unit 71A2 measures the computation elapsed time of the high-pass filter processing in the high-pass filter processing unit 71A1 as the first elapsed time. The computation elapsed time of the high-pass filter processing means the time elapsed from the start of the computation of the high-pass filter processing. In the high-pass filter processing, the computation may be reset. When the computation of the high-pass filter processing is reset, the measurement of the elapsed time is also reset. Therefore, when the computation of the high-pass filter processing is reset, the elapsed time from the start of the computation again after the reset is measured as the first elapsed time. Information on the first elapsed time (computation elapsed time of the high-pass filter processing) measured by the first elapsed time measurement unit 71A2 is output to the output signal determination unit 71A5.

[0072] The imaging information acquisition unit 71A3 acquires imaging information. The imaging information includes information on the actual imaging and information on the exposure time during the actual imaging. The "actual imaging" refers to imaging for recording. The information on the actual imaging includes information on the start and end of the actual imaging. In the digital camera 1 of this embodiment, the actual imaging is performed by fully pressing the shutter button. The digital camera 1 prepares for imaging by half pressing the shutter button (S1 on), and performs imaging for recording by fully pressing the shutter button (S2 on). In the preparation for imaging, AE (Automatic Exposure) and AF (Autofocus) are performed. The exposure conditions during the actual imaging are determined by AE. Also, the main subject is focused by AF. Note that these processes are well-known techniques, and therefore detailed explanations thereof will be omitted. The imaging information acquisition unit 71A3 acquires exposure time (shutter speed) information obtained by AE, and acquires information on the exposure time during the actual imaging. The imaging information acquisition unit 71A3 outputs the acquired imaging information to the output signal determination unit 71A5.

[0073] When a signal that has not been subjected to high-pass filtering is output during exposure in the actual imaging, the second elapsed time measurement unit 71A4 measures the elapsed time from the start of output of the signal as the second elapsed time. Information on the second elapsed time measured by the second elapsed time measurement unit 71A4 (information on the elapsed time from the start of output of a signal that has not been subjected to high-pass filtering) is output to the output signal determination unit 71A5.

[0074] The output signal determination unit 71A5 determines which signal is to be output from the filter processing unit 71A, a signal not subjected to high-pass filtering (hereinafter referred to as a high-passless signal) or a signal subjected to high-pass filtering (hereinafter referred to as a high-pass signal). The output signal determination unit 71A5 determines the signal to be output based on information on the first elapsed time (calculation elapsed time of high-pass filtering) output from the first elapsed time measurement unit 71A2, imaging information output from the imaging information acquisition unit 71A3, and information on the second elapsed time (elapsed time from the start of output of the high-passless signal) output from the second elapsed time measurement unit 71A4. Specifically, the high-pass signal is output except during exposure for the actual imaging. On the other hand, during exposure for the actual imaging, the high-passless signal or the high-passless signal is selectively output. Hereinafter, a method of determining the signal to be output from the filter processing unit 71A during exposure for the actual imaging will be described. Whether or not exposure for the actual imaging is in progress is determined based on imaging information acquired by the imaging information acquisition unit 71A3. That is, whether or not exposure for actual imaging is in progress is determined based on information on the start and end of actual imaging included in the imaging information. The period from the start to the end of actual imaging is considered to be exposure for actual imaging, and the remaining period is considered to be non-exposure.

[0075] FIG. 8 is a conceptual diagram showing how to determine the signal to be output from the filter processing section during exposure for actual imaging.

[0076] When performing main imaging, the signal output from the filter processing unit 71A during exposure is determined based on the exposure time and the first elapsed time (calculation elapsed time of high-pass filter processing).

[0077] If the exposure time is equal to or less than the first threshold (if step S1 in FIG. 8 is "Yes"), the signal output from the filter processing unit 71A during the exposure of the actual image capture is a high-passless signal regardless of the first elapsed time. In this case, the calculation elapsed time of the high-pass filter processing (first elapsed time) does not matter.

[0078] When the exposure time is equal to or less than the first threshold, it means that the exposure time is relatively short. When the exposure time is short, a high-pass filterless signal is output during exposure. This allows low-frequency blurring to be appropriately detected.

[0079] When the exposure time exceeds the first threshold and the first elapsed time exceeds the second threshold (when step S2 in FIG. 8 is "Yes"), the signal output from the filter processing unit 71A during exposure is set as a high-pass signal.

[0080] The case where the exposure time exceeds the first threshold value means that the exposure time is relatively long. In this case, if the amount of blur is calculated based on the high-passless signal, an error may occur due to the influence of the offset. Therefore, when the exposure time is long, it is better to calculate the amount of blur based on the high-pass signal, since the amount of blur can be calculated without being influenced by the offset. However, if sufficient time has not passed since the start of the calculation of the high-pass filter processing, the value of the high-pass signal will not stabilize, which may lead to an error. Therefore, the high-pass signal is output only when sufficient time has passed since the start of the calculation of the high-pass filter processing.

[0081] In view of the above, the second threshold is set to a value necessary and sufficient for the high-pass signal to stabilize. In other words, the second threshold is set to a time when the high-pass signal is stabilized when the second threshold is exceeded. The optimum value of the second threshold can be obtained, for example, by experiment or simulation. For example, the experiment and simulation can employ a method of changing the offset and the cutoff frequency of the high-pass filter processing to obtain the optimum value.

[0082] Moreover, the first threshold is set to a time when the effect of the offset is dominant when the first threshold is exceeded. In other words, when the first threshold is less than the first threshold, the time is set to a time when the effect of the offset is negligible. The optimal value of the first threshold can also be obtained, for example, by experiment or simulation. For example, the experiment and simulation can employ a method of changing the offset and the cutoff frequency of the high-pass filter processing to obtain the optimal value.

[0083] When the exposure time exceeds the first threshold and the first elapsed time is equal to or less than the second threshold (step S2 in FIG. 8 is "No"), the signal output from the filter processing unit 71A during exposure is a high-passless signal and a high-pass signal. More specifically, the high-passless signal is output at first, and then switched to and output as a high-pass signal midway.

[0084] As described above, when the exposure time is long, an error may occur in the calculated amount of blur due to the effect of offset. On the other hand, in order to realize more accurate camera shake correction, it is necessary to calculate the amount of blur based on a high-passless signal. Therefore, even if the exposure time exceeds the first threshold, if the first elapsed time is equal to or less than the second threshold, the signal is switched from the high-passless signal to a high-pass signal midway and output. This allows for more accurate camera shake correction.

[0085] The timing of switching from the high-passless signal to the high-pass signal is determined in advance. The timing of switching is specified by the elapsed time from the start of output of the high-passless signal. When the elapsed time from the start of output of the high-passless signal exceeds the third threshold, the high-passless signal is switched to the high-pass signal. As described above, the elapsed time from the start of output of the high-passless signal is measured as the second elapsed time. Therefore, when the second elapsed time exceeds the third threshold, the high-passless signal is switched to the high-pass signal. Note that since this is processing during exposure, the elapsed time from the start of output of the high-passless signal (second elapsed time) is the same as the elapsed time from the start of exposure.

[0086] The optimum value of the third threshold can be obtained, for example, by experiments or simulations. For the experiments and simulations, for example, a method of changing the offset and the cutoff frequency of the high-pass filter processing to obtain the optimum value can be adopted.

[0087] However, the calculation of the high-pass filter processing may be reset during exposure. As described above, the high-pass signal does not stabilize until a sufficient amount of time has passed since the calculation started. Therefore, if the high-pass signal continues to be output after the calculation is reset, an error will occur in the calculated amount of blur.

[0088] Therefore, in the digital camera 1 of the present embodiment, when the calculation of the high-pass filter processing is reset during exposure, the output signal is forcibly switched from a high-pass signal to a high-passless signal.

[0089] The output signal determination unit 71A5 determines whether or not a reset has occurred based on information on the first elapsed time (calculation elapsed time of the high-pass filter processing) output from the first elapsed time measurement unit 71A2, for example. That is, when the calculation of the high-pass filter processing in the high-pass filter processing unit 71A1 is reset, the measurement of the first elapsed time is also reset. Therefore, it is possible to determine whether or not the calculation of the high-pass filter processing has been reset from the first elapsed time. When the output signal determination unit 71A5 determines that the calculation of the high-pass filter processing has been reset, it forcibly switches the signal to be output to a high-pass-less signal.

[0090] FIG. 9 is a flowchart showing a processing procedure when a high-pass signal is forcibly switched to a high-passless signal during exposure in actual imaging.

[0091] The output signal determination unit 71A5 determines whether the signal output from the filter processing unit 71A is a high-pass signal (step S11). If the signal to be output is not a high-pass signal (if it is a high-passless signal), the process proceeds to step S14.

[0092] If the signal to be output is a high-pass signal, the output signal determination unit 71A5 determines whether the calculation of the high-pass filter processing has been reset (step S12). If it is determined that the calculation of the high-pass filter processing has not been reset, the process proceeds to step S14.

[0093] When it is determined that the calculation of the high-pass filter processing has been reset, the output signal determination unit 71A5 switches the signal to be output from a high-pass signal to a high-passless signal (step S13).

[0094] Thereafter, the output signal determination unit 71A5 judges whether the exposure has been completed (step S14). If it is judged that the exposure has been completed, the process ends. On the other hand, if it is judged that the exposure has not been completed, the process returns to step S11.

[0095] In this way, when a high-pass signal is output, if the calculation of the high-pass filter processing is reset during exposure, the output signal is forcibly switched to a high-passless signal, which eliminates the effect of the reset and allows the amount of blur to be calculated with higher accuracy.

[0096] When a high-pass signal is to be output, the output signal determination unit 71A5 sets the HPF use flag to "1." On the other hand, when a high-passless signal is to be output, the output signal determination unit 71A5 sets the HPF use flag to "0."

[0097] The output signal switching unit 71A6 outputs either a high-pass signal or a high-passless signal according to the decision made by the output signal deciding unit 71A5. More specifically, it outputs either a high-pass signal or a high-passless signal according to the setting of the HPF usage flag. If the HPF usage flag is "1", it outputs a high-pass signal. On the other hand, if the HPF usage flag is "0", it outputs a high-passless signal.

[0098] [Operation of the filter processing section (signal processing method)] FIG. 10 is a flowchart showing the processing procedure of the filter processing unit during exposure for main imaging.

[0099] When calculating the amount of blur, the filter processors 71A and 71B selectively output either a high-pass signal or a high-passless signal.

[0100] The amount of blur is calculated when the camera shake correction function is ON. Therefore, first, it is determined whether the camera shake correction function is ON (step S101).

[0101] If the camera shake correction function is ON, it is then determined whether or not exposure for actual imaging is in progress (step S102).

[0102] If the exposure is not being performed for the actual image capture, the amount of blur is calculated using the high-pass signal, and therefore, in this case, the HPF use flag is set to "1" (step S103).

[0103] On the other hand, during exposure for actual imaging, the HPF use flag is set based on the exposure time, the first elapsed time (the elapsed time for calculating the high-pass filter processing), and the second elapsed time (the elapsed time from the start of output of the high-passless signal).

[0104] FIG. 11 is a flowchart showing the procedure for setting the HPF use flag during exposure for main imaging.

[0105] First, it is determined whether or not the exposure time exceeds the first threshold value and the first elapsed time exceeds the second threshold value (step S111).

[0106] If the exposure time exceeds the first threshold and the first elapsed time exceeds the second threshold, the amount of blur is calculated using the high-pass signal. Therefore, in this case, the HPF use flag is set to "1" (step S115).

[0107] When the exposure time is equal to or less than the first threshold value, or when the first elapsed time is equal to or less than the second threshold value, it is first determined whether or not the exposure time exceeds the first threshold value (step S112).

[0108] Here, if the exposure time does not exceed the first threshold, that is, if the exposure time is equal to or less than the first threshold, the amount of blur is calculated using the high-pass filter-less signal. Therefore, in this case, the HPF use flag is set to "0" (step S116).

[0109] On the other hand, if the exposure time exceeds the first threshold, it is next determined whether the first elapsed time exceeds the second threshold (step S113). That is, it is determined whether the calculation elapsed time of the high-pass filter processing exceeds the second threshold. Here, if the first elapsed time does not exceed the second threshold, the amount of blur is calculated using the high-passless signal. Therefore, in this case, the HPF use flag is set to "0" (step S116). That is, the determination here is whether the calculation of the high-pass filter processing is reset during the output of the high-pass signal. If the calculation of the high-pass filter processing is reset during the output of the high-pass signal, the first elapsed time is also reset. Therefore, in this case, the signal to be output is forcibly switched from the high-pass signal to the high-passless signal.

[0110] If the first elapsed time exceeds the second threshold, it is next determined whether or not the second elapsed time exceeds the third threshold (step S114). That is, it is determined whether or not the elapsed time from the start of output of the high-passless signal exceeds the third threshold. Here, if the second elapsed time exceeds the third threshold, the HPF use flag is set to "1" (step S115). On the other hand, if the second elapsed time does not exceed the third threshold, that is, if the second elapsed time is equal to or less than the third threshold, the HPF use flag is set to "0" (step S116). That is, this determination is made to determine whether or not the timing of switching has been reached in the case of switching from the high-passless signal to the high-pass signal midway. If the timing of switching has been reached, the signal to be output is switched to the high-pass signal. That is, the HPF use flag is set to "1" (step S115).

[0111] As described above, during exposure for actual imaging, the HPF use flag is set based on the exposure time, the first elapsed time (the elapsed time from the start of the calculation of the high-pass filter processing), and the second elapsed time (the elapsed time from the start of output of the high-pass filterless signal).

[0112] Once the HPF use flag has been set, as shown in FIG. 10, it is determined whether the HPF use flag is set to "1" (step S104).

[0113] When the HPF use flag is set to "1", high-pass signals are output from the filter processing units 71A and 71B (step S105). On the other hand, when the HPF use flag is set to "0", high-passless signals are output from the filter processing units 71A and 71B (step S106).

[0114] After that, it is determined whether the image stabilization function has been turned off (step S107). If the image stabilization function has been turned off, the process ends. On the other hand, if the image stabilization function has not been turned off, the process returns to step S102 and continues. Note that if the power supply of digital camera 1 is turned off, the image stabilization function is also turned off.

[0115] As described above, in the digital camera 1 of this embodiment, the high-pass signal and the high-passless signal are selectively output from the filter processing units 71A and 71B, and the amount of blur is calculated. Except during the exposure for the actual image capture, the high-pass signal is output, and the amount of blur is calculated. During the exposure for the actual image capture, the signal to be output is determined based on the exposure time, the first elapsed time (the calculation elapsed time of the high-pass filter processing), and the second elapsed time (the elapsed time from the start of the output of the high-passless signal). If the exposure time is equal to or less than the first threshold, the high-passless signal is output. If the exposure time exceeds the first threshold and the first elapsed time exceeds the second threshold, the high-pass signal is output. If the exposure time exceeds the first threshold and the first elapsed time is equal to or less than the second threshold, the high-passless signal is output first, and is switched to the high-pass signal midway and output. Even when the high-pass signal is output, if the calculation for the high-pass filter processing is reset during the exposure, the signal to be output is forcibly switched to the high-passless signal. In this way, by appropriately controlling the signal to be output, the amount of blur can be calculated with higher accuracy. That is, it is possible to suppress the occurrence of low-frequency residual blur while suppressing the effect of offset.

[0116] As an example of the embodiment, the processing of the filter processing unit when the offset standard of the gyro sensor to be used and the offset correction calculation amount are within 14 [mdps] and the cutoff frequency in the high-pass filter processing is 0.01 [Hz] will be described. Note that the processing here is processing during the exposure of the actual image capture.

[0117] Optimal first, second and third thresholds are obtained through experiments or simulations. As a result of the experiments or simulations, it is assumed that the first threshold is set to 1 [s], the second threshold to 15 [s] and the third threshold to 1 [s].

[0118] In this case, the output of a signal from the filter processing section during exposure is controlled as follows.

[0119] (1) If the exposure time is less than 1 s, a high-pass filterless signal is output.

[0120] (2) If the exposure time exceeds 1 “s” and the calculation elapsed time of the high-pass filter processing (first elapsed time) is 15 “s” or less, a high-passless signal is output for the first 1 “s”, and a high-pass signal is output for the remaining time.

[0121] (3) When the exposure time exceeds 1 “s” and the calculation elapsed time of the high-pass filter process (first elapsed time) exceeds 15 [s], a high-pass signal is output.

[0122] (4) In the case where a high-pass signal is output during exposure, when the calculation of the high-pass filter processing is reset, the high-pass signal is switched to a high-passless signal and output.

[0123] By controlling the signal output from the filter processing unit in this manner, it is possible to suppress the occurrence of low-frequency residual blur while suppressing the effects of offset.

[0124] FIG. 12 is a graph comparing the correction effect of camera shake correction depending on the output form of the signal from the filter processing unit.

[0125] In the figure, the horizontal axis is the zero point deviation of the gyro sensor, and the vertical axis is the image stabilization step number. The image stabilization step number indicates the performance of the image stabilization, and is measured based on the method defined by the CIPA (Camera & Imaging Products Association) standard (CIPA DC-011-2015 Measurement method and description method for image stabilization performance of digital cameras (optical type)).

[0126] In the figure, signal output form A is a signal output form that always outputs a high-pass signal. Signal output form B is a signal output form that always outputs a high-passless signal. Signal output form C is a signal output form that first outputs a high-passless signal and switches to a high-pass signal after a certain time has passed. Signal output form D is a signal output form that outputs a high-passless signal when the exposure time exceeds a first threshold value and the first elapsed time exceeds a second threshold value, but otherwise outputs a high-passless signal first and switches to a high-pass signal after a certain time has passed.

[0127] As shown in the figure, signal output form A shows good performance when the offset is large. On the other hand, signal output form B shows good performance as the offset is smaller. Signal output form C shows better performance than signal output form A when the offset is small. Signal output form D shows stable and good performance both when the offset is large and when it is small.

[0128] The method of the present invention is to process the signal (angular velocity signal) from the gyro sensor in the above embodiment to calculate the amount of shake and perform camera shake correction, while the method of the conventional method is to always apply high-pass filter processing to the signal from the gyro sensor to calculate the amount of shake and perform camera shake correction.

[0129] An experiment was conducted to obtain the amount of blurring in the yaw and pitch directions of the captured images by applying a certain vibration that imitates camera shake. The cutoff frequency of the high-pass filter processing was set to 0.01 [Hz].

[0130] Fig. 13 is a graph showing the distribution of blur amounts obtained by the conventional method when the exposure time is 1 [s]. Fig. 14 is a graph showing the distribution of blur amounts obtained by the present method when the exposure time is 1 [s]. Fig. 15 is a graph comparing the variance of blur amounts obtained by the conventional method and the present method when the exposure time is 1 [s]. Fig. 16 is a graph comparing the variance of blur amounts obtained by excluding outliers from the conventional method and the present method when the exposure time is 1 [s].

[0131] As shown in FIG. 13, in the conventional method, large blurring can occur (values ​​inside the circle in FIG. 13), whereas in the present method, as shown in FIG. 14, no large blurring is observed.

[0132] Moreover, as shown in Figure 15, it is recognized that the present method has a smaller variance than the conventional method. Even when outliers are excluded, as shown in Figure 16, it is recognized that the present method has a smaller variance.

[0133] Fig. 17 is a graph showing the distribution of blur amounts obtained by the conventional method when the exposure time is set to 2 [s]. Fig. 18 is a graph showing the distribution of blur amounts obtained by the present method when the exposure time is set to 2 [s]. Fig. 19 is a graph comparing the variance of blur amounts obtained by the conventional method and the present method when the exposure time is set to 2 [s]. Fig. 20 is a graph comparing the variance of blur amounts obtained by excluding outliers from the conventional method and the present method when the exposure time is set to 2 [s].

[0134] Even when the exposure time is 2 [s], the conventional method can produce large blurring (values ​​inside the circle in Fig. 17), whereas the present method does not produce large blurring, as shown in Fig. 18. Moreover, as shown in Fig. 19, even when the exposure time is 2 [s], the present method has a smaller variance than the conventional method. Even when outliers are excluded, the present method has a smaller variance, as shown in Fig. 20.

[0135] Fig. 21 is a graph comparing the average amount of blur for each imaging timing when the exposure time is 1 [s]. Fig. 22 is a histogram of the amount of blur for each imaging timing when the exposure time is 1 [s].

[0136] With the conventional method, the amount of blurring is large immediately after the start of imaging, whereas with the present method, it can be confirmed that there is no difference in the amount of blurring depending on the timing of imaging.

[0137] Fig. 23 is a graph comparing the average amount of blur for each image capture timing when the exposure time is 2 [s]. Fig. 24 is a histogram of the amount of blur for each image capture timing when the exposure time is 2 [s].

[0138] Even when the exposure time is 2 s, the amount of blurring is large immediately after the start of imaging with the conventional method, whereas with the proposed method, there is no difference in the amount of blurring depending on the timing of shooting.

[0139] [Variations of how to determine the signal output from the filter processing unit] FIG. 25 is a conceptual diagram showing a modified example of the method for determining the signal to be output from the filter processing section during exposure for main imaging.

[0140] If the exposure time exceeds the first threshold and the first elapsed time exceeds the second threshold, a high-pass signal is output. Otherwise, a high-passless signal is output at first, and then switched to a high-pass signal midway. That is, in the above embodiment, when the exposure time is equal to or less than the first threshold, a high-passless signal is continuously output, but in this example, the signal may be switched to a high-pass signal midway.

[0141] It should be noted that even when a high-pass signal is output, if the calculation of the high-pass filter processing is reset during exposure, the signal is switched to a high-passless signal, as in the above embodiment.

[0142] FIG. 26 is a flowchart showing a processing procedure for setting the HPF use flag during exposure for main imaging in the modified example.

[0143] First, it is determined whether the exposure time exceeds the first threshold and the first elapsed time exceeds the second threshold (step S111). If the exposure time exceeds the first threshold and the first elapsed time exceeds the second threshold, the amount of blur is calculated using the high-pass signal. Therefore, in this case, the HPF use flag is set to "1" (step S115).

[0144] If the exposure time is equal to or less than the first threshold value, or if the first elapsed time is equal to or less than the second threshold value, it is determined whether or not the first elapsed time exceeds the second threshold value (step S113). That is, it is determined whether or not the calculation elapsed time of the high-pass filter processing exceeds the second threshold value. Here, if the first elapsed time does not exceed the second threshold value, the amount of blur is calculated using the high-passless signal. Therefore, in this case, the HPF use flag is set to "0" (step S116). This determination includes a determination of whether or not the calculation of the high-pass filter processing is reset during the output of the high-pass signal. If the calculation of the high-pass filter processing is reset during the output of the high-pass signal, the signal to be output is forcibly switched from the high-pass signal to the high-passless signal.

[0145] If the first elapsed time exceeds the second threshold, it is next determined whether or not the second elapsed time exceeds the third threshold (step S114). Here, if the second elapsed time exceeds the third threshold, the HPF use flag is set to "1" (step S115). On the other hand, if the second elapsed time does not exceed the third threshold, that is, if the second elapsed time is equal to or less than the third threshold, the HPF use flag is set to "0" (step S116). That is, this determination is made to determine whether or not the timing for switching has been reached when switching from a high-passless signal to a high-pass signal midway. If the timing for switching has been reached, the signal to be output is switched to a high-pass signal. That is, the HPF use flag is set to "1" (step S115).

[0146] As described above, in this modified example, if the exposure time exceeds the first threshold and the first elapsed time exceeds the second threshold, a high-pass signal is output. Otherwise, a high-passless signal is output at first, and then switched to a high-pass signal midway through. This makes it possible to suppress low-frequency residual blur while suppressing the effects of offset.

[0147] [Embodiments for optimizing each threshold] The first threshold, the second threshold, and the third threshold may be determined by simulation, etc. Furthermore, they may be optimized according to the following criteria, for example.

[0148] The standard value of the offset (zero point shift) is n [mdps]. If the standard value n of the offset is large, the cutoff frequency of the high-pass filter is large, and if the standard value n of the offset is small, the cutoff frequency of the high-pass filter is small.

[0149] If the cutoff frequency of the high-pass filter is m [Hz], the second threshold is the time when the DC component (direct current) which is the offset is removed by high-pass filter processing and the remaining component becomes within 5 [mdps].

[0150] The first threshold value is the exposure time at which n*exposure time is within 3 pixels of blur for each lens. Here, "amount of blur within 3 pixels" means that the amount of blur remaining in the captured image is equivalent to within 3 pixels. For example, if one pixel is 3.7 [μm], the amount of blur within 3 pixels is 3 × 3.7 [μm] in terms of the amount of blur on the image plane. "For each lens" means that the focal length differs depending on the lens, so it should be found according to the lens.

[0151] The exposure time at which the n*exposure time results in a blur amount of 2 pixels or less for each lens is set as the third threshold value.

[0152] The offset standard value is calculated using the following method. A gyro sensor is actually installed in a digital camera, and the time period over which the offset changes is extracted as a standard. The worst value among the extracted standards is set as the standard value.

[0153] [Optimization of each threshold based on offset] As described above, the optimum value of each threshold value used when determining the signal to be output from the filter processing unit changes depending on the offset. Meanwhile, the offset is not a constant value but a value that varies depending on the environmental temperature, etc. Therefore, it is preferable to detect the offset during startup of the digital camera and optimize each threshold value based on the detected offset.

[0154] The offset is calculated from the output of the gyro sensor, for example, by the following method.

[0155] The offset is calculated based on the extracted first and second features, by performing a process of extracting a first feature related to the offset from the output signal of the gyro sensor, a process of integrating the output signal of the gyro sensor, and a process of extracting a second feature related to the offset from the integrated value of the output signal of the gyro sensor.

[0156] Here, for the first feature amount, a process of extracting a DC component amount from the output signal of the gyro sensor is performed, and the extracted value can be used as the first feature amount.

[0157] Regarding the second feature amount, a process of calculating the slope of the cumulative integral value of the output signal of the gyro sensor is performed, and the calculated value can be used as the second feature amount.

[0158] The process of calculating the offset based on the first feature amount and the second feature amount is, for example, calculated in the following procedure. First, the absolute value of the difference between the first feature amount and the second feature amount is calculated. The calculated value is compared with a threshold value. This threshold value is set as the fourth threshold value. If the calculated value is equal to or less than the fourth threshold value, the time during which the state of being equal to or less than the fourth threshold value continues is measured. If the state of being equal to or less than the fourth threshold value continues for a specified time or more, the arithmetic mean of the first feature amount and the second feature amount (1 / 2 the sum of the first feature amount and the second feature amount) is calculated as the offset. In other words, if the state in which the absolute value of the difference between the first feature amount and the second feature amount is equal to or less than the fourth threshold value continues for a certain period or more, it is assumed that the first feature amount and the second feature amount each indicate a correct offset, and the arithmetic mean is calculated as the offset.

[0159] It is preferable to further perform a process of calculating the amount of change in the second feature amount, and initialize the process related to the calculation of the offset when the calculated amount of change in the second feature amount exceeds a specific value, thereby preventing erroneous detection.

[0160] It is also preferable that the process for calculating the offset is performed while the digital camera is stationary. The process for determining whether the digital camera is stationary is performed based on the output signal of the gyro sensor.

[0161] The calculated offset is used to optimize each of the first to third thresholds, and is also used to remove the offset. The offset correction unit has a function of calculating the offset, and performs offset correction processing using the calculated offset. An example of the configuration in this case will be described below.

[0162] [Modification of the offset correction unit] Since the two offset correction units 60A, 60B have the same functions, only the function of the offset correction unit 60A will be described here.

[0163] FIG. 27 is a block diagram of functions possessed by the offset correction unit.

[0164] As shown in the figure, the offset correction section 60A has the functions of a first subtraction section 60A1 and a second subtraction section 60A2.

[0165] The first subtraction unit 60A1 subtracts a first offset from the output signal of the first detection unit 20A. The first offset is a standard offset and is a fixed value. The first offset is set, for example, at the time of shipment from the factory. In this case, for example, the offset is detected at the time of shipment from the factory, and the detected offset is set as the first offset.

[0166] The second subtraction unit 60A2 subtracts a second offset from the signal after the first offset has been subtracted. The second offset is the latest offset. The second offset is an example of an offset of the present invention. The second offset is calculated by the second offset calculation unit 100A. The second offset calculation unit 100A calculates the second offset based on the signal after the first offset has been subtracted. Details of the second offset calculation unit 100A will be described later.

[0167] The offset storage unit 90A is configured with the auxiliary storage unit 42. The offset storage unit 90A stores information on the first offset and the second offset used by the offset correction unit 60A. Information on the second offset calculated by the second offset calculation unit 100A is stored in the offset storage unit 90A by the second offset storage control unit 110A.

[0168] FIG. 28 is a block diagram of functions included in the second offset calculation unit.

[0169] As shown in the figure, the second offset calculation section 100A has the functions of a first feature amount extraction section 101A, an integration section 102A, a second feature amount extraction section 103A, and a second offset calculation section 104A.

[0170] The first feature extraction unit 101A performs a process of extracting a first feature related to the second offset from the output signal of the first detection unit 20A after the first offset has been removed. Specifically, the first feature extraction unit 101A performs a process of calculating a DC component amount from the output signal of the first detection unit 20A after the first offset has been removed, and outputs the calculated DC component amount as the first feature amount. The first feature extraction unit 101A calculates the DC component amount, which is the first feature amount, by performing low-pass filter processing (Low-pass filter, LPF) or band-pass filter processing (Band-pass filter, BPF) on the output signal of the first detection unit 20A after the first offset has been removed. This point will be described below.

[0171] Let f(t) be the random noise generated by the gyro sensor, g(t) be the angular velocity signal generated by the gyro sensor, and O be the offset (constant) generated by the gyro sensor. The output ω_OUT from the gyro sensor is expressed by the following equation (1).

[0172] ω_OUT=f(t)+O+g(t)…(1)

[0173] The offset O generated by the gyro sensor is the first characteristic quantity to be determined. Since the offset O is a DC component, the components f(t) and g(t) can be removed by applying a low-pass filter or a band-pass filter to the above formula (1). Therefore, the first characteristic quantity can be extracted by applying a low-pass filter process or a band-pass filter process to the output signal of the first detection unit 20A after the first offset is removed.

[0174] The integrating section 102A performs a process of integrating the output signal of the first detecting section 20A after the first offset has been removed, and an angle signal is obtained by this process.

[0175] The angle signal deg_out is calculated by integrating the above equation (1) using the following equation (2).

[0176] deg_out = ∫(ω_OUT)dt =∫(g(t))dt+∫(f(t))dt+Ot+C…(2) C: Integral constant

[0177] The second feature extraction unit 103A performs a process of extracting a second feature related to the offset from the output of the integrator 102A. That is, the second feature is extracted from a value (angle signal) obtained by integrating the output signal of the first detector 20A. Specifically, the second feature is calculated as the second feature by calculating the slope of the cumulative integral from the output of the integrator 102A. For example, the second feature extraction unit 103A obtains a linear approximation curve of the cumulative integral, calculates the slope, and extracts the second feature. That is, according to the above formula (2), if Ot>>∫(g(t))dt+∫(f(t))dt after a sufficient time has passed, it is possible to estimate the offset O from the slope of the linear approximation curve of deg_out.

[0178] FIG. 29 is a conceptual diagram of the extraction process of the second feature amount performed by the second feature amount extraction unit.

[0179] Let ∫(g(t))dt = G[t], ∫(f(t))dt = F[t], and the integral value be deg_out[t], then the integral value deg_out[t] is deg_out[t] ≒ G[t] + F[t] + Ot.

[0180] The initial value is the integral value when starting the calculation of the second feature amount (the slope of the cumulative integral value). The initial value is deg_out[0].

[0181] The timer count is incremented by 1 every control period.Then, it is multiplied by the control period (Ts) [s].

[0182] Time_out = time_count * (Ts)

[0183] If the second feature amount (slope of the cumulative integral value) is O2, O2 is calculated by the following formula.

[0184] O2=(deg_out[t]-deg_out[0]) / Time_out

[0185] The second offset calculation unit 104A calculates the second offset based on the first feature amount and the second feature amount extracted by the first feature amount extraction unit 101A and the second feature amount extraction unit 103A. Specifically, it is determined whether the first feature amount and the second feature amount satisfy a certain condition, and if the condition is satisfied, the arithmetic average of both is calculated as the second offset. That is, the arithmetic average of both is calculated as the second offset. The condition that the first feature amount and the second feature amount should satisfy is that the absolute value of the difference between the first feature amount and the second feature amount is equal to or less than a fourth threshold, and the state of being equal to or less than the fourth threshold continues for a specified time or more.

[0186] FIG. 30 is a block diagram of functions included in the second offset calculation unit.

[0187] As shown in the figure, second offset calculation section 104A has the functions of difference calculation section 104A1, determination section 104A2, and calculation section 104A3.

[0188] The difference calculation section 104A1 calculates the absolute value of the difference between the first feature amount and the second feature amount.

[0189] The determination unit 104A2 determines whether the absolute value of the difference between the first feature amount and the second feature amount calculated by the difference calculation unit 104A1 becomes equal to or less than the fourth threshold value and whether this state continues for a specified time or more. In this case, the determination unit 104A2 first determines whether the absolute value of the difference between the first feature amount and the second feature amount becomes equal to or less than the fourth threshold value. If it is determined that the absolute value is equal to or less than the fourth threshold value, the determination unit 104A2 determines whether the state of being equal to or less than the fourth threshold value continues for a specified time or more. Therefore, when the absolute value of the difference between the first feature amount and the second feature amount becomes equal to or less than the fourth threshold value, the determination unit 104A2 counts (measures) the duration. The fourth threshold value and the specified time are determined in advance and stored in a memory.

[0190] When the determining unit 104A2 determines that the above condition is satisfied, the calculating unit 104A3 calculates a second offset based on the first feature amount and the second feature amount. Specifically, the calculating unit 104A3 calculates an arithmetic average of the first feature amount and the second feature amount at the time when the determining unit 104A2 determines that the above condition is satisfied, and calculates the second offset.

[0191] FIG. 31 is a conceptual diagram of the calculation process of the second offset performed by the second offset calculation unit.

[0192] The first feature amount extracted by the first feature amount extraction unit 101A is O1, and the second feature amount extracted by the second feature amount extraction unit 103A is O2. In addition, the specified time is thresh_time, and the fourth threshold is threshold.

[0193] First, the absolute value of the difference between the first feature amount O1 and the second feature amount O2 (|O2-O1|) is calculated.

[0194] In counting, |O2-O1| and thres ho ld, and |O2-O1|≦thres ho If ld, add +1 to the count. |O2-O1|>thres ho If it is ld, set the count to 0.

[0195] If the count>thresh_time, the second offset is calculated based on the first feature amount O1 and the second feature amount O2 at that time point. That is, the second offset is calculated by executing the following calculation.

[0196] Second offset = (O1 + O2) / 2

[0197] When the second offset is calculated, the process related to the calculation of the second offset is initialized (reset). That is, in the first feature extraction unit 101A, the filter process is reset. Also, in the second feature extraction unit 103A, the start position of the calculation is initialized. Specifically, t is returned to 0, and the initial value deg_out[0] is updated.

[0198] Furthermore, when the second offset is calculated, information on the calculated second offset is stored in the offset storage unit 90A by the second offset storage control unit 110A. That is, the information on the second offset stored in the offset storage unit 90A is updated. This keeps the information on the second offset stored in the offset storage unit 90A up to date. The initial value of the second offset stored in the offset storage unit 90A is 0.

[0199] In this manner, in this embodiment, two feature amounts (first feature amount and second feature amount) related to the offset are extracted from the output signal of the first detection unit 20A, and if the two extracted feature amounts satisfy a certain condition, the second offset is obtained from the two feature amounts. That is, if the two feature amounts are almost the same, the arithmetic mean is calculated to obtain the second offset. The determination made by the determination unit 104A2 is to determine whether the first feature amount and the second feature amount are the same. That is, if the absolute value of the difference between the first feature amount and the second feature amount is equal to or less than the fourth threshold value and this state continues for a specified time or more, it is assumed that the first feature amount and the second feature amount are the same, and the second offset is calculated.

[0200] In this way, the second offset can be calculated accurately by calculating two feature amounts and then calculating the second offset. That is, it is difficult to calculate the second offset accurately using only one feature amount, but the second offset can be calculated accurately by using two feature amounts.

[0201] For example, consider a case where the second offset is calculated from only the first feature amount. In this case, since there is variation in the signal after filtering, it is difficult to accurately estimate the second offset.

[0202] Similarly, consider the case where the second offset is calculated only from the second feature amount. As shown in the above formula (2), the angle signal contains ∫(g(t))dt and ∫(f(t))dt as functions of time t. Therefore, if the second offset (slope of the cumulative integral value) is calculated only from the angle signal, an accurate value may not be calculated due to the effect of the function of time t [∫(g(t))dt+∫(f(t))dt].

[0203] However, if the two values ​​are almost the same, it is considered that they show an accurate offset value. Therefore, by calculating the second offset from the two feature amounts, the second offset can be calculated accurately.

[0204] The first feature is extracted by applying a filter to the output signal (angular velocity signal) of the first detection unit 20A, because applying a filter to the integral value (angle signal) of the output signal of the first detection unit 20A may change the value of the offset O, which is the feature to be obtained.

[0205] FIG. 32 is a flowchart showing a processing procedure for calculating the second offset.

[0206] First, a first characteristic amount O1 and a second characteristic amount O2 are extracted from the output signal of the first detection unit 20A (steps S211 and S212). Note that the output signal here is the output signal after the first offset has been removed.

[0207] The first feature amount O1 is extracted by performing a predetermined filter process on the output signals of the first detection unit 20A and the second detection unit 20B. That is, the first feature amount O1 is extracted by performing a low-pass filter process or a band-pass filter process on the output signals of the first detection unit 20A and the second detection unit 20B and calculating the amount of DC components.

[0208] The second characteristic amount O2 is extracted by integrating the output signals of the first detection section 20A and the second detection section 20B and determining the slope of the cumulative integral value.

[0209] Next, the absolute value (|O1-O2|) of the difference between the extracted first feature amount O1 and second feature amount O2 is calculated and compared with a fourth threshold value. That is, it is determined whether the absolute value (|O1-O2|) of the difference between the first feature amount O1 and the second feature amount O2 is equal to or smaller than the fourth threshold value (step S213).

[0210] When it is determined that the absolute value of the difference between the first feature amount O1 and the second feature amount O2 (|O1-O2|) is equal to or less than the fourth threshold, the duration is counted (step S214). Then, it is determined whether the counted duration is equal to or greater than a specified time (step S215).

[0211] When it is determined that the state of being equal to or less than the fourth threshold is equal to or longer than the specified time, the first characteristic amount O1 and the second characteristic amount O2 are regarded as being the same, and a second offset is calculated (step S216). That is, the arithmetic average of the first characteristic amount O1 and the second characteristic amount O2 at the time when it is determined that the state of being equal to or longer than the specified time is calculated, and this is used as the value of the second offset.

[0212] When the second offset is calculated, the calculated second offset value is stored in the offset storage units 90A and 90B (step S217). More specifically, the second offset information already stored in the offset storage units 90A and 90B is rewritten. This process updates the second offset information stored in the offset storage units 90A and 90B.

[0213] When the calculated second offset value is stored in the offset storage units 90A and 90B, the process related to the calculation of the second offset is initialized (step S218). That is, the filter process is reset for the calculation of the first feature amount, and the start position of the calculation is initialized for the calculation of the second feature amount.

[0214] The above series of processes are repeated at regular intervals during operation of the digital camera. If the second offset is not calculated during one calculation cycle (if the difference between the first feature amount and the second feature amount does not satisfy the condition), the calculation process is initialized and calculation is performed in the next cycle. In this case, the second offset maintains the value that has already been detected.

[0215] As described above, in the present embodiment, two feature amounts (a first feature amount and a second feature amount) related to the offset are extracted, and the two feature amounts are compared to determine the second offset, so that the second offset can be determined with high accuracy.

[0216] FIG. 33 is a diagram showing an example of an output signal of a gyro sensor in a stationary state.

[0217] The figure shows the output signal of the gyro sensor when the offset is 0.02 [dps]. Note that the unit [dps] stands for "degree per second."

[0218] Fig. 34 is a diagram showing the extraction result of the first feature amount. That is, it shows the result of extracting the DC component amount from the output signal of the gyro sensor shown in Fig. 33. In this example, the cutoff frequency is set to 0.1 [Hz], and low-pass filter processing is performed on the output signal to extract the first feature amount (DC component amount). As shown in the figure, it is understood that it is difficult to accurately characterize the offset using only the first feature amount.

[0219] Fig. 35 is a diagram showing the calculation result of the integral value. That is, it shows the result of integrating the output signal of the gyro sensor shown in Fig. 33. The angle is calculated by integrating the output signal of the gyro sensor. The unit of angle [deg] means "degree". The second feature amount is calculated by calculating the slope of the graph shown in the figure.

[0220] 36 is a diagram showing changes over time in the first feature amount and the second feature amount. In the figure, the symbol O1 indicates the change over time in the first feature amount. Also, the symbol O2 indicates the change over time in the second feature amount.

[0221] As shown in the figure, it can be confirmed that the offset can be accurately identified by comparing the first feature amount O1 and the second feature amount O2. In other words, it can be confirmed that the offset can be accurately identified by detecting a state in which the two match. The fact that the two match can be determined by detecting that the absolute value of the difference between the two continues to be equal to or less than the fourth threshold value for a specified period of time or more. For example, in FIG. 36, the fact that the two match can be detected by detecting the state of the area S surrounded by the dashed line.

[0222] As described above, the calculation process of the second offset can be configured to be performed when the digital camera is in a stationary state. In this case, the offset correction units 60A and 60B are provided with a function of a stationary determination unit that determines whether the digital camera is in a stationary state. The stationary determination unit determines the stationary state based on the output signals (output signals of the gyro sensor) of the first detection unit 20A and the second detection unit 20B. For example, the stationary state is determined by performing low-pass filter processing or band-pass filter processing on the output signals of the first detection unit 20A and the second detection unit 20B after the first offset has been subtracted. If the filtered signals are equal to or less than the fourth threshold for a certain period of time, the stationary state is determined.

[0223] In addition, since the offset of the gyro sensor varies depending on the environmental temperature, the offset can be managed for each temperature. In this case, when calculating the offset, temperature information is acquired, and the calculated offset is stored for each temperature. In addition, when correcting the offset, the correction process is performed using the offset according to the temperature.

[0224] Furthermore, in the above configuration, the offset is removed in two separate steps, but it is also possible to remove only the second offset.

[0225] Furthermore, when the second offset is calculated as described above and removed from the signal, a standard value of the offset is set, including the accuracy of calculating the second offset, and each of the first to third thresholds is set based on the set standard value of the offset.

[0226] The second offset can be calculated by the above method, or it can be estimated by using a Kalman filter, image processing, or the like.

[0227] [Other embodiments] (1) Apparatus to which the present invention is applied In the above embodiment, the present invention is applied to a digital camera equipped with a camera shake correction function, but the application of the present invention is not limited to this. The present invention can also be applied to various other imaging devices equipped with a camera shake correction function, such as video cameras, television cameras, and cine cameras.

[0228] The imaging device of the present invention includes not only a standalone imaging device, but also an imaging device that is integrated into a device having a function other than imaging. For example, the imaging device includes an imaging device integrated into a smartphone, a personal computer, etc.

[0229] In addition, for example, in a lens-interchangeable digital camera, when an interchangeable lens is provided with a camera shake correction function, the present invention can be applied to the interchangeable lens. In this case, for example, a system control unit (such as a microcomputer) provided in the interchangeable lens realizes the function of a signal processing device. In this case, the interchangeable lens is an example of a lens device.

[0230] In addition, the present invention is not limited to the so-called lens shift type of image stabilization, and can also be applied to the case where an image sensor shift type is adopted. The image sensor shift type is an image stabilization function that corrects image stabilization by moving the image sensor (imaging element) in response to camera shake. Furthermore, the present invention can also be applied to the case where a so-called electronic image stabilization is adopted.

[0231] (2) Shake detection sensor In the above embodiment, the case where the output signal of a gyro sensor is processed has been described as an example, but the application of the present invention is not limited to this. The present invention can be applied to the case where the output signal of a sensor containing an offset, in particular, the output signal of a sensor in which the offset cannot be removed by a fixed value, is processed.

[0232] (3) Processor Processors that realize each function of a signal processing device include a CPU (Central Processing Unit), which is a general-purpose processor, a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically to execute a specific process such as an ASIC (Application Specific Integrated Circuit).

[0233] Each function of the signal processing device may be realized by one of these various processors, or by two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA).

[0234] Also, each function of the signal processing device may be realized by one processor. As an example of realizing each function of the signal processing device by one processor, first, there is a form in which one processor is configured by combining one or more CPUs and software, and this processor realizes each function of the signal processing device. Second, there is a form in which a processor that realizes the entire function of the signal processing device by one IC (Integrated Circuit) chip is used, as typified by System On Chip (SoC).

[0235] In this way, each function of the signal processing device is realized as a hardware structure using one or more of the above-mentioned various processors.

[0236] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0237] 1. Digital camera 10. Imaging optical system 11 Zoom Lens 12 Focus Lens 13 Corrective lenses 14 Aperture 15 Zoom lens drive unit 16 Focus lens drive unit 17 Correction lens drive unit 17x 1st drive unit 17y 2nd Drive Unit 18 Aperture drive unit 20 Angular velocity detector 20A First detection section 20B Second detection section 21A Gyro sensor 21B Gyro sensor 30 Image sensor 40 Main memory 41 Digital signal processing section 42 Auxiliary storage 43 Display section 44 Control section 50 System control section 60A Offset correction section 60A1 First subtraction section 60A2 Second subtraction section 60B Offset correction section 70A 1st correction amount calculation section 70B Second correction amount calculation section 71A Filter processing section 71A1 High-pass filter processing section 71A2 First elapsed time measuring unit 71A3 Imaging information acquisition unit 71A4 Second elapsed time measuring unit 71A5 Output signal determination unit 71A6 Output signal switching unit 71B Filter processing section 72A Sensitivity correction unit 72B Sensitivity correction unit 73A Integral section 73B Integral section 74A Correction amount calculation section 74B correction amount calculation unit 80A First drive control section 80B Second drive control section 90A Offset memory section 90B Offset memory section 100A Second offset calculation section 101A First feature extraction unit 102A Integral section 103A Second feature extraction unit 104A Second offset calculation unit 104A1 Difference calculation unit 104A2 Judgment section 104A3 Calculation part 110A Second offset storage control unit Pit Pitch direction Yaw Yaw direction S1~S2 How to determine the signal output from the filter processing section during the actual imaging exposure S11 to S14: Processing procedure for forcibly switching from a high-pass signal to a high-passless signal during exposure in actual imaging S101 to S107: Processing procedure of the filter processing unit S111 to S116: Processing procedure for setting the HPF use flag during exposure of actual imaging S211 to S218: Procedure for calculating the second offset

Claims

1. A signal processing device that processes a signal output from a shake detection sensor during exposure of an image sensor, A processor is provided. The processor, A process of generating a second signal by performing a high-pass filter process on a first signal output from the shake detection sensor; a process of calculating an amount of blur based on the first signal during a period from the start of exposure to a third threshold value when an exposure time of the image sensor exceeds a first threshold value and an elapsed calculation time of the high-pass filter process is equal to or less than a second threshold value, and calculating an amount of blur based on the second signal during a period from the start of exposure to a third threshold value when an exposure time of the image sensor exceeds a first threshold value and an elapsed calculation time of the high-pass filter process is equal to or less than a second threshold value; Do the following: the first threshold, the second threshold, and the third threshold are set based on a cutoff frequency in the high-pass filter processing and an offset of the shake detection sensor; Signal processing device.

2. When the amount of blur is calculated based on the second signal, if the calculation of the high-pass filter process is reset, the processor switches from the second signal to the first signal and calculates the amount of blur. The signal processing device according to claim 1 .

3. The processor further performs a process of calculating the offset based on a signal output from the shake detection sensor.

3. A signal processing device according to claim 1 or 2.

4. In calculating the offset, The processor, A process of calculating a first characteristic amount by obtaining a DC component of a signal output from the shake detection sensor; A process of calculating a second characteristic amount by obtaining a gradient of an accumulated integral value of a signal output from the shake detection sensor; calculating the offset based on the first feature amount and the second feature amount; To carry out The signal processing device according to claim 3 .

5. A shake detection sensor provided in the device body; a signal processing device according to any one of claims 1 to 4, which processes a signal output from the shake detection sensor; An imaging device comprising:

6. A shake detection sensor provided in the device body; a signal processing device according to any one of claims 1 to 4, which processes a signal output from the shake detection sensor; A lens device comprising:

7. A signal processing method for processing a signal output from a shake detection sensor during exposure of an image sensor, comprising: A process of generating a second signal by performing a high-pass filter process on a first signal output from the shake detection sensor; a process of calculating an amount of blur based on the first signal during a period from the start of exposure to a third threshold value when an exposure time of the image sensor exceeds a first threshold value and an elapsed calculation time of the high-pass filter process is equal to or less than a second threshold value, and calculating an amount of blur based on the second signal during a period from the start of exposure to a third threshold value when an exposure time of the image sensor exceeds a first threshold value and an elapsed calculation time of the high-pass filter process is equal to or less than a second threshold value; Including, the first threshold, the second threshold, and the third threshold are set based on a cutoff frequency in the high-pass filter processing and an offset of the shake detection sensor; Signal processing methods.

8. A process of acquiring a signal output from a shake detection sensor during exposure of an image sensor; A process of generating a second signal by performing a high-pass filter process on a first signal output from the shake detection sensor; a process of calculating an amount of blur based on the first signal during a period from the start of exposure to a third threshold value when an exposure time of the image sensor exceeds a first threshold value and an elapsed calculation time of the high-pass filter process is equal to or less than a second threshold value, and calculating an amount of blur based on the second signal during a period from the start of exposure to a third threshold value when an exposure time of the image sensor exceeds a first threshold value and an elapsed calculation time of the high-pass filter process is equal to or less than a second threshold value; A signal processing program for causing a computer to execute the following: the first threshold, the second threshold, and the third threshold are set based on a cutoff frequency in the high-pass filter processing and an offset of the shake detection sensor; Signal processing program.

9. A non-transitory computer-readable recording medium having the program according to claim 8 recorded thereon.

Images