Control device, imaging device, control method, and control program

Abstract

To provide a control device, imaging apparatus, control method and control program.SOLUTION: A control unit 18 derives a first shake correction amount according to the shake generated from the vibration and a second shake correction amount different from the first shake correction amount on the basis of the first output data output from a vibration detector 17, derives a first image shake amount about the shake of the image on the basis of the image data obtained in imaging by an imaging element 12, and derives an offset component included in the first output data on the basis of the first shake correction amount, second shake correction amount and first image shake amount.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device, an imaging device, a control method, and a control program. [Background technology]

[0002] Patent Document 1 describes an image stabilization device comprising: storage means for storing a first offset related to a shake detection means; calculation means for calculating a second offset related to the shake detection means based on a signal from the shake detection means and the first offset; and correction means for correcting the signal from the shake detection means using at least one of the first offset or the second offset in accordance with the elapsed time since the image stabilization device was turned on, and for performing image stabilization based on the corrected signal.

[0003] Patent Document 2 describes an imaging device that includes an imaging element that captures an image of a subject through an optical system, an image signal processing unit that has the function of combining multiple images captured while moving the imaging device into a single image using a given initial value, an attitude sensor that obtains attitude information about the imaging device, and a control unit that processes information from the attitude sensor and performs cooperative control of the processing result and the processing result of the image signal processing unit, wherein the image signal processing unit determines the relative positional relationship between the images by image recognition processing, and the control unit determines the positional relationship of the images based on the detection information of the attitude sensor, specifies the absolute positional relationship of each image based on the determined positional relationship and the relative positional relationship determined by the image signal processing unit, and supplies the absolute positional relationship to the image signal processing unit as the initial value. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-197772 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-147635 Summary of the Invention [Means for solving the problem]

[0005] A control device according to one embodiment of the technology of the present disclosure includes: Image sensor A control device for an imaging device having a processor a first vibration detector that outputs information on angular velocities around a plurality of axes; and a second vibration detector that is capable of detecting acceleration and is capable of detecting movement of the imaging device in a direction along the light receiving surface of the imaging element. and the processor receives an output from the first vibration detector. The above angular velocity information is First output data is derived from the data obtained by directly integrating , a first blur correction amount corresponding to blur caused by vibration a second shake correction amount is derived, which is information derived from data obtained by performing high-pass filtering on data obtained by integrating the first output data for a predetermined time period, or information derived from data obtained by integrating the first output data for a predetermined time period, which has been subjected to high-pass filtering; and a third shake correction amount is derived by second-order integrating the second output data, which is information about the acceleration for a predetermined time period output from the second vibration detector; The image obtained by the above image sensor Multiple Image data The motion vector between the image data is the amount of blur between the deriving a first image blur amount relating to image blur, and and The second blur correction amount From the sum of , the first image blur amount and The value obtained by subtracting the added value of the third image stabilization amount The offset component included in the first output data is derived based on the above.

[0006] An imaging device according to one embodiment of the technique of the present disclosure includes the control device described above.

[0007] A control method according to one embodiment of the technology of the present disclosure includes: A control method for an imaging device, the imaging device having an imaging element, a first vibration detector that outputs information on angular velocities around a plurality of axes, and a second vibration detector that is capable of detecting acceleration and is capable of detecting movement of the imaging device in a direction along a light receiving surface of the imaging element, The first vibration detector outputs The above angular velocity information is First output data is derived from the data obtained by directly integrating , a first blur correction amount corresponding to blur caused by vibration a second shake correction amount is derived, which is information derived from data obtained by performing high-pass filtering on data obtained by integrating the first output data for a predetermined time period, or information derived from data obtained by integrating the first output data for a predetermined time period, which has been subjected to high-pass filtering; and a third shake correction amount is derived by second-order integrating the second output data, which is information about the acceleration for a predetermined time period output from the second vibration detector; The image obtained by the above image sensor Multiple Image data The motion vector between the image data is the amount of blur between the deriving a first image blur amount relating to image blur, and and The second blur correction amount From the sum of Amount of blur in the first image above and the value obtained by subtracting the added value of the third shake correction amount from The offset component included in the first output data is derived based on the above.

[0008] A control program according to an embodiment of the technology of the present disclosure includes: A control program for an imaging device, the imaging device having an imaging element, a first vibration detector that outputs information on angular velocities around a plurality of axes, and a second vibration detector that is capable of detecting acceleration and is capable of detecting movement of the imaging device in a direction along a light receiving surface of the imaging element, The first vibration detector outputs The above angular velocity information is First output data is derived from the data obtained by directly integrating , a first blur correction amount corresponding to blur caused by vibration a second shake correction amount is derived, which is information derived from data obtained by performing high-pass filtering on data obtained by integrating the first output data for a predetermined time period, or information derived from data obtained by integrating the first output data for a predetermined time period, which has been subjected to high-pass filtering; and a third shake correction amount is derived by second-order integrating the second output data, which is information about the acceleration for a predetermined time period output from the second vibration detector;The image obtained by the above image sensor Multiple Image data The motion vector between the image data is the amount of blur between the deriving a first image blur amount relating to image blur, and and The second blur correction amount From the sum of Amount of blur in the first image above and the value obtained by subtracting the added value of the third shake correction amount from deriving an offset component included in the first output data based on the causing a processor to perform the steps; It is something. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing a schematic configuration of a digital camera 1 that is an embodiment of an imaging device of the present invention. [Figure 2] 1 is a front view showing a schematic external view of a camera body 10 of a digital camera 1. FIG. [Figure 3] 10 is a timing chart for explaining the operation of the control unit 18 during the process of deriving the offset component. [Figure 4] 1 shows the appearance of a smartphone 200. [Figure 5] FIG. 5 is a block diagram showing the configuration of the smartphone 200 shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] FIG. 1 is a diagram showing a schematic configuration of a digital camera 1, which is one embodiment of an imaging device of the present invention. FIG. 2 is a front view showing a schematic appearance of a camera body 10 of the digital camera 1. The digital camera 1 comprises the camera body 10 and a lens device 20. The lens device 20 is configured to be detachable from the camera body 10, in other words, to be replaceable. The lens device 20 may be integrated with the camera body 10.

[0011] The lens device 20 has an imaging optical system 30 and a lens control unit 40. The imaging optical system 30 includes an imaging lens 31 and an aperture mechanism (not shown). The imaging lens 31 includes, for example, a lens for adjusting the focus of the imaging optical system 30. The lens control unit 40 is mainly composed of a processor, and drives and controls the imaging optical system 30 under the control of the control unit 18 (described later).

[0012] The camera body 10 comprises an image sensor 12, an image sensor shift mechanism 13, an image sensor drive unit 14, a display unit 15 which is a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display, memory 16 which includes RAM (Random Access Memory) as a volatile memory for temporarily storing information and ROM (Read Only Memory) as a non-volatile memory for storing in advance programs and various information required for their operation, a vibration detector 17, a control unit 18 which constitutes a control device for the image pickup device, and a storage medium 19 such as a memory card which is composed of non-volatile memory.

[0013] The imaging element 12 captures an image of a subject through the imaging optical system 30. The imaging element 12 is configured with a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. As shown in FIG. 2, the imaging element 12 is rectangular. Hereinafter, the direction along the short side of the imaging element 12 will be referred to as direction Y, and the direction along the long side of the imaging element 12 will be referred to as direction X. Furthermore, the direction perpendicular to the light receiving surface of the imaging element 12 (the direction in which the optical axis K of the imaging optical system 30 extends) will be referred to as direction Z. Furthermore, the axis extending in direction X will be referred to as the yaw axis, the axis extending in direction Y will be referred to as the pitch axis, and the axis extending in direction Z will be referred to as the roll axis.

[0014] The image sensor shift mechanism 13 is a mechanism for moving the image sensor 12 within a plane perpendicular to the optical axis K of the imaging optical system 30 (hereinafter referred to as the XY plane) to prevent blurring of the image captured by the image sensor 12. The image sensor shift mechanism 13 is configured to be able to move the image sensor 12 in three directions: direction X, direction Y, and a rotational direction around a rotation axis that passes through the center of the light receiving surface of the image sensor 12 and extends along the optical axis K.

[0015] Vibration detector 17 is a sensor for detecting rotational vibrations (vibrations caused by digital camera 1 rotating around the yaw axis, pitch axis, and roll axis), and constitutes a first vibration detector. Vibration detector 17 is composed of a three-axis angular velocity sensor that outputs information on angular velocity around the yaw axis, information on angular velocity around the pitch axis, and information on angular velocity around the roll axis. Hereinafter, the information on angular velocity around the yaw axis, information on angular velocity around the pitch axis, and information on angular velocity around the roll axis output from vibration detector 17 will be referred to as yaw rotation data, pitch rotation data, and roll rotation data, respectively. The yaw rotation data, pitch rotation data, and roll rotation data each constitute first output data.

[0016] The control unit 18 controls the entire digital camera 1, and its hardware structure is made up of various processors that execute programs including a control program to perform processing.

[0017] The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes; a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture; and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration specifically designed to perform specific processes. More specifically, the structure of these various processors is an electrical circuit combining circuit elements such as semiconductor devices. The control unit 18 may be configured with one of the various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0018] The control unit 18 controls the image sensor driver 14 to cause the image sensor 12 to capture an image of the subject, and causes the image sensor 12 to output a captured image signal corresponding to the subject image formed on the light receiving surface of the image sensor 12. The control unit 18 processes the captured image signal output from the image sensor 12 to generate image data in a format such as JPEG (Joint Photographic Experts Group) that can be played back by the digital camera 1 itself or by other devices. An image based on this image data is also referred to as a captured image.

[0019] When the image sensor 12 captures an image of a subject, the control unit 18 performs shake correction to correct blur in the captured image caused by at least rotational vibration of the digital camera 1. The amount of blur in the captured image caused by rotational vibration of the digital camera 1 includes a yaw component caused by rotation around the yaw axis (yaw rotation), a pitch component caused by rotation around the pitch axis (pitch rotation), and a roll component caused by rotation around the roll axis (roll rotation).

[0020] The control unit 18 derives a yaw component based on the yaw rotation data of the vibration detector 17, a pitch component based on the pitch rotation data of the vibration detector 17, and a roll component based on the roll rotation data of the vibration detector 17. Based on the derived yaw component, the control unit 18 derives a second shake compensation amount By, which is the amount of movement of the image sensor 12 in direction Y required to cancel out the shake caused by the yaw rotation. The control unit 18 performs shake compensation by moving the image sensor 12 in direction Y by the second shake compensation amount By. Based on the derived pitch component, the control unit 18 derives a second shake compensation amount Bx, which is the amount of movement of the image sensor 12 in direction X required to cancel out the shake caused by the pitch rotation. Based on the derived roll component, the control unit 18 performs shake compensation by moving the image sensor 12 in direction X by the second shake compensation amount Bx. Based on the derived roll component, the control unit 18 derives a second shake compensation amount Bz, which is the amount of movement of the image sensor 12 in the rotation direction required to cancel out the shake caused by the roll rotation. The control unit 18 performs shake correction by rotating the image sensor 12 about the light receiving surface by the second shake correction amount Bz.

[0021] Here, blur correction is performed by moving the image sensor 12, but blur correction may also be performed by moving the vibration-proof lens included in the imaging optical system 30, or by moving both the image sensor 12 and the vibration-proof lens.

[0022] (Method of deriving the second shake correction amount) The yaw rotation data, pitch rotation data, and roll rotation data of the vibration detector 17 each contain an offset component. The control unit 18 integrates the yaw rotation data for a predetermined period of time and performs processing to remove the offset component from the integrated data, thereby deriving the rotation angle θx of the digital camera 1 about the yaw axis. The control unit 18 converts the rotation angle θx into the amount of movement of the digital camera 1 in the Y direction on the XY plane, and derives the amount of movement of the image sensor 12 in the Y direction necessary to offset this amount of movement as a second shake correction amount By corresponding to the yaw component.

[0023] Control unit 18 integrates the pitch rotation data for a predetermined time period and performs processing to remove offset components from the integrated data, thereby deriving rotation angle θy about the pitch axis of digital camera 1. Control unit 18 converts rotation angle θy into a movement amount in direction X on the XY plane, and derives the movement amount in direction X of image sensor 12 required to offset this movement amount as a second shake correction amount Bx corresponding to the pitch component.

[0024] Control unit 18 integrates the roll rotation data for a predetermined time period and performs processing to remove offset components from the integrated data, thereby deriving the rotation angle θz around the roll axis of digital camera 1. Control unit 18 converts the rotation angle θz into a movement amount in the rotation direction on the XY plane, and derives the rotation amount of image sensor 12 required to offset this movement amount as second shake compensation amount Bz corresponding to the roll component. Control unit 18 performs shake compensation by moving image sensor 12 based on the three second shake compensation amounts described above.

[0025] The process of removing the offset components is, for example, a process of performing high-pass filtering on the first output data for a predetermined time period. Note that the control unit 18 may perform a process of removing the offset components from the first output data for a predetermined time period before integration, and then integrate the processed data to derive the rotation angles θx, θy, and θz of the digital camera 1.

[0026] By performing a process to remove the offset component, it is possible to suppress the effect of the offset component included in the output of the vibration detector 17 on the accuracy of image stabilization. However, the offset component may change over time. In this embodiment, the control unit 18 performs an offset derivation process to derive the offset component of the vibration detector 17, thereby making it possible to deal with such changes in the offset component. The offset derivation process will be described in detail below.

[0027] (Offset derivation process) The control unit 18 derives the image blur amount relating to the blur of the captured image and the first blur correction amount corresponding to the blur of the captured image caused by the vibration of the digital camera 1 in order to derive the offset component.

[0028] (1st image stabilization amount) The first shake correction amount is information derived by integrating the first output data for a predetermined time period without performing processing to remove the offset component, converting the data obtained into a movement amount on the XY plane, and determining the movement amount of the image sensor 12 required to offset that movement amount.

[0029] By integrating the yaw rotation data for a predetermined period of time, control unit 18 derives the rotation angle θx around the yaw axis of digital camera 1. Control unit 18 converts the rotation angle θx into a movement amount in direction Y on the XY plane, and derives the movement amount in direction Y of image sensor 12 required to offset this movement amount as a first shake correction amount Ay corresponding to the yaw component.

[0030] By integrating the pitch rotation data for a predetermined time period, control unit 18 derives rotation angle θy about the pitch axis of digital camera 1. Control unit 18 converts rotation angle θy into a movement amount in direction X on the XY plane, and derives the movement amount in direction X of image sensor 12 required to offset this movement amount as a first shake correction amount Ax corresponding to the pitch component.

[0031] By integrating the roll rotation data for a predetermined period of time, control unit 18 derives the rotation angle θz around the roll axis of digital camera 1. Control unit 18 converts the rotation angle θz into the amount of movement in the rotation direction on the XY plane, and derives the amount of rotation of image sensor 12 required to offset this amount of movement as a first shake correction amount Az corresponding to the roll component.

[0032] The first shake compensation amount is derived without performing processing to remove the offset component from the first output data, and is therefore greater than the second shake compensation amount described above. Because the first shake compensation amount thus contains information about the amount of movement corresponding to the offset component, it is not desirable to use it for shake compensation. In digital camera 1, the first shake compensation amount is not used for shake compensation, but is used to derive the offset component.

[0033] (1st image blur amount) The first image blur amount is the motion vector (amount of blur between image data) between multiple image data obtained by imaging with the image sensor 12 during shake correction. Assume that imaging is performed at time t1 and immediately thereafter at time t2 while shake correction is being performed. The motion vectors (motion vector in direction X, motion vector in direction Y, and motion vector in the rotational direction) of the image data obtained by imaging at time t2 relative to the image data obtained by imaging at time t1 correspond to the amount of blur in the captured image that could not be fully corrected by shake correction.

[0034] The control unit 18 derives the first image blur amount by converting the motion vector between the multiple image data into the amount of movement of the image sensor 12 on the XY plane. Specifically, the control unit 18 derives the first image blur amount Cx in the X direction, the first image blur amount Cy in the Y direction, and the first image blur amount Cz in the rotational direction. The first image blur amount Cx, the first image blur amount Cy, and the first image blur amount Cz will each be zero if the process for removing the offset component described above has been performed appropriately. However, the process for removing the offset component is determined based on the offset component actually measured at the time of shipping the digital camera 1. Therefore, if the offset component has fluctuated from its state at the time of shipping, the first image blur amount Cx, the first image blur amount Cy, and the first image blur amount Cz may not be zero.

[0035] Hereinafter, the first image blur amount in direction X, which is obtained when shake correction is performed based on the first shake correction amount and is derived from image data captured during the shake correction, will be referred to as the second image blur amount Dx, the first image blur amount in direction Y will be referred to as the second image blur amount Dy, and the first image blur amount in the rotational direction will be referred to as the second image blur amount Dz. These second image blur amounts Dx, Dy, and Dz correspond to the integral data of the offset component of the pitch rotation data of vibration detector 17, the integral data of the offset component of the yaw rotation data of vibration detector 17, and the integral data of the offset component of the roll rotation data of vibration detector 17, respectively.

[0036] Furthermore, the amount of movement in direction X converted from the rotation angle around the yaw axis that actually occurred in digital camera 1 will be referred to as shake amount Tx, the amount of movement in direction Y converted from the rotation angle around the pitch axis that actually occurred in digital camera 1 will be referred to as shake amount Ty, and the amount of movement in the rotational direction converted from the rotation angle around the roll axis that actually occurred in digital camera 1 will be referred to as shake amount Tz. When defined as above, the relationships of the following six equations are obtained.

[0037] Tx-Bx=Cx (A1) Ty-By=Cy (A2) Tz-Bz=Cz (A3) Tx-Ax=Dx (B1) Ty-Ay=Dy (B2) Tz-Az=Dz (B3)

[0038] Of these six equations, the first blur correction amounts Ax, Ay, Az, the second blur correction amounts Bx, By, Bz, and the first image blur amounts Cx, Cy, Cz can be calculated during image capture. Solving equations (A1) and (B1) for the second image blur amount Dx corresponding to the offset component yields the following equation (C1). Similarly, solving equations (A2) and (B2) for the second image blur amount Dy corresponding to the offset component yields the following equation (C2). Solving equations (A3) and (B3) for the second image blur amount Dz corresponding to the offset component yields the following equation (C3).

[0039] Dx = Bx + Cx - Ax (C1) Dy = By + Cy - Ay (C2) Dz = Bz + Cz - Az (C3)

[0040] Therefore, by performing the calculations of equations (C1), (C2), and (C3), control unit 18 can derive integral data of the offset component of the yaw rotation data, integral data of the offset component of the pitch rotation data, and integral data of the offset component of the roll rotation data. By differentiating this integral data for a predetermined time, control unit 18 derives the offset component (angular velocity) of the yaw rotation data, offset component (angular velocity) of the pitch rotation data, and offset component (angular velocity) of the roll rotation data. Using the offset components derived in this manner, for example, it is possible to appropriately change parameters used in the process of removing the offset components or correct the first shake correction amount, thereby improving the accuracy of shake correction based on the first shake correction amount.

[0041] FIG. 3 is a timing chart illustrating the operation of the control unit 18 during the process of deriving the offset component. FIG. 3 shows an example of the operation when the offset component of the yaw rotation data is derived. When yaw rotation data is output from the vibration detector 17 at time T1, a first shake correction amount Ay and a second shake correction amount By are derived based on this yaw rotation data, and shake correction is performed based on the second shake correction amount By. After shake correction starts, imaging by the image sensor 12 begins at time T2. When the control unit 18 acquires image data from this imaging, a first image blur amount Cy is derived. Then, a second image blur amount Dy is derived based on the first shake correction amount Ay, the second shake correction amount By, and the first image blur amount Cy derived up to this point. By repeating the above process, multiple second image blur amounts Dy for a predetermined period of time are derived, and the multiple second image blur amounts Dy are time-differentiated to derive the offset component of the yaw rotation data.

[0042] Control unit 18 executes the process of deriving such offset components at a predetermined timing, such as when the imaging mode of digital camera 1 starts. This makes it possible to maintain the accuracy of shake correction regardless of fluctuations in the offset components due to changes over time, etc. Digital camera 1 can derive the offset components based on the image data obtained while performing shake correction based on the first output data from which the offset components have been removed and the first output data from vibration detector 17, so that the offset components can be derived without degrading the quality of the live view image.

[0043] Note that, if the multiple image data obtained during shake correction include image data that does not satisfy a predetermined condition, the control unit 18 preferably derives the first image blur amount based on image data excluding that image data. The predetermined condition is, for example, a decrease in the accuracy of deriving a motion vector, such as when the image data is significantly blurred. In this way, by not using image data for which the accuracy of deriving a motion vector may be decreased in deriving the first image blur amount, the offset component can be derived with high accuracy.

[0044] Furthermore, when deriving the first image blur amount, it is preferable to set the shutter speed of the image sensor 12 when capturing images to obtain each of the multiple image data used to obtain the first image blur amount faster than the shutter speed of the image sensor 12 when capturing images without deriving the first image blur amount. This makes it possible to reduce the blur in each image data, and to derive the first image blur amount with high accuracy.

[0045] Furthermore, if high-frequency vibrations equal to or greater than the imaging frame rate occur in digital camera 1 during the imaging period for acquiring the first image blur amount, it is difficult to accurately obtain a motion vector. Therefore, it is preferable to apply low-pass filtering, with a cutoff frequency corresponding to the imaging frame rate (30 Hz if the imaging frame rate is 30 fps), to the first output data of vibration detector 17 and the first image blur amount. In this way, the effects of high-frequency vibrations can be eliminated, and the offset component can be derived with high accuracy.

[0046] Furthermore, it is preferable not to execute the offset derivation process in situations where the vibration of the digital camera 1 is large, such as when the user of the digital camera 1 is walking or when the user is performing pan / tilt. For example, it is preferable that the control unit 18 determines whether or not a specific point is being continuously captured or whether or not there is little fluctuation in the angle of view, and executes the offset derivation process only when it determines that such a situation exists. In this way, the offset component can be derived with high accuracy.

[0047] The information required to calculate Equations (C1), (C2), and (C3) (the first shake correction amount, the second shake correction amount, and the first image blur amount) is derived based on output data from different devices, namely, the image sensor 12 and the vibration detector 17. Therefore, it is necessary to synchronize the outputs of the image sensor 12 and the vibration detector 17. The control unit 18 determines a synchronized combination of image data and first output data based on information (referred to as synchronization information) indicating the difference between the start timing of image capture by the image sensor 12 (time T2 in FIG. 3 ) and the output timing of the first output data used for shake correction performed immediately before or during the image capture (time T1 in FIG. 3 ). Then, the control unit 18 derives the information required to calculate Equations (C1), (C2), and (C3) (the first shake correction amount, the second shake correction amount, and the first image blur amount) from the image data and first output data of the determined combination. This synchronization information is experimentally determined in advance and stored in memory 16.

[0048] The control unit 18 may generate the synchronization information rather than obtain it from the memory 16. For example, the control unit 18 sets the synchronization information to a predetermined initial value (e.g., n seconds), starts continuous imaging by the image sensor 12, derives the first shake correction amount Ay, the second shake correction amount By, and the first image blur amount Cy for a predetermined time, and derives the second image blur amount Dy based on the first shake correction amount Ay, the second shake correction amount By, and the first image blur amount Cy that are synchronized based on the synchronization information of this initial value.

[0049] The control unit 18 determines the similarity between a function obtained by subtracting a function f(Dy) representing the time change of the derived second image blur amount Dy from a function f(By-Ay) representing the time change of the value obtained by subtracting a first shake correction amount Ay synchronized with the second shake correction amount By, and the function f(By) representing the time change of the second shake correction amount By. The control unit 18 repeatedly performs the process of obtaining this similarity while gradually shifting the synchronization information from the initial value, and stores the value of the synchronization information when this similarity value is minimum in memory as the final value of the synchronization information. Thereafter, the control unit 18 determines the combination of synchronized image data and first output data based on this final value.

[0050] It is preferable that the process of generating such synchronization information be performed at a specified timing, such as when the digital camera 1 is started up or when the imaging mode is changed. Furthermore, it is preferable that the control unit 18 not share the same synchronization information for the yaw rotation data, pitch rotation data, and roll rotation data, but rather generate separate synchronization information for synchronizing the output timing of the yaw rotation data with the imaging timing, synchronization information for synchronizing the output timing of the pitch rotation data with the imaging timing, and synchronization information for synchronizing the output timing of the roll rotation data with the imaging timing. This allows for more accurate generation of synchronization information. The above synchronization information constitutes the first synchronization data.

[0051] (Modification of offset derivation process) It is preferable that digital camera 1 further includes an acceleration sensor as a second vibration detector that detects translational vibrations (vibrations caused by digital camera 1 moving in at least one of the X and Y directions while the light receiving surface of image sensor 12 remains parallel to the XY plane). The acceleration sensor is capable of detecting movement of digital camera 1 in a direction along the light receiving surface of image sensor 12, and detects movement in both the X and Y directions. Information about movement in the X direction output from the acceleration sensor is referred to as x output data, and information about movement in the Y direction output from the acceleration sensor is referred to as y output data. The x output data and y output data each constitute second output data.

[0052] Hereinafter, the amount of movement of the digital camera 1 in the X direction derived based on the x output data will be referred to as the first movement amount Mx, and the amount of movement of the image sensor 12 required to correct blur in the captured image due to the first movement amount Mx will be referred to as the third shake correction amount Gx. Furthermore, the amount of movement of the digital camera 1 in the Y direction derived based on the y output data will be referred to as the first movement amount My, and the amount of movement of the image sensor 12 required to correct blur in the captured image due to the first movement amount My will be referred to as the third shake correction amount Gy. If the digital camera 1 further includes an acceleration sensor, it is preferable to modify the offset derivation process as follows: The third shake correction amount Gx is obtained by double-integrating the x output data for a predetermined time period. The third shake correction amount Gy is obtained by double-integrating the y output data for a predetermined time period.

[0053] The above-mentioned amounts of shake Tx, Ty, and Tz that actually occur in digital camera 1 are described on the assumption that only rotational vibration occurs in digital camera 1. Therefore, if translational vibration occurs in digital camera 1, the first shake correction amount must be corrected taking into account the blur in the captured image caused by this rotational vibration. Specifically, formulas (B1) and (B2) must be replaced with the following formulas (B1a) and (B2a):

[0054] Tx-(Ax+Gx)=Dx (B1a) Ty-(Ay+Gy)=Dy (B2a)

[0055] When equations (B1a) and (A1) are solved for the second image blur amount Dx corresponding to the offset component, the following equation (C1a) is obtained. Similarly, when equations (B2a) and (A2) are solved for the second image blur amount Dy corresponding to the offset component, the following equation (C2a) is obtained.

[0056] Dx = Bx + Cx - (Ax + Gx) (C1a) Dy = By + Cy - (Ay + Gy) (C2a)

[0057] Based on the second output data output from the acceleration sensor, the control unit 18 derives the first movement amounts Mx and My of the digital camera 1, derives the third shake correction amounts Gx and Gy required to correct for shake in the captured image due to the first movement amounts Mx and My, and performs the calculations of equations (C1a) and (C2a) to derive the offset components of the yaw rotation data and the pitch rotation data. Note that the derivation of the offset component of the roll rotation data is the same as when an acceleration sensor is not used.

[0058] In this way, by using the output data of the acceleration sensor, it is possible to correct the first shake correction amount based on the amount of blur in the captured image caused by translational vibration, and it is possible to derive the offset component more accurately.

[0059] Next, the configuration of a smartphone, which is another embodiment of the imaging device of the present invention, will be described.

[0060] Fig. 4 shows the external appearance of smartphone 200. Smartphone 200 shown in Fig. 4 has a flat housing 201, and is provided on one surface of housing 201 with display panel 202 as a display unit and display input unit 204 which is an integrated unit of operation panel 203 as an input unit.

[0061] Such housing 201 also includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. The configuration of housing 201 is not limited to this, and for example, it is also possible to adopt a configuration in which the display unit and the input unit are independent, or a configuration having a foldable structure or a sliding mechanism.

[0062] FIG. 5 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0063] As shown in FIG. 5, the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a memory unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0064] The smartphone 200 also has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0065] The wireless communication unit 210 performs wireless communication with a base station device BS accommodated in the mobile communication network NW in accordance with instructions from the main control unit 220. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0066] The display input unit 204 is a so-called touch panel that displays images (still images and moving images) or text information, etc. under the control of the main control unit 220 to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 202 and an operation panel 203.

[0067] The display panel 202 uses an LCD (Liquid Crystal Display), an OLED (Organic Electro-Luminescence Display), or the like as a display device.

[0068] The operation panel 203 is placed so that an image displayed on the display surface of the display panel 202 can be seen, and is a device that detects one or more coordinates operated by a user's finger or a stylus. When this device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 220. Next, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0069] As shown in FIG. 5, the display panel 202 and operation panel 203 of a smartphone 200, which is exemplified as one embodiment of the imaging device of the present invention, are integrated to form a display input unit 204, and the operation panel 203 is positioned so as to completely cover the display panel 202.

[0070] When such an arrangement is adopted, operation panel 203 may also have a function to detect user operations in areas outside display panel 202. In other words, operation panel 203 may have a detection area for the overlapping portion that overlaps display panel 202 (hereinafter referred to as a display area), and a detection area for the other outer edge portion that does not overlap display panel 202 (hereinafter referred to as a non-display area).

[0071] The size of the display area and the size of the display panel 202 may be completely the same, but they do not necessarily have to be the same. Also, the operation panel 203 may have two sensitive areas: an outer edge portion and an inner portion other than the outer edge portion. Furthermore, the width of the outer edge portion is designed appropriately depending on the size of the housing 201, etc.

[0072] Furthermore, the position detection method used in the operation panel 203 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, or the like, and any method may be used.

[0073] The call unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0074] Also, as shown in FIG. 4, for example, speaker 205 can be mounted on the same surface as display input unit 204, and microphone 206 can be mounted on the side surface of housing 201.

[0075] The operation unit 207 is a hardware key using a key switch or the like, and receives instructions from a user. For example, as shown in Fig. 4, the operation unit 207 is a push-button switch mounted on the side of the housing 201 of the smartphone 200, which is turned on when pressed with a finger or the like, and turned off by the restoring force of a spring or the like when the finger is released.

[0076] The storage unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, data of sent and received e-mails, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data, etc. The storage unit 212 is composed of an internal storage unit 217 built into the smartphone and an external storage unit 218 having a removable external memory slot.

[0077] The internal memory unit 217 and the external memory unit 218 constituting the memory unit 212 are realized using storage media such as a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., MicroSD (registered trademark) memory, etc.), a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

[0078] The external input / output unit 213 serves as an interface with all external devices connected to the smartphone 200, and is used to directly or indirectly connect to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or a network (e.g., Ethernet (registered trademark), wireless LAN (Local Area Network), etc.).

[0079] Examples of external devices that can be connected to the smartphone 200 include wired / wireless headsets, wired / wireless external chargers, wired / wireless data ports, memory cards connected via card sockets, SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) cards, external audio / video devices connected via audio / video I / O (Input / Output) terminals, wirelessly connected external audio / video devices, wired / wirelessly connected smartphones, wired / wirelessly connected personal computers, wired / wirelessly connected personal computers, earphones, etc.

[0080] The external input / output unit 213 can transmit data received from such external devices to each component inside the smartphone 200, or transmit data inside the smartphone 200 to external devices.

[0081] The GNSS receiver 214 receives GNSS signals transmitted from GNSS satellites ST1 to STn in accordance with instructions from the main controller 220, executes positioning calculation processing based on the received multiple GNSS signals, and detects a position consisting of the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiver 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), it can also detect the position using the position information.

[0082] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with instructions from the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0083] The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with instructions from the main control unit 220.

[0084] The main control unit 220 includes a microprocessor, operates according to the control program and control data stored in the memory unit 212, and controls all the units of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the control unit 18. The main control unit 220 also has a mobile communication control function that controls all the units of the communication system to perform voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0085] The application processing function is realized by the main control unit 220 operating in accordance with the application software stored in the storage unit 212. Examples of the application processing function include an infrared communication function that controls the external input / output unit 213 to perform data communication with a partner device, an email function that sends and receives emails, and a web browsing function that views web pages.

[0086] The main control unit 220 also has an image processing function for displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0087] The image processing function refers to a function in which the main control unit 220 decodes the image data, performs image processing on the decoded result, and displays the image on the display input unit 204.

[0088] Furthermore, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting user operations via the operation unit 207 and the operation panel 203 .

[0089] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for starting application software, or displays a window for creating an e-mail.

[0090] The scroll bar refers to a software key for receiving an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​the display panel 202.

[0091] In addition, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the above icons and input of character strings into the input field of the above window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0092] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is an overlapping portion (display area) that overlaps the display panel 202 or an outer edge portion (non-display area) that does not overlap the display panel 202, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 203 or the display position of the software key.

[0093] Furthermore, the main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in response to the detected gesture operation.

[0094] Gesture operation is not a simple touch operation as in the past, but rather an operation in which a trajectory is drawn with a finger or the like, multiple positions are specified simultaneously, or a combination of these is used to draw a trajectory for at least one of multiple positions.

[0095] The camera unit 208 includes the lens device 20, the image sensor 12, the image sensor shift mechanism 13, the image sensor drive unit 14, and the vibration detector 17 shown in FIG.

[0096] The captured image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210 .

[0097] In the smartphone 200 shown in FIG. 5, the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may be mounted on the back surface of the display input unit 204.

[0098] Furthermore, the camera unit 208 can be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or an image from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0099] Furthermore, when the GNSS receiver 214 detects a position, it can also detect the position by referring to an image from the camera unit 208. Furthermore, it can also refer to an image from the camera unit 208 to determine the optical axis direction of the camera unit 208 of the smartphone 200 or determine the current usage environment, without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 208 can also be used in application software.

[0100] In addition, image data of still or video images can be added with location information acquired by the GNSS receiving unit 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., and stored in the memory unit 212, or output via the external input / output unit 213 or wireless communication unit 210.

[0101] As explained above, this specification describes at least the following:

[0102] (1) A control device for an imaging device having a first vibration detector and an imaging element, a processor; The processor is deriving a first shake compensation amount corresponding to a shake caused by vibration and a second shake compensation amount different from the first shake compensation amount based on first output data output from the first vibration detector; deriving a first image blur amount relating to image blur based on image data obtained by imaging using the imaging element; deriving an offset component included in the first output data based on the first blur correction amount, the second blur correction amount, and the first image blur amount; Control device.

[0103] (2) The control device according to (1), The second shake correction amount is smaller than the first shake correction amount. Control device.

[0104] (3) The control device according to (2), the first blur correction amount is information not used in blur correction performed based on the first output data, The second blur correction amount is information used for the blur correction. Control device.

[0105] (4) The control device according to (2) or (3), the first blur correction amount is based on data obtained by integrating the first output data, the second image blur correction amount is based on data obtained by integrating data obtained by performing a process to remove the offset component from the first output data, or is based on data obtained by performing a process to remove the offset component from data obtained by integrating the first output data. Control device.

[0106] (5) The control device according to any one of (1) to (4), the processor derives the offset component based on a value obtained by subtracting the first image blur correction amount from a sum of the first image blur amount and the second image blur correction amount; Control device.

[0107] (6) The control device according to any one of (1) to (5), The first vibration detector is capable of detecting angular velocity. Control device.

[0108] (7) The control device according to any one of (1) to (4), the first vibration detector is capable of detecting an angular velocity, the imaging device further includes a second vibration detector capable of detecting acceleration; the second vibration detector is capable of detecting a movement of the imaging device in a direction along a light receiving surface of the imaging element; The processor is deriving a first amount of motion of the imaging device based on second output data output from the second vibration detector; deriving the offset component based on the first blur correction amount, the second blur correction amount, the first image blur amount, and a third blur correction amount required to correct the blur caused by the first motion amount; Control device.

[0109] (8) (7) The control device according to the present invention, the processor derives the offset component based on a value obtained by subtracting the sum of the first image blur amount and the third blur compensation amount from the sum of the first image blur amount and the second blur compensation amount; Control device.

[0110] (9) A control device according to any one of (1) to (8), the processor derives the amount of blur between the plurality of image data as the first image blur amount, and if the plurality of image data includes image data that does not satisfy a predetermined condition, derives the first image blur amount based on image data excluding the image data that does not satisfy the predetermined condition; Control device.

[0111] (10) The control device according to any one of (1) to (9), the processor sets a shutter speed of the image sensor when the offset component is derived to be faster than a shutter speed when the offset component is not derived; Control device.

[0112] (11) The control device according to any one of (1) to (10), The processor is acquiring first synchronization data relating to synchronization between an imaging timing by the imaging element and an output timing of the first output data; determining a synchronized combination of the image data and the first output data based on the first synchronization data; deriving the offset component based on the first blur correction amount, the second blur correction amount, and the first image blur amount obtained from the determined combination of data; Control device.

[0113] (12) An imaging device comprising the control device according to any one of (1) to (11).

[0114] (13) A method for controlling an imaging device having a first vibration detector and an imaging element, comprising: deriving a first shake compensation amount corresponding to a shake caused by vibration and a second shake compensation amount different from the first shake compensation amount based on first output data output from the first vibration detector; deriving a first image blur amount relating to image blur based on image data obtained by imaging using the imaging element; deriving an offset component included in the first output data based on the first blur correction amount, the second blur correction amount, and the first image blur amount; Control method.

[0115] (14) A control program for an imaging device having a first vibration detector and an imaging element, deriving a first shake compensation amount corresponding to a shake caused by vibration and a second shake compensation amount different from the first shake compensation amount based on first output data output from the first vibration detector; deriving a first image blur amount relating to image blur based on image data obtained by imaging using the imaging element; deriving an offset component included in the first output data based on the first blur correction amount, the second blur correction amount, and the first image blur amount; A control program that causes a processor to execute steps. [Explanation of symbols]

[0116] 1. Digital camera 10 Camera body 12 Image sensor 13 Image sensor shift mechanism 14 Image sensor drive unit 15 Display section 16 memory 17 Vibration detector 18 Control Unit 19 Storage medium 20 Lens device 30 Imaging optical system 31 Imaging lens 40 Lens control unit 200 smartphones 201 Case 202 Display Panel 203 Operation Panel 204 Display and input section 205 Speaker 206 Microphone 208 Camera Club 210 Radio Communication Department 211 Telephone section 212 Storage section 213 External input / output section 214 GNSS receiver 215 Motion sensor unit 216 Power supply section 217 Internal storage 218 External Memory Unit 220 Main control unit

Claims

1. A control device for an imaging device having an imaging element, a processor; a first vibration detector that outputs information about angular velocities around a plurality of axes; a second vibration detector capable of detecting acceleration and detecting a movement of the imaging device in a direction along the light receiving surface of the imaging element; Equipped with The processor: deriving a first shake correction amount corresponding to shake caused by vibration, the first shake correction amount being derived from data obtained by directly integrating first output data, which is information about the angular velocity output from the first vibration detector; deriving a second image blur correction amount, which is information derived from data obtained by performing high-pass filtering on data obtained by integrating the first output data for a predetermined time period, or information derived from data obtained by integrating the first output data for a predetermined time period, deriving a third shake correction amount by performing a second integration on second output data, which is information on the acceleration for a predetermined time period output from the second vibration detector; deriving a first image blur amount relating to image blur, which is an inter-image data blur amount that is a motion vector between a plurality of image data obtained by image capture using the image sensor during image blur correction; deriving an offset component included in the first output data based on a value obtained by subtracting the sum of the first image blur amount and the third image blur compensation amount from the sum of the first image blur compensation amount and the second image blur compensation amount; Control device.

2. A control device according to claim 1, the processor derives the amount of blur between the plurality of image data as the first image blur amount, and if the plurality of image data includes image data that does not satisfy a predetermined condition, derives the first image blur amount based on image data excluding the image data that does not satisfy the predetermined condition. Control device.

3. A control device according to claim 1 or 2, the processor sets a shutter speed of the image sensor to be set when the offset component is derived faster than a shutter speed to be set when the offset component is not derived; Control device.

4. A control device according to any one of claims 1 to 3, The processor: acquiring first synchronization data relating to synchronization between an imaging timing by the imaging element and an output timing of the first output data; determining a synchronized combination of the image data and the first output data based on the first synchronization data; deriving the offset component based on the first blur correction amount, the second blur correction amount, and the first image blur amount obtained from the determined combination of data; Control device.

5. A control device for an imaging device having a first vibration detector and an imaging element that outputs information on angular velocities around multiple axes, a processor; The processor: deriving a first shake correction amount corresponding to shake caused by vibration, the first shake correction amount being derived from data obtained by directly integrating first output data, which is information about the angular velocity output from the first vibration detector; deriving a second image blur correction amount, which is information derived from data obtained by performing high-pass filtering on data obtained by integrating the first output data for a predetermined time period, or information derived from data obtained by integrating the first output data for a predetermined time period, deriving a first image blur amount relating to image blur, which is an inter-image data blur amount that is a motion vector between a plurality of image data obtained by image capture using the image sensor during image blur correction; deriving an offset component included in the first output data based on a value obtained by subtracting the first image blur amount from the sum of the first blur correction amount and the second blur correction amount; a shutter speed of the image sensor set when the offset component is derived is set to be faster than a shutter speed set when the offset component is not derived; Control device.

6. A control device according to claim 5, the processor derives the amount of blur between the plurality of image data as the first image blur amount, and if the plurality of image data includes image data that does not satisfy a predetermined condition, derives the first image blur amount based on image data excluding the image data that does not satisfy the predetermined condition. Control device.

7. A control device according to claim 5 or 6, The processor: acquiring first synchronization data relating to synchronization between an imaging timing by the imaging element and an output timing of the first output data; determining a synchronized combination of the image data and the first output data based on the first synchronization data; deriving the offset component based on the first blur correction amount, the second blur correction amount, and the first image blur amount obtained from the determined combination of data; Control device.

8. An imaging device equipped with a control device described in any one of claims 1 to 7.

9. A control method for an imaging device, comprising: the imaging device includes an imaging element, a first vibration detector that outputs information on angular velocities around a plurality of axes, and a second vibration detector that is capable of detecting acceleration and is capable of detecting movement of the imaging device in a direction along a light receiving surface of the imaging element; deriving a first shake correction amount corresponding to shake caused by vibration, the first shake correction amount being derived from data obtained by directly integrating first output data, which is information about the angular velocity output from the first vibration detector; deriving a second image blur correction amount, which is information derived from data obtained by performing high-pass filtering on data obtained by integrating the first output data for a predetermined time period, or information derived from data obtained by integrating the first output data for a predetermined time period, deriving a third shake correction amount by performing a second integration on second output data, which is information on the acceleration for a predetermined time period output from the second vibration detector; deriving a first image blur amount relating to image blur, which is an inter-image data blur amount that is a motion vector between a plurality of image data obtained by image capture using the image sensor during image blur correction; deriving an offset component included in the first output data based on a value obtained by subtracting the sum of the first image blur amount and the third image blur compensation amount from the sum of the first image blur compensation amount and the second image blur compensation amount; Control method.

10. A control program for an imaging device, comprising: the imaging device includes an imaging element, a first vibration detector that outputs information on angular velocities around a plurality of axes, and a second vibration detector that is capable of detecting acceleration and is capable of detecting movement of the imaging device in a direction along a light receiving surface of the imaging element; deriving a first shake correction amount corresponding to shake caused by vibration, the first shake correction amount being derived from data obtained by directly integrating first output data, which is information about the angular velocity output from the first vibration detector; deriving a second image blur correction amount, which is information derived from data obtained by performing high-pass filtering on data obtained by integrating the first output data for a predetermined time period, or information derived from data obtained by integrating the first output data for a predetermined time period, deriving a third image blur amount by performing a second-order integration on second output data, which is information on the acceleration for a predetermined time period output from the second vibration detector, and deriving a first image blur amount related to image blur, which is an inter-image data blur amount, which is a motion vector between a plurality of image data obtained by imaging using the image sensor during image blur correction; deriving an offset component included in the first output data based on a value obtained by subtracting the sum of the first image blur amount and the third image blur compensation amount from the sum of the first image blur compensation amount and the second image blur compensation amount; A control program that causes a processor to execute steps.

Images