Imaging device

Abstract

The present invention provides an imaging device that can efficiently correct image distortion caused by camera shake while suppressing a reduction in the field of view. [Solution] The imaging device comprises an image sensor that captures an image of a subject through an optical system and generates image data, a detection unit that detects the amount of blur of the imaging device, an image processing unit that performs image blur correction by adjusting the part of the image data that outputs an image according to the amount of blur detected by the detection unit, and a control unit that controls the image blur correction performed in the image processing unit. In the image processing unit, distortion correction according to the distortion aberration of the optical system is performed on the image region indicated by the image data. The control unit changes the ratio between the first and second image blur corrections performed in the image region that has been distortion corrected by the image processing unit according to the focal length of the optical system, the first image blur correction corrects the distortion of the image in the distortion-corrected image region, and the second image blur correction moves the part of the image that outputs an image in the said image region.

Description

[Technical Field]

[0001] This disclosure relates to an imaging device having an image stabilization function. [Background technology]

[0002] Patent Document 1 discloses an imaging device aimed at efficiently utilizing the imaging area of ​​an image sensor to perform shake correction. This imaging device extracts an image of an extracted region that has been rotated or moved according to the amount of shake correction (correction angle) from an image in which distortion aberration generated by the imaging optical system has been corrected for the captured image. In this way, the imaging device of Patent Document 1 uses an image area that has been stretched outward from the center of the image by the correction of distortion aberration, and by rotating or moving the extracted region, shake in the rotational direction around the optical axis of the imaging optical system, or in the pan and tilt directions, is corrected. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2010-273245 [Overview of the project] [Problems that the invention aims to solve]

[0004] This disclosure provides an imaging device that can efficiently correct image distortion caused by camera shake while suppressing a reduction in the field of view. [Means for solving the problem]

[0005] An imaging device in one aspect of the present disclosure includes an image sensor having an imaging area in which an image of a subject is formed via an optical system, an image sensor that captures an image of a subject and generates image data, a detection unit that detects the amount of blur of the imaging device, an image processing unit that performs image blur correction by adjusting the portion of the image data that outputs an image according to the amount of blur detected by the detection unit, and a control unit that controls the image blur correction performed in the image processing unit. In the image processing unit, distortion correction according to the distortion aberration of the optical system is performed on the image area indicated by the image data. The control unit changes the ratio between the first and second image blur corrections performed in the image area that has been distortion corrected by the image processing unit according to the focal length of the optical system, the first image blur correction corrects the distortion of the image in the distortion-corrected image area, and the second image blur correction moves the portion of the image that outputs an image in the said image area. [Effects of the Invention]

[0006] According to the imaging device described herein, image distortion caused by camera shake can be efficiently corrected while suppressing a reduction in the field of view. [Brief explanation of the drawing]

[0007] [Figure 1] Perspective view of a digital camera according to Embodiment 1 of this disclosure [Figure 2] Block diagram showing the configuration of the digital camera according to Embodiment 1 [Figure 3] Block diagram showing the configuration of the BIS processing unit in the digital camera of Embodiment 1 [Figure 4] A diagram illustrating the correction mode using the EIS function of the digital camera in Embodiment 1. [Figure 5] A diagram illustrating the EIS function with cropping in digital cameras. [Figure 6] A diagram illustrating the correction required for distortion aberration in the optical system of a digital camera. [Figure 7] A diagram illustrating camera shake caused by tilt-shift in digital cameras. [Figure 8]Figure for explaining the crop-res EIS function in the digital camera of Embodiment 1 [Figure 9] Flowchart exemplifying the overall operation related to camera shake correction of the digital camera of Embodiment 1 [Figure 10] Figure for explaining the calculation process of distortion correction parameters in a digital camera [Figure 11] Flowchart exemplifying the calculation process of roll correction parameters in a digital camera [Figure 12] Figure for explaining the calculation process of the surplus area in the calculation process of roll correction parameters [Figure 13] Flowchart exemplifying the operation of the digital camera according to the modified example of Embodiment 1 [Figure 14] Figure for explaining the operation of the digital camera according to Embodiment 2 [Figure 15] Flowchart exemplifying the operation of the digital camera according to Embodiment 2

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments in the present disclosure will be described with appropriate reference to the drawings. However, in the detailed description, unnecessary parts of the description regarding the prior art and substantially the same configurations may be omitted. This is for the purpose of simplifying the description. Also, the following description and the attached drawings are disclosed so that those skilled in the art can fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

[0009] (Embodiment 1) In Embodiment 1, as an example of an imaging device, an example of a digital camera having a camera shake correction function will be described.

[0010] 1. Configuration FIG. 1 is a perspective view of a digital camera 1 according to Embodiment 1. FIG. 2 is a block diagram showing the configuration of the digital camera 1. The digital camera 1 is composed of a camera body 100 and an interchangeable lens 200 that is detachable therefrom.

[0011] In the following description, the function of moving the imaging element within the camera body 100 to correct blurring is referred to as the "BIS (Body Image Stabilizer) function". Also, the function of moving the correction lens within the interchangeable lens 200 to correct blurring is referred to as the "OIS (Optical Image Stabilizer) function". Further, the function of adjusting the area for outputting an image in the image data generated by the imaging element to correct blurring is referred to as the "EIS (Electronic Image Stabilizer) function". In the following description, the BIS function and the OIS function may be collectively referred to as the optical shake correction function, and the EIS function may be referred to as the electronic shake correction function.

[0012] Also, in the following description, the rotational directions corresponding to the horizontal and vertical directions of the imaging element in the digital camera 1 are referred to as the yaw direction and the pitch direction, respectively, and the rotational direction about the rotation axis along the optical axis of the digital camera 1 is referred to as the roll direction (see FIG. 1).

[0013] 1-1. Camera Body The camera body 100 (an example of an imaging device) includes an image sensor 110, a liquid crystal monitor 120, an operation unit 130, a camera control unit 140, a body mount 150, and a card slot 170. Also, the camera body 100 includes an image correction unit 143 that implements the EIS function as a functional configuration of, for example, the camera control unit 140.

[0014] The camera control unit 140 controls the operation of the entire digital camera 1 by controlling components such as the image sensor 110 in response to an instruction from the release button. The camera control unit 140 transmits a vertical synchronization signal to the timing generator 112. In parallel with this, the camera control unit 140 generates an exposure synchronization signal. The camera control unit 140 periodically transmits the generated exposure synchronization signal to the lens control unit 240 via the body mount 150 and the lens mount 250. The camera control unit 140 uses the DRAM 141 as a work memory during control operations and image processing operations.

[0015] The image sensor 110 is an example of an image sensor that captures an image of a subject incident through the interchangeable lens 200 and generates image data. The image sensor 110 is, for example, a CCD, CMOS image sensor, or NMOS image sensor. The generated image data is digitized by the AD converter 111. The digitized image data is subjected to predetermined image processing by the camera control unit 140. These predetermined image processing processes include, for example, gamma correction, white balance correction, scratch correction, YC conversion, electronic zoom, and JPEG compression.

[0016] The image sensor 110 operates at timings controlled by the timing generator 112. The image sensor generates still images, moving images, or through images for recording. Through images are mainly moving images and are displayed on the liquid crystal monitor 120 for the user to determine the composition for capturing still images.

[0017] The liquid crystal monitor 120 displays images such as through images and various information such as menu screens. The liquid crystal monitor 120 is an example of a display unit in this embodiment. Other types of display devices, such as an organic EL display device, may be used instead of the liquid crystal monitor.

[0018] The control unit 130 includes various operating components such as a release button for instructing the start of shooting, a mode dial for setting the shooting mode, and a power switch. The control unit 130 also includes a touch panel superimposed on the LCD monitor 120.

[0019] The card slot 170 can accommodate a memory card 171 and controls the memory card 171 based on control from the camera control unit 140. The digital camera 1 can store image data in the memory card 171 and read image data from the memory card 171.

[0020] The body mount 150 is mechanically and electrically connectable to the lens mount 250 of the interchangeable lens 200. The body mount 150 can send and receive data to and from the interchangeable lens 200 via the lens mount 250. The body mount 150 transmits the exposure synchronization signal received from the camera control unit 140 to the lens control unit 240 via the lens mount 250. It also transmits other control signals received from the camera control unit 140 to the lens control unit 240 via the lens mount 250. Furthermore, the body mount 150 transmits signals received from the lens control unit 240 via the lens mount 250 to the camera control unit 140.

[0021] Furthermore, the camera body 100 is configured to implement the BIS function and further includes a gyro sensor 184 (shake detection unit) for detecting camera body 100 shake, and a BIS processing unit 183 for controlling shake correction processing based on the detection result of the gyro sensor 184. In addition, the camera body 100 is equipped with a sensor drive unit 181 for moving the image sensor 110 and a position sensor 182 for detecting the position of the image sensor 110.

[0022] The sensor drive unit 181 can be implemented, for example, by a magnet and a flat coil. The sensor drive unit 181 may also include other motors or actuators. The position sensor 182 is a sensor that detects the position of the image sensor 110 in a plane perpendicular to the optical axis of the optical system. The position sensor 182 can be implemented, for example, by a magnet and a Hall element.

[0023] The BIS processing unit 183 controls the sensor drive unit 181 based on signals from the gyro sensor 184 and the position sensor 182 to shift the image sensor 110 in a plane perpendicular to the optical axis to compensate for camera body 100 shake. There are mechanical limitations to the range in which the image sensor 110 can be driven by the sensor drive unit 181. In the BIS function, the range in which the image sensor 110 can be driven by the sensor drive unit 181 is called the "element drive range".

[0024] 1-2. Interchangeable lenses The interchangeable lens 200 comprises an optical system, a lens control unit 240, and a lens mount 250. The optical system includes a zoom lens 210, an OIS (Optical Image Stabilizer) lens 220, a focus lens 230, and an aperture 260.

[0025] The zoom lens 210 is a lens for changing the magnification of the subject image formed by the optical system. The zoom lens 210 consists of one or more lenses. The zoom lens 210 is driven by a zoom drive unit 211. The zoom drive unit 211 includes a zoom ring that can be operated by the user. Alternatively, the zoom drive unit 211 may include a zoom lever and an actuator or motor. The zoom drive unit 211 moves the zoom lens 210 along the optical axis of the optical system in response to user operation.

[0026] The focus lens 230 is a lens used in the optical system to change the focus state of the subject image formed on the image sensor 110. The focus lens 230 is composed of one or more lenses. The focus lens 230 is driven by the focus drive unit 233.

[0027] The focus drive unit 233 includes an actuator or motor and moves the focus lens 230 along the optical axis of the optical system based on the control of the lens control unit 240. The focus drive unit 233 can be implemented using a DC motor, stepping motor, servo motor, or ultrasonic motor, etc.

[0028] The OIS lens 220 is a lens used in the OIS function to correct blur in the subject image formed by the optical system of the interchangeable lens 200. The OIS lens 220 reduces blur in the subject image on the image sensor 110 by moving in a direction that cancels out the blur of the digital camera 1. The OIS lens 220 is composed of one or more lenses. The OIS lens 220 is driven by the OIS drive unit 221.

[0029] The OIS drive unit 221 shifts the OIS lens 220 in a plane perpendicular to the optical axis of the optical system, under control from the OIS processing unit 223. There are mechanical limitations to the range in which the OIS lens 220 can be driven by the OIS drive unit 221. The range in which the OIS lens 220 can be driven by the OIS drive unit 221 is called the "lens drive range". The OIS drive unit 221 can be implemented, for example, by a magnet and a flat coil. The position sensor 222 is a sensor that detects the position of the OIS lens 220 in a plane perpendicular to the optical axis of the optical system. The position sensor 222 can be implemented, for example, by a magnet and a Hall element. The OIS processing unit 223 controls the OIS drive unit 221 based on the output of the position sensor 222 and the output of the gyro sensor 224 (shake detection unit).

[0030] The aperture 260 adjusts the amount of light incident on the image sensor 110. The aperture 260 is driven by an aperture drive unit 262, which controls the size of its opening. The aperture drive unit 262 includes a motor or actuator.

[0031] The gyro sensor 184 or 224 detects shake (vibration) in the yaw, pitch, and roll directions based on the angular change per unit time, i.e., angular velocity, of the digital camera 1. The gyro sensor 184 or 224 outputs an angular velocity signal indicating the amount of shake (angular velocity) detected to the BIS processing unit 183 or OIS processing unit 223. The angular velocity signal output by the gyro sensor 184 or 224 may contain a wide range of frequency components due to camera shake, mechanical noise, etc. Other sensors capable of detecting shake in the digital camera 1 can be used instead of the gyro sensor. Furthermore, the gyro sensor 224 of the interchangeable lens 200 does not need to detect shake in the roll direction.

[0032] The camera control unit 140 and the lens control unit 240 may be configured as hardwired electronic circuits or as a microcomputer using a program. For example, the camera control unit 140 and the lens control unit 240 can be implemented using various processors such as a CPU, MPU, GPU, DSU, FPGA, or ASIC.

[0033] 1-3. Configuration of the image stabilization function The configuration for realizing various image stabilization functions of the digital camera 1 in this embodiment will be explained with reference to Figures 3 to 5.

[0034] 1-3-1. BIS Processing Unit The configuration of the BIS processing unit 183 in the camera body 100 will be explained using Figure 3. Figure 3 is a block diagram showing the configuration of the BIS processing unit 183 in the digital camera 1 of this embodiment. The BIS processing unit 183 includes an HPF (high-pass filter) 406, a phase compensation unit 407, an integrator 408, and a PID control unit 410. For example, the BIS processing unit 183 receives a signal from the gyro sensor 184 at a predetermined time interval (e.g., 4 kilohertz).

[0035] The HPF406 blocks drift components by, for example, blocking predetermined low-frequency components contained in the signal received from the gyro sensor 184.

[0036] The phase compensation unit 407 corrects the phase delay caused by the sensor drive unit 181 and other components in the signal received from the HPF 406.

[0037] The integrator 408 integrates the signal indicating the angular velocity of the vibration input from the phase compensation unit 407 to generate a signal indicating the angle of the vibration (hereinafter referred to as the "vibration detection signal"). The vibration detection signal from the integrator 408 is input to the PID control unit 410. Here, the BIS processing unit 183 may use or add filter configurations other than the above configuration, such as a notch filter for noise processing.

[0038] The PID control unit 410 generates a drive signal to shift the image sensor 110 based on the output from the position sensor 182 and the output from the integrator 408, and outputs it to the sensor drive unit 181. The sensor drive unit 181 drives the image sensor 110 based on the drive signal. Specifically, the sensor drive unit 181 translates the image sensor 110 horizontally or vertically in the imaging plane, or rotates the image sensor 110 with the optical axis as the axis of rotation, within the element's movable range.

[0039] The BIS processing unit 183 is configured to communicate data with the camera control unit 140. For example, the BIS processing unit 183 starts / stops the image stabilization operation in response to a control signal from the camera control unit 140. The BIS processing unit 183 also transmits various information related to the image stabilization operation to the camera control unit 140.

[0040] For example, the BIS processing unit 183 may calculate the horizontal and vertical shake correction amounts for the imaging plane, respectively, as the amount of movement of the image sensor 110 by the sensor drive unit 181, based on the yaw and pitch shake angles indicated by the generated shake detection signal. The BIS processing unit 183 may also obtain the focal length from the interchangeable lens 200 via the camera control unit 140 according to the zoom state, and use the obtained focal length, etc., to calculate the shake correction amount by converting the correction angle that cancels out the shake angle into the amount of movement of the image sensor 110.

[0041] The OIS processing unit 223 can be configured to drive the OIS drive unit 221 instead of the sensor drive unit 181, for example, in a configuration similar to that of the BIS processing unit 183 described above. Furthermore, the OIS processing unit 223 operates using the detection result of the gyro sensor 224 in the interchangeable lens 200 instead of the gyro sensor 184 in the camera body 100. The gyro sensor 224 in the interchangeable lens 200 does not need to detect roll-direction shake.

[0042] 1-3-2. Correction Modes Using EIS Function The digital camera 1 of this embodiment has multiple correction modes as operating modes for correcting blur using the image stabilization function. Figure 4 is a diagram illustrating the correction modes by the EIS function of the digital camera 1 of this embodiment.

[0043] In the digital camera 1 of this embodiment, the correction mode used for image stabilization by the EIS function can be selected by the user. Figure 4 shows an example of the menu screen for setting the correction mode when shooting video in the digital camera 1. In the example in Figure 4, the LCD monitor 120 displays "Large Crop," "Small Crop," "Cropless," and "OFF" as menu items corresponding to each correction mode. When the "OFF" menu item is selected, the EIS function is disabled.

[0044] In the "large crop" and "small crop" correction modes, the image correction unit 143 corrects camera shake by changing the area from which the image is cropped within the imaging area of ​​the image sensor 110 according to the degree of camera shake. This cropped EIS function will be explained using Figure 5. Figure 5 is a diagram illustrating the cropped EIS function in the digital camera 1.

[0045] The image correction unit 143 performs a process to extract an image of a narrower region from the entire image data 10 generated by the image sensor 110 by a preset extraction amount Eo, as shown in Figure 5, for example. For example, according to a predetermined extraction ratio, the extraction amount Eo is calculated for each of the horizontal X and vertical Y directions of the image data 10 based on the number of pixels, and an image is extracted with the number of pixels reduced by the calculated extraction amount Eo for each direction X and Y. Various image processing performed by the camera control unit 140 for recording the shooting results is applied to the image data after extraction. For example, electronic zoom processing may be applied so that the extracted image is the same size as the image before extraction.

[0046] For example, depending on the user's operation, such as selecting the "Large Crop" or "Small Crop" menu item, a predetermined cropping amount Eo, pre-stored in the flash memory 142 or the like, is set in the image correction unit 143 for each correction mode with cropping. In the "Large Crop" correction mode, a larger cropping amount Eo is set than in the "Small Crop" correction mode. For example, in the "Large Crop" correction mode, a cropping amount Eo is set such that the number of pixels in the cropped image decreases by 20-30% compared to the image before cropping. In the "Small Crop" correction mode, a cropping amount Eo is set such that the number of pixels decreases by approximately 8% before and after cropping.

[0047] The image correction unit 143 calculates a blur correction amount as an adjustment amount for the cropping position based on the blur detection signal input from the integrator 408 of the BIS processing unit 183. The image correction unit 143 realizes the EIS function in the "large crop" or "small crop" correction mode by adjusting the cropping position by the calculated blur correction amount. For example, a reference area 21 is set, which is the position of the cropped image when the blur correction amount by the EIS function with cropping is zero, with the center position of the entire image in the image data 10 as the reference. For example, the reference area 21 is arranged along the horizontal X and vertical Y directions of the image data 10. For example, areas in the image data 10 other than the reference area 21 are examples of correction areas in this embodiment.

[0048] The image correction unit 143 translates the image region 22, which is cut out from the reference region 21, in the horizontal direction X according to the amount of horizontal shake correction of the image sensor 110 acquired from the BIS processing unit 183. Similarly, the image correction unit 143 translates the image region 22 in the vertical direction Y according to the amount of vertical shake correction of the image sensor 110. In addition, the image correction unit 143 rotates the orientation of the image region 22 from the orientation of the reference region 21 on the XY plane according to the amount of shake correction in the roll direction.

[0049] The EIS function with cropping allows for the position adjustment of the image region 22 as described above, within the range of the cropping amount Eo. Specifically, the translation of the image region 22 is performed within the range obtained by subtracting the amount for roll as a margin Er from the cropping amount Eo. The rotation of the image region 22 is performed within the range of rotation angles that fit within the roll margin Er in the cropping amount Eo. The roll margin Er is determined by the camera control unit 140 according to, for example, the lens condition of the interchangeable lens 200. For example, when the interchangeable lens 200 is wide-angle, the amount of movement of the image region 22 that corrects the amount of blur in the yaw and pitch directions becomes small, so the shorter the focal length, the larger the roll margin Er is determined to be.

[0050] In contrast, in the digital camera 1 of this embodiment, when the "cropped" correction mode is selected on the menu screen shown in Figure 4, a cropless EIS function is executed in the image data 10 without setting the cropping amount Eo described above within the range corresponding to the imaging area. In this case, the corrected image has the same field of view (number of pixels in that range) as when the EIS function is "OFF," i.e., disabled. In this way, the cropless EIS function suppresses the reduction in the number of pixels due to image cropping in the image data 10, and can perform image stabilization while maintaining the field of view at the time of imaging. The image correction unit 143 implements the cropless EIS function according to the lens characteristics of the interchangeable lens 200, for example, as described later.

[0051] 2. Operation The operation of the digital camera 1, which is configured as described above, will be explained below.

[0052] 2-1. Correction of distortion In the digital camera 1 of this embodiment, for example, the image correction unit 143 corrects distortion aberration caused by the optical system of the interchangeable lens 200 in the captured image, in addition to the operation of image stabilization by the EIS function.

[0053] Figure 6 is a diagram illustrating the correction required for distortion aberration in the optical system of the digital camera 1. For example, as shown in Figure 6(A), distortion aberration caused by the lens characteristics of the interchangeable lens 200 can result in image distortion in the image data 10 generated by the image sensor 110. In the image data 10, the entire image area corresponding to the imaging area of ​​the image sensor 110 constitutes an image formation area 20 corresponding to the number of output pixels in the recording of the shooting result when various image corrections are not applied.

[0054] Figure 6(A) shows an example where the optical system of interchangeable lens 200 has negative distortion. In the captured image shown in image data 10 of Figure 6(A), so-called barrel distortion occurs, where the periphery of the image is compressed relative to the central position Pc of the entire image due to negative distortion.

[0055] Figure 6(A) also illustrates distortion characteristic data 30 showing the distortion characteristics of the interchangeable lens 200. The distortion characteristic data 30 relates the image height relative to the center position Pc and the distortion rate. For example, the distortion rate indicates the rate of change of the image height in which the subject image is formed on the imaging surface of the image sensor 110, from the state where there is no distortion. The sign of the distortion rate is positive if the direction in which the image formation position of the subject on the imaging surface changes from the state where there is no distortion is in the direction of increasing image height, and negative if the direction of change is in the direction of decreasing image height. For example, wide-angle lenses, which have a relatively wide angle of view as a lens characteristic, are prone to barrel distortion, which is a type of distortion that exhibits a negative distortion rate.

[0056] The characteristics of distortion aberration also change depending on the focal length corresponding to the zoom state of the interchangeable lens 200, and the focus position corresponding to the focus state. For example, barrel distortion is more likely to occur at the wide-angle end when the focal length is relatively short.

[0057] The image correction unit 143 corrects distortion aberration by deforming the image data 10 in Figure 6(A), for example as shown in Figure 6(B), by stretching the peripheral portion outward from the image forming region 20 relative to the central position Pc. The image correction unit 143 performs this distortion correction based on the distortion correction data 31 shown in Figure 6(B), for example. The distortion correction data 31 relates the image height relative to the central position Pc to the correction rate. For example, the correction rate indicates the degree of image deformation from the pre-correction image before distortion correction.

[0058] As a result of the distortion correction described above, an enlarged image area 13 is obtained in the image data 10, which is enlarged from the image forming area 20 corresponding to the rectangular imaging area. When the amount of blur correction by the EIS function is zero or the EIS function is disabled, the image correction unit 143 cuts out an image of the same size as the range corresponding to the imaging area from the enlarged image area 13 in the image data 10, centered on the central position Pc, and outputs it. In the digital camera 1 of this embodiment, the image correction unit 143 realizes a cropless EIS function by using the enlarged image area 13 from before correction in the image data 10 through this distortion correction.

[0059] The above describes the case where barrel distortion is corrected as a distortion correction method. However, in the case of pincushion distortion, which exhibits a positive distortion rate, the captured image will be one in which the peripheral part is stretched outward relative to the central position Pc. The digital camera 1 performs distortion correction for such pincushion distortion by compressing the peripheral part of the image. In this case, if the enlarged image area 13 cannot be obtained by distortion correction, the digital camera 1 of this embodiment does not perform, for example, a cropless EIS function. On the other hand, in the case of canopy distortion, which occurs as a combination of barrel distortion and pincushion distortion, for example, similar to the case of barrel distortion correction, if the enlarged image area 13 can be obtained by distortion correction, a cropless EIS function may be applied.

[0060] 2-2. Correction of tilt blur In addition to the distortion aberrations described above, in situations where camera shake is relatively large, such as when the user (photographer) is shooting video while walking, distortion may occur in the peripheral areas of the captured image due to camera shake caused by changes in the posture of the digital camera 1. Correction of such lateral movement shake will be explained using Figures 7 and 8.

[0061] Figure 7 is a diagram illustrating camera shake caused by tilt-shift in the digital camera 1. Figures 7(A) to 7(D) show the orientation of the digital camera 1 and the captured image Im of the same subject taken by the digital camera 1 in that orientation.

[0062] Figure 7(A) shows an example where no tilt blur occurs. Figure 7(B) shows an example where the orientation of the digital camera 1 changes in the pitch direction from the state in Figure 7(A). For example, in the captured image Im when tilt blur occurs in the pitch direction as shown in Figure 7(B), trapezoidal distortion occurs in which the subject image is stretched in the horizontal direction X and compressed in the vertical direction Y.

[0063] Figure 7(C) shows an example where the orientation of the digital camera 1 changes from the state in Figure 7(A) to the yaw direction instead of the pitch direction as shown in Figure 7(B). For example, as shown in Figure 7(C), when yaw-direction tilt shake occurs, the captured image Im exhibits trapezoidal distortion, where the subject image is stretched in the vertical Y direction and compressed in the horizontal X direction. Figure 7(D) shows an example where the orientation of the digital camera 1 changes in both the pitch and yaw directions from the state in Figure 7(A). In this case, the captured image Im exhibits trapezoidal distortion in both the horizontal X and vertical Y directions. It is known that this trapezoidal distortion due to various types of tilt shake is particularly noticeable in the captured image Im during wide-angle photography, where the imaging range is relatively wide.

[0064] For example, in the central part of the captured image Im, the translation and rotation of the image sensor 110 by the BIS function and / or the translation and rotation of the image region 22 extracted by the EIS function can be performed with respect to the central position Pc, thereby reducing blur in the yaw, pitch, and roll directions. On the other hand, even if the above-mentioned translation and rotation correction is performed on the captured image Im, trapezoidal distortion due to tilt blur remains, and there are concerns that this will be particularly noticeable in the peripheral areas far from the central position Pc.

[0065] Therefore, the digital camera 1 corrects trapezoidal distortion caused by tilt-shift blur by performing an image deformation process in the image data 10 that corrects the trapezoidal distortion. Such trapezoidal correction of images requires, for example, the use of an image area wider than the corrected area for deformation processing that projectively transforms the coordinates on the image according to the trapezoidal distortion. Therefore, the digital camera 1 of this embodiment corrects trapezoidal distortion caused by tilt-shift blur by using a cropless EIS function to use an enlarged image area 13, as shown in Figure 6(B), for trapezoidal correction.

[0066] Figure 8 is a diagram illustrating the cropless EIS function in the digital camera 1 of this embodiment. In the digital camera 1 of this embodiment, the image correction unit 143 sets a reference area 11 to be deformed by trapezoidal correction from the enlarged image area 13 by distortion correction in the image data 10. The image correction unit 143 then outputs an image of the output area 12 as the corrected area deformed from the reference area 11. The output area 12 is the area in the image data 10 that is recorded on, for example, the memory card 171 or displayed on the liquid crystal monitor 120, and has a number of pixels corresponding to the resolution of the image recorded as a shooting result.

[0067] As described above, the digital camera 1 of this embodiment corrects tilt blur in image data 10 in which distortion aberrations such as barrel distortion have occurred, by using the enlarged image area 13 obtained by distortion correction. In this way, by using the enlarged image area 13, which is wider than the image forming area 20 (Figure 6) corresponding to the imaging area of ​​the image sensor 110, to correct trapezoidal distortion caused by tilt blur, an image of the output area 12 having the same number of pixels as the image forming area 20 can be output. As a result, tilt correction that corrects such image distortion can be achieved while suppressing the reduction of the field of view due to image cropping.

[0068] For example, in an EIS function with cropping, as shown in Figure 5, which crops the image region 22 from a reference region 21 according to the cropping amount Eo within the image formation region 20, the number of pixels in the output image region 22 decreases from the number of pixels in the image formation region 20. Therefore, if correction is performed using an EIS function with cropping, for example, the output image of the image region 22 will have a narrower field of view than before the correction. In contrast, a cropless EIS function can achieve both tilt correction and maintenance of the field of view, as described above.

[0069] In the cropless EIS function, an example was described above in which an image from the output area 12 having the same number of pixels as the image forming area 20 is output. However, it is sufficient if the number of pixels between the output area 12 and the image forming area 20 is approximately the same. For example, even when the number of pixels decreases due to various image processing by the camera control unit 140, the cropless EIS function may be executed such that the rate of decrease in the number of pixels is one-tenth or less of the rate of decrease in the "small crop" correction mode. With such a cropless correction mode, image stabilization can be performed without substantially reducing the number of pixels before and after correction.

[0070] 2-3. Overall Operation The digital camera 1 of this embodiment performs image stabilization using the BIS function in combination with distortion and tilt correction by the image correction unit 143 as described above. The overall operation of the image stabilization of the digital camera 1 of this embodiment will be explained with reference to Figure 9. Figure 9 is a flowchart illustrating the overall operation of the image stabilization of the digital camera 1 of this embodiment.

[0071] The process shown in the flowchart of Figure 9 is initiated, for example, when the interchangeable lens 200 is attached to the camera body 100 and the cropless correction mode is selected by user operation using the menu screen in Figure 4. Each process in this flowchart is executed in parallel with operations such as video recording by, for example, the camera control unit 140. The camera control unit 140 repeatedly executes the processes shown in this flowchart at a predetermined period. The predetermined period is, for example, the frame period, and is, for example, 1 / 30 to 1 / 60 of a second.

[0072] The camera control unit 140 communicates with the lens control unit 240 of the interchangeable lens 200 via the body mount 150 and lens mount 250 to acquire lens status data (S1). The lens control unit 240 reads the lens status data in response to a request from the camera control unit 140, for example, and transmits it to the camera body 100.

[0073] Lens status data, for example, includes the focal length corresponding to the zoom state of the interchangeable lens 200 and the focus position corresponding to the focus state, and is stored in the RAM 241 of the interchangeable lens 200.

[0074] The camera control unit 140 calculates distortion correction parameters to be used for distortion correction based on lens state data acquired in step S1 (S2). For example, the distortion correction parameters include distortion correction coefficients used for coordinate transformation of each pixel that deforms the image by distortion correction.

[0075] Figure 10 is a diagram illustrating the calculation process (S2) for distortion correction parameters in the digital camera 1. In distortion correction, for example, as shown in Figure 10(A), in the image data 10, the coordinates of pixels R are calculated based on the distance d from the distortion correction center C, which corresponds to the intersection point of the optical axis and the imaging plane of the image sensor 110, to each pixel P, and are transformed from the coordinates of pixels P according to the deformation of the image.

[0076] In Figure 10(A), (Cx,Cy) represents the coordinates of the distortion correction center C, (Px,Py) represents the coordinates of pixel P, and (Rx,Ry) represents the coordinates of pixel R. For example, in image data 10, pixel P is located in the image region output by distortion correction, and pixel R is located in the image formation region 20 referenced in distortion correction (see Figure 6).

[0077] In step S2, the camera control unit 140 calculates a distortion correction coefficient e for each pixel P, based on the distance d from the distortion correction center C and the distortion correction data 31 as shown in Figure 6(B), for example, as shown in Figure 10(B). For example, the central position Pc of the entire image in the image data 10 is used as the distortion correction center C. The distance d is calculated, for example, as the Euclidean distance and indicates the number of pixels corresponding to the image height in the distortion correction data 31 in Figure 6(B).

[0078] For example, before executing the process in this flowchart, distortion characteristic data 30, as shown in Figure 6(A), may be acquired for each combination of focal length and focus position through data communication with the interchangeable lens 200 and stored in RAM 141 or the like. In step S2, the camera control unit 140 may read the distortion characteristic data 30 corresponding to the focal length and focus position in the lens state data, and acquire distortion correction data 31 as the inverse characteristic of the distortion aberration characteristics from the distortion characteristic data 30 by interpolation processing or the like as appropriate.

[0079] The camera control unit 140 sets various parameters in the BIS processing unit 183 (Figure 3) to control the image stabilization operation performed by the BIS processing unit 183 in the digital camera 1 (S3). The camera control unit 140 may calculate the angle range that can be corrected by the BIS function in the pitch and yaw directions based on information indicating the element drive range in the BIS function and the focal length of the interchangeable lens 200 in the lens status data, and set this in the BIS processing unit 183. The information indicating the element drive range may be stored in advance in the flash memory 142 or the like. The camera control unit 140 may obtain the current image stabilization amount from the BIS processing unit 183, such as the amount of image stabilization at the time of execution of step S3, as the amount of image stabilization corresponding to the state in which blur is being corrected by the BIS function.

[0080] The BIS processing unit 183 generates shake detection signals in the pitch, yaw, and roll directions using the integrator 408 based on the detection results of the gyro sensor 184 of the camera body 100. Based on the generated shake detection signals, the BIS processing unit 183 calculates shake correction amounts for each direction as horizontal and vertical movement amounts of the image sensor 110, respectively, according to the amount of shake indicated by the shake detection signals in the yaw and pitch directions. Furthermore, the BIS processing unit 183 calculates a shake correction amount in the roll direction as the amount of movement of the image sensor 110 in the roll direction. The BIS processing unit 183 may limit the calculated shake correction amounts to, for example, the correctable angle range set as described above.

[0081] The BIS processing unit 183 causes the sensor drive unit 181 to translate the image sensor 110 according to the calculated horizontal and vertical shake correction amounts. The BIS processing unit 183 causes the sensor drive unit 181 to rotate the image sensor 110 according to the shake correction amount calculated for the amount of shake in the roll direction. The BIS processing unit 183 performs the above image stabilization operations as needed, for example, according to the detection results of the gyro sensor 184, based on various parameters set by the camera control unit 140.

[0082] The camera control unit 140 acquires, for example, the amount of tilt-shift caused by camera body 100 shaking, as indicated by the yaw and pitch shake detection signals (S4), from the BIS processing unit 183. For example, this tilt-shift shake amount is acquired based on the exposure shake detection signal corresponding to the exposure synchronization signal.

[0083] The camera control unit 140 performs calculation processing for tilt correction parameters to be used for tilt correction based on the acquired amount of tilt blur, etc. (S5). The tilt correction parameters include, for example, parameters for projection transformation that deform the image according to the amount of tilt blur. In step S5 of this embodiment, the camera control unit 140 does not calculate tilt correction parameters if the enlarged image area 13 after distortion correction cannot be obtained. This determination is made according to the distortion correction parameters, etc. calculated in step S2. Details of the tilt correction parameter calculation process (S5) will be described later.

[0084] The camera control unit 140 performs an image correction unit 143 to deform the image using the distortion correction parameter calculated in step S2 and the tilt correction parameter calculated in step S5 (S6). In this embodiment, the image correction unit 143 applies distortion correction and tilt correction calculations simultaneously to the image data 10 on which image stabilization operation by the BIS function has been performed in accordance with the image stabilization control (S3). Furthermore, the image correction process in this embodiment calculates a coordinate transformation for each pixel to deform the image, so as to determine the corresponding pixel in the image formation region 20 before distortion correction from each pixel in the output region 12 after tilt correction.

[0085] In tilt correction, the projection transformation is calculated according to equation (1) below, using the tilt correction parameters calculated in step S5. The tilt correction parameters include the rotation matrix R and translational displacement t of the digital camera 1. In equation (1), for example, the coordinates (xa, ya, za) represent the 3D coordinates projected onto the coordinates (xb, yb) on the 2D image by the projection transformation. The coordinates (xb, yb) represent the coordinates in the output region 12 (Figure 8) after tilt correction. f represents the focal length in the lens state data of the interchangeable lens 200.

number

[0086] Furthermore, for example, the coordinates (x,y) corresponding to the coordinates (Px,Py) of pixel P in distortion correction are calculated from the coordinates (xa,ya,za) according to the following equation (2).

number

[0087] In step S6, the image correction unit 143 performs calculations using equations (1) and (2) for, for example, the number of pixels in the output area 12. If, for example, the tilt correction parameter has not been calculated, the image correction unit 143 does not perform the calculations using equations (1) and (2), and only performs the distortion correction calculation.

[0088] For example, in distortion correction, the coordinates (Rx, Ry) of pixel R in the image formation region 20 are calculated from the coordinates (Px, Py) of pixel P in the tilt correction reference region 11 using the distortion correction coefficient e calculated in step S2, according to the following equation (3). The image correction unit 143 performs this calculation for the number of pixels in the output region 12, for example, in Figure 8. Rx=e(Px-Cx)+Cx,Ry=e(Py-Cy)+Cy (3)

[0089] Following the above processing, distortion correction parameters are calculated according to the lens condition data acquired from the interchangeable lens 200 (S1, S2). Furthermore, based on the amount of blur correction in the horizontal, vertical, and roll directions of the image sensor 110, image stabilization operation by the BIS function is performed according to the image stabilization control (S3). Then, the amount of tilt blur in the yaw and pitch directions is acquired (S4), and the tilt correction parameters are calculated according to the amount of tilt blur, etc. (S5). Based on the calculated correction parameters, the image correction unit 143 performs correction on the image data 10 from the image sensor 110 (S6). As a result, when the cropless correction mode is selected in the EIS function, distortion correction and tilt correction can be efficiently performed by deforming the image using the calculated correction parameters.

[0090] For example, in this embodiment, the BIS function performs image stabilization, correcting the horizontal and vertical directions of the image sensor 110 according to the amount of shake in the yaw and pitch directions, and correcting the amount of shake in the roll direction, after which the image correction unit 143 performs tilt correction, etc. (S6). As a result, in addition to distortion correction, the image correction unit 143 performs tilt correction only using the cropless EIS function as image stabilization, thereby correcting image distortion caused by tilt shake while suppressing a reduction in the field of view.

[0091] In step S6 described above, an example was shown in which distortion correction and tilt correction are applied to the image data 10 at the same time. However, tilt correction may also be performed on the image data 10 after distortion correction has been applied. In this case, distortion correction may be performed before the calculation process of the tilt correction parameters (S5). For example, in step S6, the distortion correction calculation may be performed in the same way as the process in step S21 described later. Also, in the digital camera 1, the distortion correction function may be forcibly enabled when the cropless correction mode is selected.

[0092] 2-4. Calculation process for tilt correction parameters The calculation process for the tilt correction parameter in step S5 of Figure 9 will be explained using Figures 11 and 12.

[0093] Figure 11 is a flowchart illustrating the calculation process (S5) for tilt correction parameters in the digital camera 1. The process shown in this flowchart starts, for example, when lens condition data of the interchangeable lens 200 is acquired in step S1 of Figure 9, distortion correction parameters are calculated in step S2, and tilt blur amount is acquired in step S4.

[0094] First, the camera control unit 140 calculates the excess region resulting from distortion correction based on the coordinates indicating the position of the image forming region 20 and the distortion correction parameters in the image data 10 from the image sensor 110 (S21). Figure 12 is a diagram illustrating the calculation process of the excess region (S21) in the calculation process of the tilt correction parameters (S5). Figure 12 illustrates the excess region 15 resulting from distortion correction in the image data 10. In the image data 10, the excess region 15 is the region of the image area 13 after distortion correction that does not include the image forming region 20.

[0095] In this embodiment, the camera control unit 140 calculates the distortion-corrected coordinates for each of the eight pixels at the vertices 41 and midpoints 42 of each side of the rectangular image forming region 20, for example, as shown in Figure 12 (S21). In this way, the coordinates of the region that may become the surplus region 15 are partially calculated. This improves the calculation speed for calculating the surplus region 15 and reduces the processing load on the camera control unit 140.

[0096] In step S21, for example, in the same operation as the distortion correction formula (3) of the image correction process (S6), the coordinates of each pixel to be calculated are input as (Px, Py) from the image formation region 20, and coordinate conversion is performed so that the coordinates in the surplus region 15 are output as (Rx, Ry). In this case, the distortion correction coefficient e changes so as to increase according to the distance d from the distortion correction center C to each pixel to be calculated, contrary to the example of FIG. 10(B), and e>1 in the peripheral part of the image in the correction of barrel distortion.

[0097] Next, the camera control unit 140 determines whether a surplus region 15 has occurred outside the range of the image formation region 20 in the image data 10 by comparing each coordinate calculated in step S21 with the coordinate corresponding to the coordinate in the image formation region 20 (S22). For example, the camera control unit 140 determines that there is a surplus region 15 (YES in S22) when all the calculated coordinates are outside the range of the image formation region 20.

[0098] If there is a surplus region 15 (YES in S22), the camera control unit 140 calculates the panning correction parameters (S23). The camera control unit 140 calculates, for example, the rotation matrix R and the translational movement amount t (t x ,t y ,t z ) used for projective transformation according to the following formulas (4) and (5). In formulas (4) and (5), pitch and yaw respectively indicate the amounts of panning blur (angles) in the pitch direction and the yaw direction. The x c ,y c in formula (5) indicate, for example, the positions in the horizontal direction X and the vertical direction Y of the optical axis center corresponding to the intersection of the optical axis of the optical system and the imaging surface of the image sensor 110, based on the center position Pc of the entire image in the image data 10. The camera control unit 140 obtains x c ,y c from, for example, the BIS processing unit 183. For example, x c ,y cThis value is calculated as the amount of movement on the imaging surface obtained by converting the output from the integrator 408, or based on the output from the position sensor 182.

number

number

[0099] After calculating the tilt correction parameters (S23), the camera control unit 140 terminates the processing in this flowchart and proceeds to step S6 in Figure 9.

[0100] Furthermore, if there is no surplus area 15 (NO in S22), the camera control unit 140 does not calculate the tilt correction parameters (S23) and terminates the processing of this flowchart.

[0101] According to the above process, the excess region is calculated based on the image formation region 20 in the image data 10 and the distortion correction parameter (S21). If there is an excess region 15 resulting from distortion correction (YES in S22), the tilt correction parameter is calculated (S23). On the other hand, if there is no excess region 15 (NO in S22), the tilt correction parameter is not calculated. As a result, when an excess region 15 is generated outside the range of the image formation region 20 due to distortion correction, tilt correction can be performed in addition to distortion correction in the image correction process (S6 in Figure 9) using the calculated tilt correction parameter.

[0102] If there is an excess area 15 (YES in S22), tilt correction can be performed using the cropless EIS function with the image area 13 enlarged by distortion correction from the image forming area 20 by the amount of the excess area 15. This makes it possible to perform tilt correction while suppressing a reduction in the field of view.

[0103] In step S23, when calculating the tilt correction parameters, an upper limit may be set on the amount of blur correction due to tilt correction. For example, if an upper limit on the amount of blur correction is set according to the size of the excess region 15, and the amount of blur correction that cancels out all of the tilt blur acquired in step S4 of Figure 9 is greater than the upper limit, the amount may be limited to that upper limit. Furthermore, the pixels for which the coordinates after distortion correction are calculated from the image forming region 20 in the excess region calculation process (S21) are not limited to the eight points mentioned above. For example, by performing the calculation on more than eight pixels, the upper limit on the amount of blur correction can be set with greater precision.

[0104] 2-5. Variations In the above Embodiment 1, an example was described in which image stabilization operation by the BIS function and tilt correction operation by image correction processing (S6) were performed. Such image stabilization operation may also be performed by the coordinated operation of the BIS function and the OIS function. A modification of this Embodiment 1 will be explained with reference to Figure 13.

[0105] Figure 13 is a flowchart illustrating the operation of a modified digital camera 1 according to Embodiment 1. In addition to the same operations as the digital camera 1 of Embodiment 1 (S1-S2, S4-S6), the digital camera 1 of this modified version also performs a calculation process (S30) for image stabilization parameters used for image stabilization operation using the BIS function and OIS function. In this modified version, the digital camera 1 controls the image stabilization operation using the BIS function and OIS function (S3A) instead of controlling the image stabilization operation using the BIS function in Embodiment 1 (S3 in Figure 9).

[0106] For example, the camera control unit 140 calculates image stabilization parameters (S30) based on lens status data of the interchangeable lens 200 acquired in step S1, information indicating the element drive range in the BIS function, and information indicating the lens drive range in the OIS function. The image stabilization parameters include, for example, a correction distribution that distributes the amount of image stabilization between the BIS processing unit 183 and the OIS processing unit 223. The correction distribution is calculated for the horizontal, vertical, and roll directions of the image sensor 110. The image stabilization parameters may also include frequency bands corresponding to the division of responsibility between the BIS processing unit 183 and the OIS processing unit 223 for the frequency components included in the angular velocity signal from the gyro sensor 184 or 224.

[0107] The camera control unit 140 controls the image stabilization operation by the BIS function and OIS function using the calculated image stabilization parameters (S3A). For example, the camera control unit 140 sets the gain indicating the allocation of the BIS processing unit 183 in the calculated stabilization distribution to the BIS processing unit 183. The BIS processing unit 183 calculates the amount of image stabilization by multiplying the shake detection signal from the integrator 408 by the gain. The camera control unit 140 may also set a frequency band to be cut off in the HPF 406, etc., of the BIS processing unit 183. In addition, in this modified example, when the tilt correction parameters are calculated (S5, S23), the camera control unit 140 further calculates x in equation (5) based on the information obtained from the OIS processing unit 223. c ,y c You may calculate this.

[0108] Furthermore, the camera control unit 140 transmits the calculated image stabilization parameters to the interchangeable lens 200 via the body mount 150 and the lens mount 250. The camera control unit 140 may also set the image stabilization parameters to the OIS processing unit 223 via the lens control unit 240.

[0109] Following the above processing, image stabilization parameters are calculated (S3A, S30) to coordinately control the BIS and OIS functions, and image stabilization operations are performed by each function according to the image stabilization parameters. This allows, for example, the optical image stabilization function to perform image stabilization using the translational and roll stabilization amounts of the image sensor 110, in addition to tilt correction, and the image correction processing (S6) to perform tilt correction using the cropless EIS function.

[0110] 3. Summary As described above, the digital camera 1 (imaging device) in this embodiment includes an image sensor 110 (image sensor) that has an imaging area in which a subject image is formed via an optical system and captures the subject image to generate image data, gyro sensors 184, 224 (detection units) that detect the amount of blur of the digital camera 1, and an image correction unit 143 (image processing unit) that corrects image blur by adjusting the part of the image data that outputs an image according to the amount of blur detected by the detection unit. In the image correction unit 143, distortion correction according to the distortion aberration of the optical system is performed on the image area indicated by the image data 10 (S2, S6). In the image area in which distortion correction has been performed, the image correction unit 143 performs image blur correction using a cropless EIS function as an example of image blur correction, without using a correction area provided to cut out the image of a part such as the image of the output area 12 inside the image formation area 20 (range corresponding to the imaging area) (S5, S6) (see Figure 8).

[0111] With the above imaging device, in the image region where distortion correction has been performed, image stabilization is performed by a cropless EIS function without using a correction region provided inside the image forming region 20. This allows for correction of image distortion due to camera shake while suppressing a reduction in the field of view. For example, the output region 12 produced by the cropless EIS function can be output as a region in which the number of pixels is not substantially reduced from the image forming region 20, similar to the case where a cropping amount Eo is not set in the reference region 21 setting of the image data 10 shown in Figure 5.

[0112] In this embodiment, the digital camera 1 further includes an operation unit 130 that inputs a user operation to select a correction mode to be used for image blur correction from a plurality of correction modes (image blur correction modes). The plurality of correction modes include a "cropless" correction mode (first image blur correction mode) that performs image blur correction without using a correction area provided inside the image forming area 20 (range corresponding to the imaging area), and "large crop" and "small crop" correction modes, i.e., correction modes with cropping (second image blur correction mode), that perform image blur correction using the said correction area (see Figure 4). As a result, the cropless correction mode can be selected by the user operation, and the cropless EIS function can be executed according to the user's selection.

[0113] In this embodiment, the cropless image blur correction mode performs image blur correction at a rate smaller than the rate at which the number of pixels decreases before and after image blur correction in the cropped correction mode, and outputs an image (partial image) of the output area 12. As a result, the cropless correction mode can output a higher resolution image than the cropped correction mode.

[0114] In this embodiment, the image correction unit 143 uses the enlarged image area 13 (an area enlarged beyond the range corresponding to the imaging area) which is expanded beyond the range of the image forming area 20 in the image area where distortion correction has been performed, to adjust the shape of the reference area 11 (an area referenced for outputting the image of the output area 12) according to the detected amount of blur. This allows for cropless EIS image correction without using a correction area corresponding to a predetermined cropping amount Eo provided inside the image forming area 20 (S5, S6) (see Figures 8 and 5). This avoids, for example, the narrowing of the output area 12 by the amount of the correction area, and allows for correction of image distortion due to camera shake while suppressing a reduction in the field of view. Furthermore, in this embodiment, when a type of blur occurs that changes the orientation of the digital camera 1 (imaging device) relative to the subject corresponding to the subject image, such as tilt blur, the image correction unit 143 (image processing unit) adjusts the shape of the reference area 11 (reference area) to cancel out the distortion of the output area 12 (partial image) caused by the orientation blur, thereby performing image blur correction (image blur correction) using cropless EIS function (S5, S6).

[0115] In this embodiment, the digital camera 1 further includes a sensor drive unit 181 (element drive unit) that performs optical image blur correction by moving the image sensor 110 in a plane perpendicular to the optical axis of the optical system, and an OIS drive unit 221 (lens drive unit) that performs optical image blur correction by moving the OIS lens 220 (correction lens) included in the optical system in a plane perpendicular to the optical axis. In optical image blur correction, the image sensor 110 or both the image sensor 110 and the OIS lens are moved to cancel out the detected amount of blur. The image correction unit 143 performs image blur correction in the image region where distortion correction has been performed by adjusting the shape of the reference region 11 that is referenced to output the image of the output region 12, without moving the part that outputs the image for the amount of blur that has been canceled out by optical image blur correction among the detected amount of blur (S5, S6).

[0116] As described above, the image correction unit 143 does not perform further correction by the EIS function for the amount of blur that has been offset by the optical image stabilization (BIS, OIS) function, which performs optical image blur correction. For example, the EIS function can perform tilt correction only for tilt blur, separate from the amount of blur correction by the optical image stabilization function. This allows for correction of image distortion caused by camera shake, such as tilt blur, while suppressing a reduction in the field of view.

[0117] In this embodiment, the distortion aberration of the optical system is negative at least in the peripheral part of the imaging area, and the image correction unit 143 performs distortion correction by expanding at least the peripheral area of ​​the image area indicated by the image data 10 to outside the range of the image forming area 20 corresponding to the imaging area (S2, S6). As a result, an expanded image area 13 is obtained by distortion correction, and cropless EIS function can be performed using the expanded image area 13.

[0118] In this embodiment, the digital camera 1 further includes a body mount 150 and a lens mount 250 (communication unit) that communicate with the optical system, and a camera control unit 140 (control unit) that controls the communication unit and the image correction unit 143. The camera control unit 140 acquires distortion characteristic data 30 (information on distortion aberration) of the optical system from the optical system via the body mount 150 and the lens mount 250, and based on the acquired information and information indicating the imaging area, when an excess area 15 outside the range of the image forming area 20 corresponding to the imaging area is detected in the image area where distortion correction has been performed (YES in S22), the camera control unit 140 causes the image correction unit 143 to perform image blur correction without using the correction area provided inside the image forming area 20 (S23, S5, S6) (see Figure 12). As a result, when an excess area 15 is detected, the image distortion due to camera shake can be corrected while suppressing a reduction in the angle of view, depending on the enlarged image area 13 obtained by distortion correction.

[0119] (Embodiment 2) Embodiment 2 of this disclosure will be described below with reference to Figures 14 to 15. Embodiment 1 described an example in which the digital camera 1 performs image stabilization operations other than tilt correction using an optical image stabilization function such as the BIS function, and performs tilt correction using a cropless EIS function. Embodiment 2 describes a digital camera 1 that, when operating in such a cropless correction mode, also performs image stabilization operations other than tilt correction using the EIS function.

[0120] Hereinafter, descriptions of the configuration and operation similar to those of the digital camera 1 according to Embodiment 1 will be omitted as appropriate, and the digital camera 1 according to this embodiment will be described.

[0121] 1. Overview In the image Im captured by the digital camera 1, trapezoidal distortion due to various types of tilt blur, such as changes in the viewing direction from the digital camera 1 to the subject, tends to occur in the peripheral areas of the image, for example, in wide-angle photography (see Figures 7(B) to (D)). Furthermore, in wide-angle photography with a relatively short focal length of the interchangeable lens 200, the correction angle corresponding to the amount of movement of the image sensor 110 by the BIS function becomes larger, which can also increase the peripheral distortion in the captured image Im. On the other hand, the inventors of this application have focused on the tendency for distortion to be less noticeable in telephoto photography with a relatively long focal length of the interchangeable lens 200.

[0122] The inventors of this invention have diligently conducted studies based on these considerations and have come to create the digital camera 1 of this embodiment. For example, as described above, when the focal length is relatively long, the effect of distortion due to tilt blur is relatively small in the captured image Im, and it is assumed that such distortion can be corrected even if a portion of the surplus area 15 (see Figure 12) generated by distortion correction is used for tilt correction.

[0123] Therefore, the digital camera 1 of this embodiment utilizes, for example, the area of ​​the surplus area 15 that is not used for tilt correction for other image stabilization functions using EIS, such as translation and / or rotation of the image area 22 as illustrated in Figure 5. This allows for efficient use of the enlarged image area 13, such as the surplus area 15 due to distortion correction, while suppressing the reduction of the field of view due to image cropping, and enabling other image stabilization functions using EIS in addition to tilt correction.

[0124] In this embodiment, the digital camera 1 performs image stabilization by changing the ratio between tilt correction in the EIS function and other image stabilization methods such as translation / rotation correction as described above, according to the focal length of the interchangeable lens 200. In this embodiment, the digital camera 1 dynamically determines the image correction ratio by the EIS function according to the focal length when the interchangeable lens 200 is zoomed in. Figure 14 is a diagram illustrating the operation of the digital camera 1 according to this embodiment.

[0125] Figure 14 illustrates the ratio data D1 that the digital camera 1 references in determining the image correction ratio. In Figure 14, the horizontal axis represents the focal length of the interchangeable lens 200, and the vertical axis represents the image correction ratio in the EIS function. For example, as shown in Figure 14, the ratio data D1 shows the image correction ratio for each focal length. In this embodiment, the ratio data D1 is generated such that as the focal length increases within a predetermined range (for example, from f1 to f2 in Figure 14), the image correction ratio R1 for tilt correction decreases and the image correction ratio R2 for translation / rotation correction increases. The operation of the digital camera 1 with respect to these image correction ratios R1 and R2 will be described below.

[0126] 2. Operation Figure 15 is a flowchart illustrating the operation of the digital camera 1 according to Embodiment 2. In addition to the same processing (S1-S6) as in Embodiment 1 (Figure 9), or alternatively, the digital camera 1 of this embodiment performs processing related to translation / rotation correction in addition to tilt correction by the EIS function (S10, S11-S12, S3B, S5A). The processing in this flowchart starts, for example, when the digital camera 1 is started up.

[0127] The camera control unit 140 of the digital camera 1 generates ratio data D1 as shown in Figure 14 (S10) based on, for example, distortion characteristic data 30 of the interchangeable lens 200 (see Figure 6(A)). In step S10, the camera control unit 140 acquires distortion characteristic data 30 corresponding to the focal length from the interchangeable lens 200 via the body mount 150 and the lens mount 250. The camera control unit 140 generates ratio data D1 by setting image correction ratios R1 and R2 based on, for example, the distortion characteristics shown in the distortion characteristic data 30 for each focal length. The image correction ratios R1 and R2 are set according to the focal length under normalization conditions such as the sum being "1".

[0128] For example, the image correction ratios R1 and R2 at each focal length are set from the perspective of the amount of horizontal, vertical, and roll shake of the image sensor 110, which can be offset within the element drive range by the BIS function, in addition to the effect of image distortion due to tilt shake. Below, we will explain an example in which the maximum or minimum values ​​of each image correction ratio R1 and R2 are set at focal lengths f1 and f2 in Figure 14, assuming a situation where there is relatively large camera shake, such as when a user is shooting a video while walking. In this example, the image correction ratios R1 and R2 are set so that the correction angle is 1.4 degrees or more in the translational directions, such as the horizontal and vertical directions of the image sensor 110, and so that the correction angle is 1.0 degree or more in the roll direction, based on the above assumption.

[0129] For example, at a focal length f1 (e.g., 20 millimeters), if the correction angle calculated from the amount of movement of the image sensor 110 within the element drive range is 3 degrees or more, the lower limit of the correction angle in the walking shooting situation described above can be ensured by the BIS function alone. For this reason, in this example, at a focal length f1, the image correction ratio R1 for tilt correction is set to a maximum value of 100%, and the image correction ratio R2 for translation / rotation correction is set to a minimum value of 0%.

[0130] On the other hand, for example, at a focal length f2 (e.g., 50 millimeters), if the correction angle within the element drive range by the BIS function is about 1.4 degrees, it may become difficult to secure the lower limit of the correction angle with the BIS function alone. Also, as mentioned above, as the focal length increases, the effect of distortion due to tilt blur becomes less apparent in the peripheral parts of the captured image Im. In this example, for a focal length f2, the image correction ratio R1 for tilt correction is set to 0%, and the image correction ratio R2 for translation / rotation correction is set to 100%.

[0131] In this example, the image correction ratios R1 and R2 are set to change linearly between their respective set values ​​for each focal length f1 and f2 within the range of focal lengths f1 to f2. In this example, the camera control unit 140 sets the image correction ratios R1 and R2 for each focal length as the ratio of the area that can be used for tilt correction and translation / rotation correction, respectively, in the surplus area 15 by the cropless EIS function. The camera control unit 140 generates ratio data D1 for the EIS function based on the setting of the image correction ratios R1 and R2 as described above, and stores the generated ratio data D1 in RAM 141 or the like (S10). The image correction ratios R1 and R2 are not limited to the example of changing linearly as described above, but may also be set to change non-linearly (for example, curvilinearly) according to the focal length.

[0132] After generating EIS ratio data D1 (S10), the camera control unit 140 acquires lens state data from the interchangeable lens 200 and calculates distortion correction parameters (S2), in parallel with operations such as video recording, similar to the operation in Embodiment 1 (Figure 9). The lens state data includes the focal length corresponding to the zoom state of the interchangeable lens 200 by the zoom lens 210.

[0133] In this embodiment, the camera control unit 140 determines the image correction ratios R1 and R2 corresponding to the focal length in the ratio data D1 for tilt correction and translation / rotation correction by the EIS function, based on the focal length of the acquired lens state data (S11).

[0134] Furthermore, the camera control unit 140 performs a process to determine the correction distribution between the BIS processing unit 183 and the image correction unit 143 for each of the horizontal, vertical, and roll directions of the image sensor 110 (S12). For example, in step S12, the camera control unit 140 acquires information such as the element drive range of the BIS function from the flash memory 142 or the like. In addition to the acquired information and lens state data, the camera control unit 140 determines the correction distribution based on the translation / rotation correction image correction ratio R2 (S12). For example, depending on the image correction ratio R2, the cropping amount Eo by the EIS function may be set outside the range of the image forming region 20 as shown in Figure 5, and the set cropping amount Eo may be used to determine the correction distribution.

[0135] The above correction distribution includes a BIS ratio indicating the distribution to the BIS processing unit 183 and an EIS ratio indicating the distribution to the image correction unit 143, as ratios for pre-distributing the amount of blur correction in the horizontal, vertical, and roll directions of the image sensor 110. The correction distribution is determined, for example, under normalization conditions where the sum of the BIS ratio and the EIS ratio is "1", according to the upper limit of the amount of blur correction that can be distributed to the BIS function and the EIS function, respectively. For example, the BIS ratio and the EIS ratio of the correction distribution can be determined as a ratio similar to the respective upper limits (S12).

[0136] The camera control unit 140 controls the image stabilization operation using the BIS function according to the BIS ratio in the determined correction distribution (S3B). The camera control unit 140 sets, for example, a gain indicating the BIS ratio in the BIS processing unit 183. The BIS processing unit 183 calculates the amount of image stabilization by multiplying the shake detection signal from the integrator 408 by the gain set for each period in this flowchart. The gains for the shake detection signals in the yaw, pitch, and roll directions may be the same or may be set separately.

[0137] Subsequently, the camera control unit 140 obtains the amount of tilt shake from the BIS processing unit 183, for example, as in the example in Figure 9 (S4).

[0138] The camera control unit 140 calculates image correction parameters, such as tilt correction parameters and parameters used for translation / rotation correction by the EIS function, based on the image correction ratios R1, R2, etc., determined in step S12 (S5A).

[0139] For example, the camera control unit 140 calculates tilt correction parameters (S5A) from projection transformation parameters calculated based on the amount of tilt shake, etc., similar to step S5 in Figure 9, and restricts them to values ​​that allow image deformation within the range of image correction ratio R1 in the surplus region 15. The camera control unit 140 also calculates translation / rotation correction parameters (S5A) from gains, for example, that represent the EIS ratio of the correction distribution determined in step S12, and restricts them to values ​​that allow movement of the image region 22 within the range of image correction ratio R2 in the surplus region 15. For example, the gain calculated according to the EIS ratio as a translation / rotation correction parameter is multiplied by the shake detection signal input from the integrator 408 of the BIS processing unit 183 in the image correction unit 143.

[0140] The camera control unit 140 performs calculation processing as an image correction unit 143 using the distortion correction parameters calculated in step S2 and the image correction parameters calculated in step S5A, instead of the tilt correction parameters of Embodiment 1 (S6). In step S6 of this embodiment, in addition to the processing of deforming the image as distortion correction and tilt correction, the image region 22 is moved by at least one of translational movement and rotational movement as translation / rotation correction. After executing such image correction processing (S6), the camera control unit 140 terminates the processing of this flowchart.

[0141] Based on the above processing, for example, ratio data D1 including image correction ratios R1 and R2 for each focal length in the EIS function is generated from distortion characteristic data 30 corresponding to the focal length of the interchangeable lens 200 (S10). Based on this ratio data D1 and the focal length in the lens state data of the interchangeable lens 200, the image correction ratio R1 for tilt correction and the image correction ratio R2 for translation / rotation correction in the EIS function are determined (S11). For example, for the horizontal, vertical, and roll directions of the image sensor 110, the correction distribution between the BIS processing unit 183 and the image correction unit 143 is determined according to the image correction ratio R2, etc. (S12).

[0142] Furthermore, based on the BIS ratio in the determined correction distribution, image stabilization control is performed by the BIS function (S3B), and the amount of tilt blur is acquired (S4). Then, based on the amount of tilt blur and the image correction ratios R1 and R2, tilt correction parameters and translation / rotation correction parameters are calculated (S5A), and image correction processing is performed using these image correction parameters and distortion correction parameters (S6). As a result, for example, in the excess region 15 created by distortion correction, tilt correction and translation / rotation correction by the EIS function can be performed according to the image correction ratios R1 and R2, respectively.

[0143] For example, at a focal length f12 as shown in Figure 14, even if the EIS function performs tilt correction using the excess region 15 generated by distortion correction within the range of the image correction ratio r1, based on ratio data D1 corresponding to the distortion characteristics of the interchangeable lens 200, the amount of tilt blur can be offset. On the other hand, at a focal length f12, even if there is excess region 15 that is not used for tilt correction, the EIS function can efficiently utilize the excess region 15, etc., generated by distortion correction by performing translation / rotation correction within the range of the image correction ratio r2. In this way, for example, tilt correction can be performed efficiently while suppressing the reduction of the angle of view by using a cropless EIS function.

[0144] In the above example, an example was described in which the ratio data D1 of the EIS function is generated to include image correction ratios R1 and R2 (S10). The ratio data D1 may include only one of the image correction ratios R1 or R2, and the other may be appropriately calculated from, for example, the relationship of how the two ratios R1 and R2 change with respect to focal length. In addition, the ratio data D1 may be generated in advance, for example, before the execution of this flowchart and stored in flash memory 142 or the like.

[0145] The above describes an example in which image stabilization control (S3B) and image correction processing (S6) are performed using the BIS function at the BIS ratio and EIS ratio of the correction distribution determined in step S12. The corrections performed by the BIS processing unit 183 and the image correction unit 143 are not limited to the above example. For example, if a remaining correction amount remains even after performing translation / rotation correction using the EIS function in image correction processing (S6), image stabilization may be performed using the BIS function for that remaining correction amount according to the element drive range, etc.

[0146] The above describes an example in which the EIS function performs tilt correction and translation / rotation correction based on image correction ratios R1 and R2. In addition to tilt correction according to such image correction ratios, the EIS function may also perform image stabilization operations using image processing other than translation / rotation correction.

[0147] The above example illustrates a scenario where the processing in this flowchart begins when the digital camera 1 is turned on. However, this processing may also begin, for example, when the interchangeable lens 200 is attached to the camera body 100.

[0148] 3. Summary As described above, the digital camera 1 (an example of an imaging device) in this embodiment includes an image sensor 110 (an example of an image sensor) having an imaging area in which a subject image is formed via an optical system, and generating image data by capturing a subject image; gyro sensors 184 and 224 (each an example of a detection unit) for detecting the amount of camera shake of the digital camera 1; an image correction unit 143 (an example of an image processing unit) that corrects image shake by adjusting the part of the image data that outputs an image according to the amount of shake detected by the detection unit; and a camera control unit 140 (an example of a control unit) that controls the image shake correction performed by the image correction unit 143. In the image correction unit 143, distortion correction according to the distortion aberration of the optical system is performed on the image area indicated by the image data. The camera control unit 140 changes the ratio between tilt correction and translation / rotation correction (each an example of first and second image shake correction, respectively) performed in the image area that has been distortion corrected by the image correction unit 143 in the EIS function, for example, according to the focal length of the optical system (S11). Tilting correction corrects the distortion of the image in the distortion-corrected image area (see Figure 8). Translation / rotation correction moves the part of the image output area within that image region (see Figure 5).

[0149] According to the digital camera 1 described above, for example, depending on the focal length of the optical system in the interchangeable lens 200, the ratio between tilt correction and translation / rotation correction performed in the distortion-corrected image area, such as the excess area 15, by the EIS function is changed. This allows for efficient correction of image distortion caused by camera shake while suppressing a reduction in the angle of view. For example, depending on the focal length at which the effect of image distortion due to tilt shake changes, the excess area 15 created by distortion correction can be efficiently utilized by changing the ratio between tilt correction and translation / rotation correction in the cropless EIS mode.

[0150] In this embodiment, the optical system includes a zoom lens 210 of the interchangeable lens 200 (see Figure 2). When the focal length changes due to the zoom lens 210, the camera control unit 140 changes the image correction ratios R1 and R2 (an example of the ratio between tilt correction and translation / rotation correction, respectively) (S11, see Figure 14). This allows the image correction ratios R1 and R2 to be changed according to the zoom state of the zoom lens 210, for example.

[0151] In this embodiment, the camera control unit 140 changes the image correction ratios R1 and R2 such that as the focal length increases within a predetermined range, the image correction ratio R2 (an example of the ratio of the second image blur correction in the ratio between the first image blur correction and the second image blur correction) increases (S11, see Figure 14). As a result, for example, as the focal length increases, the upper limit of the correction angle by the BIS function decreases, while the effect of image distortion due to tilt blur tends to become less apparent, allowing tilt correction and translation / rotation correction by the EIS function to be performed efficiently in the surplus region 15.

[0152] In this embodiment, the camera control unit 140 causes the image correction unit 143 to perform tilt correction (S4-S6) to correct image distortion caused by changes in posture, where the line of sight from the digital camera 1 to the subject is tilted, based on the detected amount of blur. This allows for correction of image distortion caused by tilt blur during image capture.

[0153] In this embodiment, the image correction unit 143 performs tilt correction using a cropless EIS function in the distortion-corrected image region without using a correction region provided within the image formation region 20 (the range corresponding to the imaging region) to crop out parts of the image such as the image region 22 (S5A, S6) (see Figures 5 and 8). This allows for correction of image distortion due to tilt blur while suppressing a reduction in the field of view.

[0154] In this embodiment, the digital camera 1 further includes a sensor drive unit 181 (element drive unit) that performs optical image blur correction by moving the image sensor 110 in a plane perpendicular to the optical axis of the optical system, and an OIS drive unit 221 (lens drive unit) that performs optical image blur correction by moving the OIS lens 220 (correction lens) included in the optical system in a plane perpendicular to the optical axis. In this embodiment, in optical image blur correction, the image sensor 110 is moved to cancel out the detected amount of blur (S3B). The image correction unit 143 performs translation / rotation correction by EIS function by moving the image area 22 (an example of the part that outputs the image) in the distortion-corrected image area according to the detected amount of blur (S5A, S6). As a result, similar to translation / rotation correction by EIS function, the translational and roll directions of the image sensor 110 can also be corrected by, for example, the BIS function by the sensor drive unit 181. Furthermore, the EIS function may perform correction in only one of the translational or rotational directions; for example, the correction of the other direction may be performed by the BIS function.

[0155] (Other embodiments) As described above, Embodiment 1 has been explained as an example of the technology disclosed in this application. However, the technology in this disclosure is not limited to this and can be applied to embodiments that are modified, substituted, added, or omitted as appropriate. Furthermore, it is possible to create new embodiments by combining the components described in Embodiment 1 above. Therefore, other embodiments are described below as examples.

[0156] In the above embodiment 1, an example was described in which the digital camera 1, for example, displays the option to select a cropless correction mode on the menu screen shown in Figure 4, regardless of the interchangeable lens 200, when selecting various correction modes using the EIS function. In this embodiment, the enablement and disablement of the cropless correction mode in the digital camera 1 may be switched depending on the attached interchangeable lens 200. For example, the camera control unit 140 may obtain information from the interchangeable lens 200 indicating the type of interchangeable lens 200 and / or lens characteristics, and when it determines that an enlarged image area 13 can be obtained by distortion correction, it may display the option to select a cropless correction mode.

[0157] In the above embodiment, an example was described in which, when selecting various correction modes using the EIS function, the availability of the cropless correction mode is switched based on information obtained from the interchangeable lens 200. In this embodiment, for example, if information indicating lens characteristics such as distortion cannot be obtained from the interchangeable lens 200, the EIS function in that correction mode does not need to be executed even if the cropless correction mode is selected, and the cropless correction mode may be made unavailable.

[0158] Furthermore, not limited to the above example, information indicating whether the cropless EIS function can be used may be used for each interchangeable lens 200 to manage whether the function is enabled or disabled, or whether the cropless correction mode can be selected. Such information may be stored in the flash memory 142 of the camera body 100, or it may be obtained from the interchangeable lens 200. For example, the above-mentioned enable / disable status may be set according to the peripheral light falloff limit of the interchangeable lens 200. Also, in the digital camera 1, a function to correct peripheral light falloff may be enabled when the cropless EIS function is enabled. In addition, for example, the element drive range of the BIS function may be limited when the amount of peripheral light falloff due to vignetting of the optical system exceeds a predetermined allowable value.

[0159] In the above embodiment 2, an example was described in which the digital camera 1 changes the image correction ratios R1 and R2 of tilt correction and translation / rotation correction in the EIS function according to the focal length of the interchangeable lens 200. In this embodiment, for example, correction by the EIS function may be performed on a portion of the amount of blur that has not been offset by the BIS function. The camera control unit 140 may obtain the remaining amount of correction by the BIS function from the BIS processing unit 183 for each amount of blur in the yaw direction and pitch direction, and correct the remaining amount of correction by, for example, the EIS function with cropping. In this embodiment, for example, the cropless EIS function may be executed when the decrease in the number of pixels of the image output for recording and / or display before and after tilt correction is one-tenth or less of the decrease in the correction mode of "small crop".

[0160] In the above embodiment 2, an example was described in which translation / rotation correction by the EIS function is performed in cropless correction mode. Translation / rotation correction by the EIS function may also be performed in cropped correction mode, and may be performed not only in the excess area 15 due to distortion correction but also in the enlarged image area 13.

[0161] In this embodiment, for example, when cropless and cropped EIS functions are used in combination, the image area 22 output by the cropped EIS function does not need to be moved for the amount of blur that has been offset by an optical image stabilization function such as the BIS function. This also allows for correction of image distortion due to camera shake while suppressing the reduction of the field of view by the cropped EIS function. Furthermore, even with the cropped EIS function, the enlarged image area 13 after distortion correction can be used to crop the image area 22.

[0162] In the above embodiment, an example was described in which, in addition to tilt correction by cropless EIS function, the amount of blur in the yaw, pitch, and roll directions is corrected by cropped EIS function. In this embodiment, furthermore, image distortion due to, for example, the rolling shutter phenomenon may be corrected by the EIS function. For example, such correction may be performed using multiple projection transformation matrices corresponding to the characteristics of such distortion as correction parameters.

[0163] In the above embodiment 2, an example was described in which camera shake correction control by the BIS function (S3B) and image correction processing by the EIS function (S6) are performed based on the correction distribution determined between the BIS processing unit 183 and the image correction unit 143 (S11). In this embodiment, for example, camera shake correction may be further performed by the OIS processing unit 223 for the amount of translational shake of the image sensor 110. For example, the BIS ratio in step S11 of Figure 15 may be further distributed by the correction distribution between the BIS processing unit 183 and the OIS processing unit 223, similar to the shake correction parameters in the modified example of embodiment 1 (S30 in Figure 13). In this embodiment, in optical image shake correction, at least one of the image sensor 110 and the OIS lens 220 is moved to cancel out the detected amount of shake.

[0164] In the above-described embodiment 2, an example was given in which the interchangeable lens 200 includes a zoom lens 210. In this embodiment, for interchangeable lenses such as prime lenses that do not include a zoom lens, tilt correction and translation / rotation correction may be performed in the EIS function according to the focal length of the interchangeable lens, similar to embodiment 2, using image correction ratios R1 and R2. In this embodiment, for example, only the image correction ratios R1 and R2 for the focal length of the interchangeable lens may be calculated, and the ratio data D1 for the EIS function may not be generated.

[0165] In each of the embodiments described above, examples were given in which the digital camera 1 performs image stabilization different from tilt correction by cropless EIS function, using either the BIS function alone or both the BIS function and the OIS function. In this embodiment, for example, the OIS function alone may be used to perform image stabilization different from tilt correction. Thus, the digital camera 1 of this embodiment performs image stabilization using at least one of the OIS function and the BIS function, with an amount of image stabilization different from the amount of image stabilization used in tilt correction. The digital camera 1 of this embodiment is not limited to the example in Figure 1, and may have only one of the BIS function or the OIS function.

[0166] In other words, in this embodiment, the digital camera 1 may include at least one of a sensor drive unit 181 (element drive unit) that performs optical image blur correction by moving the image sensor 110 in a plane perpendicular to the optical axis of the optical system, and an OIS drive unit 221 (lens drive unit) that performs optical image blur correction by moving the OIS lens 220 (correction lens) included in the optical system in a plane perpendicular to the optical axis. In this embodiment, in optical image blur correction, at least one of the image sensor 110 and the OIS lens may be moved to cancel out the detected amount of blur.

[0167] In each of the embodiments described above, an example was explained in which, in calculating the distortion correction parameter (S2), the center position Pc of the entire image in the image data 10 is used as the distortion correction center. In this embodiment, for example, depending on the amount of blur correction by the optical image stabilization function, a position obtained by shifting the center position Pc in the image data 10 may be used as the distortion correction center. This allows for the acquisition of an enlarged image region 13 by distortion correction according to the lens characteristics of the interchangeable lens 200, even when, for example, the center position Pc is shifted from the intersection point of the optical axis and the imaging plane of the image sensor 110 due to the optical image stabilization function.

[0168] In each of the embodiments described above, an example was given in which the digital camera 1 performs cropless EIS function during video recording according to the selected EIS function correction mode. The digital camera of this embodiment may also perform cropless EIS function when shooting still images.

[0169] In the embodiments described above, a lens-interchangeable digital camera was explained as an example of an imaging device, but the imaging device in these embodiments may be a digital camera that is not particularly lens-interchangeable. Furthermore, the concept of this disclosure is not limited to digital cameras, but can also be applied to movie cameras, and to various electronic devices with imaging functions such as camera-equipped mobile phones, smartphones, or PCs.

[0170] As described above, embodiments have been explained as examples of the technology in this disclosure. For this purpose, accompanying drawings and a detailed description have been provided.

[0171] Therefore, the components described in the attached drawings and detailed descriptions may include not only components essential for solving the problem, but also components that are not essential for solving the problem, provided that they illustrate the technology described above. For this reason, the mere presence of these non-essential components in the attached drawings and detailed descriptions should not be immediately assumed to mean that they are essential.

[0172] Furthermore, since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents.

[0173] (Summary of characteristics) The various aspects of this disclosure are listed below.

[0174] A first aspect of the present disclosure is an imaging device comprising: an image sensor having an imaging area through an optical system in which an image of a subject is formed, and generating image data by capturing an image of a subject; a detection unit for detecting the amount of blur of the imaging device; and an image processing unit that performs image blur correction by adjusting the portion of the image data that outputs an image according to the amount of blur detected by the detection unit. In the image processing unit, distortion correction according to the distortion aberration of the optical system is performed on the image area indicated by the image data. In the image area in which distortion correction has been performed, the image processing unit performs image blur correction without using a correction area provided for cutting out a portion of the image within the range corresponding to the imaging area.

[0175] In the second embodiment, the imaging apparatus of the first embodiment further includes an operating unit for inputting a user operation to select an image blur correction mode to be used for image blur correction from a plurality of image blur correction modes. The plurality of image blur correction modes include a first image blur correction mode that performs image blur correction without using a correction area provided within a range corresponding to the imaging area, and a second image blur correction mode that performs image blur correction using the correction area.

[0176] In the third embodiment, in the imaging device of the second embodiment, the first image blur correction mode performs image blur correction at a rate smaller than the rate at which the number of pixels decreases before and after image blur correction in the second image blur correction mode, and outputs a partial image.

[0177] In the fourth embodiment, in the imaging device according to any of the first to third embodiments, the image processing unit performs image blur correction without using a correction area provided within the range corresponding to the imaging area by adjusting the shape of the reference area for outputting a partial image according to the detected amount of blur, using an area that is expanded beyond the range corresponding to the imaging area in the image area on which distortion correction has been performed.

[0178] In the fifth embodiment, the imaging device according to any of the first to fourth embodiments further comprises at least one of an element drive unit that performs optical image blur correction by moving an image sensor in a plane perpendicular to the optical axis of the optical system, and a lens drive unit that performs optical image blur correction by moving a corrective lens included in the optical system in a plane perpendicular to the optical axis. In optical image blur correction, at least one of the image sensor and the corrective lens is moved to cancel out the detected amount of blur. In the image region where distortion correction has been performed, the image processing unit does not move the part that outputs the image for the amount of blur that has been canceled out by optical image blur correction among the detected amount of blur, and performs image blur correction by adjusting the shape of the reference region for outputting the image of the part.

[0179] In the sixth embodiment, in the imaging device according to any of the first to fifth embodiments, the distortion aberration of the optical system is negative at least in the peripheral part of the imaging area, and the image processing unit performs distortion correction by expanding at least the area corresponding to the peripheral part of the image area indicated by the image data to an area outside the range corresponding to the imaging area.

[0180] In the seventh embodiment, the imaging device according to any of the first to sixth embodiments further comprises a communication unit that communicates with an optical system, and a control unit that controls the communication unit and the image processing unit. The control unit obtains information regarding the distortion aberration of the optical system from the optical system via the communication unit, and based on the obtained information and information indicating the imaging area, when an area outside the range corresponding to the imaging area is detected in the distortion-corrected image area, it causes the image processing unit to perform image blur correction without using a correction area provided inside the range corresponding to the imaging area.

[0181] An eleventh aspect of the present disclosure is an imaging device comprising: an image sensor having an imaging area in which an image of a subject is formed via an optical system, and generating image data by capturing an image of a subject; a detection unit for detecting the amount of blur of the imaging device; an image processing unit that performs image blur correction by adjusting the portion of the image data that outputs an image according to the amount of blur detected by the detection unit; and a control unit that controls the image blur correction performed in the image processing unit. In the image processing unit, distortion correction according to the distortion aberration of the optical system is performed on the image area indicated by the image data. The control unit changes the ratio between the first and second image blur corrections performed in the image area that has been distortion corrected by the image processing unit according to the focal length of the optical system, the first image blur correction corrects the distortion of the image in the distortion-corrected image area, and the second image blur correction moves the portion of the image that outputs an image in the said image area.

[0182] The movement in the second image blur correction includes, for example, both translational and rotational movement. The movement in the second image blur correction may be either translational or rotational movement.

[0183] In the twelfth aspect, the imaging device of the eleventh aspect includes a zoom lens in the optical system. The control unit changes the ratio between the first image blur correction and the second image blur correction when the focal length changes due to the zoom lens.

[0184] In the 13th embodiment, in the imaging apparatus of the 11th or 12th embodiment, the control unit changes the ratio such that the proportion of the second image blur correction in the ratio between the first image blur correction and the second image blur correction increases as the focal length increases within a predetermined range.

[0185] In the 14th embodiment, in the imaging device according to any of the 11th to 13th embodiments, the control unit causes the image processing unit to perform a first image blur correction to correct the image distortion caused by a change in posture in which the line of sight from the imaging device to the subject is tilted, based on the detected amount of blur.

[0186] In the 15th embodiment, in an imaging device according to any of the 11th to 14th embodiments, the image processing unit performs first image blur correction in the distortion-corrected image region without using a correction region provided for cutting out a portion of the image within the range corresponding to the imaging region.

[0187] In the sixteenth embodiment, in the imaging apparatus of the fifteenth embodiment, the image processing unit performs first image blur correction without using a correction area provided inside the range corresponding to the imaging area by adjusting the shape of the reference area for outputting a partial image according to the detected amount of blur, using an area that is expanded beyond the range corresponding to the imaging area in the distortion-corrected image area.

[0188] In the 17th embodiment, the imaging device according to any of the 11th to 16th embodiments further comprises at least one of an element drive unit that performs optical image blur correction by moving an image sensor in a plane perpendicular to the optical axis of the optical system, and a lens drive unit that performs optical image blur correction by moving a corrective lens included in the optical system in a plane perpendicular to the optical axis. In optical image blur correction, at least one of the image sensor and the corrective lens is moved to cancel out the detected amount of blur. The image processing unit performs a second image blur correction in the distortion-corrected image region, such that the portion of the image output is moved according to the detected amount of blur.

[0189] In the 18th embodiment, in the imaging device according to any of the 11th to 17th embodiments, the distortion aberration of the optical system is negative at least in the peripheral part of the imaging area, and the image processing unit performs distortion correction by expanding at least the area corresponding to the peripheral part of the image area indicated by the image data to an area outside the range corresponding to the imaging area. [Industrial applicability]

[0190] The concept of this disclosure can be applied to electronic devices with imaging capabilities that include image stabilization (imaging devices such as digital cameras and camcorders, mobile phones, smartphones, etc.). [Explanation of Symbols]

[0191] 1 Digital camera 100 Camera Body 110 Image Sensor 140 Camera Control Unit 143 Image Correction Unit 181 Sensor drive unit 183 BIS Processing Unit 184 Gyroscope Sensor 200 interchangeable lenses 220 OIS lens 221 OIS drive unit 223 OIS Processing Unit 224 Gyroscope Sensor

Claims

1. An imaging device, An image sensor having an imaging region in which an image of a subject is formed via an optical system, and generating image data by capturing the image of the subject, A detection unit for detecting the amount of blur of the imaging device, An image processing unit performs image blur correction by adjusting the portion of the image data that outputs an image according to the amount of blur detected by the detection unit. The system includes a control unit that controls image blur correction performed in the image processing unit, In the image processing unit, distortion correction corresponding to the distortion aberration of the optical system is performed on the image region indicated by the image data. The control unit changes the ratio between the first and second image blur corrections performed by the image processing unit in the distortion-corrected image region, according to the focal length of the optical system. The first image blur correction corrects the distortion of the image in the distortion-corrected image region, The second image blur correction involves moving the part that outputs the image in the image region. Imaging device.

2. The optical system includes a zoom lens, The control unit changes the ratio between the first image blur correction and the second image blur correction when the focal length changes due to the zoom lens. The imaging apparatus according to claim 1.

3. The control unit changes the ratio such that the proportion of the second image blur correction in the ratio between the first image blur correction and the second image blur correction increases as the focal length increases within a predetermined range. The imaging apparatus according to claim 1.

4. The control unit causes the image processing unit to perform the first image blur correction in order to correct the distortion of the image caused by a change in posture in which the line of sight from the imaging device to the subject is tilted, based on the detected amount of blur. The imaging apparatus according to claim 1.

5. The image processing unit performs the first image blur correction in the distortion-corrected image region without using a correction region provided within the range corresponding to the imaging region to cut out the image of the portion. The imaging apparatus according to claim 1.

6. The image processing unit performs the first image blur correction without using the correction region provided within the range corresponding to the imaging region by adjusting the shape of the region referenced for outputting the image of the portion, according to the detected amount of blur, using a region that is expanded beyond the range corresponding to the imaging region within the distortion-corrected image region. The imaging apparatus according to claim 5.

7. The system further comprises at least one of the following: an element drive unit that performs optical image blur correction by moving the image sensor in a plane perpendicular to the optical axis of the optical system; and a lens drive unit that performs optical image blur correction by moving a corrective lens included in the optical system in a plane perpendicular to the optical axis. In the optical image blur correction described above, at least one of the image sensor and the corrective lens is moved to cancel out the detected amount of blur. The image processing unit performs the second image blur correction in the distortion-corrected image region, such that it moves the portion of the image output according to the detected amount of blur. The imaging apparatus according to claim 1.

8. The distortion of the optical system is negative at least in the peripheral portion of the imaging area. The image processing unit performs distortion correction by expanding at least the area corresponding to the peripheral portion of the image region indicated by the image data to an area outside the range corresponding to the imaging region. The imaging apparatus according to claim 1.

Images