Imaging device, correction control method, and program

Abstract

This imaging device includes a control unit that executes control in which a correction operation, performed by moving a focus lens in response to optical axis-direction oscillation of a device body, serves as a correction operation which is in accordance with the following: a first amount of resolution degradation of a captured image due to oscillation; and a second amount of resolution degradation of the captured image due to a change in zoom ratio caused by moving the focus lens in a direction for cancelling the oscillation.

Description

Imaging device, correction control method, and program

[0001] The present technology relates to an imaging device, a correction control method, and a program, and in particular to a technology for correction processing for fluctuations in the optical axis direction of an imaging device.

[0002] For example, Patent Documents 1 and 2 disclose technologies that detect shaking in the optical axis direction of an imaging device (camera) based on signal information from an inertial sensor, and correct shake by driving a focus lens mechanism in a direction that cancels it.

[0003] JP 2020-071265 A JP 2012-128356 A

[0004] Japanese Patent Application Laid-Open Nos. 2003-209499 and 2004-103262 propose basic techniques for correcting blur by driving a focus lens mechanism. However, they do not address specific problems that arise when this technique is actually incorporated into a lens optical system. Specifically, in an actual lens optical system, driving the focus lens mechanism causes a change in image magnification in response to a change in the focal position. When the image magnification changes, the image expands or contracts radially around the optical axis, and therefore, driving the focus lens mechanism during exposure results in a deterioration of peripheral resolution.

[0005] Therefore, this disclosure proposes a technique for correcting for fluctuations in the optical axis direction due to camera shake or the like, while taking into consideration degradation in resolution due to changes in image magnification in response to movement of the focus lens.

[0006] The imaging device according to the present technology includes a control unit that executes control such that a correction operation by moving a focus lens in response to a swing of a device body in a direction along an optical axis becomes a correction operation according to a first amount of resolution degradation of a captured image due to the swing and a second amount of resolution degradation of a captured image due to a change in image magnification caused by moving the focus lens in a direction that cancels the swing. The correction amount of image stabilization by moving the focus lens is controlled so that the amount of resolution degradation caused in an image due to swing in the direction along the optical axis due to camera shake or the like and the amount of resolution degradation caused by a change in angle of view in response to movement of the focus lens to cancel the resolution degradation are each reduced.

[0007] 1 is a block diagram of an imaging device according to an embodiment of the present technology; FIG. 2 is an explanatory diagram of a configuration related to shake correction control according to an embodiment; FIG. 3 is an explanatory diagram of an in-focus image; FIG. 4 is an explanatory diagram of an image with resolution degradation due to movement of the focal plane during exposure; FIG. 5 is an explanatory diagram of an image with resolution degradation due to a change in image magnification during exposure; FIG. 6 is an explanatory diagram of the amount of blurring when oscillation in the optical axis direction is not corrected; FIG. 7 is an explanatory diagram of the amount of blurring when oscillation in the optical axis direction is completely corrected; FIG. 8 is an explanatory diagram of the amount of blurring when correction is performed by setting a correction strength for oscillation in the optical axis direction; FIG. 9 is an explanatory diagram of the amount of blurring when the F-number is large; and FIG. 10 is an explanatory diagram of the amount of blurring for each image height. FIG. 11 is a flowchart of shake correction processing according to an embodiment; and FIG. 12 is a flowchart of shake correction processing according to a modified example of an embodiment.

[0008] The following describes the embodiment in the following order: <1. Image capture device configuration> <2. Image stabilization processing> <3. Summary and modifications>

[0009] 1. Configuration of Imaging Device Fig. 1 shows an example of the configuration of an imaging device 1. The imaging device 1 in Fig. 1 is a so-called digital still camera or digital video camera, and is capable of capturing still images and moving images. Note that in this disclosure, the term "image" is used to include both still images and moving images.

[0010] The device main body of the imaging device 1 is shown as a camera body 2. The lens barrel 3 functions as a so-called interchangeable lens and is detachable from the camera body 2 of the imaging device 1. The user can change the lens barrel 3 depending on the use case. Note that the embodiment assumes such an interchangeable lens type imaging device 1, but the technology of the present disclosure can also be applied to a type in which the lens barrel 3 cannot be removed from the camera body 2.

[0011] A lens system 10 having a plurality of optical components is formed in the lens barrel 3. For example, the lens system 10 includes a zoom lens 10a, an iris mechanism 10b, a focus lens 10c, and an image stabilization lens 10d.

[0012] The image stabilization lens 10d is a mechanism that reduces shaking that occurs in an image by mechanically driving the lens in response to camera shake. For example, the image stabilization lens 10d performs camera shake correction in response to shaking in any or all of the yaw, pitch, and roll directions. Note that while this embodiment describes shaking along the optical axis, when shaking in the optical axis direction occurs due to camera shake or the like, shake correction is performed by driving the focus lens 10c back and forth (in the optical axis direction). Oscillation in the optical axis direction is a change in the subject distance, which causes the image to become out of focus and blurred. In response to this, the focus lens 10c is moved in the optical axis direction to reduce the blur.

[0013] Light (incident light) from a subject is focused onto the image sensor unit 11 via the lens system 10. The image sensor unit 11 includes an image sensor (image sensor), such as a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) type. The image sensor unit 11 photoelectrically converts the light received by the image sensor to obtain an electrical signal, which is then subjected to processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control), and then A / D (Analog / Digital) conversion. The image sensor then outputs the resulting digital image signal to a camera signal processor 13.

[0014] The camera signal processing unit 13 is configured as an image processor, for example, using a DSP (Digital Signal Processor). This camera signal processing unit 13 performs various types of signal processing on the digital signal (captured image signal) from the imaging element unit 11. For example, as camera processes, the camera signal processing unit 13 performs preprocessing, synchronization processing, YC generation processing, various correction processing, resolution conversion processing, codec processing, etc. The captured image signal processed by the camera signal processing unit 13 is sent to the recording control unit 14, display unit 15, and output unit 16 in appropriate data formats.

[0015] The camera control unit 18 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 19 stores information and the like used for processing by the camera control unit 18. The illustrated memory unit 19 comprehensively includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like.

[0016] The RAM in the memory unit 19 is used to temporarily store data, programs, etc. as a working area when the CPU of the camera control unit 18 processes various data. The ROM and flash memory (non-volatile memory) in the memory unit 19 are used to store the OS (Operating System) used by the CPU to control each unit, content files such as image files, application programs for various operations, firmware, etc. The memory unit 19 may be a memory area built into a microcomputer chip that serves as the camera control unit 18, or may be configured as a separate memory chip.

[0017] The camera control unit 18 executes programs stored in the ROM, flash memory, or the like of the memory unit 19, thereby controlling the entire imaging device 1 and the lens barrel 3. For example, the camera control unit 18 controls the operation of each necessary unit regarding control of the shutter speed of the image sensor unit 11, instructions for various signal processing in the camera signal processing unit 13, imaging operations and recording operations in response to user operations, playback operations of recorded image files, operations of the lens system 10 such as zoom, focus, and aperture adjustment in the lens barrel 3, user interface operations, etc. The camera control unit 18 also communicates with a lens control unit 20 on the lens barrel 3 side.

[0018] The recording control unit 14 performs recording and playback on a recording medium, such as a nonvolatile memory. The recording control unit 14 performs processing to record image files, such as video data and still image data, and thumbnail images, on the recording medium. The actual configuration of the recording control unit 14 is conceivable in a variety of ways. For example, the recording control unit 14 may be configured as a flash memory and its write / read circuit built into the imaging device 1, or may be configured as a card recording and playback unit that performs recording and playback access on a recording medium that can be attached to or detached from the imaging device 1, such as a memory card (such as a portable flash memory). The recording control unit 14 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0019] The display unit 15 displays various information to the user. The display unit 15 may be, for example, a display panel or viewfinder using a display device such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display disposed on the housing of the imaging device 1. The display unit 15 displays various information on the display screen based on instructions from the camera control unit 18. For example, the display unit 15 displays a playback image of image data read from a recording medium by the recording control unit 14. The display unit 15 may also receive image data of a captured image whose resolution has been converted for display by the camera signal processing unit 13, and may display information based on the image data of the captured image in response to instructions from the camera control unit 18. This allows a so-called through image, which is an image captured during composition confirmation, to be displayed. The display unit 15 also displays various operation menus, icons, messages, and the like, i.e., a graphical user interface (GUI), on the screen based on instructions from the camera control unit 18.

[0020] The output unit 16 performs wired or wireless data communication and network communication with external devices. For example, it transmits and outputs captured image data (still image files and video files) to an external display device, recording device, playback device, etc. The output unit 16 may also be a network communication unit that performs communication via various networks such as the Internet, a home network, a LAN (Local Area Network), etc., and transmits and receives various data to and from servers, terminals, etc. on the network.

[0021] The operation unit 17 collectively represents input devices for the user to input various operations. Specifically, the operation unit 17 represents various operators (keys, dials, touch panel, touch pad, etc.) provided on the housing of the imaging device 1. The operation unit 17 detects user operations, and sends signals corresponding to the input operations to the camera control unit 18.

[0022] The subject distance detection unit 12 transmits information related to the subject distance to the camera control unit 18. This is because information on the distance from the imaging device 1 to the subject (subject distance) is used in the shake correction process described below. The subject distance detection unit 12 transmits to the camera control unit 18 information that is detection information for autofocus control and can determine the subject distance, such as information detected by image plane phase difference pixels formed in the imaging element unit 11.

[0023] The camera control unit 18 calculates the subject distance using the information detected by the subject distance detection unit 12, and sends a command for the focus lens position to the lens control unit 20 via communication. This performs autofocus control. Also, in the shake correction process for fluctuations in the optical axis direction, which will be described later, information on the distance to the subject is sent from the camera control unit 18 to the lens control unit 20. Note that the subject distance is necessary for shake correction, but for this purpose information from another sensor, such as distance measurement information from a TOF (Time of Flight) sensor, may be used.

[0024] A lens control unit 20, which is implemented by, for example, a microcomputer, is mounted in the lens barrel 3. When the lens barrel 3 is attached to the camera body 2, the camera control unit 18 and the lens control unit 20 can communicate with each other.

[0025] The lens control unit 20 and the camera control unit 18 constantly exchange bidirectional data communications at a certain communication speed. For example, the camera control unit 18 issues drive instructions to the lens control unit 20 for the zoom lens 10a, focus lens 10c, iris mechanism 10b, and image stabilization lens 10d. The lens control unit 20 executes operations of the lens system 10 in accordance with these drive instructions. The lens control unit 20 also transmits lens distortion correction information, focal length information, the position of the image stabilization lens 10d, and the like to the camera control unit 18.

[0026] Lens barrel 3 is provided with, for example, a zoom driver 21 having a motor and motor driver for driving zoom lens 10a, an iris driver 22 having a motor and motor driver for driving iris mechanism 10b, a focus driver 23 having a motor and motor driver for driving focus lens 10c, and an image stabilization driver 24 having a motor and motor driver for driving image stabilization lens 10d. Zoom driver 21, iris driver 22, focus driver 23, and image stabilization driver 24 apply drive currents to the corresponding motors in response to instructions from lens control unit 20, which are based on instructions from camera control unit 18. This allows zooming, iris opening / closing, focusing, and optical image stabilization to be performed.

[0027] The memory unit 26 stores information and the like used by the lens control unit 20 for processing. The memory unit 26 is a comprehensive term that refers to, for example, ROM, RAM, flash memory, and the like. The RAM in the memory unit 26 is used as a working area for the CPU of the lens control unit 20 to temporarily store data, programs, and the like when processing various data. The ROM and flash memory (non-volatile memory) in the memory unit 26 are used to store an OS (Operating System) for the CPU to control each unit, programs for various processes, and the like. The memory unit 26 may also be used to temporarily store information that the lens control unit 20 transmits to the camera control unit 18. The memory unit 26 may be a memory area built into a microcomputer chip that serves as the lens control unit 20, or may be configured as a separate memory chip.

[0028] The camera shake signal detection unit 25 outputs a signal detecting shaking occurring in the imaging device 1 to the lens control unit 20. The camera shake signal detection unit 25 detects angular velocity and acceleration, for example, along three axes (pitch, yaw, and roll) from a signal from an inertial sensor 27, such as an IMU (inertial measurement unit), and outputs these to the lens control unit 20. The inertial sensor 27 also detects inertia in the optical axis direction, and the camera shake signal detection unit 25 also outputs a signal indicating shaking in the optical axis direction to the lens control unit 20. The lens control unit 20 controls the camera shake correction driver 24 based on the signal detecting shaking in the pitch, yaw, and roll directions to drive the camera shake correction lens 10d. The lens control unit 20 controls the focus driver 23 based on the signal detecting shaking in the optical axis direction to drive the focus lens 10c.

[0029] The camera shake signal detection unit 25 and the inertial sensor 27 may be mounted on the camera body 2. In the case of an imaging device 1 with an integrated lens, the camera control unit 18 and the lens control unit 20 may be realized as functions of a single microcomputer.

[0030] 2 shows the functional configuration within the lens control unit 20, which is related to the blur correction process by moving the focus lens 10c in response to the blur caused by the oscillation in the optical axis direction. The lens management unit 30 has the function of managing all the control units within the lens, and sends commands to the focus control unit 32 and the iris control unit 31 regarding the blur correction process for the blur in the optical axis direction.

[0031] The focus control unit 32 controls the focus drive unit 23 based on instructions from the lens management unit 30. At the same time, the focus control unit 32 also controls the focus drive unit 23 based on a signal indicating fluctuation in the optical axis direction from the camera shake signal detection unit 25. The iris control unit 31 controls the iris drive unit 22 based on instructions from the lens management unit 30, and at the same time provides information about the F-number to the focus control unit 32. This allows the focus control unit 32 to know the current F-number.

[0032] 2. Blur Correction Processing In the imaging device 1 configured as described above, blur correction processing is performed by the focus lens 10c to compensate for fluctuations in the optical axis direction, and degradation of resolution due to breathing during this processing is suppressed.

[0033] Generally, when the image magnification of a lens optical system is increased and the F-number is bright, the depth of field becomes shallow. During handheld shooting, shaking of the imaging device 1 also occurs in the optical axis direction due to camera shake, so under conditions where the depth of field is shallow, if the shaking of the imaging device 1 exceeds one depth, a blurred image will be exposed, leading to a deterioration in resolution of the captured image. This is particularly noticeable during macro photography, but to suppress this, the focus lens 10c is driven to cancel the shaking of the imaging device 1, thereby suppressing the deterioration in resolution.

[0034] However, in an actual lens optical system, when the focus lens 10c is driven, the image magnification also changes in accordance with the change in the focal position. In other words, the angle of view changes. When the image magnification changes, the image expands or contracts radially around the optical axis, so if the focus lens 10c is moved during exposure, this causes a deterioration in peripheral resolution.

[0035] To be more specific, Figure 3 shows an image in focus. It is assumed to be an image with an array of dots. This is an image that would be captured if neither the subject nor the camera moved during exposure, and is an image taken under the most ideal circumstances.

[0036] Next, Figure 4 shows an image with resolution degradation caused by movement of the focal plane during exposure. This occurs when the subject or the camera moves along the optical axis during exposure, causing the subject to lose focus during exposure and resulting in a blurred image. To suppress this degradation in resolution caused by the subject losing focus, blur correction processing for the optical axis movement is performed by detecting the amount of movement along the optical axis due to camera shake or other factors during exposure and driving the focus lens 10c to cancel it.

[0037] On the other hand, Figure 5 shows an image with resolution degradation due to a change in image magnification during exposure. Although the image remained in focus during exposure, the change in optical image magnification causes blurring in the periphery and resolution degradation due to breathing. This is a drawback of image stabilization along the optical axis.

[0038] The resolution-degraded image in Figure 4 is characterized by resolution degradation occurring throughout the image, not just in the center or periphery. On the other hand, the resolution-degraded image in Figure 5 has resolution in the center, but resolution degradation due to image flow occurs in the periphery.

[0039] However, it is possible to compare the magnitude of either type of resolution degradation using the numerical value of the amount of blur (width) at the same image height. In this embodiment, the magnitude of these numerical values ​​is compared based on the amount of change in image magnification and the depth of field, and the presence or absence or strength of image stabilization in the optical axis direction is controlled.

[0040] That is, a first amount of resolution degradation and a second amount of resolution degradation are taken into consideration when correcting the image by moving the focus lens 10c in response to the movement of the image pickup device 1 in the optical axis direction. The first amount of resolution degradation is the amount of image resolution degradation caused by a change in subject distance due to camera movement and a loss of focus, as shown in Figure 4. The second amount of resolution degradation is the amount of resolution degradation that becomes particularly noticeable in the peripheral areas of the image due to a change in image magnification caused by moving the focus lens 10c, as shown in Figure 5. The strength of the image blur correction caused by moving the focus lens is controlled to be appropriate depending on these two amounts of resolution degradation.

[0041] The strength of the image stabilization is maximum when the amount of shift in subject distance due to camera shake or other factors is completely canceled by moving the focus lens, and the strength decreases as the amount of shift canceled decreases.The strength is minimum (zero) when the focus lens is not moved.

[0042] A specific example of a process for reflecting the first and second amounts of resolution degradation in image blur correction will be described below. Although each process will be described, the variables used in the calculation formulas will be listed below.

[0043] bs: Amount of resolution degradation (first amount of resolution degradation: amount of blur due to shaking in the optical axis direction) bb: Amount of resolution degradation (second amount of resolution degradation: amount of blur due to breathing) DoF: Depth of field δ: Allowable diameter of circle of confusion F: F-number β: Image magnification Δβ(x): Rate of change of image magnification at focus lens position x x: Focus lens position x ０ : Focus lens position at the start of exposure x ｉ : Focus lens position to be corrected Δx: Shake correction amount (amount of focus lens movement for shake correction) Δx ｉ : Current (time point i) shake correction amount s ０ : Subject distance at the start of exposure s ｉ : current (time point i) subject distance Δs: deviation amount (deviation amount of subject distance (= DoF / 2)) h: image height α: correction strength coefficient α ｉ : Current correction strength coefficient α ０ : Correction intensity coefficient at the start of exposure γ: Weighting coefficient

[0044] Generally, when the object distance is short and the image magnification β becomes high, the depth of field DoF is expressed by the following formula (1): where δ is the diameter of the permissible circle of confusion and F is the F-number.

[0045]

[0046] If this formula (1) is transformed into a formula for deriving the diameter δ of the permissible circle of confusion and only the depth of field (DoF / 2) on one side is treated, the formula will take the form of formula (2-1).

[0047]

[0048] If DoF / 2 is the deviation amount Δs from the subject distance and the allowable diameter of the circle of confusion δ is the amount of resolution degradation bs in the planar direction of the image sensor unit on the image plane, then equation (2-1) can be rewritten as the following equation (2-2).

[0049]

[0050] This formula (2-2) represents the amount of resolution degradation bs as the amount of blur (shake width) on the image plane when the subject moves by the amount of displacement Δs within a range where the image magnification β does not change significantly.

[0051] The amount of image flow (amount of resolution degradation bb) caused by the change in image magnification caused by moving one of the focus lenses 10c is expressed by equation (3).

[0052]

[0053] The amount of resolution degradation bb is the amount of blur due to breathing, Δβ(x) is the rate of change of image magnification at focus lens position x, Δx is the amount of focus lens movement, and h is the image height. Δβ(x) is expressed as a function because its characteristics differ depending on the optical design.

[0054] If the imaging device 1 oscillates in the optical axis direction during exposure, the amount of deviation Δs increases, and the amount of resolution degradation bs increases according to equation (2-2). If no shake correction in the optical axis direction is performed, the amount of resolution degradation bs directly becomes the amount of blur (shake width) on the image plane, but if shake correction in the optical axis direction is performed to move the focus lens 10c by the shake correction amount Δx and reduce the amount of deviation Δs in equation (2-2), the amount of resolution degradation bs can be kept small as a result. Equation (2-3) is shown below.

[0055]

[0056] On the other hand, if the focus lens 10c is moved to reduce the deviation Δs in equation (2-2), the shake correction amount Δx increases, and accordingly the amount of resolution degradation bb in equation (3) increases.

[0057] The amount of resolution degradation bs and the amount of resolution degradation bb derived by these equations are compared, and the presence or absence or strength of blur correction in the optical axis direction is controlled depending on the magnitude relationship, thereby making it possible to suppress overall resolution degradation.

[0058] The phenomenon is summarized below using graphs with the amount of deviation Δs due to blurring in the optical axis direction on the horizontal axis and the amount of resolution degradation (b, b) according to equations (2-3) and (3) on the vertical axis. In each of Figures 6 to 10, the solid line indicates the amount of resolution degradation bb according to equation (3), and the dotted line indicates the amount of resolution degradation bs according to equation (2-3).

[0059] [When no correction is made (amount of image stabilization Δx = 0)] A graph is shown in Figure 6. When no correction is made to shake in the optical axis direction, the amount of resolution degradation bs increases in proportion to the amount of deviation Δs in the subject distance according to equation (2-3). On the other hand, in equation (3), the amount of image stabilization Δx = 0, so the amount of resolution degradation bb is fixed at zero. This is a phenomenon that occurs in systems that do not generally incorporate image stabilization in the optical axis direction.

[0060] [When Δx = Δs Correction Is Performed] To completely eliminate shaking in the optical axis direction, consider a situation in which the image blur correction amount Δx is perfectly equal to the object distance deviation amount Δs. In this case, the deviation amount Δs is substituted for the image blur correction amount Δx in equation (2-3), resulting in the equation shown in FIG. 7. In other words, the line of equation (3) always exceeds the line of equation (2-3), and image blurring in the optical axis direction becomes noticeable. Because the vertical axis of equation (3) expands and contracts with the image height h, the effect of image blurring can be felt near the center of the image, but the higher the image height, the more noticeable blurring due to image breathing becomes. Although image blurring caused by focus deviation is eliminated by image blurring in the optical axis direction, the image blurring becomes noticeable to the user.

[0061] [When Intensity is Controlled by Δx = α·Δs (Correction Intensity Coefficient α<1)] Consider the case where the image blur correction amount Δx is not perfectly equal to the deviation amount Δs, but rather correction is suppressed at a certain intensity. Figure 8 shows a graph of this case. The amount of resolution degradation bs in equation (2-3) increases in proportion to the deviation amount Δs in the optical axis direction due to residual correction. The residual correction refers to the amount of deviation Δs that cannot be completely canceled out, where the absolute value of the image blur correction amount Δx is smaller than the absolute value of the deviation amount Δs. However, because the deviation amount Δs is reduced by the amount of image blur correction Δx, the slope of the line in equation (2-3) is smaller than that in Figure 6. In other words, blurring due to focus deviation is reduced. On the other hand, because the increase in the image blur correction amount Δx is suppressed in equation (3), the curve in Figure 8 is a curve that is stretched to the right from the curve in Figure 7. As a result, the curve in equation (3) is lower than the curve in equation (2-3). In other words, blurring caused by breathing is less noticeable.

[0062] Therefore, it can be said that performing image blur correction by moving the focus lens while controlling the correction strength is effective in preventing degradation of the resolution of the captured image, and this type of control is adopted in the present embodiment. The positional relationship between the curve of equation (3) and the curve of equation (2-3) can be changed by setting the correction strength, allowing the user to select the shape that is most desirable.

[0063] [When the F-number is large] Incidentally, equation (2-3) includes the F-number, a parameter that can be operated by the user. As the F-number increases, the depth of field (DoF) widens, and as a result, the amount of resolution degradation bs in equation (2-3) decreases. This means that the importance of performing image stabilization in the optical axis direction decreases. If image stabilization in the optical axis direction is performed in the same way as when the F-number is small, only the adverse effects of breathing become noticeable, because equation (3) remains unchanged and independent of the F-number.

[0064] FIG. 9 shows a graph when the F-number is large. Because blurring due to focus shift is less likely to occur when the F-number is large, the slope of the line in equation (2-3) is smaller than in the case of FIG. 8. The curve in equation (3) in FIG. 9 shows the same correction strength as in FIG. 8, but in this case, blurring due to breathing becomes more noticeable than blurring due to focus shift. In this case, it is advisable to reduce the amount of blur correction Δx in the optical axis direction in inverse proportion to the rate of increase in the F-number. In other words, the correction strength coefficient α is reduced so that the curve in equation (3) is lower than the curve in equation (2-3). This makes blurring due to breathing less noticeable, and blur correction becomes effective.

[0065] [When the image height is high] Equation (3) indicates that the blur width also changes depending on the image height h. Fig. 10 shows that the magnitude relationship between the amount of resolution degradation bb in equation (3) and the amount of resolution degradation bs in equation (2-3) changes depending on several image heights h. Here, the image heights h are h1, h2, and h3. Image height h1 is the highest image height, and image height h3 is the lowest image height.

[0066] As described above, since the amount of blur caused by breathing differs depending on the image height h, it is advisable to perform control in accordance with the user's preferences and the product's merits. For example, it is possible to set the correction intensity coefficient α in accordance with the image height h that is prioritized by the user or manufacturer. Specifically, it is possible to use a design method in which an image height that is prioritized in terms of product merits is fixed, or to use control in which the correction intensity coefficient α is varied in accordance with the image height of the autofocus frame, for example.

[0067] Next, a method for deriving the correction intensity coefficient α will be described. In this embodiment, as described above, Δx = α·Δs (where α<1) is used to control the correction intensity of image stabilization. For this reason, it is necessary to set an appropriate correction intensity coefficient α.

[0068] As a specific method for deriving the correction intensity coefficient α, consider the condition under which the amount of blurring is minimized in equations (2-3) and (3), that is, the amount of resolution degradation bs = the amount of resolution degradation bb. Rearranging the equations, we obtain the following equation (4-1).

[0069]

[0070] In this formula (4-1), the coefficient multiplied by the deviation amount Δs corresponds to the correction intensity coefficient α. This makes it possible to derive the correction intensity coefficient α that minimizes the amount of blur according to the combination of the rate of change Δβ(x) of image magnification at the focus lens position x, the F-number F, the image magnification β, and the image height h.

[0071] On the other hand, since the amount of resolution degradation bs, which is the amount of blur due to focus shift, and the amount of resolution degradation bb, which is the amount of blur due to breathing, are qualitatively different, there are cases where it is desired to select which one to suppress more. In such cases, instead of setting the amount of resolution degradation bs = the amount of resolution degradation bb, a weighting coefficient γ is used as a ratio of how much importance is given to the amount of resolution degradation bb over the amount of resolution degradation bs, and equation (4-2) is used where bs = γ bb.

[0072]

[0073] In equation (4-2), the coefficient multiplied by the deviation amount Δs corresponds to the correction intensity coefficient α. This makes it possible to derive the correction intensity coefficient α from the rate of change Δβ(x) of image magnification at focus lens position x, the F-number F, the image magnification β, the image height h, and the weighting coefficient γ.

[0074] Below, we will explain an example of control processing executed by the lens control unit 20 or the camera control unit 18. Below, the processing entity will be referred to as the "control unit," which means that the processing entity may be either the lens control unit 20 or the camera control unit 18, or may be considered to be processing performed by the cooperation of both.

[0075] The control unit repeatedly performs the process of FIG. 11 during the period from when exposure in the image sensor unit 11 starts to when exposure ends.

[0076] In step S101, the control unit acquires the image height h of the coordinate position of the autofocus frame. This is to be used as the image height h to calculate the correction intensity coefficient α. Because the autofocus frame is the area that the user is focusing on, this is to prevent resolution degradation due to breathing from becoming noticeable at the image height of that area of ​​focus. Note that the image height h may be a preset fixed value.

[0077] In step S102, the control unit acquires the subject distance s0, which is the start position at the time of exposure start, and the focus lens position x0. The exposure start time here refers to the time when exposure starts when the subject is in focus through autofocus control, for example. In other words, the control unit acquires the subject distance s0 and focus lens position x0 in the in-focus state that serves as the basis for shake correction. The control unit acquires the subject distance s0 based on information from the subject distance detection unit 12, for example, and acquires the focus lens position x0 from detection information from a lens position sensor provided in the focus drive unit 23.

[0078] Thereafter, the control unit repeats the processes of steps S103 to S108 at each time point i until exposure is completed.

[0079] In step S103, the control unit acquires the rate of change Δβ(x) of image magnification at the current focus lens position xi, the F-number, and the image magnification β.

[0080] In step S104, the control unit calculates the correction intensity coefficient αi to be used at the current time. For example, the coefficient portion of the deviation amount Δs in equation (4-1) is calculated using the rate of change Δβ(x) of image magnification obtained in step S103, the F-number, the image magnification β, and the image height h set in step S101. In this case, if a weighting coefficient γ is set by a user setting operation or as a fixed setting, the correction intensity coefficient αi can be calculated as the coefficient portion of the deviation amount Δs in equation (4-2).

[0081] In step S105, the control unit acquires the current subject distance si, and then calculates the deviation amount Δsi (Δsi=si−s0) using the subject distance si and the subject distance s0 at the start of exposure acquired in step S101.

[0082] In step S106, the control unit calculates the shake correction amount Δxi, which corresponds to the amount of movement of the focus lens 10c that is actually controlled. In the correction operation, the focus lens 10c is moved so as to cancel the deviation amount Δsi, so if the shake correction amount Δxi=the deviation amount Δsi, the maximum correction strength described in Fig. 7 is achieved. However, in this embodiment, the correction strength is controlled by the correction strength coefficient α. That is, (shake correction amount Δxi)=(correction strength coefficient αi)·(deviation amount Δsi).

[0083] In step S107, the control unit calculates xi = x0 + Δxi and causes the focus driver 23 to perform shake correction. That is, the position obtained by adding the shake correction amount Δxi to the focus lens position x0 at the start of exposure is set as the focus lens position xi to be corrected, and the focus lens 10c is moved to this focus lens position xi.

[0084] In step S108, the control unit determines whether exposure is complete, and if exposure is in progress, the process returns to step S103 and repeats the same process. If exposure has ended, the process in FIG. 11 from step S108 ends.

[0085] 11, the correction operation by moving the focus lens 10c in response to the swing in the optical axis direction is controlled to be a correction operation corresponding to the amount of resolution degradation bs of the captured image due to the swing and the amount of resolution degradation bb of the captured image due to a change in image magnification caused by moving the focus lens 10c. Specifically, the strength of the correction operation by moving the focus lens 10c is controlled to be a correction operation with an intensity corresponding to the amount of resolution degradation bs and the amount of resolution degradation bb.

[0086] Fig. 12 shows another example of processing. Note that the same steps as those in Fig. 11 are given the same step numbers, and redundant explanations will be avoided.

[0087] In the processing example of Figure 12, the control unit performs steps S101 and S102 at the start of exposure, and then in step S103A, acquires the rate of change Δβ(x) of image magnification at the focus lens position x0 at the start of exposure, the F-number, and the image magnification β.

[0088] Then, in step S104A, the control unit calculates a correction intensity coefficient α0. That is, the correction intensity coefficient α0 is calculated as the coefficient portion of the deviation amount Δs in equation (4-1) or equation (4-2) using the rate of change Δβ(x0) of image magnification at the focus lens position x0 at the start of exposure, the F-number F, the image magnification β0, and the image height h.

[0089] Thereafter, the control unit repeats the processes of steps S105 to S108 at each time point i until exposure is completed.

[0090] In step S105, the control unit acquires the current subject distance si, and then calculates the deviation amount Δsi using the subject distance si and the subject distance s0 at the start of exposure (Δsi=si−s0).

[0091] In step S106A, the control unit calculates the shake correction amount Δxi. In this case, the correction intensity coefficient α0 calculated in step S104A is used. That is, (shake correction amount Δxi) = (correction intensity coefficient α0) · (deviation amount Δsi). Then, in step S107, the control unit calculates the target focus lens position xi = x0 + Δxi, and causes the focus driver 23 to perform shake correction.

[0092] In step S108, the control unit determines whether exposure is complete, and if exposure is in progress, the process returns to step S105 and repeats the same process. If exposure has ended, the process in FIG. 12 ends from step S108.

[0093] In the above processing example, once the correction intensity coefficient α0 is calculated at the start of exposure, the image blur correction amount Δxi is calculated using that correction intensity coefficient α until the exposure is completed. This reduces the computational processing load. Steps S103 and S104 in FIG. 11 involve processes with a relatively high computational load, so the computational load during exposure is reduced by fixing the correction intensity coefficient α to the correction intensity coefficient α0 calculated using the optical parameters (Δβ(x), F0, β0) at the focus lens position x0 at the start of exposure. This processing is preferably employed when the amount of change in the optical parameters near the focus lens position x0 at the start of exposure is relatively small and the effect on calculation errors is small.

[0094] 11 and 12 show processing examples, but many more processing examples are possible. In step S101 in FIGS. 11 and 12, the image height h is determined according to the autofocus frame coordinates, but this may be a fixed value set in advance in product design. In that case, it is advisable to determine the image height up to which breathing is guaranteed, taking into account the product's suitability.

[0095] 11 , the calculation load can be reduced by using table data. For example, table data may be provided that stores, as optical parameters, the rate of change Δβ(x) of image magnification at focus lens position x, the F-number F, the image magnification β, and a correction intensity coefficient α corresponding to the image height h, and the control unit may refer to the table data to obtain the correction intensity coefficient α. This reduces the calculation load.

[0096] 3. Summary and Modifications The imaging device 1 according to the above embodiment provides the following effects.

[0097] The imaging device 1 of the embodiment includes a control unit (lens control unit 20 or camera control unit 18) that controls the correction operation by moving the focus lens 10c in response to shaking in the optical axis direction to be a correction operation corresponding to a first amount of resolution degradation bs of the captured image due to shaking and a second amount of resolution degradation bb of the captured image due to a change in image magnification caused by moving the focus lens 10c in a direction that cancels the shaking. That is, the correction amount of image stabilization by moving the focus lens is controlled so as to reduce the amount of resolution degradation bs caused in the image due to shaking in the optical axis direction caused by camera shake or the like and the amount of resolution degradation bb caused by a change in the angle of view caused by moving the focus lens to eliminate that resolution degradation. This suppresses resolution degradation due to breathing when moving the focus lens for shake correction, and enables a captured image with an appropriate overall resolution to be obtained without burdening the user with operations.

[0098] In this embodiment, the control unit controls the correction operation by moving the focus lens 10c to be a correction operation with an intensity corresponding to the amount of resolution degradation bs and the amount of resolution degradation bb. That is, the degree of shake correction by moving the focus lens is determined based on the amount of resolution degradation bs and the amount of resolution degradation bb. By adjusting the intensity of shake correction by moving the focus lens, correction can be performed that maintains a captured image with as little resolution degradation as possible, according to the optical parameters at the time of exposure.

[0099] In the embodiment, an example was given in which the control unit controls the correction strength by setting the ratio of the movement amount of the cancellation movement by the focus lens 10c (blur correction amount Δx) to the deviation amount Δs of the subject distance due to shaking. In other words, rather than always completely canceling the amount of shaking by moving the focus lens, the process adjusts the amount of cancellation by moving the focus lens relative to the amount of shaking. This makes it possible to perform a compensation operation with an appropriate correction strength, taking into account both resolution degradation due to camera shake and resolution degradation due to focus lens movement.

[0100] In the embodiment, the first amount of resolution degradation bs is a value calculated based on the image magnification β, the amount of deviation Δs in the subject distance due to fluctuation in the optical axis direction, the movement amount of the focus lens 10c (the shake correction amount Δx), and the F-number F. The second amount of resolution degradation bb is a value calculated based on the rate of change Δβ(x) of image magnification at the focus lens position x, the movement amount of the focus lens 10c (the shake correction amount Δx), and the image height h. For example, the first amount of resolution degradation bs is set to a value corresponding to the above-mentioned formula (2-3), and the second amount of resolution degradation bb is set to a value corresponding to the above-mentioned formula (3). As a result, the first amount of resolution degradation bs is obtained as the amount of blur in the imager plane direction, with the allowable circle of confusion diameter δ representing the resolution degradation due to camera shake or the like in the optical axis direction, and the second amount of resolution degradation bb is obtained as the amount of blur due to breathing at a certain image height. These values ​​enable quantitative comparison of the resolution degradation caused by fluctuations in the optical axis direction and the resolution degradation caused by breathing, and correction control can be performed.

[0101] In the embodiment, an example was given in which the control unit controls the intensity of the correction operation by moving the focus lens so that the amount of resolution degradation bs and the amount of resolution degradation bb are equal. That is, the correction intensity coefficient α is set using the above-mentioned equation (4-1). This realizes a correction operation that minimizes the resolution degradation of the captured image in equations (2-3) and (3).

[0102] In the embodiment, an example was given in which the control unit sets a weighting factor γ between the amount of resolution degradation bs and the amount of resolution degradation bb, and controls the intensity of the correction operation by moving the focus lens so that the amount of resolution degradation bs and the amount of resolution degradation bb are equal after assigning the weighting factor γ to one of them. For example, using the above-mentioned equation (4-2), the weighting factor γ is set for the amount of resolution degradation bb relative to the amount of resolution degradation bs, and the correction intensity factor α is set. This makes it possible to set whether the correction operation should be prioritized, resolution degradation due to fluctuation in the optical axis direction or resolution degradation due to breathing. By allowing the user to variably set the weighting factor γ, for example, correction operations that meet the user's needs can be realized. Note that the weighting factor γ may be set for the amount of resolution degradation bs relative to the amount of resolution degradation bb, and the correction intensity factor α may be set.

[0103] In the embodiment, the control unit sets an image height h, acquires the rate of change Δβ(x) of image magnification at a focus lens position x, an F-number F, and an image magnification β, calculates a correction intensity coefficient α using the image height h, the rate of change Δβ(x), the F-number F, and the image magnification β, calculates a blur correction amount Δx based on the deviation amount Δs of the subject distance due to shaking and the correction intensity coefficient α, and controls the focus lens 10c to move to a focus lens position xi based on the blur correction amount Δx. In other words, the blur correction process shown in FIGS. 11 and 12 is performed. This achieves blur correction by moving the focus lens so that the amount of resolution degradation bs and the amount of resolution degradation bb are equal. In particular, by repeatedly performing the above process during exposure, blur correction that suppresses resolution degradation due to breathing while responding to camera shake in the optical axis direction is possible.

[0104] In the embodiment, an example was described in which the control unit acquires the correction intensity coefficient α by referencing table data that stores the correction intensity coefficient α in accordance with the image height h, the rate of change Δβ(x) of image magnification at the focus lens position x, the F-number F, and the image magnification β, from the start of exposure to the end of exposure. For example, in equation (4-1), the coefficient portion of the deviation amount Δs of the subject distance becomes the correction intensity coefficient α, and this correction intensity coefficient α is determined in the processing of steps S103 and S104 of FIG. 11, which is a calculation process with a relatively heavy computational load. Therefore, table data that stores the correction intensity coefficient α in accordance with each parameter (h, Δβ(x), F, β) is prepared, and the correction intensity coefficient α is acquired by referencing this table data. This reduces the processing load of FIG. 11.

[0105] In the embodiment, an example was given in which the control unit calculates the shake correction amount Δx by fixing the correction intensity coefficient α0 calculated from the image height h, the rate of change Δβ(x0) of image magnification at the focus lens position x0 at the start of exposure, the F-number F, and the image magnification β0 until the end of exposure. For example, as shown in Figure 12, the correction intensity coefficient α0 calculated initially is used as a fixed coefficient until the end of exposure. This reduces the calculation load for calculating the shake correction amount Δx at each point during exposure.

[0106] In the embodiment, an example has been given in which the control unit sets the image height h based on the coordinates of the autofocus frame. Because the amount of resolution degradation bb varies depending on the image height h, it is necessary to set an appropriate image height h. Because the AF frame is the position at which the user focuses, setting the image height h according to the coordinates of the AF frame makes it possible to perform blur correction taking into account resolution degradation due to breathing at the image height at which the user focuses.

[0107] The program of the embodiment is a program that causes an arithmetic processing device such as a CPU or a DSP (digital signal processor) to execute the processes shown in Figures 11 and 12. That is, the program of the embodiment is a program that causes the control unit of the imaging device 1 to execute control such that the correction operation by moving the focus lens 10c in response to rocking of the imaging device body in the optical axis direction becomes a correction operation according to a first amount of resolution degradation bs of the captured image due to the rocking and a second amount of resolution degradation bb of the captured image due to a change in image magnification caused by moving the focus lens 10c in a direction that cancels the rocking. The imaging device 1 of the embodiment can be realized by having the camera control unit 18 or the lens control unit 20 execute such a program.

[0108] Such a program can be pre-recorded on a hard disk drive (HDD) or a ROM in a microcomputer having a CPU, which serves as a built-in recording medium in a computer or other device. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. In addition to being installed into the imaging device 1 from a removable recording medium, such a program can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0109] Furthermore, such a program is suitable for widespread provision of the imaging device 1 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, or the like, these devices can function as the imaging device 1 of the present disclosure.

[0110] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0111] The present technology may also be configured as follows. (1) An imaging device including a control unit that executes control such that a correction operation by moving a focus lens in response to a rocking movement of a device body in the optical axis direction becomes a correction operation according to a first amount of resolution degradation of a captured image due to the rocking movement and a second amount of resolution degradation of a captured image due to a change in image magnification caused by moving the focus lens in a direction that cancels the rocking movement. (2) The imaging device described in (1) above, wherein the control unit executes control such that the correction operation by moving the focus lens becomes a correction operation with an intensity according to the first amount of resolution degradation and the second amount of resolution degradation. (3) The imaging device described in (2) above, wherein the control unit controls the intensity by setting a ratio of the amount of movement of the cancellation movement by the focus lens to the amount of deviation in subject distance due to the rocking movement. (4) The imaging device according to any one of (1) to (3) above, wherein the first amount of resolution degradation is a value calculated based on image magnification, an amount of shift in subject distance due to the swing, an amount of movement of the focus lens, and an F-number, and the second amount of resolution degradation is a value calculated based on a rate of change of image magnification at the position of the focus lens, an amount of movement of the focus lens, and an image height. (5) The imaging device according to any one of (1) to (4) above, wherein the control unit controls the intensity of the correction operation caused by movement of the focus lens so that the first amount of resolution degradation and the second amount of resolution degradation are equal. (6) The imaging device according to any one of (1) to (4) above, wherein the control unit sets a weighting coefficient between the first amount of resolution degradation and the second amount of resolution degradation, and assigns a weighting coefficient to one of them, and controls the intensity of the correction operation caused by movement of the focus lens so that the first amount of resolution degradation and the second amount of resolution degradation are equal.(7) The imaging device according to any one of (1) to (6), wherein the control unit sets an image height, acquires a rate of change of image magnification at a focus lens position, an F-number, and an image magnification, calculates a correction intensity coefficient using the image height, the rate of change of image magnification at the focus lens position, the F-number, and the image magnification, calculates a shake correction amount based on an amount of deviation in subject distance due to the swing and the correction intensity coefficient, and controls to move the focus lens to a focus lens position based on the shake correction amount. (8) The imaging device according to (7), wherein the control unit acquires the correction intensity coefficient by referring to table data that stores correction intensity coefficients according to the image height, the rate of change of image magnification at the focus lens position, the F-number, and the image magnification, between the start of exposure and the end of exposure. (9) The imaging device according to (7), wherein the control unit calculates the shake correction amount while keeping the correction intensity coefficient calculated from the image height, the rate of change of image magnification at the focus lens position at the start of exposure, the F-number, and the image magnification fixed until the end of exposure. (10) The imaging device according to any one of (7) to (9), wherein the control unit sets the image height based on the coordinates of an autofocus frame. (11) A correction control method, wherein the control unit of the imaging device executes control such that a correction operation by moving the focus lens in response to rocking of the device body in the optical axis direction becomes a correction operation corresponding to a first amount of resolution degradation of the captured image due to the rocking and a second amount of resolution degradation of the captured image due to a change in image magnification caused by moving the focus lens in a direction that cancels the rocking. (12) A program, wherein the control unit of the imaging device executes control such that a correction operation by moving the focus lens in response to rocking of the device body in the optical axis direction becomes a correction operation corresponding to the first amount of resolution degradation of the captured image due to the rocking and a second amount of resolution degradation of the captured image due to a change in image magnification caused by moving the focus lens in a direction that cancels the rocking.

[0112] REFERENCE SIGNS LIST 1 imaging device 2 camera body 3 lens barrel 10 lens system 10c focus lens 11 imaging element section 12 subject distance detection section 18 camera control section 20 lens control section 23 focus drive section 25 camera shake signal detection section 30 lens management section 31 iris control section 32 focus control section

Claims

1. An imaging device equipped with a control unit that executes control so that the correction operation by moving the focus lens in response to the oscillation of the device body in the optical axis direction becomes a correction operation according to a first amount of resolution degradation of the captured image due to the oscillation and a second amount of resolution degradation of the captured image due to a change in image magnification caused by moving the focus lens in a direction that cancels the oscillation.

2. The imaging device according to claim 1, wherein the control unit controls the correction operation by moving the focus lens to be a correction operation of an intensity according to the first amount of resolution degradation and the second amount of resolution degradation.

3. The imaging device according to claim 2, wherein the control unit controls the intensity by setting a ratio of the amount of cancellation movement by the focus lens to the amount of deviation in subject distance due to the swing.

4. The imaging device according to claim 1, wherein the first amount of resolution degradation is a value calculated based on the image magnification, the amount of deviation in subject distance due to the swing, the amount of movement of the focus lens, and the F-number, and the second amount of resolution degradation is a value calculated based on the rate of change in image magnification at the position of the focus lens, the amount of movement of the focus lens, and the image height.

5. The imaging device according to claim 1, wherein the control unit controls the strength of the correction operation performed by moving the focus lens so that the first amount of resolution degradation and the second amount of resolution degradation are equal.

6. The imaging device according to claim 1, wherein the control unit sets a weighting coefficient between the first amount of resolution degradation and the second amount of resolution degradation, and controls the strength of the correction operation by moving the focus lens so that the first amount of resolution degradation and the second amount of resolution degradation are equal after giving a weighting coefficient to one of them.

7. The imaging device according to claim 1, wherein the control unit sets an image height, acquires a rate of change of image magnification at a focus lens position, an F-number, and an image magnification, calculates a correction strength coefficient using the image height, the rate of change of image magnification at a focus lens position, the F-number, and the image magnification, calculates a shake correction amount based on the amount of deviation in subject distance due to the swing and the correction strength coefficient, and controls the focus lens to move to a focus lens position based on the shake correction amount.

8. The imaging device according to claim 7, wherein the control unit acquires the correction intensity coefficient by referring to table data that stores correction intensity coefficients according to image height, rate of change of image magnification at focus lens position, F-number, and image magnification, during the period from the start of exposure to the completion of exposure.

9. The imaging device according to claim 7, wherein the control unit calculates the amount of image blur correction by fixing the correction intensity coefficient calculated from the image height, the rate of change of image magnification at the focus lens position at the start of exposure, the F-number, and the image magnification until the end of exposure.

10. The imaging device according to claim 7, wherein the control unit sets the image height based on the coordinates of an autofocus frame.

11. A correction control method in which a control unit of an imaging device executes control such that a correction operation by moving a focus lens in response to a rocking movement of the device body in the optical axis direction becomes a correction operation according to a first amount of resolution degradation of the captured image due to the rocking movement and a second amount of resolution degradation of the captured image due to a change in image magnification caused by moving the focus lens in a direction that cancels the rocking movement.

12. A program that causes the control unit of an imaging device to execute control so that the correction operation by moving the focus lens in response to the oscillation of the device body in the optical axis direction becomes a correction operation according to a first amount of resolution degradation of the captured image due to the oscillation and a second amount of resolution degradation of the captured image due to a change in image magnification caused by moving the focus lens in a direction that cancels the oscillation.

Images