Imaging apparatus and control method thereof

Abstract

To provide an imaging apparatus and a control method thereof capable of effectively correcting image blurring caused by panning a moving subject whose speed is not constant.SOLUTION: An imaging apparatus predicts the movement of a subject during an exposure period on the basis of the movement of a subject acquired before the exposure period. The imaging apparatus also detects the movement of the subject during the exposure period. The imaging apparatus reflects a difference between the predicted movement of the subject and the detected movement of the subject in subject blur correction control based on the predicted movement of the subject, which is executed during the exposure period.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a control method thereof. [Background technology]

[0002] 2. Description of the Related Art Imaging devices that utilize an image blur correction function to assist in panning are known (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-54698 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, it is assumed that a moving subject moves linearly at a constant speed, so the movement of the moving subject can be predicted with high accuracy, and image blurring that occurs during panning can be effectively corrected. However, if the moving speed of the moving subject is not constant, the effect of correcting image blurring that occurs during panning decreases.

[0005] In view of the problems with the conventional technology, the present invention provides, in one embodiment, an imaging apparatus capable of effectively correcting image blurring that occurs when panning a moving subject whose speed is not constant, and a control method thereof. [Means for solving the problem]

[0006] In one aspect, the present invention provides an imaging device comprising: a prediction means for predicting a movement of a subject during an exposure period based on the movement of the subject acquired before the exposure period; a first detection means for detecting the movement of the subject during the exposure period; an acquisition means for acquiring a difference between the movement of the subject predicted by the prediction means and the movement of the subject detected by the first detection means; and a control means for performing image blur correction control in accordance with a correction amount based on the difference between the movement of the subject predicted by the prediction means and the movement of the subject acquired by the acquisition means. Effect of the Invention

[0007] According to the present invention, it is possible to provide an imaging apparatus and a control method thereof that can effectively correct image blurring that occurs when panning a moving subject whose speed is not constant. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera that is an example of an imaging apparatus according to an embodiment; [Diagram 2] FIG. 2 is a block diagram showing in more detail the configuration related to image blur correction in FIG. 1; [Diagram 3] FIG. 1 is a diagram for explaining a method for detecting a subject vector in an embodiment; [Figure 4] FIG. 1 is a diagram for explaining subject blur correction control in an embodiment; [Diagram 5] FIG. 1 is a diagram for explaining an example of a method for detecting the movement of a subject during an exposure period in an embodiment. [Figure 6] Flowchart regarding image blur correction control in the embodiment [Figure 7] FIG. 1 is a diagram for explaining an example of a method for sharing image blur correction in an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in a digital camera with an interchangeable lens that can be attached and detached. However, the present invention can also be implemented in any electronic device having an imaging function. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0011] 1 is a block diagram showing an example of the configuration of a lens-interchangeable digital camera as an example of an imaging device according to an embodiment of the present invention. The digital camera 1 has a body 100 and a lens unit 200 (interchangeable lens) that is detachable from the body 100. The body 100 and the lens unit 200 are connected via a mount that mechanically engages with each other so as to be detachable. The mount provided on the lens unit 200 and the mount provided on the body 100 are provided with terminals that are configured to come into contact when the lens unit 200 is attached to the body 100. The terminals include a terminal that supplies power from the body 100 to the lens unit 200, and a terminal for communication between the camera side communication unit 135 and the lens side communication unit 128.

[0012] The lens unit 200 has an imaging optical system (or imaging lens) 150 that generates an optical image of a subject on the imaging plane of an imaging element 106 included in the main body 100. The imaging optical system 150 has a plurality of optical lenses including a movable lens, and an aperture 104. For the sake of convenience, Fig. 1 shows only the zoom lens 101, the image blur correction lens 102, and the focus lens 103, which are movable lenses among the optical lenses. Note that each of these movable lenses may actually be composed of a plurality of lenses.

[0013] The zoom lens 101 is driven in the optical axis direction by a zoom lens drive unit 124 to change the focal length (angle of view) of the imaging optical system 150. The image blur correction lens 102 can be moved in a direction perpendicular to the optical axis by a correction lens drive unit 122 to optically correct (suppress) image blur caused by the movement of the digital camera 1. The focus lens 103 is driven in the optical axis direction by a focus lens drive unit 121 to change the distance at which the imaging optical system 150 is in focus. The aperture 104 has an opening amount controlled by an aperture drive unit 120 to adjust the amount of light entering the body 100 from the imaging optical system 150.

[0014] The lens control unit 160 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing a program. The lens control unit 160 reads a program stored in the ROM 142 into the RAM 141 and executes the program, thereby controlling the operation of the lens unit 200. The lens control unit 160 also controls the operation of the lens unit 200 and outputs information about the lens unit 200 to the camera control unit 115 in response to instructions or requests from the camera control unit 115.

[0015] In the figure, the functional blocks described inside the lens control unit 160 are schematic representations of various functions that are realized by the lens control unit 160 executing a program. Therefore, the operations of the functional blocks inside the lens control unit 160 in this specification are actually performed by the lens control unit 160. Details of the functional blocks shown inside the lens control unit 160 will be described later.

[0016] The ROM 142 is a rewritable non-volatile memory, and stores programs executed by the lens control unit 160, information and setting values ​​of the lens unit 200, etc. The RAM 141 is used to read programs executed by the lens control unit 160 and to temporarily store information of the lens unit 200. At least one of the ROM 142 and the RAM 141 may be integrated with the lens control unit 160.

[0017] The shutter 105 is a mechanical focal plane shutter. Note that an electronic shutter may be used instead of or in addition to the shutter 105. The electronic shutter is a shutter operation realized by controlling the operation of the image sensor 106.

[0018] The imaging element 106 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element 106 has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A pixel signal group (analog image signal) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0019] In this embodiment, the image sensor 106 is configured to be movable in a direction perpendicular to the optical axis of the imaging optical system 150. The position of the image sensor 106 is controlled by the sensor vibration isolation driving unit 130. In this manner, the digital camera 1 according to this embodiment has an optical image blur correction function in each of the lens unit 200 and the body 100.

[0020] The AD converter 107 applies noise removal processing, gain adjustment processing, and AD conversion processing to the analog image signal read out from the image sensor 106 to generate a digital image signal (image data). The AD converter 107 outputs the digital image data to the image processing circuit 109. The AD converter 107 may be provided in the image sensor 106.

[0021] A timing generator 108 follows instructions from a camera control unit 115 and supplies signals for controlling the operation timing to the image sensor 106 and the AD converter 107 .

[0022] As described later, in this embodiment, the image sensor 106 may be a stacked image sensor in which one or more substrates each having a memory, a signal processing circuit, and an output circuit are stacked on a substrate having a pixel array. Also, there are two output circuits, each of which has the function of the AD converter 107. Therefore, the image sensor 106 can output separate images, such as a still image and a video, in parallel from the two output circuits.

[0023] The image processing circuit 109 applies predetermined image processing to the image data output by the AD converter 107, generates signals and image data according to the application, and acquires and / or generates various information. The image processing circuit 109 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing circuit 109 may be configured to realize a specific function by a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executing software. The image processing circuit 109 outputs the acquired or generated information and data to the camera control unit 115, RAM 110, etc. according to the application.

[0024] The image processing applied by the image processing circuit 109 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on an image sensor, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system 150 (image restoration), correction of the effect of peripheral light falloff of the imaging optical system 150, color correction, and the like. The detection process may include detection of feature regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. The data processing may include processes such as area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. The generation of image data for display (including image data for live view display) and image data for recording is also included in the data processing. The evaluation value calculation process can include processes such as generation of a signal and evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are examples of processes that can be applied by the image processing circuit 109, and do not limit the processes that can be applied by the image processing circuit 109. In addition, some of the image processing listed here may be executed by a signal processing circuit incorporated in the image sensor 106.

[0025] The camera control unit 115 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing a program. The camera control unit 115 reads a program stored in the ROM 113 into the RAM 110 and executes it, thereby controlling the operation of the main body 100 and the lens unit 200 and realizing the functions of the digital camera 1. The camera control unit 115 controls the operation of the lens unit 200 through communication with the lens control unit 160.

[0026] In the figure, the functional blocks described inside camera control unit 115 are schematic representations of various functions realized by camera control unit 115 executing programs. Therefore, the operations of the functional blocks inside camera control unit 115 in this specification are actually performed by camera control unit 115.

[0027] ROM 113 is a rewritable non-volatile memory, and stores programs executed by camera control unit 115, various setting values ​​of digital camera 1, GUI data, etc. RAM 14 is used to read programs executed by camera control unit 115, and for image processing circuit 109 to temporarily store data to be processed, intermediate data, data resulting from processing, etc. RAM 110 is used as a buffer memory for image data, and as a video memory for display unit 111.

[0028] The display unit 111 is provided, for example, on the surface of the housing of the digital camera 1, and displays a live view image, information about the digital camera 1, information about the captured scene, a menu screen, etc. The display unit 111 may be a touch display.

[0029] The operation unit 114 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to the digital camera 1. The input devices constituting the operation unit 114 have names according to the functions assigned to them. For example, the operation unit 114 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, direction keys, a decision key, etc. The release switch is a switch for recording still images, and the camera control unit 115 recognizes the half-pressed state of the release switch as an instruction to prepare for shooting, and the full-pressed state as an instruction to start shooting. In addition, the camera control unit 115 recognizes the video recording switch, when pressed in a shooting standby state, as an instruction to start recording a video, and recognizes the video recording switch, when pressed during video recording, as an instruction to stop recording. Note that the functions assigned to the same input device may be variable.

[0030] When the camera control unit 115 detects an operation on the operation unit 114, it executes an operation corresponding to the detected operation. For example, when the camera control unit 115 detects an operation to change the angle of view of the lens unit 200, it periodically transmits an angle of view change instruction to the lens control unit 160 while the operation is being detected. Each time the lens control unit 160 (zoom lens driving unit 124) receives an angle of view change instruction, it drives the zoom lens 101 by a fixed amount.

[0031] The exposure control unit 139 determines the exposure conditions (aperture value, shutter speed (or exposure time), and imaging sensitivity) based on, for example, an evaluation value for AE generated by the image processing circuit 109 and a setting of the digital camera 1 (for example, an imaging mode). When the exposure conditions for still image capturing are determined, the exposure control unit 139 outputs the shutter speed to the shutter driving unit 136 and the aperture value to the lens control unit 160. When the exposure control unit 139 determines the exposure conditions for video shooting, the exposure control unit 139 outputs the exposure time to the timing generator 108. When the aperture value is changed, the exposure control unit 139 outputs the aperture value to the lens control unit 160. The exposure control unit 139 outputs the imaging sensitivity to the image sensor 106 or the image processing circuit 109.

[0032] The lens control unit 160 (aperture driving unit 120) drives the aperture 104 in accordance with the aperture value received from the camera control unit 115.

[0033] The focus lens control unit 140 determines the drive amount and drive direction for moving the focus lens 103 from the current position to the target position based on, for example, an evaluation value for AF generated by the image processing circuit 109. If the evaluation value for AF is a contrast evaluation value, the focus lens control unit 140 determines the drive amount and drive direction with the focus lens position at which the evaluation value is maximized as the target position. If the evaluation value for AF is a defocus amount, the focus lens control unit 140 converts the defocus amount into a drive amount and drive direction. The focus lens control unit 140 transmits the determined drive amount and drive direction to the lens control unit 160. The lens control unit 160 (focus lens driving unit 121) drives the focus lens 103 according to the drive amount and drive direction received from the camera control unit 115.

[0034] The following describes the functional blocks related to image blur correction. When capturing a still image, digital camera 1 corrects both image blur (camera shake) caused by the movement of body 100 and lens unit 200, and image blur (subject blur) caused by the movement of the subject during exposure.

[0035] Camera-side motion detection unit 134 detects the motion of main body 100, and outputs a signal corresponding to the detected motion to camera-side anti-shake control unit 133. Lens-side motion detection unit 125 detects the motion of lens unit 200, and outputs a signal corresponding to the detected motion to lens-side anti-shake control unit 126. Camera-side motion detection unit 134 and lens-side motion detection unit 126 may be, for example, a gyro sensor that outputs a signal corresponding to the angular velocity around each axis of a Cartesian coordinate system, or a combination of a gyro sensor and an acceleration sensor that outputs a signal corresponding to the acceleration in each axial direction of a Cartesian coordinate system.

[0036] In this embodiment, main body 100 and lens unit 200 each have a separate motion detection unit (second detection means), but a motion detection unit may be provided in either main body 100 or lens unit 200. For example, if lens unit 200 does not have a motion detection unit, an output signal from the motion detection unit of main body 100 can be sent to both camera-side anti-shake drive unit 133 and lens-side anti-shake control unit 126.

[0037] The image sensor vibration isolation position detector 132 detects the current position of the image sensor 106. The current position may be, for example, a relative displacement with respect to a reference position. The reference position may be a position where the optical axis intersects with the center of the pixel array of the image sensor 106 at right angles. The displacement may be in two directions parallel to the short and long sides of the image sensor.

[0038] The sensor vibration compensation control unit 133 determines the amount of correction by which to move the image sensor 106 so as to cancel the movement detected by the camera-side motion detection unit 134. The sensor vibration compensation control unit 133 determines the amount of correction within a range not exceeding the maximum amount of movement in each direction relative to the reference position. The sensor vibration compensation control unit 133 then compensates for image blur by moving the image sensor 106 via the sensor vibration compensation drive unit 130 according to the determined amount of correction.

[0039] The correction lens position detection unit 123 detects the current position of the image blur correction lens 102. The current position may be, for example, a relative displacement with respect to a reference position. The reference position may be a position where the optical axis intersects with the center of the image blur correction lens 102 at right angles. The displacement may be in two directions parallel to the short side and the long side of the image sensor.

[0040] The lens vibration reduction control unit 126 determines the amount of correction by which to move the image blur correction lens 102 so as to cancel the movement detected by the lens-side motion detection unit 125. The lens vibration reduction control unit 126 determines the amount of correction within a range not exceeding the maximum amount of movement in each direction relative to the reference position. The lens-side vibration reduction control unit 126 then corrects image blur by moving the image blur correction lens 102 via the correction lens driving unit 122 according to the determined amount of correction.

[0041] The image blur correction described here may be performed by either the main body 100 or the lens unit 200, or by both. When image blur correction is performed by both the main body 100 and the lens unit 200, for example, the amount of movement of one unit that exceeds the maximum amount of movement of the other unit is compensated for by the other unit. Alternatively, the required amount of movement may be allocated to the main body 100 and the lens unit 200 at a predetermined ratio.

[0042] Instead of or in addition to optical image stabilization that moves an anti-shake member (here, one or both of the image stabilization lens 102 and the image sensor 106), electronic image stabilization can be performed. Electronic image stabilization can be realized by moving the position at which an effective area is cut out from a captured image so as to cancel out the movement of the digital camera 1. For example, the amount of movement of the image sensor 106 determined by the camera-side anti-shake control unit 133 can be converted into the amount of movement on the pixel array, and the cut-out position can be changed. In this embodiment, the image synthesis processing unit 131 converts the amount of movement and performs the cut-out.

[0043] The motion vector detection unit 143 (first detection means) detects the movement between frames as a motion vector, for example, by a template matching method. The motion vector detection unit 143 can also be used to detect the movement of the digital camera 1, but in this embodiment it is mainly used to detect subject blur. The method of detecting subject blur will be described later. When detecting the motion vector of the entire frame, electronic image blur correction can be performed by changing the cut-out position based on the amount of movement represented by the motion vector.

[0044] The camera information management unit 137 is used to store information about the main body 100, such as setting information for the main body 100, the current position of the image sensor 106, driving limits (maximum amount of movement in each direction), and subject blur correction amount, as well as information acquired by the main body 100.

[0045] The lens information management unit 129 is used to store information about the lens unit 200 and information acquired by the lens unit 200, such as the optical characteristics, current position, and driving limit (maximum movement amount) of the image blur correction lens 102.

[0046] FIG. 2 is a block diagram for further explaining the configuration related to image blur correction in the lens unit 200 and the main body 100. In the figure, the functional blocks described inside the lens side anti-shake driving unit 126 are schematic diagrams showing various functions realized by the lens control unit 160 executing a program. Therefore, the operation of the functional blocks inside the lens side anti-shake control unit 126 described below is actually performed by the lens control unit 160. Similarly, the operation of the functional blocks described inside the camera side anti-shake driving unit 133 is actually performed by the camera control unit 115. The camera control unit 115 and the lens control unit 160 function as control means for cooperatively performing image blur correction control. The lens control unit 160 may be configured to function as the lens anti-shake driving unit 126 or as a part of the functional blocks inside the lens anti-shake driving unit 126 by being controlled by the camera control unit 115 via the camera side communication unit 135 and the lens side communication unit 128.

[0047] First, there will be described a configuration relating to image blur correction in the lens unit 200. The lens side integration section 151 integrates the angular velocity signal output from the lens side motion detection section 125 and converts it into an angle signal.

[0048] A lens-side blur correction amount determination unit 152 determines the amount of correction corresponding to the amount of movement of the image blur correction lens 102 required to correct the image blur, based on the angle signal output by the lens-side integration unit 151 .

[0049] Adder 153 reflects (adds) the amount of subject blur compensation determined by main body 100 (camera control unit 115) and acquired by subject blur compensation amount acquisition unit 157, to the amount of compensation determined by lens side blur compensation amount calculation unit 152. A method for determining the amount of subject blur compensation will be described later.

[0050] Lens side ratio determination section 154 determines the ratio of the amount of movement that lens unit 200 is responsible for when the total amount of movement required to correct camera shake and subject blur is 100%. Note that when only one of lens unit 200 and main body 100 is used, the ratio determined by lens side ratio determination section 154 is 100% or 0%.

[0051] Whether image blur correction is performed by both lens unit 200 and main body 100 or only one of them can be determined according to any condition. For example, it may be according to a user setting, or it may be determined to be performed by both main body 100 and lens unit 200 when the total movement amount exceeds the maximum movement amount of correction lens 102 or image sensor 106. Lens side ratio determination section 154 can make the determination based on information stored in camera information management section 137 and lens information management section 129.

[0052] The lens side driving range limiter 155 limits the amount of correction so that the image blur correction lens 102 moves within a range not exceeding the maximum amount of movement.

[0053] The correction lens controller unit 156 controls the position of the image blur correction lens 102 via the correction lens drive unit 122. The correction lens controller unit 156 performs feedback control based on the current position of the image blur correction lens 102 detected by the correction lens position detection unit 123 so that the image blur correction lens 102 moves to a target position according to the amount of movement.

[0054] Next, there will be described a configuration relating to image blur correction in the main body 100. The camera-side integration section 161 integrates the angular velocity signal output from the camera-side motion detection section 131, and converts it into an angle signal.

[0055] Based on the angle signal output by camera side integrator 161, camera side blur correction amount determination unit 162 determines the amount of correction corresponding to the amount of movement of image sensor 106 required to correct image blur.

[0056] Subject vector detection section 163 detects a subject vector that represents subject blur based on the inter-frame motion vector detected by motion vector detection section 143. The subject vector detection operation will be described in detail later.

[0057] The subject blur correction amount determination section 164 determines the amount of correction for correcting subject blur based on the subject vector.

[0058] Adder 165 adds the amount of correction for camera shake correction determined by camera side shake correction amount determination section 162 and the amount of subject shake correction calculated by subject shake correction amount calculation section 164 .

[0059] Camera side ratio determination section 166 determines the proportion of the amount of movement that main body 100 is responsible for when the total amount of movement required to correct camera shake and subject shake is 100%. When only one of lens unit 200 and main body 100 is used, the proportion determined by camera side ratio determination section 166 is 100% or 0%. Camera side ratio determination section 166 can make its determination based on information stored in camera information management section 137 and lens information management section 129.

[0060] The camera-side driving range limiter 167 limits the amount of correction so that the image sensor 106 moves within a range that does not exceed the maximum amount of movement.

[0061] The camera-side controller unit 168 controls the position of the image sensor 106 through the sensor vibration isolation driving unit 122. The camera-side controller unit 168 performs feedback control based on the current position of the image sensor 106 detected by the sensor position detection unit 132 so that the image sensor 106 moves to a target position according to the amount of movement.

[0062] Here, a method for detecting a subject vector in subject vector detection section 163 will be described with reference to Fig. 3. Note that a subject is an object that exists in front of the background, and is generally a person. Here, an area that moves between frames is detected as a subject area, and a vector that represents the movement of the subject area is detected as a subject vector.

[0063] The motion vector detection unit 143 detects a motion vector between the reference frame and each region set in the target frame. Fig. 3(A) shows an example of setting a region in the target frame for detecting a motion vector. In this case, in order to reduce the processing load, a motion vector detection range 401 is set in a part of the target frame, and a motion vector is detected for each region obtained by dividing the detection range 401. The detection of the motion vector can be performed by template matching, which uses an image of each region as a template and detects a region in the reference frame with the highest correlation as a destination region.

[0064] Fig. 3(B) is a histogram of the magnitude of the motion vector detected for each region shown in Fig. 3(A). Subject vector detection unit 163 detects the subject vector based on this histogram. Subject vector detection unit 163 converts the angle signal output by camera side integration unit 151 or lens side integration unit 161 into movement amount 402 on the imaging plane. This conversion can be performed using, for example, the focal length of imaging optical system 150, the imaging time difference or frame rate between the target frame and the reference frame, and the pixel pitch of imaging element 106.

[0065] The output origin of camera side motion detection section 134 or lens side motion detection section 125 fluctuates due to disturbances such as temperature drift (offset phenomenon). For this reason, subject vector detection section 163 determines a predetermined range centered on movement amount 402 as background determination area 403. Then, subject vector detection section 163 determines a motion vector whose size is included in background determination area 403 as background vector 404. Moreover, subject vector detection section 163 determines a motion vector whose size is not included in background determination area 403 as subject vector 405.

[0066] The subject vector 405 can be used to correct subject blur, and the background vector 404 can be used to correct camera shake.

[0067] Although the method of separating the motion vector into the subject vector 405 and the background vector 404 based on the angular velocity signal (or angle signal) has been described here, other methods may be used. For example, if the movement of the digital camera 1 or the subject is small, it becomes difficult to separate the subject vector from the background vector using a method based on the angular velocity or angle.

[0068] For example, by using the subject distance information, the motion vector can be separated into a subject vector and a background vector even when the digital camera 1 or the subject moves little. The subject distance information may be, for example, a depth map indicating the subject distance for each region obtained by dividing the imaging range or for each pixel. Here, as a simpler method, an example of separating the subject region and the background region based on the AF frame with the highest degree of focus will be described. In FIG. 3C, 406 indicates a plurality of AF frames (focus detection regions) that can be set in the digital camera 1. Also, 407 indicates the AF frame with the highest degree of focus among the plurality of AF frames 406. In this case, the subject vector detection unit 163 sets the subject region 408 based on the subject distance of the AF frame 407, and sets the motion vector detected within the subject region 408 as the subject vector, and the motion vector detected in the other regions as the background vector. The subject vector detection unit 163 can set, for example, an AF frame whose difference in focus level with the AF frame 407 is within a predetermined threshold range, or an AF frame whose difference in focus level with the AF frame 407 is within a predetermined threshold range (for example, ±2 depth) based on the depth of field of the AF frame 407 as the subject area 408.

[0069] The method using the movement of the digital camera 1 and the method using the subject distance information may be used in combination. In this case, a representative value (average value, mode, etc.) of the same type of vector determined by each method may be used. Also, the subject region may be detected based on machine learning or a known feature region detection technique, and the motion vector of the subject region may be detected as subject blur. FIG. 3(D) shows a schematic diagram of a state in which a predetermined subject region (here, a human face region) 409 has been detected.

[0070] The detection of subject blur can be performed using frames of a moving image captured for live view display. In this embodiment, a stacked image sensor 106 that supports multi-streaming and can capture a moving image for live view display in parallel with capturing (exposing) a still image is used, so that subject blur during the exposure period of a still image can be detected.

[0071] 4 is a schematic diagram for explaining detection and correction of subject blurring during the process of capturing a still image from a standby state for still image capture and returning to the standby state. The image sensor 106 has a configuration in which a pixel array 501 and a signal processing circuit 502, which are provided on separate substrates, are stacked. The image sensor 106 can output two systems of image data, such as a moving image and a still image, in parallel by dividing the image signal read out from the pixel array 501 into two systems by the signal processing circuit 502. Here, an example is shown in which image data for recording 503 and image data for evaluation 504 are output in parallel.

[0072] The recording image data 503 is image data for recording, including still image data. On the other hand, the evaluation image data 504 is image data used for live view display, generation of evaluation values ​​for AF and AE, detection of characteristic regions, detection of motion vectors, etc. The evaluation image data 504 can have a lower resolution than the recording image data 503. The remainder of FIG. 4 will be described below in conjunction with FIG. 6.

[0073] It is also possible to detect subject blur during the exposure period without using a stacked imaging element. For example, a LiDAR sensor or the like is used to obtain distance information of a range including an imaging range, for example, at a period corresponding to the frame rate of the evaluation image data 504. Then, the amount of movement of the subject in a direction parallel to the imaging surface during one period is detected. The amount of movement can be calculated based on the horizontal resolution of the LiDAR sensor, etc.

[0074] Fig. 5 shows a schematic diagram of a case where a subject moving in a direction parallel to the imaging plane is detected by comparing two successive pieces of distance information. For example, if distance information is acquired at 1 / 30 second intervals, the time t required for the subject to move from position A to B is 33.3 ms.

[0075] 5, if the principal point (assuming there is only one) of the imaging optical system 150 is C, a triangle ABC with one side being the distance (movement amount Δ) between positions A and B is similar to a triangle abC with one side being the movement amount δ of the subject on the imaging surface. In addition, the ratio between triangles ABC and abC is (Lf):f, where L is the subject distance and f is the focal length of the imaging optical system 150.

[0076] therefore, Amount of movement of the subject on the imaging plane δ = Amount of movement of the subject Δ × f / (Lf) This relationship holds even if the subject's speed is different between point A and point B.

[0077] The amount of movement on the imaging surface can be converted into angular velocity by treating it as a horizontal motion vector, for example, and executing the reverse procedure for converting the movement of the digital camera 1 into the amount of movement on the imaging surface. Note that the method described here is merely an example, and the movement of the subject during the exposure period may be detected by other methods without using the image sensor 106.

[0078] Next, the image stabilization operation in this embodiment will be further described with reference to the flowchart shown in Fig. 6. Here, it is assumed that digital camera 1 is in a standby state for still image shooting and is shooting a moving image for live view display.

[0079] In S301, the camera-side motion detector 134 and the lens-side motion detector 125 detect the motion (camera shake) of the digital camera 1 as an angular velocity signal.

[0080] In S302, camera side blur compensation amount determination unit 162 and lens side blur compensation amount determination unit 152 determine the amount of compensation corresponding to the amount of movement of image sensor 106 and image blur compensation lens 102 required to cancel the movement detected in S301.

[0081] In S303, subject vector detection section 163 detects a subject vector representing subject blur using any of the methods described above.

[0082] In S304, subject blur correction amount determination section 164 converts the subject vector detected in S303 into the angular velocity of the subject. This conversion can be performed by reversing the procedure for converting the movement of digital camera 1 into the amount of movement on the imaging surface. The angular velocity of the subject is used to predict the movement of the subject within a frame. For this reason, subject blur correction amount determination section 164 stores in RAM 110 the angular velocities of the subject obtained in, for example, the most recent predetermined number of frames.

[0083] In S305, the camera control unit 115 determines whether or not an instruction to start shooting has been detected, and if it is determined that an instruction to start shooting has been detected, the process proceeds to S306. On the other hand, if it is not determined that an instruction to start shooting has been detected, the camera control unit 115 returns the process to S301. The processes of S301 to S304 are repeatedly executed every time one frame of a moving image for live view display is shot.

[0084] S306 to S311 are executed for each frame of the evaluation image data 504 during the exposure period of the still image (frames N+1 to N+3 in FIG. 4). In S306, subject blur correction amount determination unit 164 predicts angular velocity 509 of the subject during the exposure period of the still image from subject blur 505-508 detected as angular velocity of the subject during the shooting standby state (1st to Nth frames in FIG. 4). Note that only the most recent predetermined number of angular velocities need to be used for the prediction. The prediction can be performed using a known method such as the least squares method.

[0085] Then, subject blur correction amount determination unit 164 determines the correction amount for correcting subject blur based on predicted subject angular velocity 509. Note that the correction amount determined here corresponds to the movement amount according to the vibration isolation member (imaging element 106 and / or image blur correction lens 102) to be moved for image blur correction.

[0086] Based on this correction amount and the correction amount for camera shake correction, the camera-side image stabilization control unit 133 and / or the lens-side image stabilization control unit 126 drive the imaging element 106 and / or the lens image shake correction lens 102 as described above. When the image shake correction lens 102 is driven based on the correction amount determined here, information indicating the correction amount for correcting subject shake determined by the subject shake correction amount determination unit 164 is transmitted to the subject shake correction amount acquisition unit 157 of the lens-side image stabilization control unit 126. In this way, the camera-side image stabilization control unit 133 controls the subject shake correction using the correction lens 102.

[0087] In S307, subject vector detection section 163 detects a subject vector that represents subject blur, similarly to S303.

[0088] Furthermore, in S308, subject blur correction amount determination unit 164 converts the subject vector detected in S307 into the angular velocity of the subject in the same manner as in S304, and stores it in RAM 110. Reference numerals 510 to 512 in Fig. 4 indicate the angular velocity of the subject obtained in each frame (N+1 to N+3 frames) of evaluation image data 504 during the exposure period.

[0089] In S309, subject blur correction amount determination unit 164 calculates the deviation (difference) between the angular velocity of the subject detected in S308 and the angular velocity of the subject predicted in S306. This deviation is the difference between the actual movement of the subject during the exposure period and the predicted movement, and is the cause of subject blur.

[0090] In S310, the subject blur correction amount determination unit 164 determines an additional correction amount in the same manner as in S306, based on the deviation of the angular velocity of the subject calculated in S309. Note that the additional correction amount determined here is a correction amount corresponding to the movement amount according to the member (imaging element 106 and / or image blur correction lens 102) to be moved for image blur correction.

[0091] Here, an example of a method for determining which image blur correction member (the image pickup element 106 and / or the image blur correction lens 102) to drive in order to perform additional subject blur correction according to the deviation will be described with reference to FIG.

[0092] 7 shows an example of the relationship between the correctable angle [deg] of the image stabilization lens 102 and the image sensor 106 and the focal length [mm] of the imaging optical system 150. In the focal length range less than f1, the image sensor 106 has a larger correctable angle than the image stabilization lens 102. Also, in the focal length range of f1 or more, the image stabilization lens 102 has a larger correctable angle than the image sensor 106. Note that f1 can change depending on the conditions, but generally, the image sensor 106 has a larger correctable angle than the image stabilization lens 102 on the wide-angle side.

[0093] For example, subject blur compensation amount determination unit 164 predicts the maximum deviation during the exposure period by multiplying the deviation between the subject's angular velocity predicted before exposure and the subject's angular velocity detected in the first frame of the exposure period by the shutter speed (exposure time). Then, subject blur compensation amount determination unit 164 can determine the type of blur compensation member to be driven so as to cover the compensation angle corresponding to the maximum deviation.

[0094] For example, suppose that the focal length of the imaging optical system 150 when capturing a still image is f2. In this case, the product 603 of the predicted angular velocity 509 of the subject and the shutter speed exceeds the correctable angle 602 of the image sensor 106. On the other hand, it is smaller than the correctable angle of the image blur correction lens 102. In this case, the subject blur correction amount determination unit 164 determines to use at least the image blur correction lens 102.

[0095] For example, because control is simple, the subject blur compensation amount determination unit 164 determines that the predicted subject blur should be compensated for only by the image blur compensation lens 102. Also, the sum of the product 604 of the deviation in the angular velocity of the subject and the shutter speed and the product 603 of the predicted angular velocity 509 of the subject and the shutter speed is smaller than the correctable angle of the image blur compensation lens 102. Therefore, the subject blur compensation amount determination unit 164 determines that additional correction of subject blur according to the deviation should also be performed only by the image blur compensation lens 102.

[0096] Note that the product 604 of the deviation in the angular velocity of the subject and the shutter speed is smaller than the correctable angle 602 of the image sensor 106 at the focal length f2. Therefore, the subject blur correction amount determination section 164 may determine that additional correction of subject blur according to the deviation should be performed by the image sensor 106. Basically, the type of blur correction member to be driven can be determined so that both the predicted value of the movement and the deviation between the predicted value and the actual measured value can be corrected.

[0097] When the sum of the product 604 of the deviation in the angular velocity of the object and the shutter speed and the product 603 of the predicted angular velocity 509 of the object and the shutter speed is smaller than the correctable angle of the image blur correction lens 102, image blur correction can be performed only by the image blur correction lens 102. However, image blur correction can also be performed by using the image sensor 106 in combination. For example, the predicted value of object blur can be corrected using the image blur correction lens 102, and the deviation between the predicted value of object blur and the measured value can be corrected using the image sensor 106.

[0098] Image blur correction that also uses the image sensor 106 has at least the following two advantages. The first is that it does not require communication between the main body 100 and the lens unit 200. For example, when correcting the deviation between a predicted value and an actual measured value of subject blur using the image blur correction lens 102, it is necessary to transmit the amount of correction based on the deviation from the main body 100 to the lens unit 200 (subject blur correction amount acquisition unit 157).

[0099] When communication between the main body 100 and the lens unit 200 is performed periodically, the period of image blur correction using communication is limited to the communication period. Therefore, when the communication period is slower than the period for acquiring the deviation of the subject blur, image blur correction for the deviation of the subject blur can be performed in a shorter period by using the image sensor 106 than by using the image blur correction lens 102. Also, since information other than the amount of correction is exchanged in the communication between the main body 100 and the lens unit 200, it is not always possible to transmit the amount of correction for each communication. Therefore, even if the period for acquiring the deviation of the subject blur is equal to the communication period, the period of image blur correction using communication can be longer than the communication period.

[0100] For example, suppose that the period during which subject blur correction amount calculation unit 164 obtains the correction amount based on the deviation is 60 times / second, and the period during which communication between main body 100 and lens unit 200 is 30 times / second. In this case, the execution period of image blur correction using the correction amount based on the deviation is limited to 30 times / second when image blur correction lens 102 is used. However, when image sensor 106 is used, this limitation does not apply, and correction can be executed at a maximum period of 60 times / second.

[0101] Another advantage of image stabilization using the image sensor 106 in combination is that even if the deviation between the predicted value and the actual measured value suddenly becomes large, the image stabilization capacity of the image sensor 106 can be fully utilized for correcting the deviation. This increases the possibility of correcting even if the change in the subject's movement is large. This is particularly advantageous when the subject suddenly moves erratically during the exposure period, causing the deviation to suddenly increase.

[0102] As described above, according to this embodiment, the subject movement detected during the exposure period is reflected in the image blur correction control based on the subject movement predicted based on the subject movement before the exposure period of the still image. This makes it possible to appropriately correct subject blur even if the subject movement changes during the exposure period, thereby improving the performance of the panning support function.

[0103] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0104] The disclosure of the present embodiment includes the following imaging device, a control method for the imaging device, and a program. (Item 1) a prediction means for predicting a motion of a subject during an exposure period based on a motion of the subject acquired before the exposure period; a first detection means for detecting a motion of the subject during the exposure period; an acquisition means for acquiring a difference between the motion of the subject predicted by the prediction means and the motion of the subject detected by the first detection means; a control unit that performs image blur correction control in accordance with a correction amount based on the movement of the subject predicted by the prediction unit and the difference acquired by the acquisition unit; An imaging device comprising: (Item 2) 2. The imaging device according to item 1, wherein the first detection means detects the movement of the subject during the exposure period based on a moving image captured during the exposure period. (Item 3) 3. The imaging device according to item 2, wherein the imaging device captures the moving image during the exposure period using an imaging element capable of capturing a plurality of images in parallel. (Item 4) 2. The imaging device according to item 1, wherein the first detection means detects the movement of the subject during the exposure period based on distance information of an imaging range acquired during the exposure period. (Item 5) Further comprising a second detection means for detecting a motion of the imaging device, the control means performs image blur correction control in accordance with a correction amount that reflects the movement of the imaging device detected by the second detection means. 5. The imaging device according to any one of items 1 to 4. (Item 6) 6. The imaging device according to claim 1, wherein the control unit performs image blur correction control by controlling one or more positions of an imaging element and a correction lens in accordance with the correction amount. (Item 7) 7. The imaging device according to item 6, wherein the control means controls a position of the compensation lens based on the movement of the subject predicted by the prediction means, and controls a position of the imaging element based on the difference acquired by the acquisition means. (Item 8) 8. The imaging device according to item 7, wherein the interchangeable lens is detachable, and the control means controls the position of the compensation lens by transmitting information indicating the movement of the subject predicted by the prediction means to the interchangeable lens. (Item 9) 7. The imaging device according to item 6, wherein when the amount of correction cannot be realized by one of the imaging element and the correction lens, the control means realizes the amount of correction by using the other or both. (Item 10) Further comprising a second detection means for detecting a motion of the imaging device, the control means performs the image blur correction control so that a correction amount based on the movement of the subject predicted by the prediction means and the difference acquired by the acquisition means is realized in one of the image sensor and the compensation lens, and a correction amount based on the movement of the image pickup device detected by the second detection means is realized in the other of the image sensor and the compensation lens. 7. The imaging device according to item 6, (Item 11) A control method executed by an imaging device, comprising: predicting a motion of a subject during an exposure period based on a motion of the subject acquired before the exposure period; detecting a motion of the subject during the exposure period; obtaining a difference between a predicted motion of the object and a detected motion of the object; performing an image blur correction control in accordance with a correction amount based on the predicted movement of the subject and the difference; 13. A method for controlling an imaging apparatus comprising: (Item 12) A program for causing a computer included in an imaging apparatus to function as each of the means included in the imaging apparatus according to any one of items 1 to 10.

[0105] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0106] 100: main body, 115: camera control unit, 126: lens side vibration control unit, 133: camera side vibration control unit, 143: motion vector detection unit, 160: lens control unit, 200: lens unit

Claims

1. a prediction means for predicting a motion of a subject during an exposure period based on a motion of the subject acquired before the exposure period; a first detection means for detecting a motion of the subject during the exposure period; an acquisition means for acquiring a difference between the motion of the subject predicted by the prediction means and the motion of the subject detected by the first detection means; a control unit that performs image blur correction control in accordance with a correction amount based on the movement of the subject predicted by the prediction unit and the difference acquired by the acquisition unit; An imaging device comprising:

2. 2. The imaging apparatus according to claim 1, wherein the first detection means detects the movement of the subject during the exposure period based on a moving image captured during the exposure period.

3. 3. The imaging device according to claim 2, wherein the imaging device captures the moving image during the exposure period using an imaging element capable of capturing a plurality of images in parallel.

4. 2. The imaging apparatus according to claim 1, wherein the first detection means detects the movement of the subject during the exposure period based on distance information of an imaging range acquired during the exposure period.

5. a second detection means for detecting a motion of the imaging device; the control means performs image blur correction control in accordance with a correction amount that reflects the movement of the imaging device detected by the second detection means. The imaging device according to claim 1 .

6. 2. The image pickup apparatus according to claim 1, wherein the control means performs image blur correction control by controlling at least one position of an image pickup element and a correction lens in accordance with the correction amount.

7. 7. The imaging device according to claim 6, wherein the control means controls a position of the compensation lens based on the movement of the subject predicted by the prediction means, and controls a position of the imaging element based on the difference acquired by the acquisition means.

8. The interchangeable lenses are removable, 8. The imaging apparatus according to claim 7, wherein the control means controls the position of the compensation lens by transmitting information indicating the movement of the subject predicted by the prediction means to the interchangeable lens.

9. 7. The imaging apparatus according to claim 6, wherein, when the amount of correction cannot be achieved by one of the imaging element and the correction lens, the control means achieves the amount of correction by using the other or both of the imaging element and the correction lens.

10. a second detection means for detecting a motion of the imaging device; the control means performs the image blur correction control so that a correction amount based on the movement of the subject predicted by the prediction means and the difference acquired by the acquisition means is realized in one of the image pickup element and the correction lens, and a correction amount based on the movement of the image pickup device detected by the second detection means is realized in the other of the image pickup element and the correction lens.

7. The imaging device according to claim 6.

11. A control method executed by an imaging device, comprising: predicting a motion of a subject during an exposure period based on a motion of the subject acquired before the exposure period; detecting a motion of the subject during the exposure period; obtaining a difference between a predicted motion of the object and a detected motion of the object; performing an image blur correction control in accordance with a correction amount based on the predicted movement of the subject and the difference; 13. A method for controlling an imaging apparatus comprising:

12. A program for causing a computer included in an imaging apparatus to function as each of the means included in the imaging apparatus according to any one of claims 1 to 10.

Images