Imaging device and its control method, program, and storage medium

Abstract

To provide an image capturing apparatus capable of recognizing a state of blur of a main subject and a panning speed of the image capturing apparatus, in image capturing for performing optical blur correction control different from that during preparatory image capturing, during main image capturing.SOLUTION: An image capturing apparatus includes: first motion information acquisition means configured to acquire a first captured image obtained by a first capturing with a first capturing parameter and motion information of a subject in the first captured image; second motion information acquisition means configured to acquire the motion information of the image capturing apparatus in the first capturing; setting means configured to set a second capturing parameter independent from the first capturing parameter; driving information acquisition means configured to acquire driving information for a capturing optical system or capturing means; motion blur estimation means configured to calculate estimated motion blur information obtained by converting motion blur of the subject in the first captured image into motion blur of the subject in a second captured image obtained when second capturing is performed with the second capturing parameter; and motion blur informing means configured to give information on the estimated motion blur information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device, and particularly to a technique for checking subject blur.

Background Art

[0002] In order to image a moving main subject without subject blur using an imaging device such as a digital camera, it is necessary to set an appropriate shutter speed for imaging or to follow the imaging device with the imaging device to the moving subject for imaging. Since the latter method requires advanced operation of the imaging device during shooting, various auxiliary functions and techniques have been proposed.

[0003] For example, in Patent Document 1, during preparatory shooting, in order for the user to visually confirm the area where blur occurs, a technique is disclosed for detecting the area where blur occurs between time-series images captured during preparatory shooting and highlighting that area. Here, preparatory shooting refers to shooting while adjusting the composition and setting shooting conditions while looking at the electronic viewfinder or the rear liquid crystal of the imaging device. Further, in Patent Document 2, from the motion information detected during preparatory shooting, the difference in angular velocity between the motion of the subject and the motion of the imaging device is calculated, and an optical correction lens or an imaging element of the imaging device is driven so as to reduce the speed difference, and a technique for assisting subject tracking in actual shooting is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The display method described in Patent Document 1 makes it possible to determine whether the panning speed of the imaging device is appropriate for the entire main subject area. However, there was a problem in that it was not possible to determine whether the panning speed of the imaging device was appropriate for each sub-area when the speed and direction of movement differed in each part of the main subject area. Similarly, while the display method described in Patent Document 2 makes it possible to understand the difference in panning direction and speed between the subject and the imaging device during preparatory shooting, and the tracking range in which optical blur correction control is possible, it was difficult to intuitively grasp the state of subject blur during the actual shooting.

[0006] In view of the aforementioned problems, the present invention aims to provide an imaging device that makes it easy to check the blur state of the main subject during the main shooting and the appropriate panning speed during the preparation shooting, particularly in shooting where different optical blur correction controls are performed during the main shooting and during the preparation shooting. [Means for solving the problem]

[0007] The imaging apparatus of the present invention comprises: imaging means for acquiring an image by forming an image of a subject on an image sensor via an imaging optical system; first motion information acquisition means for acquiring motion information of a subject from a first captured image obtained by first imaging with first imaging parameters; second motion information acquisition means for acquiring motion information of the imaging apparatus during the first imaging; setting means for setting second imaging parameters independently of the first imaging parameters; drive information acquisition means for acquiring drive information of the imaging optical system or the imaging means; motion blur estimation means for calculating estimated motion blur by converting the motion blur of the subject in the first captured image to the motion blur of the subject in a second captured image obtained when second imaging is performed with the second imaging parameters; and motion blur notification means for providing notification based on the estimated motion blur. The motion blur estimation means calculates a pre-correction blur amount by converting the motion information of the subject in the first captured image to motion blur of the subject in the second captured image, based on the time interval between images in the first imaging and the exposure time as the second shooting parameter, and calculates the estimated motion blur of the subject in the second captured image by subtracting the blur correction amount obtained based on the drive information from the pre-correction blur amount. It is characterized by the following: [Effects of the Invention]

[0008] According to the present invention, motion blur during preparatory shooting is converted to the equivalent of motion blur during the main shooting using optical image stabilization information, and this is notified during preparatory shooting. This makes it easier and more intuitive for the user to grasp the blur status of the main subject and the appropriate panning speed. [Brief explanation of the drawing]

[0009] [Figure 1] Figure showing a first configuration example of the present invention. [Figure 2] Diagram showing the processing flow of the imaging device 100 [Figure 3] Diagram showing the processing flow of the motion vector calculation unit 122. [Figure 4] Diagram showing how to calculate motion vectors [Figure 5] Diagram showing preparatory images and motion vectors. [Figure 6] This figure shows the estimated motion blur during the actual filming. [Figure 7] A diagram illustrating a motion tremor notification method according to an embodiment of the present invention. [Figure 8] Figure showing a second configuration example of the embodiment of the present invention. [Modes for carrying out the invention]

[0010] The embodiments will be described below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted. [Examples]

[0011] In the embodiment of the present invention, motion blur is notified based on motion information of the subject calculated from the image during preparatory shooting, motion information of the imaging device, and the amount of motion blur correction.

[0012] FIG. 1 is a block diagram showing a configuration example of an embodiment of the present invention.

[0013] In FIG. 1, an imaging device 100 is connected to an optical system 101 described later, and images an object image on an imaging surface formed by an imaging element of an imaging unit 114 described later through the optical system 101 to perform imaging.

[0014] The optical system 101 (imaging optical system) includes a zoom lens, a lens group 102 including a focus lens, a correction lens 103, an aperture 104, an optical system control unit 105, and an optical system communication unit 106.

[0015] When the optical system communication unit 106 is connected to the imaging device 100, it exchanges information necessary for imaging. Specifically, it mutually transmits and receives optical control information such as the focal length of the optical system 101 and the aperture 104, and imaging control information such as the exposure time and imaging timing of the imaging device 100.

[0016] In addition, the optical system 101 in this embodiment has an optical blur correction function. This optical blur correction function is realized by the optical system control unit 105 driving and controlling the correction lens 103 in a direction perpendicular to the optical axis so that the imaging position on the imaging surface of the imaging element does not change. This control method will be described later in conjunction with the processing of the imaging device 100.

[0017] The imaging device 100 includes a control unit 110, a bus 111, a ROM 112, a RAM 113, an imaging unit 114, an image processing unit 115, a recording unit 116, a display unit 117, an instruction input unit 118, a motion detection unit 119, a communication unit 120, and a blur correction information acquisition unit 121.

[0018] The control unit 110 is, for example, a CPU, reads a control program for each block included in the imaging device 100 from the ROM 112 described later, and develops and executes it in the RAM 113 described later.

[0019] Thereby, the control unit 110 controls the operations of each block included in the imaging device 100 via the bus 111.

[0020] ROM 112 is an electrically erasable and recordable non-volatile memory that stores the operation programs for each block of the imaging device 100, as well as parameters necessary for the operation of each block.

[0021] RAM113 is a rewritable volatile memory used for deploying programs executed by the control unit 110, etc., and for temporarily storing data generated by the operation of each block in the imaging device 100.

[0022] The imaging unit 114 comprises an image sensor, such as a CCD or CMOS sensor, and an A / D conversion unit. The imaging unit 114 photoelectrically converts the optical image formed on the imaging surface composed of the image sensor via the aforementioned optical system 101, and inputs the resulting analog image signal to the A / D conversion unit. The A / D conversion unit converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit is then temporarily stored in the RAM 113.

[0023] The image processing unit 115 applies various image processing functions, such as white balance adjustment, color interpolation, and gamma processing, to the image data stored in the RAM 113. The image processing unit 115 also includes a motion vector calculation unit 122, a motion blur estimation unit 123, and a motion blur notification image generation unit 124, which generate motion blur notification images for the images stored in the RAM 113, allowing for easy confirmation of motion blur. Details of the processing will be described later.

[0024] The recording unit 116 is a removable storage medium such as a memory card. The recording unit 116 records the image data processed by the image processing unit 115 as a recorded image via the RAM 113.

[0025] The display unit 117 is a display device such as an LCD, and it displays images stored in the RAM 113 and images recorded in the recording unit 116, as well as an operation user interface for receiving instructions from the user. The display unit 117 also displays images captured by the imaging unit 114 for composition adjustment during preparation shooting, and motion blur notification images generated by the image processing unit 115. Furthermore, the display unit 117 may have multiple display devices, such as an EVF (electronic viewfinder) and a rear monitor provided on the photographer's side (rear). The display unit 117 may be capable of simultaneous output to multiple display devices, or it may be configured to switch and selectively display.

[0026] The instruction input unit 118 is an interface for receiving user instructions via a touch panel, buttons, mouse, etc., and transmitting them to the imaging device 100. For example, the exposure time for the main image and the focal length of the optical system 101 can be set via the instruction input unit 118.

[0027] The motion detection unit 119 is, for example, a gyro sensor, which periodically detects the angular velocity representing the amount and direction of movement of the imaging device 100, converts it into an electrical signal, and outputs it to the RAM 113 and the image processing unit 115.

[0028] The communication unit 120 communicates with the optical system communication unit 106 provided in the aforementioned optical system 101.

[0029] The image stabilization information acquisition unit 121 acquires optical image stabilization information from the optical system 101 via the aforementioned communication unit 120 and outputs it to the motion blur estimation unit 123. This optical image stabilization information includes at least the drive amount of the correction lens 103. It may also include information regarding the movable range of the correction lens 103 and parameters necessary for calculating the drive amount of the correction lens 103.

[0030] Furthermore, the imaging device 100, under the control of the control unit 110, performs preparatory shooting (live view shooting), which sequentially displays the analog image signals output from the imaging unit 114 on a display device via the A / D conversion unit, RAM 113, image processing unit 115, and display unit 117. During preparatory shooting, it is possible to prepare for the actual shooting, such as adjusting the composition for recording to a recording medium or output to an external device, and changing the shooting parameters for the actual shooting, such as exposure time (Tv value), aperture value (Av value), and ISO sensitivity.

[0031] Next, the processing of the imaging device 100 will be explained in detail with reference to the flowchart in Figure 2. Each step in this flowchart is executed by the control unit 110 controlling each part of the imaging device 100 and the optical system 101 according to the program stored in the ROM 112.

[0032] The shooting scene in this embodiment is assumed to be a so-called panning shot, where the imaging device 100 tracks a moving main subject. Therefore, the auxiliary display in this embodiment focuses on the motion blur state of the main subject area, and motion blur detection and auxiliary display of the background area are unnecessary.

[0033] First, in the case of ideal panning, the movement speed of the main subject and the panning speed of the imaging device 100 are perfectly matched, and the magnitude of the motion vector in the main subject area is zero. However, in reality, if the panning speed of the imaging device 100 is too fast or too slow relative to the movement speed of the main subject, motion blur occurs in relation to the main subject.

[0034] Therefore, the system controls the system to provide motion blur notification in areas where the magnitude of the motion vector of the main subject calculated by the motion vector calculation unit 122 is smaller than the magnitude of the motion vector of the imaging device 100 calculated by the motion detection unit 119.

[0035] Furthermore, the target area for motion blur notification can be determined by identifying the main subject using a known method, and the motion vector calculation process and motion blur notification process described later can be controlled to be performed only on the main subject area.

[0036] Figure 2 is a flowchart of the process in this embodiment. In step S201, the user turns on the power to the imaging device 100 and starts preparatory shooting, such as composing the shot. During this preparatory shooting period, the imaging device 100 sequentially captures images and displays them on the display unit 117. This allows the user to composing the shot while checking the sequentially displayed preparatory images. The series of processes from steps S202 to S210, described later, are performed during the preparatory shooting period.

[0037] In step S202, the user uses the instruction input unit 118 to set the exposure time for the actual shooting as an independent shooting parameter. Alternatively, the control unit 110 may automatically set the exposure time for the actual shooting.

[0038] In step S203, the imaging device 100 determines whether motion blur notification is set to ON or OFF. The user sets whether motion blur notification is ON or OFF using the instruction input unit 118. Once motion blur notification is set to ON or OFF, that setting is retained.

[0039] If motion blur notification is set to ON in step S203, proceed to step 204; if motion blur notification is set to OFF, proceed to step S209.

[0040] In step S204, at the instruction of the control unit 110, the motion vector calculation unit 122 calculates the motion vector information of the main subject in the preparation image and stores it in the RAM 113. Details of this process will be described later.

[0041] In step S205, the control unit 110 uses the motion detection unit 119 to acquire motion information of the imaging device 100 and stores it in the RAM 113.

[0042] First, the motion detection unit 119 acquires angular velocity information of the imaging device 100 and calculates the motion vector of the imaging device 100 on the prepared image.

[0043] Equations (1) and (2) show approximate conversion formulas for converting angular velocity into motion information on an image.

[0044]

number

[0045]

number

[0046] MOV_yaw represents the amount of movement in the yaw direction, and MOV_pitch represents the amount of movement in the pitch direction. f represents the focal length, ω_yaw represents the angular velocity in the yaw direction, ω_pitch represents the angular velocity in the pitch direction, fps represents the frame rate of the preparation image, and pp represents the pixel pitch of the imaging unit 114. In the conversion formulas shown in equations (1) and (2), the amount of movement on the imaging plane is calculated based on the angle and focal length moved during the time interval between preparation images, and the amount of movement on the image (number of moved pixels) is calculated by dividing by the pixel pitch. Note that the amount of movement on the image calculated here is not different for each pixel, but is a uniform amount of movement for all pixels.

[0047] Furthermore, the amount of movement in the yaw direction is considered as the amount of movement in the horizontal direction, and the amount of movement in the pitch direction is considered as the amount of movement in the vertical direction, and these are output to RAM113 as a uniform motion vector for the imaging device across all pixels.

[0048] Alternatively, the motion vector of the imaging device 100 may be calculated from the motion vector of the subject in the background region of the preparatory image. That is, the subject in the background region may be considered stationary, and the motion vector of the imaging device 100 may be obtained by reversing the start and end points of the motion vector of the subject calculated in the background region.

[0049] Before describing the process in step S206, we will first describe the motion blur correction function of the optical system 101, specifically the operation of calculating the corrective lens drive information for the actual shooting during the preparation shooting. Note that the drive information referred to here includes at least one of the drive amount and drive direction.

[0050] During the preparation phase, two calculation processes are performed regarding the drive information of the corrective lens during the actual shooting. One is the calculation of the target angular velocity to match the movement speed of the main subject. The other is the calculation of the amount of drive of the corrective lens 103 during the actual shooting, and the determination of the drive direction, in order to bridge the difference between the target angular velocity and the current angular velocity.

[0051] First, the method for calculating the target angular velocity will be explained. The optical system control unit 105 acquires angular velocity information from the imaging device 100 at an arbitrary sampling period T via the optical system communication unit 106. From the acquired angular velocity information, the stability of subject tracking is determined and the target angular velocity is calculated. For example, subject tracking is considered stable when the rate of change of angular velocity over a predetermined period falls within a predetermined ratio. Then, the average value of the angular velocity over the predetermined period is calculated and used as the target angular velocity. In addition to the average value of the angular velocity, a weighted average value with weights on the time axis or the most frequent angular velocity that occurred most frequently during the period may also be calculated as the target angular velocity.

[0052] Next, we will explain how to calculate the amount of correction lens drive during actual shooting for the correction lens 103. Approximate conversion formulas for converting angular velocity to the amount of correction lens drive during actual shooting are shown in equations (3) and (4).

[0053]

number

[0054]

number

[0055] DRV_yaw indicates the drive amount in the yaw direction, and DRV_pitch indicates the drive amount in the pitch direction. W_yaw and W_pitch are the target angular velocities in the yaw and pitch directions, respectively, while ω_yaw and ω_pitch indicate the current angular velocities in the yaw and pitch directions. Then, f indicates the focal length, T is an arbitrary sampling period, pp is the pixel pitch of the imaging unit 114, and exp_time indicates the exposure time during this imaging.

[0056] As shown in equations (3) and (4), the amount of correction lens drive during this shooting is calculated by converting the angular velocity difference between the target angular velocity and the current angular velocity into motion information on the image, and multiplying it by the exposure time of this shooting to estimate the amount of movement (blur) on the image during this shooting. The direction of drive is the direction that cancels out the aforementioned angular velocity difference.

[0057] Furthermore, if the corrective lens 103 has a maximum drive range, the amount of drive of the corrective lens during actual shooting may be determined by comparing the maximum drive range with the aforementioned drive amount. For example, if the drive amount exceeds the maximum drive range, the drive amount may be set to the maximum drive range.

[0058] Furthermore, the drive amount may be calculated using angular acceleration in addition to angular velocity. For example, it may be possible to determine whether the subject is moving with constant acceleration and then determine the drive amount of the corrective lens 103 and the control method, taking the acceleration into consideration. The above describes the operation of the optical system 101 during the preparation shooting for the motion blur correction function.

[0059] Next, the processing of the imaging device 100 from step S206 onwards will be described.

[0060] In step S206, the control unit 110 acquires motion blur correction information for the optical system 101 from the image stabilization information acquisition unit 121. The motion blur correction information includes at least the correction lens drive amount during the actual shooting described above.

[0061] If the optical system 101 is not equipped with a motion blur correction function, or if the motion blur correction function is turned OFF, motion blur correction information will not be acquired correctly. In that case, the control unit 110 will determine that it will not use the motion blur correction information (drive information). Then, based on the instructions of the control unit 110, in the motion blur information estimation process during actual shooting in step S207 described later, the motion blur estimation unit 123 may perform the estimation process with the correction amount of the motion blur correction information set to 0, or it may switch to an estimation process that does not use motion blur correction information.

[0062] The ON / OFF status of the motion image stabilization function can be specified by the user via the instruction input unit 118, for example, or by the control unit 110 according to the shooting mode.

[0063] Furthermore, it may be possible to specify ON / OFF for each direction of movement of the imaging device, for example, ON in the horizontal direction and OFF in the vertical direction.

[0064] In step S207, at the instruction of the control unit 110, the motion blur estimation unit 123 estimates the motion blur information for the main shooting using the motion vector information from the preparation shooting, the exposure time during the main shooting, the motion information of the imaging device 100, and the motion blur estimation information. Details of the conversion process will be described later with reference to Figure 6.

[0065] In step S208, the motion blur notification image generation unit 124 generates a motion blur notification image by superimposing a motion blur notification plane onto the preparation image. The motion blur notification plane is generated using the motion blur information from the actual shooting estimated in step S207. Details of the motion blur notification plane generation process will be described later with reference to Figure 7.

[0066] In step S209, the control unit 110 displays the preparatory image on the display unit 117. When motion blur notification is set to ON, this display allows the user to check the motion blur status of the main subject and a guideline for the panning speed by looking at the motion blur notification image displayed on the display unit 117. If the panning speed of the imaging device 100 does not match the movement speed of the main subject, the user can adjust the panning speed of the imaging device 100 before issuing the main shooting command, thereby enabling proper panning photography.

[0067] In step S210, it is determined whether or not the user has issued a shooting instruction via the instruction input unit 118. If a shooting instruction has been issued in step S210, the process proceeds to step S211; otherwise, the process returns to step S202.

[0068] In step S211, the imaging device 100 and the optical system 101 perform the main shooting process at the instruction of the control unit 110. Upon receiving the main shooting instruction, the optical system 101 drives the correction lens 103 using the correction lens drive amount during the main shooting as described above, thereby realizing the blur correction function. The imaging unit 114 then acquires the main shooting image exposed for the exposure time set in step S202, performs predetermined image processing, and records it as the main shooting record image in the recording unit 116. The above is a basic overview of the processing in this embodiment.

[0069] The method for calculating the motion vector in step S204 will be explained in detail with reference to Figures 3, 4, and 5.

[0070] Figure 3 is a flowchart showing the motion vector calculation process by the motion vector calculation unit 122. Figure 4 is a diagram showing the motion vector calculation method. Figure 5 is a diagram showing the preparation image and motion vector.

[0071] In this invention, the block matching method is used as an example to explain the method for calculating motion vectors, but the method for calculating motion vectors is not limited to this example; for example, the gradient method may also be used.

[0072] In step S301, the motion vector calculation unit 122 receives two temporally adjacent preparatory images. The motion vector calculation unit 122 then sets the M-th frame preparatory image as the reference frame and the M+1-th frame preparatory image as the reference frame.

[0073] In step S302, the motion vector calculation unit 122 places an N×N pixel reference block 402 in the reference frame 401, as shown in Figure 4.

[0074] In step S303, the motion vector calculation unit 122 sets the search range 405 for the reference frame 403 as (N+n)×(N+n) pixels surrounding the center coordinate 404 of the reference block 402 of the reference frame 401, as shown in Figure 5.

[0075] In step S304, the motion vector calculation unit 122 performs a correlation calculation between the reference block 402 of the reference frame 401 and the reference block 406 of different N×N pixels at different coordinates within the search range 405 of the reference frame 403, and calculates a correlation value. The correlation value is calculated based on the sum of the absolute values ​​of the inter-frame differences for the pixels of the reference block 402 and the reference block 406. In other words, the coordinate with the smallest sum of the absolute values ​​of the inter-frame differences is the coordinate with the highest correlation value.

[0076] Note that the method for calculating the correlation value is not limited to calculating the sum of absolute values ​​of inter-frame differences; for example, it may also be calculated using methods such as the sum of squared inter-frame differences or the normal cross-correlation value. In the example in Figure 4, it is assumed that reference block 406 has the highest correlation.

[0077] In step S305, the motion vector calculation unit 122 calculates a motion vector based on the reference block coordinates showing the highest correlation value obtained in step S304. In the example in Figure 4, within the search range 405 of the reference frame 403, the motion vector is determined based on the same coordinate 404 corresponding to the center coordinate of the reference block 402 of the reference frame 401 and the center coordinate of the reference block 406. In other words, the distance and direction between the same coordinate 404 and the center coordinate of the reference block 406 are determined as the motion vector.

[0078] In step S306, the motion vector calculation unit 122 determines whether or not it has calculated motion vectors for all pixels of the reference frame 401. If the motion vector calculation unit 122 determines in step S306 that it has not calculated motion vectors for all pixels, it returns to step S302. Then, in step S302, an N×N pixel reference block 402 is placed in the aforementioned reference frame 401, centered on the pixels for which motion vectors have not been calculated, and the processing from steps S303 to S305 is performed as described above. That is, the motion vector calculation unit 122 calculates motion vectors for all pixels of the reference frame 401 by repeating the processing from steps S302 to S305 while moving the reference block 402 shown in Figure 4.

[0079] Examples of motion vectors calculated in this way are shown in Figures 5(b) and (c). In Figures 5(b) and (c), the cases where the direction of the motion vector is (b) to the right and (c) to the left are illustrated as examples.

[0080] Figure 5(b) shows that the main subject, dog 501, is shifted to the right compared to Figure 5(a), indicating that the panning speed of the imaging device 100 is faster than the appropriate panning speed that matches the movement speed of the main subject. In this case, as shown in Figure 5(d), the imaging device motion vector 503 and the subject motion vector 504 are in opposite directions (the absolute value of the angle θ is greater than 90 degrees). Figure 5(c) shows that the main subject, dog 501, is shifted to the left compared to Figure 5(a), indicating that the panning speed of the imaging device 100 is slower than the appropriate panning speed that matches the movement speed of the main subject. In this case, as shown in Figure 5(e), the imaging device motion vector 503 and the subject motion vector 505 are in the same direction (the absolute value of the angle θ is less than 90 degrees). The figures illustrate the case where the direction of the motion vectors is constant, but it is also acceptable for the direction of the motion vectors to differ from region to region. Furthermore, the motion vector calculation unit 122 may calculate motion vectors for predetermined pixels rather than calculating motion vectors for all pixels. The process by which the motion vector calculation unit 122 calculates motion vectors has been described above.

[0081] Next, the method for estimating motion blur during the main shooting in step S207 will be explained in detail with reference to Figure 6. Figure 6 shows the motion vector in the preparation shooting and the estimated motion blur of the main shooting. For the purpose of this explanation, Figure 6 shows the time interval between images in the preparation shooting as 1 / 60 second, and the exposure times for the main shooting as 1 / 120 second, 1 / 30 second, and 1 / 15 second as examples.

[0082] The motion blur estimation unit 123 estimates the motion vector for each pixel as the motion blur of the actual shooting based on the conversion formulas shown in equations (5), (6), and (7) below. CNV_GAIN=EXP_TIME / INT_TIME···Formula (5) CNV_PRE_BLUR=VEC_LEN×CNV_GAIN...Formula (6) CNV_BLUR=CONV_PRE_BLUR-ADJ_BLUR...Formula (7)

[0083] Here, in equation (5), CNV_GAIN represents the estimated gain for converting the motion vector of the preparation shot to the motion vector of the main shot, EXP_TIME represents the exposure time of the main shot, and INT_TIME represents the time interval between images of the preparation shot. In equation (6), CNV_PRE_BLUR represents the estimated motion blur before image stabilization in the main shot, and VEC_LEN represents the length of the motion vector in the preparation shot. Also, in equation (7), CNV_BLUR represents the estimated motion blur of the main shot, and ADJ_BLUR represents the amount of motion blur correction by the image stabilization function.

[0084] In equation (5), the estimated gain is calculated by dividing the exposure time of the main shot by the time interval between the images of the preparation shot. In equation (6), the estimated motion blur before image stabilization in the main shot is estimated by multiplying the length of the motion vector by the estimated gain.

[0085] Specifically, as shown in Figure 6(a), if the length of the motion vector in the preparation shot is 2 pixels, then according to equations (5) and (6), the estimated motion blur for the main shot with an exposure time of 1 / 120 second will be 1 pixel because the estimated gain is halved. If the estimated motion blur for the main shot with an exposure time of 1 / 30 second will be 4 pixels because the estimated gain is doubled, and if the main shot with an exposure time of 1 / 15 second will be 8 pixels because the estimated gain is quadrupled.

[0086] Furthermore, by subtracting the motion blur correction amount from the estimated motion blur before image stabilization in the actual shooting using equation (7), it is possible to appropriately estimate the estimated motion blur in the actual shooting even when optical image stabilization is enabled.

[0087] For example, when the maximum movable range of the corrective lens 103 is 4 pixels, the estimated motion blur in this shooting will be 0 pixels for exposure times of 1 / 120 second and 1 / 30 second, and 4 pixels for exposure time of 1 / 15 second, as shown in Figure 6(b). The process by which the motion blur estimation unit 123 estimates the estimated motion blur in this shooting has been explained.

[0088] Next, the method for generating motion blur notification images by the motion blur notification image generation unit 124 in step S208 will be explained in detail with reference to Figure 7. Figure 7 shows an example of the motion blur notification image displayed on the display unit 117.

[0089] As a first example of display, Figures 7(a) and 7(b) show an example of notifying motion blur using a motion blur frame display. Figures 7(a) and 7(b) correspond to Figures 5(b) and 5(c), respectively.

[0090] Here, we will explain how to generate motion blur notification images using frame displays. Based on the result of step S305, the motion blur notification image generation unit 124 compares the number of estimated motion blur pixels that indicate a panning speed faster than the appropriate speed with the number of estimated motion blur pixels that indicate a panning speed slower than the appropriate speed among the pixels included in each divided region. The motion blur notification image generation unit 124 determines the motion blur state with the higher number of pixels as the state of motion blur of the subject within that divided region, and creates a motion blur frame 901 based on the result. That is, the motion blur frame changes its display style depending on whether the panning speed is faster or slower than the appropriate speed. Then, for regions where the magnitude of the estimated motion blur exceeds the motion blur display threshold, the motion blur notification image generation unit 124 creates the motion blur frame 901 as a motion blur notification plane and superimposes it on the prepared captured image to generate motion blur notification images as shown in Figures 7(a) and 7(b).

[0091] In this embodiment, the motion blur display threshold is set to 0 pixels or higher, but the motion blur display threshold can be any value and depends on the acceptable degree of blur in the recorded image during shooting. Therefore, it may be changed as appropriate according to user instructions or recording resolution.

[0092] As a result, as shown in Figures 7(a) and 7(b), if the panning speed is faster (slower) than the appropriate panning speed, the motion blur frame is displayed with a solid line (dotted line), allowing the user to understand whether the current panning speed is too fast or too slow.

[0093] Next, as a second display example, Figures 7(c) and 7(d) are used to show an example of highlighting the edges of a subject where motion blur has occurred. Figures 7(c) and 7(d) correspond to Figures 5(b) and 5(c), respectively. Here, we will explain how to generate a motion blur notification image by highlighting motion blur edges.

[0094] The motion blur notification image generation unit 124 detects the edge intensity of the prepared image. The edge intensity is calculated using an existing method such as a Sobel filter, and the explanation is omitted. The motion blur notification image generation unit 124 then extracts pixels in which the edge intensity is greater than or equal to a predetermined value and the estimated motion blur is greater than or equal to a predetermined value. For the extracted pixels, a motion blur notification plane is created that highlights the motion blur edges, as shown in 702 of Figures 7(c) and 7(d), and this plane is superimposed on the prepared image to generate a motion blur notification image like the one in Figure 7(c). The example in Figure 7(c) 702 shows an example in which the motion blur edges are made thicker.

[0095] In addition, similar to the first display example, the motion blur notification image generation unit 124 determines, based on the comparison result in step S406, whether the motion blur state of the target pixel is faster or slower than the appropriate panning speed, and changes the degree of edge enhancement according to the result. For example, in Figures 7(c) and 7(d), if the panning speed is faster (slower) than the appropriate panning speed, the motion blur edges are displayed as thicker solid (dotted) lines.

[0096] Another example of a highlighting method is to extract pixels where the edge intensity is above a predetermined value and the estimated motion blur is above a predetermined value, and then color the extracted pixels. In this case, the color should be changed based on the comparison results of the motion vectors.

[0097] In addition, areas where motion blur occurs may also be highlighted, including flat areas where the edge strength is below a predetermined value. Specifically, the motion blur notification plane creation unit 305 highlights pixels where the estimated motion blur for each pixel is greater than or equal to a predetermined value by changing the color according to the direction of the motion blur. By highlighting not only the edge areas but also areas outside the edge areas in this way, the entire subject is highlighted, making it easier to check for motion blur.

[0098] Furthermore, as a method for notifying motion blur, the system may be controlled so that motion blur is not notified in areas where the direction of the motion vector of the imaging device 100 and the direction of the motion vector of the subject differ by a predetermined angle or more. In this way, when panning with a subject whose direction of movement differs between areas, such as the torso and limbs of a running person, motion blur notification will only be displayed in the area that the user is focusing on and tracking with the imaging device 100. This makes it easier for the user to check for motion blur in the area of ​​focus.

[0099] Next, as a third display example, we will explain how to display the degree of motion blur using the relative positions of icons, using Figures 7(e) and 7(f). Figures 7(e) and 7(f) correspond to Figures 5(b) and 5(c), respectively.

[0100] The third display example includes a reference icon 903 and a blur icon 904, and the relative positions of the two icons indicate the degree of motion blur during the actual shooting. For example, Figure 7(e) is a display example where, as in Figure 5(b), the main subject moves from right to left on the screen, and the panning speed of the imaging device 100 is faster than an appropriate panning speed that matches the movement speed of the main subject.

[0101] The reference icon 903 determines whether the movement direction component of the subject is horizontal or vertical relative to the detection frame 905 of the main subject, and is positioned in the center of a direction side different from the direction of movement (in Figure 7(e), since the subject is moving horizontally from right to left, it is positioned at the top and bottom edges of the frame).

[0102] The blur icon 904 is positioned on the axis of the direction of movement of the subject on which the reference icon 903 is placed. The blur icon 904 is positioned such that the distance between the reference icon 903 and the blur icon 904 is directly proportional to the magnitude of the estimated motion blur by a predetermined proportionality coefficient.

[0103] Here, the estimated motion blur only needs to be represented by the distance between two icons, for example, if they are directly proportional. The proportionality constant may be changed depending on the recording resolution.

[0104] If the panning speed of the imaging device 100 is slower than the appropriate panning speed that matches the movement speed of the main subject, the result will be as shown in Figure 7(f). If the panning speed of the imaging device 100 is the appropriate panning speed that matches the movement speed of the main subject, the reference icon 903 and the blur icon 904 will be as shown in Figure 7(g). Therefore, the positional relationship between the reference icon and the motion blur icon can indicate the timing of the shot without impairing the visibility of the main subject, enabling intuitive operation assistance.

[0105] Furthermore, the amount of motion blur can be represented by the motion blur icon not only by its positional relationship to a reference icon, but also by changes in its shape, size, and color. The motion blur notification image generation process of the motion blur notification image generation unit 124 has now been explained.

[0106] In this invention, three examples of motion blur notification methods have been described: displaying a motion blur frame, highlighting motion blur edges, and displaying the positional relationship of motion blur icons. However, these are not the only methods of motion blur notification.

[0107] In motion blur notification, the display examples described above may be switched depending on the shooting scene and shooting mode. For example, when prioritizing the visibility of the main subject, such as in panning shots, the display may be shown as in the third example, while when checking the blur of the main subject in detail, the display may be shown as in the second example.

[0108] Furthermore, the display may be switched depending on the resolution of the recorded image being captured, and in addition to the motion blur notification mentioned above, the blur corrected by the motion blur correction function may be displayed in a way that allows for distinction between the two.

[0109] As described above, the present invention makes it possible to notify the user of the expected motion blur during the preparatory shooting. This allows the user to more intuitively check the degree of subject blur and whether the imaging device 100 is tracking the subject at an appropriate speed during the preparatory shooting. Furthermore, if the main shooting is for still images, it becomes possible to check in advance the motion blur that may occur in the image acquired during the main shooting.

[0110] In this invention, an example of notifying motion blur has been described in which a display unit 117 is used to display information about motion blur. However, this is not the only method for notifying motion blur. For example, motion blur may be notified by sound. In this case, the imaging device 110 can be equipped with a device capable of providing sound notification, such as a speaker (notification sound generation unit). Specifically, when it is determined that the proportion of pixels showing estimated motion blur of a predetermined value or more among the estimated motion blur for each pixel is greater than or equal to a predetermined proportion of the entire screen, the control unit 110 instructs the notification sound generation unit to generate a motion blur notification sound. In addition, it is also possible to change the notification sound to be generated based on the comparison result of the direction of motion blur.

[0111] Furthermore, the estimated motion blur in the actual shooting, estimated by the motion blur estimation unit 123, may be recorded as a motion blur log and associated with the actual captured image. For example, the estimated motion blur immediately following the shooting instruction, or for a predetermined period, may be recorded as a motion blur log. The motion blur log may then be used to highlight motion blur during playback of the captured image, or as rating information for image selection based on blur.

[0112] Next, as another example of this embodiment, we will show an example in which motion blur correction is performed within the imaging device during actual shooting. Figure 8 is a block diagram showing the second configuration of the present invention, and will be described below with reference to it.

[0113] The second imaging device 800 has the same configuration as the imaging device 100, plus a mechanism for correcting motion blur. Hereafter, configurations equivalent to the imaging device 100 will be omitted, and only the differences will be described. The second imaging device 800 is connected to the optical system 101 and performs imaging in the same manner as described above. Furthermore, if the optical system 101 is equipped with a correction lens 103, optical blur correction control may be performed in coordination with the second imaging device 800.

[0114] For example, if it is desired to perform blur correction exceeding the amount that the second imaging device 800 can correct, the optical system 101 may also be controlled to perform blur correction control. In addition, to facilitate motion blur correction during actual shooting by the imaging device 800, the optical system 101 may be assigned specific roles in processing, such as performing camera shake correction during preparation shooting or correcting high-frequency blur.

[0115] The image stabilization control unit 801 drives the image sensor of the imaging unit 114 to realize an optical image stabilization function. Specifically, it uses motion information acquired by the motion vector calculation unit 122 and the motion detection unit 119 during the preparation shooting to calculate the amount of drive of the image sensor during the actual shooting, and outputs motion image stabilization information by the image sensor to the RAM 113 and the image stabilization information acquisition unit 121. This image sensor image stabilization information includes, for example, information on the direction and amount of drive of the image sensor in a direction perpendicular to the optical axis. The formula for calculating the amount of drive of the image sensor during the actual shooting is the same as the calculation formulas (3) and (4) for the motion image stabilization function of the optical system 101 described above. Then, during the actual shooting, the image sensor of the imaging unit 114 is driven using the aforementioned amount of drive of the image sensor during the actual shooting, thereby realizing the motion image stabilization function.

[0116] Furthermore, the difference in the second configuration is that in step S206, the blur correction information acquisition unit 121 moves and acquires the blur correction information from the blur correction control unit 801.

[0117] [Other embodiments] The object of the present invention can also be achieved as follows: a storage medium containing program code for software describing the procedures for realizing the functions of each embodiment described above is supplied to a system or device. The computer (or CPU, MPU, etc.) of the system or device then reads and executes the program code stored on the storage medium.

[0118] In this case, the program code read from the storage medium itself realizes the novel function of the present invention, and the storage medium and program that store that program code constitute the present invention.

[0119] Furthermore, storage media for supplying program code include, for example, flexible disks, hard disks, optical disks, and magneto-optical disks. CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, and ROMs can also be used.

[0120] Furthermore, the functions of each of the embodiments described above are realized by making the program code read by the computer executable. In addition, this also includes cases in which the OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program code, and the functions of each of the embodiments described above are realized through that processing.

[0121] Furthermore, the following cases are also included: First, program code read from a storage medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions of that program code, the CPU or other components of that function expansion board or function expansion unit perform some or all of the actual processing. [Explanation of Symbols]

[0122] 100 Imaging device 101 Optical system 102 Lens Group 103 Correction Lens 104 aperture 105 Optical System Control Unit 106 Optical Communication Department 110 Control Unit 111 Bus 112 ROM 113 RAM 114 Imaging Unit 115 Image Processing Unit 116 Records Section 117 Display section 118 Instruction Input Section 119 Motion detection unit 120 Communications Department 121 Image stabilization information acquisition unit 122 Motion Vector Calculation Unit 123 Motion blur estimation unit 124 Motion blur notification image generation unit 800 Second imaging device 801 Image stabilization control unit

Claims

1. An imaging device, An imaging means that acquires an image by forming an image of a subject onto an image sensor via an imaging optical system, A first motion information acquisition means for acquiring motion information of a subject from a first captured image obtained by first imaging with first shooting parameters, A second motion information acquisition means for acquiring motion information of the imaging device during the first imaging, A setting means for setting a second shooting parameter independently of the first shooting parameter, A drive information acquisition means for acquiring drive information of the imaging optical system or the imaging means, Motion blur estimation means for calculating estimated motion blur by converting the motion blur of the subject in the first captured image into the motion blur of the subject in the second captured image obtained when the second image is taken with the second shooting parameters, The system includes motion sway notification means that provides notification based on the estimated motion sway, The motion blur estimation means is characterized by calculating a pre-correction blur amount by converting the motion information of the subject in the first captured image into motion blur of the subject in the second captured image based on the time interval between images in the first imaging and the exposure time as a second shooting parameter, and then calculating the estimated motion blur of the subject in the second captured image by subtracting the blur correction amount obtained based on the drive information from the pre-correction blur amount.

2. The imaging optical system includes a corrective lens and a first drive control means for driving and controlling the corrective lens in a direction perpendicular to the optical axis. The first drive control means drives and controls the corrective lens using drive information calculated using the motion information of the imaging device and the second shooting parameters. The imaging apparatus according to claim 1, characterized in that the drive information acquisition means acquires drive information of the corrective lens by communicating with the imaging optical system.

3. The image sensor is provided with a second drive control means for driving and controlling the image sensor in a direction perpendicular to the optical axis, The second drive control means drives and controls the image sensor using drive information calculated using the motion information of the imaging device, the second shooting parameters, and the drive information of the shooting optical system. The imaging apparatus according to claim 1 or 2, characterized in that the drive information acquisition means acquires drive information of the image sensor by communicating with the second drive control means.

4. The imaging apparatus according to any one of claims 1 to 3, characterized in that the drive information includes at least information on the amount of drive and the direction of drive of the image sensor.

5. The system includes a determination means for determining whether to use the drive information to estimate the motion wobble, The imaging apparatus according to claim 4, characterized in that the motion blur estimation means switches the estimation process according to the determination result of the determination means.

6. The imaging device according to claim 5, characterized in that the determination means determines whether to use the drive information to estimate motion blur according to the shooting mode of the imaging device.

7. The imaging apparatus according to claim 6, characterized in that the determination means determines whether to use the drive information for estimating motion blur depending on the presence or absence of the corrective lens and drive control means of the imaging optical system.

8. The imaging device according to any one of claims 5 to 7, wherein the determination means determines whether to use the drive information to estimate the motion blur according to the direction of movement of the imaging device.

9. The imaging apparatus according to any one of claims 1 to 8, characterized in that the second imaging is the main imaging, and the first imaging is a preparatory imaging performed before the main imaging.

10. The imaging device according to any one of claims 1 to 9, further comprising a display means for displaying information based on the estimated motion blur.

11. The motion disturbance notification means is, The imaging apparatus according to claim 10, characterized in that motion blur is notified by displaying an image with a motion blur frame drawn on the display means.

12. The motion disturbance notification means is, The imaging apparatus according to claim 10, characterized in that motion blur is notified by displaying an image drawn with enhanced motion blur edges on the display means.

13. The motion disturbance notification means is, The imaging apparatus according to claim 10, characterized in that motion blur is notified by displaying an image on the display means that shows the positional relationship between a reference icon and a motion blur icon.

14. The imaging apparatus according to claim 10, characterized in that the motion blur notification means distinguishes and notifies the blur corrected by the motion blur correction and the motion blur itself.

15. The motion disturbance notification means is, The imaging device according to any one of claims 12 to 14, characterized in that it switches the notification form and / or degree of emphasis according to the shooting mode and recording resolution of the imaging device.

16. The imaging apparatus according to any one of claims 12 to 14, characterized in that the motion blur notification means notifies motion blur during the first imaging.

17. It is equipped with a motion shake notification sound generating means that generates a motion shake notification sound, The imaging device according to claim 1, characterized in that the motion blur notification means notifies motion blur by sound.

18. The imaging apparatus according to any one of claims 1 to 17, characterized in that the second imaging is at least still image capture.

19. A method for controlling an imaging device, The imaging step involves using an imaging means to capture an image of a subject via an imaging optical system and then projecting the image onto an image sensor to acquire an image, A first motion information acquisition step involves acquiring motion information of the subject from a first captured image obtained by first imaging using first shooting parameters, and A second motion information acquisition step for acquiring motion information of the imaging device during the first imaging, A setting step for setting a second shooting parameter independently of the first shooting parameter, A drive information acquisition step for acquiring drive information of the imaging optical system or the imaging means, A motion blur estimation step that calculates estimated motion blur by converting the motion blur of the subject in the first imaging to the motion blur of the subject when a second imaging is performed with the second imaging parameters, A motion variability notification step that provides notification based on the estimated motion variability, It has, A control method for an imaging device, wherein in the motion blur estimation step, a pre-correction blur amount is calculated by converting the motion information of the subject in the first captured image to motion blur of the subject in the second captured image based on the time interval between images in the first imaging and the exposure time as a second shooting parameter, and the estimated motion blur of the subject in the captured image in the second imaging is calculated by subtracting the blur correction amount obtained based on the drive information from the pre-correction blur amount.

20. A program for causing a computer to execute each step of the control method described in claim 19.

21. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method described in claim 19.

Images