Image blur correction device and imaging device

Abstract

To provide smooth video even when the photographer changes the main subject while subject blur stabilization control is in place. [Solution] The image stabilization control device includes: subject detection means for detecting the main subject; first image stabilization amount acquisition means for acquiring a first image stabilization amount for correcting image blur of the main subject; second image stabilization amount acquisition means for acquiring a second image stabilization amount for correcting blur of the imaging device; image stabilization control means for controlling the image stabilization means based on at least one of the first image stabilization amount and the second image stabilization amount; and determination means for determining whether or not the main subject has changed between a first timing and a second timing. If the determination means determines that the main subject has changed, the image stabilization control means changes control from image stabilization based on the first image stabilization amount to image stabilization based on the second image stabilization amount.

Description

[Technical Field]

[0001] This invention relates to an image blur correction device that controls the correction of image blur in a subject. [Background technology]

[0002] Image blur in captured images can be caused by movement of the imaging device, such as camera shake, or by movement of the subject. To correct image blur and track the subject intended by the photographer (main subject) to provide a stable composition, it is necessary to identify the main subject and detect its movement, separating it from the movement of other subjects such as the background. Furthermore, it is necessary to correct the image so that the main subject remains in approximately the same position on the screen.

[0003] Patent Document 1 describes image blur correction during panning photography. Specifically, when panning is detected, the ideal angular velocity for panning (reference angular velocity for panning) is calculated, and during exposure, image blur is corrected based on the difference between the reference angular velocity for panning and the detected angular velocity of the imaging device. This makes it possible to correct image blur occurring in the subject while retaining the effect of panning to blur the background. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-95630 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, Patent Document 1 focuses on panning photography as the subject of photography and does not consider cases where the subject is intentionally changed. Therefore, it is difficult to apply image blur correction when the user intentionally changes the main subject during live view display, video recording, or continuous shooting. Accordingly, the present invention aims to provide an image blur correction method that can provide smooth images even when the main subject is changed, and an image blur correction device that performs said image blur correction method. [Means for solving the problem]

[0006] A blur correction control device as one aspect of the present invention comprises: a subject detection means for detecting a main subject; a first motion detection means for detecting the motion vector of the main subject; a second motion detection means for detecting the movement of an imaging device; a first blur correction amount acquisition means for acquiring a first blur correction amount to correct the image blur of the main subject based on the motion vector of the main subject detected by the first motion detection means; a second blur correction amount acquisition means for acquiring a second blur correction amount to correct the blur of the imaging device based on the movement of the imaging device detected by the second motion detection means; a blur correction control means for controlling the blur correction means based on at least one of the first blur correction amount and the second blur correction amount; and a determination means for determining whether or not the main subject has changed between a first timing when a first image is captured and a second timing when a second image is captured, wherein the blur correction control means changes control from blur correction based on the first blur correction amount to blur correction based on the second blur correction amount if the determination means determines that the main subject has changed.

[0007] Other aspects of the present invention will be revealed in the embodiments described below. [Effects of the Invention]

[0008] According to the present invention, even when the photographer changes the main subject during subject blur suppression control, it becomes possible to provide smooth video. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing the imaging system according to Example 1. [Figure 2] A flowchart showing the main subject detection operation procedure in Example 1. [Figure 3] A flowchart showing the shooting procedure in Example 1. [Figure 4] A flowchart showing the operation procedure of the image stabilization process in Example 1. [Figure 5] Diagram illustrating the setting of the vector detection frame in Example 1. [Figure 6] A diagram illustrating the clustering operation in Example 1. [Figure 7] A flowchart showing the operation procedure of the image stabilization process in Example 2. [Modes for carrying out the invention]

[0010] The following embodiments of the present invention will be described in detail with reference to the drawings. Note that the following embodiments do not limit the invention as defined in the claims. While multiple features are described in the embodiments, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] [Example 1] Figure 1 is a block diagram showing the configuration of an imaging device according to Embodiment 1 of the present invention. In Figure 1, the imaging system 100 is configured such that an interchangeable lens 31 is detachably attached to the imaging device body (hereinafter referred to as imaging device) 1.

[0012] The components of the imaging device 1 will now be explained. In the imaging device 1, the lens mount 2 is positioned for mounting the interchangeable lens 31. The image sensor 3 can be a CMOS sensor or a CCD sensor, and it photoelectrically converts the image of the subject formed by light transmitted through the imaging optical system in the interchangeable lens 31. The imaging circuit 4 generates a predetermined image signal (hereinafter referred to as the image signal) by applying various image processing to the electrical signal photoelectrically converted by the image sensor 3. The A / D conversion circuit 5 converts the analog image signal generated by the imaging circuit 4 into a digital image signal (image data). The memory (VRAM) 6, consisting of a buffer memory or the like, receives the output of the A / D conversion circuit 5 and temporarily stores this image signal. The D / A conversion circuit 7 reads the image signal stored in the VRAM 6 and converts it into an analog signal in a form suitable for playback output. The image display device (hereinafter referred to as LCD) 8, such as a liquid crystal display device (LCD), displays this image signal. The storage memory 10 consists of a semiconductor memory or the like and stores the image signal. The compression / decompression circuit 9 includes a compression circuit and a decompression circuit. The compression circuit reads the image signal temporarily stored in the VRAM 6 and performs compression and encoding processing of the image data to make it suitable for storage in the storage memory 10. The decompression circuit performs decoding and decompression processing to make the image signal stored in the storage memory 10 suitable for playback and display. The AE processing circuit 11 receives the output from the A / D conversion circuit 5 and performs automatic exposure (AE) processing. The AF processing circuit 12 receives the output from the A / D conversion circuit 5, generates an AF evaluation value for autofocus (AF) processing, and further detects the amount of defocus.

[0013] The blur detection sensor 14 detects movement of the imaging system 100, such as camera shake. The blur detection sensor 14 can be composed of inertial sensors such as a gyro sensor or accelerometer, and by using multiple sensors, multi-axis blur can be detected. The blur detection circuit 13 processes the signals from the blur detection sensor 14. The CPU 15 is a microcomputer with built-in memory for calculations that control the entire imaging system 100. The timing generator (hereinafter TG) 16 generates predetermined timing signals. The image sensor driver 17 controls the driving of the image sensor 3. The operation switch 18 consists of various switches. The operation switch 18 includes a main power switch for starting the imaging device 1 and supplying power, a recording start switch for starting video shooting (video recording), and a release switch for starting still image shooting (still image recording). Furthermore, it also includes a playback switch for starting playback, an exposure compensation amount change dial, an exposure time change dial, and an aperture value change dial. The release switch consists of a two-stage switch having a first stroke (hereinafter SW1) and a second stroke (hereinafter SW2). When SW1 is turned ON, an instruction signal is generated to start the AE processing and AF processing that are performed prior to the shooting operation. When SW2 is turned ON, an instruction signal is generated to start the actual exposure operation.

[0014] The EEPROM 19 is an electrically rewritable read-only memory in which programs for performing various controls and data for causing various operations to be performed are stored in advance. The battery 20 is the power source for the entire imaging system 100. The communication driver 21 is a circuit for the imaging device 1 to communicate with the interchangeable lens 31. The LED 22 is a display element for performing warning displays and the like. The imaging element moving motor 25 is a drive source (actuator) for moving the sensor in the horizontal and vertical rotation directions. The imaging element movement control circuit 24 controls the movement of the imaging element 3 by the imaging element moving motor 25. The motion vector detection circuit 27 performs a process of detecting the motion vector of the subject in response to the output from the A / D conversion circuit 5. The main subject detection circuit 26 performs a main subject detection process in response to the outputs from the motion vector detection circuit 27, the A / D conversion circuit 5, and the CPU 15. The image deformation and extraction circuit 28 performs image processing such as rotation, scaling, and trimming (extraction) of the captured image. The speaker 23 is a sound source for performing focus notification, out-of-focus warning, and the like.

[0015] Each component of the interchangeable lens 31 will be described. In the interchangeable lens 31, the blur correction lens 32 is an optical element for moving the subject image on the image plane of the imaging element in order to correct image blur. By moving the blur correction lens in the xy plane perpendicular to the optical axis, the subject moves. The focus lens 33 is an optical element for adjusting the imaging position of the subject image in the optical axis direction to focus. The aperture 34 is a light amount adjusting means for controlling the amount of light beam transmitted through the imaging optical system composed of the blur correction lens 32, the focus lens 33, and the like. The communication driver 35 is a circuit for the interchangeable lens 31 to communicate with the imaging device 1. The control circuit 36 controls the respective drives of an aperture drive motor (not shown) for driving the aperture 34, a focus drive motor (not shown) for driving the focus lens 33, and a blur correction lens drive motor (not shown) for driving the blur correction lens 32. The EEPROM 37 is an electrically rewritable read-only memory in which data for causing various operations to be performed is stored in advance.

[0016] Note that the storage memory 10, which is a storage medium for image data and the like, can use a fixed semiconductor memory such as a flash memory, or a semiconductor memory such as a card-shaped or stick-shaped flash memory that is detachably formed for various devices. Alternatively, various forms such as a magnetic storage medium such as a hard disk or a floppy (registered trademark) disk can be applied.

[0017] The operation of the imaging system 100 of this embodiment configured as described above will be described below.

[0018] First, the light beam from the subject whose light amount has been adjusted after passing through the interchangeable lens 31 is imaged on the light receiving surface of the imaging device 3. This subject image is converted into an electrical signal by the photoelectric conversion process of the imaging device 3 and output to the imaging circuit 4. In the imaging circuit 4, various signal processes are performed on the input signal, and a predetermined image signal is generated. This image signal is output to the A / D conversion circuit 5, converted into a digital signal (image data), and then temporarily stored in the VRAM 6. The image data stored in the VRAM 6 is output to the D / A conversion circuit 7, converted into an analog signal suitable for display, and then displayed as an image on the LCD 8 (live view display).

[0019] The image data stored in the VRAM 6 is also output to the compression / expansion circuit 9. After the compression process is performed by the compression circuit in this compression / expansion circuit 9, it is converted into image data in a form suitable for storage and stored in the storage memory 10.

[0020] Also, for example, when a playback switch (not shown) among the operation switches 18 is operated and turned on, the playback operation is started. Then, the image data stored in the storage memory 10 in a compressed form is output to the compression / expansion circuit 9. After decoding and expansion processes and the like are performed in the expansion circuit, it is output to the VRAM 6 and temporarily stored. Further, this image data is output to the D / A conversion circuit 7, converted into an analog image signal suitable for display, and then displayed as an image on the LCD 8.

[0021] The TG16 outputs a predetermined timing signal to the CPU15, imaging circuit4, and image sensor driver17. The CPU15 performs various controls in synchronization with this timing signal. The imaging circuit4 receives the timing signal from the TG16 and performs various image processing, such as separating color signals, in synchronization with it. Furthermore, the image sensor driver17 receives the timing signal from the TG16 and drives the image sensor3 in synchronization with it to acquire an image signal.

[0022] The image data digitized by the A / D conversion circuit 5 is output separately to the AE processing circuit 11, AF processing circuit 12, motion vector detection circuit 27, main subject detection circuit 26, and image deformation and cropping circuit 28, in addition to the VRAM 6 mentioned above.

[0023] In the AE processing circuit 11, upon receiving the input digital image signal, calculations such as cumulative addition are performed on the brightness values ​​of the image data for one screen. As a result, an AE evaluation value corresponding to the brightness of the subject is calculated and output to the CPU 15. Based on the input AE evaluation value, the CPU 15 calculates the exposure time of the image sensor 3 and the aperture value of the aperture 34, and transmits this information to the interchangeable lens 31 via the communication driver 21. Accordingly, aperture drive processing is performed on the interchangeable lens 31 side, and the aperture amount of the aperture 34 is adjusted to be appropriate.

[0024] The AF processing circuit 12 performs image correction on the image signal acquired by the image sensor 3, which has imaging pixels for focus adjustment, and performs a correlation calculation using the corrected image signal to detect the amount of defocus. The imaging pixels for focus adjustment consist of a pair of imaging pixels designed to receive light beams from a first region of the exit pupil of the optical system and a second region of the exit pupil of a different optical system. The AF processing circuit 12 corrects the images of a reference image (image A) composed of the output signal from the imaging pixel that receives the light beam from the first region, and a reference image (image B) composed of the output signal from the imaging pixel that receives the light beam from the second region. Then, it performs a correlation calculation between image A and image B, and calculates the amount of defocus by multiplying the obtained image shift amount between image A and image B by a conversion coefficient (K value). This enables on-sensor phase-detection AF.

[0025] The CPU 15 determines the amount and direction of drive of the focus lens 33 and transmits this information to the interchangeable lens 31 via the communication driver 21. The interchangeable lens 31 then performs the drive processing for the focus lens 33, enabling AF control to achieve focus.

[0026] The motion vector detection circuit 27 receives the input digital image signal (reference image) and performs a correlation calculation with the digital image signal (reference image) from the previous frame, according to the regions divided by the instructions from the CPU 15. This determines the motion vector of the subject within the divided region.

[0027] In other words, by shifting the reference image by a predetermined number of pixels horizontally and vertically, the difference calculation between the reference image and the reference image is performed, and the pixel shift amount that yields the highest correlation (smallest difference amount) is defined as the amount of motion of the subject in that region. The horizontal and vertical pixel shift directions at that time are defined as the motion directions. From this, the motion vector of the subject within the region between frames can be obtained. Note that the method for determining the motion vector is described in patent 3143173, etc., so a detailed explanation will be omitted.

[0028] The main subject detection circuit 26 acts as a main subject detection means and detects the position of the main subject within the screen as follows. Details of the method for detecting the main subject area will be described later using the flowchart in Figure 2, so only an overview is provided here.

[0029] First, it is determined whether the main subject area has been designated by the photographer. For example, the main subject area is determined to be the area where the AF point has been designated by the photographer, or, if the LCD8 has a touch panel, the area where the photographer has performed a touch operation for a predetermined time.

[0030] Furthermore, upon receiving the output from the A / D conversion circuit 5, the system searches the image for features that characterize the face, such as the pupils and eyebrows. Based on their positional relationships, the system detects the position of the person's face in the image. The size and tilt of the face are then determined from the positional relationships of the features that characterize the face, and the detected facial region is designated as a candidate for the main subject region. In addition, upon receiving the output from the A / D conversion circuit 5, the system searches the image for parts with shapes corresponding to features that characterize a person, such as the head and torso, and evaluates their positional relationships to detect the position of a person present in the image. For example, if a shape close to a circle is detected, and a first rectangular shape exists below it, and a second rectangle with a shorter side than the first rectangle exists adjacent to the first rectangle, then it can be determined that a person is present. The region in which a person is detected in this way is also designated as a candidate for the main subject region.

[0031] Furthermore, upon receiving the output from the A / D conversion circuit 5, the system detects clusters of similar color and brightness, determines their size and position on the screen, and estimates the degree to which they are the main subject. For example, if a cluster of similar color and brightness of a certain size or larger exists near the center of the screen, that cluster is judged to be a candidate for the main subject. Then, from among those that satisfy the predetermined conditions for size and position on the screen, the region with the highest degree of being the main subject, calculated from the centroid coordinates with the center of the screen as the origin and the size of the cluster, is designated as the main subject region. The degree of being the main subject increases as the centroid coordinates are closer to the center and as the size of the cluster increases.

[0032] Furthermore, the processing results of the AF processing circuit 12 are used to obtain the distance or defocus amount for each AF point within the screen. Then, using the processing results of the motion vector detection circuit 27 and the results of the blur detection circuit 13, the camera detects the moving subject that the photographer is trying to capture within the screen. Here, a moving subject refers not to a subject moving on the image, but to a subject moving in real space.

[0033] Furthermore, the CPU 15 acquires information such as the processing results from the AE processing circuit 11, AF point information set by the photographer (including settings made by touching the LCD screen), shooting mode, shutter speed, and aperture value.

[0034] The main subject detection circuit 26 detects multiple main subject regions from the detection results obtained in this way, ranks them, and sends the results to the CPU.

[0035] Using Figure 2, we will explain the specific procedure for detecting the main subject by the main subject detection circuit 26. In this procedure, the main subject that is detected earlier has a higher priority.

[0036] When main subject processing begins, the main subject detection circuit 26 first turns off all main subject detection flags, and then in step S201, it determines whether the main subject area has been specified by the photographer. This is done by examining the AF point information sent from the CPU 15 and determining whether an arbitrary AF point has been specified by the photographer via the menu or touch operation on the LCD screen. If it has been specified, the process proceeds to step S202.

[0037] In step S202, if an AF point is specified in the menu or elsewhere, that AF point will be designated as the main subject area. If no AF point is specified, and the photographer selects the intended main subject by touching the LCD screen, that area and the surrounding areas containing similar subjects will be designated as the main subject area.

[0038] For example, it determines whether face detection or similar actions have occurred within the selected area closest to the center of the selected area. The size of the area selected by touch can be predetermined. This size may also be variable depending on the focal length of the interchangeable lens 31.

[0039] If a face is detected within a predetermined area, the main subject area is determined based on the center coordinates and size of the detected face. If multiple faces are detected, such as if part of one face is within the predetermined area, the face whose center coordinates are closest to the center of the predetermined area is selected. If the proximity is the same, the face with the larger size is selected. If the sizes are the same, the face detected earlier is selected. If the detected face is large and the pupils are detectable, the area containing both pupils or the area containing the largest pupil is designated as the main subject area.

[0040] In step S203, it is checked whether face detection has been performed. If a face has been detected, the face detection flag is turned on, and then the process proceeds to step S221. In step S221, it is determined whether the detected face can be considered identical to the main subject up to the previous frame. If they can be considered identical, the process proceeds to step S204, and the detected face region is designated as the main subject region. If they cannot be considered identical, the process proceeds to step S205. The determination of whether they can be considered identical is made using the "function to track a subject presumed to be the same" of the main subject detection circuit 26, which will be described later.

[0041] In step S205, it is checked whether a person has been detected. If a person has been detected, the person detection flag is turned on, and then the process proceeds to step S222. In step S222, it is determined whether the detected person can be considered the same as the main subject up to the previous frame. If they can be considered the same, the process proceeds to step S206, and the detected person area is designated as the main subject area. If they cannot be considered the same, the process proceeds to step S207. The determination of whether they can be considered the same is made using the "function to track a subject that is presumed to be the same," similar to step S221.

[0042] In step S207, the system checks whether a moving subject exists. If a moving subject exists, it turns on the motion detection flag and proceeds to step S223. In step S223, it determines whether the detected moving subject can be considered identical to the subject that was the main subject up to the previous frame. If they can be considered identical, the system proceeds to step S208, and that moving subject is designated as the main subject area. If they cannot be considered identical, the system proceeds to step S209. The determination of whether they can be considered identical is made using the "function to track a subject that is presumed to be the same" in the same way as in step S221.

[0043] The presence or absence of a moving subject is determined using the output of the blur detection circuit 13, the output of the motion vector detection circuit 27, and the output of the AF processing circuit 12. If the output of the blur detection circuit 13 is small, that is, if the photographer is not intentionally moving the imaging device 1 (so-called panning operation), the motion vector detection circuit 27 determines whether there is a region where the amount of motion of the motion vector detected is greater than or equal to a predetermined value. Note that a small output of the blur detection circuit 13 refers to a case where the detected values ​​for all multiple axes are less than the predetermined value. If there is a region where the amount of motion of the motion vector is greater than or equal to the predetermined value, that region is considered to be the region of a moving subject. The amount of motion of the motion vector can be calculated by taking the square root of the sum of the squared value of the horizontal motion vector and the squared value of the vertical motion vector.

[0044] If the output of the blur detection circuit 13 is large (if the detected value of any of the multiple axes is greater than or equal to a predetermined value), that is, if the photographer is intentionally moving the imaging device 1, the region that is moving approximately the same as this intentional movement will be designated as the main subject region. In other words, if there is a region where the amount of motion of the motion vector detected by the motion vector detection circuit 27 is less than or equal to a predetermined value, that region will be designated as the region of the moving subject. The image will be divided into multiple regions, and if the amount of motion of the motion vector detected in each region is less than or equal to a predetermined value, that region will be designated as the region of the moving subject that the photographer is following, and if regions with an amount of motion less than or equal to a predetermined value are adjacent, they will be merged. If there are multiple regions where the amount of motion of the motion vector detected by the motion vector detection circuit 27 is less than or equal to a predetermined value, the one closest to the center of the screen will be selected as the region of the moving subject.

[0045] If a moving subject cannot be detected by processing using the output of the motion vector detection circuit 27, it is determined whether or not there is a moving subject moving in the distance direction (optical axis direction). This is done by determining whether or not there are AF points (for example, AF points that are getting closer for 5 consecutive frames) whose distance or defocus amount changes over time in the same direction, based on the distance or defocus amount obtained from the AF processing circuit 12. The change in distance or defocus amount for each frame is examined for all AF points obtained from the AF processing circuit 12, and AF points that have changed by a predetermined amount in the same direction over a predetermined number of frames or more are extracted. These AF points are designated as regions where moving objects exist, and adjacent AF points are merged. If there are multiple AF point regions that have changed by a predetermined amount in the same direction over a predetermined number of frames or more, the AF point region closest to the center of the screen is selected as the moving subject region.

[0046] In step S209, the system checks whether there is a region that can be considered a main subject region, which has a high degree of main subject characteristics among a group of similar colors and brightness. If such a region exists, the main subject detection flag is turned on, and the system proceeds to step S224. In step S224, it is determined whether the main subject included in the detected main subject region can be considered identical to the main subject up to the previous frame. If they can be considered identical, the system proceeds to step S210, and that region is designated as the main subject region. If they cannot be considered identical, the system proceeds to step S225. The determination of whether they can be considered identical is made using the "function to track subjects that are presumed to be the same," similar to step S221.

[0047] The degree to which a group of objects is the primary subject is determined by the position and size of the group on the screen, based on their similar color and brightness. Groups that do not touch two of the four sides of the screen are selected, and those larger than a certain size are considered to have a high degree of primary subject importance. If multiple such groups exist, the group whose center of gravity is closest to the center of the image is selected.

[0048] In step S225, the system checks whether a candidate for the main subject has been detected in the previous processing by checking the detection flags. If any of the detection flags are set to ON, it is determined that a candidate for the main subject has been detected, and the system proceeds to step S226.

[0049] In step S226, the detection area with the highest priority among the detection flags that are turned on is selected and designated as the main subject area. As mentioned above, the priority corresponds to the detection order in this flow. In other words, if a face is detected, the face detection area is designated as the main subject area; if a face is not detected but a person is detected, the person detection area is designated as the main subject area; and if neither a face nor a person is detected but a moving subject is detected, the moving subject area is designated as the main subject area. If none of the above are detected, the detection area judged to have a high degree of main subject potential is designated as the main subject area. In the first frame, regardless of which subject is detected, the determination of whether it is the same subject or not (step S221, etc.) is NO, and the process reaches this step. However, if a candidate for the main subject is detected, this step sets the area with the highest priority as the main subject area.

[0050] If the detection flag is not turned on in step S226, it means that no main subject candidate has been detected in the processing up to that point, so the process proceeds to step S211. In step S211, the processing results of the AF processing circuit 12 are used to determine whether there is a subject with a high proportion of AF points within the AF-capable screen and a significant distance difference. If the conditions are met, the process proceeds to step S212, and the AF point showing the closest AF result among the multiple AF points is designated as the main subject area. In this case, since the entire screen is not the subject, such as in landscape photography, but rather in commemorative photography with a landscape in the background, the closest subject is designated as the main subject area. If the conditions are not met, the process proceeds to step S213, where the main subject area is determined from the AE processing results, shooting mode, shutter speed, aperture value, and flash ON / OFF information obtained from the CPU 15.

[0051] Let's explain the process in step S213. First, the main subject area is determined based on the shooting mode, as shown in the table below.

[0052] [Table 1]

[0053] If the above conditions are not met, determine the main subject area as shown in the table below.

[0054] [Table 2]

[0055] The main subject detection circuit 26 has the function of tracking a subject that is presumed to be the same as the subject detected as the main subject in the previous frame. With this function, if the subject detected as the main subject in the above procedure is presumed to be the same as the main subject up to the previous frame, that subject will be designated as the highest-ranking main subject. As shown in the flowchart in Figure 2, even if there is a main subject detected earlier in the above procedure, the main subject that is presumed to be the same will take precedence. In other words, even if a face is detected, if it is not the same subject, and a person is detected and it is determined to be the same subject, the person will be designated as the highest-ranking main subject. However, this excludes cases where the photographer has explicitly designated an AF point in the menu or elsewhere to specify the main subject, and the selected area or its vicinity will be designated as the main subject area regardless of whether it is the same subject or not. Therefore, it will be determined that a new main subject (second main subject) has been detected in the following two cases. The first is when the photographer has explicitly designated the main subject. The second case is when the first main subject moves significantly, and the photographer intentionally fails to track that subject, causing its position on screen to change drastically. In such cases, the main subject is no longer presumed to be the same as in the previous frame, and at the same time, a higher-ranking main subject is detected.

[0056] Next, we will explain the "function to track a subject that is presumed to be the same as the subject detected as the main subject in the previous frame." The method for estimating the same subject differs depending on which of the above conditions was used to detect the main subject area.

[0057] If the photographer has specified an arbitrary AF point, the area containing that AF point will always be considered the main subject area. In other words, the system does not attempt to determine if it is the same subject, but rather treats the subject the photographer intended as the main subject. However, since the main subject may temporarily shift to an adjacent AF point, the system also acquires AF information from adjacent AF points to the top, bottom, left, and right. If the AF information of the specified AF point changes significantly, and nearly identical AF information is obtained from an adjacent AF point, the system will then consider the adjacent AF point as the main subject area.

[0058] If the main subject area is determined based on the face detection result or person detection result, the following processing is performed: A face or person is detected, and if a face or person of approximately the same size as the detected one is in the same position as in the previous frame (the difference in the center coordinates of the detection result is within a predetermined value), it is estimated to be the same subject, and the newly detected area is designated as the main subject area.

[0059] If a moving subject is designated as the main subject area, the area of ​​the moving subject is detected using the procedure described above, and the main subject area is selected from the detected position (coordinates of the center of the area) and the motion vector of that area. If the position and motion vector of the detected moving subject area are the same as those detected in the previous frame, it is presumed to be the same subject, and the newly detected area is designated as the main subject area. Similarly, for a moving subject in the distance direction, if the position and motion vector of the newly detected moving subject area are the same as those detected in the previous frame, it is presumed to be the same subject, and the newly detected area is designated as the main subject area.

[0060] Alternatively, the motion of the subject can be estimated by approximating it with a quadratic function from the motion vectors (amount of motion and direction) over several frames. If the difference in the amount of motion obtained from the motion vector relative to the estimated amount of motion is within a predetermined value, it can be estimated to be the same subject, and that region can be designated as the main subject region.

[0061] Similarly, when a group of objects with similar color and brightness is identified as the main subject area, if the size and position of a newly detected group of objects with similar color and brightness are the same as those detected in the previous frame, it is presumed to be the same subject. This newly detected area is then designated as the main subject area.

[0062] If the nearest subject is designated as the main subject area (Figure 2, step S212), a new main subject area is searched for without determining whether or not it is a subject that is presumed to be the same as the subject.

[0063] Furthermore, if recording stops, panning beyond the field of view (including tilting), extreme brightness changes occur, the mode is changed to playback, or the power is turned off, it is determined that shooting of the main subject up to that point has ended. The main subject area information detected up to that point is then erased.

[0064] The image deformation and cropping circuit 28 corrects the changes in the image based on the output of the main subject detection circuit 26 and the motion vector detection circuit 27, according to the information about the up-and-down and left-and-right movement of the main subject on the image and the rotation of the imaging device 1 calculated by the CPU 15. This correction is a so-called electronic blur correction, performed by image processing such as image rotation or cropping of a portion of the image.

[0065] For example, the position of a person's face detected as the main subject may change significantly on screen due to the person's movement or the photographer's camera shake, or the main subject may move diagonally. When such blurring or movement of the main subject occurs between frames of a video, horizontally, vertically, or diagonally, the main subject may not be in the position on screen that the photographer desired, or the main subject may move unnaturally. If such phenomena occur frequently between frames, the recorded video becomes very difficult to watch.

[0066] Therefore, in this embodiment, motion vector detection circuit 27 detects motion vectors representing the horizontal and vertical movement of the main subject between frames, and CPU 15 calculates information for correcting the image (hereinafter referred to as the correction amount) from the detected motion vectors. Then, the image is deformed and corrected by image deformation and extraction circuit 28 according to the calculated correction amount, and the corrected image is recorded in a predetermined area of ​​VRAM 6 to perform electronic blur correction.

[0067] In this embodiment, the motion vector of the main subject detected by the main subject detection circuit 26 is detected by the motion vector detection circuit 27, and then the blur occurring in the main subject is corrected using the detected motion vector. If multiple main subject candidates are detected, the main subject candidate with the highest ranking is selected as the main subject, and subject blur correction is performed so that the blur of the main subject in the image is reduced.

[0068] Then, the amount of change in the position of the main subject is calculated from the output of the main subject area detection means. If the amount of change is greater than a predetermined amount, or if the difference between the movement of the main subject calculated from the motion vector and inertial sensor and the movement of the imaging device is greater than a predetermined value, it is determined that the photographer has intentionally changed the main subject, and subject blur correction is stopped, with only camera shake correction in the predetermined direction being performed.

[0069] Next, the actual shooting operation of the imaging system 100 in this embodiment will be explained using the flowchart shown in Figure 3.

[0070] The imaging processing sequence is executed when the main power switch of imaging device 1 is ON and the operating mode of the imaging device is in shooting (recording) mode.

[0071] First, in step S301, after performing processes such as initializing variables and moving the drive member to its initial position, it is determined whether or not the interchangeable lens 31 is attached. If it is attached, information regarding the image stabilization lens 32, the focus lens 33, and the aperture 34 is obtained. If no lens is installed, the process of acquiring replacement lens information is skipped and the process proceeds to step S302.

[0072] In step S302, the CPU 15 displays the image formed on the image sensor 3, which has passed through the interchangeable lens 31, as an image on the LCD 8.

[0073] If the system is set to video recording mode and image stabilization is enabled, it is acceptable to display the image with image stabilization applied on LCD8.

[0074] In step S303, the CPU 15 checks whether the system is set to video recording mode or still image shooting mode. If it is set to video recording mode, it proceeds to step S304; if it is set to still image shooting mode, it proceeds to step S321.

[0075] In step S321, the state of the release switch is checked. When the CPU 15 confirms that the release switch has been operated by the photographer and SW1 (the first stroke of the release switch) is in the ON state, the process proceeds to the next step S322. In step S322, if the photographer has set up image stabilization processing, image stabilization processing is performed. This process detects the blur applied to the imaging system 100 with the blur detection sensor 14 and performs blur stabilization processing (movement of at least one of the blur stabilization lens and the image sensor) based on the detection result. For example, the conventional blur stabilization processing in still image shooting described in Japanese Patent Application Publication No. 5-161053 can be applied, so a detailed explanation of the processing is omitted.

[0076] In step S323, AF and AE processing are performed, driving the focus lens 33 to the focus position, determining the aperture value and exposure time for still image shooting, and then proceeding to step S324. In step S324, the AE processing results (high brightness, low brightness warning, etc.) and AF processing results (AF success or failure, etc.) are superimposed on the subject image formed on the image sensor 3 and displayed as an image on the LCD 8. The AE and AF processing results may also be indicated to the photographer by lighting or blinking the LED 22, or by emitting focus and out-of-focus sounds from the speaker.

[0077] In step S325, the SW2 (second stroke of the release switch) is checked. If SW2 is ON, the process proceeds to step S326 to perform the actual exposure process. If the photographer has set up image stabilization during this exposure process, the image stabilization process is performed in the same manner as in step S322. After the exposure process is complete, the process proceeds to step S310. In this specification, "exposure" refers to the imaging process (exposure) for capturing a record image, unless otherwise specified, and does not include the imaging process for acquiring a live view image.

[0078] On the other hand, if the video recording mode was set in step S303, the process proceeds to step S304. In step S304, the CPU 15 performs AF processing and AE processing, drives the focus lens 33 to the focus position, determines the aperture value and exposure time, and controls the driving of the aperture 34 and the exposure time (accumulation time) of the image sensor 3.

[0079] In step S305, it is determined whether the photographer has enabled image stabilization (IS) processing. If image stabilization is enabled (IS ON), image stabilization is performed in step S306. On the other hand, if image stabilization is not enabled (IS OFF), the process proceeds to step S307. Details of the processing in step S306 will be described later.

[0080] In step S307, the image is displayed on LCD8 with the AE processing results (high brightness, low brightness warnings, etc.) and AF processing results (AF success or failure, etc.) superimposed. If image stabilization processing is set, this image is the image with image stabilization applied, created in step S306; if image stabilization processing is not set, this is the image read out in step S302. As with step S324, the photographer can be notified of the AE processing results and AF processing results by a method other than displaying them on LCD8.

[0081] In step S308, it is checked whether a video recording instruction has been given by the photographer (i.e., whether it has been input to the imaging device 1). If a video recording instruction has been given, the process proceeds to step S309, where the video recording process is executed, and then the process proceeds to step S310. In the video recording process in step S309, if the image stabilization processing setting has been enabled, the image with image stabilization created in the process in step S306 is recorded. If no video recording instruction has been given, the process proceeds to step S310.

[0082] In step S310, the status of the main power switch, the playback switch, and whether a lens has been changed are checked. If any of these actions have been performed, the process terminates. Otherwise, the process returns to step S302.

[0083] Here, we will explain the image stabilization process performed in step 306 using Figures 4 to 6. Figure 4 shows a flowchart of the image stabilization process performed in step S306.

[0084] In step S401, the CPU 15 determines whether or not panning (hereinafter including tilting) is occurring. The presence or absence of panning is determined based on the output of the main subject detection circuit 26, the output of the motion vector detection circuit 27, and the output from the blur detection circuit 13. Specifically, based on the output of the main subject detection circuit 26, the CPU calculates the amount of change in the position of the main subject from the previous frame (first image) to the current frame (second image), and if the amount of change is greater than or equal to a first predetermined amount, it is determined that panning has occurred. In addition, the amount of movement of the main subject within the screen is obtained from the motion vector output from the motion vector detection circuit 27, and if this amount is greater than or equal to a second predetermined value, it is also determined that panning has occurred. Furthermore, if the amount of movement of the imaging device 1 output from the blur detection circuit 13 is greater than or equal to a third predetermined value, it is also determined that panning has occurred. It is preferable that the first to third predetermined values ​​be set according to the focal length of the interchangeable lens. For example, the first to third predetermined values ​​are based on the value at a focal length of 50mm, with values ​​proportional to the focal length between 24mm and 200mm, the same value as the 24mm value for focal lengths less than 24mm, and the same value as the 200mm value for focal lengths exceeding 200mm.

[0085] If it is determined in this step that panning is occurring, the process proceeds to step S402, where the CPU 15 turns on the panning occurrence flag and proceeds to step S403.

[0086] Steps S403 and beyond involve correcting image blur caused by the movement of the imaging device 1 during the panning period, specifically correcting camera shake only in the direction perpendicular to the direction of movement of the imaging device due to panning.

[0087] In step S403, the CPU 15 sets the direction for correcting camera shake. From the amount of shake in the pitch direction and the amount of shake in the yaw direction output from the shake detection circuit 13, it determines the direction of movement of the imaging device 1 (the direction of panning). Then it calculates the direction on the image sensor that is perpendicular to the direction of movement of the imaging device 1, sets that direction as the shake correction direction while panning is occurring, and proceeds to step S404.

[0088] In step S404, the CPU 15 instructs the main subject detection circuit 26 to perform main subject detection and obtains the position of the detected main subject. Alternatively, the main subject detection itself may be performed immediately after the frame to be processed is acquired, prior to this step, and this step may only involve obtaining the detection result.

[0089] The process proceeds to step S405, where the vector detection frame is set. In step S405, if the CPU 15 determines that the main subject exists in step S404 (processing in the flowchart of Figure 2) and has been able to obtain the position of the main subject, it sets a vector detection frame of a predetermined size in the area excluding the main subject area. If it is determined that the main subject does not exist (for example, if the AF area is determined in step S213 in the flowchart of Figure 2), it sets a predetermined number of vector detection frames of a predetermined size across the entire screen.

[0090] The CPU 15 sets the vector detection frame, for example, as shown in Figure 5. If it is determined that the main subject exists, the CPU 15 divides the entire screen into a predetermined number of blocks of a predetermined size (here, 7 x 9) as shown in Figure 5(A), and sets the blocks. Of each block, the blocks that partially overlap with the area included in the main subject region detected in step S404 (the gray blocks in Figure 5(A)) are excluded (the white blocks in Figure 5(A)) and these blocks are designated as the vector detection frame. On the other hand, if the CPU 15 determines that the main subject is not present, it sets a predetermined number of vector detection frames (in this case, 7 x 9) of a predetermined size across the entire screen, as shown in Figure 5(B).

[0091] In step S406, vector detection is performed within the frame set in step S405. Then, in step S407, the vector detection is clustered, and the vectors detected in the vector detection frame excluding the main subject area are separated into vectors caused by camera shake and vectors in which motion components due to the movement of the subject itself are superimposed. Then, the amount of camera shake is detected by extracting the motion vectors caused by camera shake from the separated vectors.

[0092] For example, as shown in Figure 5(C), if a part of the main subject exists around the gray area near the center, which is designated as the main subject area, the vector detected in that vector detection frame will have a different direction and magnitude from many of the vectors detected in other vector detection frames. In the example in Figure 5(C), the vector detection frame where the moving main subject exists detects a vector pointing to the left of the page, while the other vector detection frames detect a vector pointing to the upper right of the page. Therefore, in step S407, camera shake is detected by excluding those whose direction and magnitude differ by more than a certain threshold and then taking the average.

[0093] Subsequently, in step S408, a correction amount is calculated based on the camera shake detected in step S407. This allows obtaining a correction amount (second shake correction amount) to correct image blur (i.e., camera shake) caused by the movement of the imaging device 1. The method for obtaining the correction amount will be explained below.

[0094] If a correction direction is set in step S403, this step acquires a correction amount corresponding only to the amount of shake in the set direction. For example, if the hand shake acquired in step S407 is the vector L shown in Figure 5(C), and the angle between vector L and the x-axis direction (horizontal direction of the paper) is θ, the correction amount v acquired from the motion vector L is calculated by the following formula. Note that the amount of shake correction in the x-axis direction is v(x), and the amount of shake correction in the y-axis direction is v(y). |v|=L cosθsinθ, L=√(x1·x1+y1·y1) v(x) =|v| cosθ = L cosθsinθcosθ v(y) =|v| sinθ = L cosθsinθsinθ

[0095] On the other hand, if the process proceeds from step S420 to step S405 and then to step S408, as described later, the direction of blur correction is not set because step S403 has not been passed. In that case, i.e., when correcting camera shake in all directions, the amount of blur in the rotational direction (roll direction) of the imaging device 1 is detected by the blur detection sensor 14 and the blur detection circuit 13, and the image rotation angle for correcting rotational blur is obtained from the detected value using the frame rate of the video recording. Here, "all directions" refers to all directions that the image stabilization means can correct. The acquired image rotation angle is the rotation angle with respect to the center of the screen, and the correction amount is obtained based on the image rotation angle and the camera shake (L) obtained in step S407. If the direction of image stabilization is not set, the amount of shake in the pitch, yaw, and roll directions may be obtained from the image stabilization detection circuit 13, and the image stabilization amount may be obtained based on these.

[0096] In step S409, the CPU 15 controls the image deformation and cropping circuit 28 to generate a blur-corrected image by cropping the image based on the correction amount acquired in step S408. However, when correcting blur in all correctable directions (when the process reaches step S409 from step S420), the correction rotation angle for correcting the amount of blur in the rotational direction of the imaging device 1 is determined from the signals output from the blur detection sensor 14 and the blur detection circuit 13. Then, correction of that component by geometric deformation (roll correction) is performed before cropping the image. Since the size of the cropped image is constant, if the cropping range becomes less than the predetermined size when the main subject position is set to the predetermined position, the cropping range is adjusted to the predetermined size.

[0097] In this way, when panning occurs, it is determined that the main subject has been intentionally changed, and camera shake, which is blurring in the image captured due to the movement of the imaging device 1, is corrected.

[0098] If it is determined in step S401 that panning has not occurred, the process proceeds to step S411. In step S411, the CPU 15 controls the main subject detection circuit 26 to perform the main subject detection shown in Figure 2, obtains information from the output of the main subject detection circuit 26 indicating whether or not the main subject has been detected, and proceeds to step S412. In step S412, the CPU 15 determines whether or not the main subject has been detected based on the information obtained in step S411. If the main subject has not been detected, the process proceeds to step S420, where, if the panning occurrence flag was set to ON in step S402 of the previous frame, the flag is set to OFF, and the process proceeds to step S405. The processing from step S405 onward is as described above.

[0099] On the other hand, if the main subject is detected in step S412, the process proceeds to step S413 to determine whether the panning flag is turned on or off. If the flag is turned on, the process proceeds to step S414, where the panning flag is turned off, and then the process proceeds to step S415. If the flag is not turned on, the process proceeds to step S421.

[0100] The processing from step S415 onward is a process that gradually transitions from image stabilization performed during panning to subject blur correction (correction of blur in the image caused by camera shake and subject movement). This subsequent processing is performed until a predetermined time has elapsed after the end of panning. First, in step S415, the period for this transition (hereinafter referred to as the transition period) Tch is set. Step S415 is reached if panning was occurring up to the previous frame. Therefore, it is preferable to determine the transition period Tch based on the duration of the panning that occurred up to the previous frame. For example, by setting the transition period Tch to the same duration as the panning that occurred, a smooth image can be obtained. However, if the panning speed is fast, setting the transition period Tch to the duration of the panning may result in unstable framing after panning. Therefore, a reference panning speed is predetermined, and if the speed of the panning that occurred exceeds the reference speed, the speed is limited to a maximum value and then made a proportional multiple of the reference speed. The panning speed can be obtained from the absolute value of the value detected by the blur detection circuit 13. It is preferable to determine the reference speed considering the focal length. For example, first determine the reference speed for a focal length of 50mm, and then make the reference speed for focal lengths between 24mm and 200mm proportional to the reciprocal of the ratio of the focal lengths to 50mm. Then, for focal lengths less than 24mm, use the same value as the reference speed for 24mm, and for focal lengths exceeding 200mm, use the same value as the reference speed for 200mm. In this case, the standard speed at a focal length of 100mm is half the value at 50mm, and at 25mm it is twice the value at 50mm. The maximum speed limit may be uniform regardless of focal length, or it may be set separately for each focal length.

[0101] When the transition period Tch is set, the vector detection frame is set in step S416. In this step, a vector detection frame is set for both the main subject area and the area other than the main subject area. For example, as shown in Figure 5(A), the entire screen is divided into a predetermined number of blocks of a predetermined size. Of these blocks, the block that partially overlaps with the area included in the main subject area detected in step S411 (the gray block in Figure 5(A)) is set as the detection frame for detecting the subject vector. Then, each of the remaining blocks (the white block in Figure 5(C)) is set as the background vector detection frame. The movement of the subject is detected by the subject vector, and the movement of the imaging device 1 is detected by the background vector.

[0102] In step S417, vector detection is performed within the frame set in S416, and the process proceeds to step S418. In step S418, clustering of the detected vectors is performed to detect the amount of camera shake and the amount of subject blur. The method for detecting the amount of camera shake is the same as in step S407. The vectors detected within a vector detection frame that does not overlap with the main subject area are separated into those caused by camera shake and those caused by the movement of the subject itself. By extracting the motion vectors caused by camera shake, the amount of camera shake is detected.

[0103] Subject blur is also detected by performing clustering in a similar manner. Vectors detected in the vector detection frame within the frame overlapping with the main subject area are separated into vectors that contain the subject's motion component and vectors that do not. By extracting the main subject motion vector that contains the subject's motion component, the amount of subject blur (camera shake + subject motion) is detected.

[0104] For example, as shown in Figure 5(D), there is a possibility that background other than the main subject exists within the dotted frame 501 near the center, which is designated as the main subject area. Therefore, among the vectors detected in the vector detection frame within the dotted frame 501, there are some that differ in direction and magnitude from many of the vectors detected in other vector detection frames within the dotted frame 501. In step S418, the amount of subject blur caused by both camera shake and the movement of the subject itself is detected by excluding those that differ in direction and magnitude by more than a standard value and then taking the average.

[0105] Then, in step S419, the correction amount is obtained by taking a weighted average of the amount of camera shake and the amount of subject blur detected in step S418. The weights of the weighted average (hereinafter referred to as the correction ratio) are determined based on the set transition period Tch and the elapsed time since the start of the transition process. For example, the correction ratio is determined using the transition period Tch obtained in step S415 and the elapsed time Tpst since the start of the transition process by the following formula. Image stabilization ratio = 1 - Tpst / Tch Subject blur correction ratio = Tpst / Tch The sum of the correction ratio for camera shake and the correction ratio for subject blur is 1.

[0106] Each image stabilization amount is expressed by the following formula using the amount of camera shake and the amount of subject blur obtained in step S418. The final image stabilization amount is obtained by adding the amount of camera shake and the amount of subject blur correction. Image stabilization amount = Correction ratio of image stabilization amount × Image stabilization amount Subject blur amount = Subject blur correction ratio × Subject blur amount

[0107] Subsequently, in step S409, the CPU 15 controls the image deformation and cropping circuit 28 to generate a blur-corrected image by cropping the image based on the correction amount acquired in step S419.

[0108] Thus, immediately after panning is complete, the CPU 15 performs a transition process from correcting camera shake caused by the movement of the imaging device 1 to correcting subject blur caused by the main subject of the captured image.

[0109] On the other hand, if the panning occurrence flag was turned off, and the process proceeds from step S413 to step S421, then no panning has occurred since the previous frame (either panning was completed before the previous frame or no panning has occurred at all). Therefore, the CPU 15 determines whether the transition period from image stabilization to subject blur stabilization has ended. The CPU 15 compares the elapsed time Tpst since the start of the transition process with the transition period Tch set in step S415, and if Tpst ≥ Tch, it determines that the transition period has ended.

[0110] If the transition period has ended, proceed to step S422 and perform motion blur correction processing. If the transition period has not ended, proceed to step S416 and perform the transition process for subject blur correction described above.

[0111] Once the transition period ends and the process proceeds to step S422, the CPU 15 sets a vector detection frame in the main subject area. For example, as shown in Figure 5(A), the entire screen is divided into a predetermined number of blocks of a predetermined size (here, 7 x 9). Of each block, the area that partially overlaps with the area included in the main subject area (the gray block in Figure 5(A)) is set as the vector detection frame for correcting subject blur.

[0112] In step S423, the CPU 15 performs vector detection within the frame set in step S422. Then, in step S424, it performs clustering of the vector detections and extracts the main subject vector that contains the motion component caused by the movement of the main subject, thereby detecting the amount of subject blur (amount of camera shake + amount of subject movement).

[0113] In step S425, the CPU 15 acquires a correction amount (first correction amount) to correct the blur occurring in the subject of the captured image, based on the amount of subject blur detected in step S424. Then, in step S409, the CPU 15 controls the image deformation and cropping circuit 28 to generate a blur-corrected image by cropping the image based on the correction amount acquired in step S425.

[0114] Thus, if panning has not occurred at all, or if a predetermined period has elapsed since panning was completed, the system corrects for blur (subject blur) that occurs in the subject of the captured image.

[0115] Once the processing in step S409 is completed, the process returns to step S307 in Figure 3, and the processing from step S401 is executed again for the next frame.

[0116] Here, we will explain the clustering process performed in step S407 and other steps using Figure 6.

[0117] By detecting motion vectors, setting cluster values, separating vectors, and determining the separation results, motion vectors are separated when there are multiple subjects with different movements, such as the main subject vector and the background vector.

[0118] The dashed line in Figure 6(A) represents the tracking frame 601, and the main subject is located within this frame. The main subject moves as indicated by the arrows in the figure, and the background also moves as indicated by the arrows. Motion vectors in the horizontal (X-axis) and vertical (Y-axis) directions are detected for each. When the detection results are plotted on a graph with the Y-direction motion vector on the vertical axis and the X-direction motion vector on the horizontal axis, it looks like the right-hand figure in Figure 6(A). The motion vector of the main subject is due to the movement of the subject itself and the photographer's intentional panning to track it, while the motion vector of the background is mainly due to camera shake. Therefore, there is a clear difference between the two as shown in the graph. In this example, the main subject is moving in the negative horizontal direction (left), so the X-motion vector is distributed in the negative region and the Y-motion vector is distributed near zero. The background is moving in the positive direction (right), opposite to the main subject in the horizontal direction, and in the positive direction (up) in the vertical direction, so the X-motion vector is distributed in the positive region, and the Y-motion vector is also distributed in the positive region.

[0119] We then separate the two by setting cluster values. First, we calculate the average value of all X motion vectors (XmeanAll) and the average value of all Y motion vectors (YmeanAll), and use these as initial cluster values ​​to separate the detected motion vectors. Using the two calculated average values, we separate the motion vectors shown in the graph of Figure 6(A) into four regions.

[0120] In the example in Figure 6(A), examining the motion vectors in the regions divided by the dashed lines on the graph reveals that the background motion vector is in the upper left region and the main subject motion vector is in the lower right region. Thus, separation is possible when two motion vectors exist within the same frame. However, as shown in Figure 6(B), when evaluating within the tracking frame, separation is difficult because most of the detected motion vectors correspond to the subject's movement.

[0121] Therefore, taking detection errors into consideration, and assuming a distribution range (XΔ·YΔ) where identical motion vectors are found, if the number of vectors present in XmeanAll±XΔ and YmeanAll±YΔ exceeds a predetermined percentage (e.g., 80%), it is determined that only a single motion vector exists within the tracking frame, and separation is not performed.

[0122] If separation is not performed, the standard deviation of the entire X motion vector (XσAll) and the entire Y motion vector (YσAll) are calculated. If both of these values ​​are less than or equal to a predetermined value (for example, XσAll ≤ 2·XΔ and YσAll ≤ 2YΔ), it is determined that it is possible to obtain the vectors within the tracking frame. Then, motion vectors within the range of XmeanAll ± XσAll and YmeanAll ± YσAll are extracted, and the mean value of the X motion vector (XmeanAllp) and the mean value of the Y motion vector (YmeanAllp) are used as the motion vectors within the tracking frame.

[0123] On the other hand, if the number of vectors present in XmeanAll±XΔ and YmeanAll±YΔ is less than a predetermined percentage (e.g., 80%), separation is considered possible, and the validity of the separation of motion vectors is determined by evaluating the number and distribution range of motion vectors in each region.

[0124] The average values ​​(Xmean[n] and Ymean[n]) of the X and Y motion vectors present in each region are calculated. Then, regions where the number of vectors in Xmean[n]±XΔ and Ymean[n]±YΔ exceeds a predetermined amount (for example, 20% of all vectors) are determined to be regions where a subject with different motions exists.

[0125] In this case, the standard deviation of the entire X motion vector (Xσ[n]) and the standard deviation of the entire Y motion vector (Yσ[n]) are calculated. If both of these values ​​are less than or equal to a predetermined value (for example, XσAll ≤ 2·XΔ and YσAll ≤ 2YΔ), it is determined that it is possible to acquire the vector for that region within the tracking frame.

[0126] Then, in the region where data acquisition is deemed possible, motion vectors within the range of Xmean[n]±Xσ[n] and Ymean[n]±Yσ[n] are extracted. The average values ​​of the extracted X motion vectors (Xmean[n]p) and Y motion vectors (Ymean[n]p) are then used as the motion vectors within the tracking frame. Multiple such vectors may be obtained.

[0127] If motion vectors can be obtained in multiple regions within the tracking frame, the Xmean[n]p·Ymean[n]p that maximizes the number of detected vectors is selected. However, if the number of extracted vectors is less than a predetermined value, motion vector detection is deemed impossible, the vector value is set to zero, and it is assumed that there are no moving subjects within the detected tracking frame.

[0128] As described above, in this embodiment, the main subject detection circuit 26 plays the role of a subject detection means for detecting the position of the main subject. The motion vector detection circuit 27 plays the role of a first motion detection means for detecting the movement of the main subject in the image based on the main subject vector, and a second motion detection means for detecting the movement of the imaging device 1 based on the background vector. The blur detection sensor 14 and the blur detection circuit 13 may also play the role of a second motion detection means for detecting the movement of the imaging device.

[0129] The CPU 15 also acts as a means for calculating the amount of change in the position of the main subject, based on the output of the main subject detection means. Furthermore, the CPU 15 calculates the movement of the main subject from the motion vector output from the motion vector detection circuit 27 and the output of the blur detection circuit 13, and calculates the movement of the imaging device from the output of the blur detection circuit 13, and then calculates the difference between the two movements. In addition, the CPU 15 acts as a determination means for detecting intentional changes in the main subject (panning) based on the amount of change in the main subject's position, the difference between the movement of the main subject and the imaging device, and the movement of the imaging device. The CPU 15 also acts as a means for acquiring a blur correction amount, calculating a correction amount to correct blur occurring in the subject of the captured image. Finally, the CPU 15 acts as a correction direction setting means for setting a predetermined direction for performing blur correction during changes in the main subject.

[0130] By performing the processing described above, in this embodiment, even if the photographer changes the main subject during subject blur correction control, smooth video acquisition is made possible by performing image stabilization in a predetermined direction during the change. In this embodiment, the videos on which the processing shown in Figure 4 is performed are the recorded video and the video displayed as a live view image before recording starts, but the processing shown in Figure 4 may be performed on only one of the videos. Furthermore, the processing in Figure 4 may be performed on the live image before still image shooting, or on multiple images recorded during continuous shooting (burst shooting). By considering each still image taken by burst shooting as corresponding to each frame of the video, this embodiment can also be applied to burst shooting. [Example 2] This embodiment differs from Embodiment 1 in its method for determining whether the photographer intentionally changed the main subject. In Embodiment 1, the presence or absence of panning was used to determine whether the photographer intentionally changed the main subject. In this embodiment, an intentional change of the main subject is determined when a change in focal length during shooting, a change in focus exceeding a certain amount, or a change in exposure exceeding a certain amount is detected. The configuration of the imaging system 100, the flow of main subject detection, and the shooting flow other than the blur correction process (step S306) are the same as in Embodiment 1, so their explanation is omitted.

[0131] Figure 7 shows the operation procedure for the image stabilization process in Example 2. Processes that are the same as in Example 1 are denoted by the same reference numerals and their descriptions are omitted.

[0132] First, in step S701, the CPU 15 determines whether the photographer has specified an arbitrary AF point through a menu or by touching the LCD screen. If it has been specified, the process proceeds to step S705, as the photographer has designated the main subject area. In the process from step S705 onward, the subject within the specified main subject area, or a nearby subject, is treated as the main subject, and motion blur correction is performed to maintain the position of the main subject in the same location on the image.

[0133] In step S705, if the system is set to automatically activate AF operation when an arbitrary AF point is specified by the photographer, the system waits for the AF operation to complete (completion of focusing on the subject at the specified AF point) and then proceeds to step S422. If the system is set not to activate AF operation, the system immediately proceeds to step S422. Then, the same processing as in Example 1, from steps S422 to S409, is performed to correct the blur (subject blur) that occurs in the main subject of the captured image.

[0134] If the CPU 15 determines in step S701 that no AF point has been specified, the process proceeds to step S702. In step S702, it is determined whether the photographer has intentionally changed the main subject, and if so, the process proceeds to step S703. In this step, the presence or absence of a change in the main subject is determined by whether the focal length, focus position, and exposure amount have been changed by the photographer by a predetermined amount or more. If one or more of the focal length, focus position, and exposure amount have been changed by a predetermined amount, it is determined that the photographer has intentionally changed the main subject. If none of them have been changed by a predetermined amount or more, it is determined that there has been no intentional change in the main subject.

[0135] The photographer can change the focal length of an interchangeable lens, for example, by operating the zoom ring (not shown) of the interchangeable lens 31. This is achieved by a mechanical structure configured such that the zoom lens (not shown), which constitutes the photographic optical system, moves in proportion to the amount the zoom ring is operated. Alternatively, the zoom lens may be moved by driving an actuator in proportion to the amount the zoom ring is operated. In this case, the operating part may be a zoom lever or a zoom button instead of a zoom ring. The photographer can change the focus position, for example, by operating the manual focus ring (not shown) that adjusts the focus of the interchangeable lens 31. This is achieved by a mechanical structure configured such that the focus lens 33 moves in proportion to the amount the manual focus ring is operated. Alternatively, the focus lens 33 may be moved by driving an actuator in proportion to the amount the manual focus ring is operated. In this case, the operating part may be a manual focus button instead of a manual focus ring. Furthermore, when the focus lens 33 is controlled electronically in this way, a single ring member may function as both a zoom ring and a manual focus ring.

[0136] In this embodiment, if the photographer changes the focal length by a predetermined amount or the focus position by a predetermined amount or more, it is determined that the photographer has intentionally changed the main subject. During this change operation, blur due to the movement of the main subject itself is not corrected, and only image stabilization is performed.

[0137] The exposure compensation amount, exposure time, and aperture value can be changed, for example, by operating the exposure compensation amount change dial, exposure time change dial, and aperture value change dial, which are operation switches 18. If the exposure compensation amount is changed, the CPU 15 calculates the exposure time of the image sensor 3 and the aperture value of the aperture 34 based on this exposure compensation amount and AE evaluation value. If the exposure time, aperture value, or both are changed, the CPU 15 calculates the exposure time or aperture value as needed based on the set value and shooting mode. Then, it transmits the aperture value information of the aperture 34 to the interchangeable lens 31 via the communication driver 21. On the interchangeable lens side, aperture drive processing is performed and the aperture amount of the aperture 34 is adjusted to be appropriate. In addition, the CPU 15 controls the exposure time of the image sensor 3 via the TG 16 and the image sensor driver 17 to adjust the exposure time to be appropriate.

[0138] In this embodiment, if the photographer changes the exposure amount by more than a predetermined amount (a change in exposure compensation amount or a change in exposure amount by a combination of exposure time and aperture value), it is determined that the photographer has intentionally changed the main subject. During this change operation, blur due to movement of the main subject itself is not corrected, and only image stabilization is performed.

[0139] If it is determined in step S702 that an intentional change of the main subject has occurred, and the process proceeds to step S703, the CPU 15 turns on a flag indicating that an intentional change of the main subject has been made by the photographer (hereinafter referred to as the subject change flag), and proceeds to step S704. In step S704, the CPU 15 sets the direction for correcting camera shake. The method for setting the correction direction differs depending on whether or not the operation to change at least one of the focal length, focus position, or exposure amount by the photographer to a predetermined value or more is accompanied by panning of the imaging device 1. If the imaging device 1 is not panning, the correction direction is set to all directions in which the imaging device 1 can correct camera shake, such as the pitch direction, yaw direction, and roll direction. On the other hand, if the imaging device is panning, similar to step S403 in Example 1, the movement direction of the imaging device is determined, the direction on the image sensor perpendicular to that movement direction is calculated, and that direction is set as the blur correction direction during panning.

[0140] Subsequently, the process from steps S404 to S409 is performed in the same manner as in Example 1, and image stabilization is performed for the correction direction set in step S704.

[0141] In this manner, if the photographer changes the focal length, focus position, or exposure amount by a predetermined amount or more, it is determined that the main subject has been intentionally changed, and the blur (camera shake) caused by the movement of the imaging device 1 in the captured image is corrected. On the other hand, if it is determined in step S702 that the photographer has not intentionally changed the main subject, the process proceeds to step S411. In the steps from step S411 onward, as in Example 1, if the main subject is not detected, camera shake correction is performed. Also, if the main subject is detected and the transition period from camera shake correction to subject blur correction has ended, subject blur correction is performed; if the transition period is still in effect, a combined transition period processing of camera shake correction and subject blur correction is performed.

[0142] In steps S411 and S412, similar to Example 1, the CPU 15 controls the main subject detection circuit 26 to perform main subject detection and determines whether or not a main subject has been detected.

[0143] If the main subject is not detected, the process proceeds to step S721. If the subject change flag was set to ON in step S703 for the previous frame, the flag is set to OFF, and the process proceeds to step S405. The processing from step S405 onward corrects camera shake in the same manner as in Example 1.

[0144] If the main subject is detected, proceed to step SS711 to determine whether the subject change flag is turned on.

[0145] If the subject change flag is turned on, proceed to step S712, turn off the subject change flag that was turned on in step S703, and then proceed to step S713.

[0146] The processing from step S713 onward is a process that gradually transitions from image stabilization performed during panning to subject blur correction (correction of blur in the image caused by camera shake and the movement of the subject itself). First, in step S713, the transition period Tch is set, similar to step S415 in Example 1. Step S713 is reached when an operation that suggests a change in the main subject (an operation to change the focal length, focus position, or exposure amount) has occurred up to the previous frame. Therefore, it is preferable to determine the transition period Tch based on the duration of the operation that occurred up to the previous frame. For example, The transition period Tch should be the same as the time Top required for the photographer to change the focal length, focus position, and exposure, thereby enabling the acquisition of smooth footage. However, since fine adjustments may be made at the end of component manipulation, this time should be taken into consideration, and Top should be set to 10-20% of the time required for average changes. + The time including that period will be considered the transition period.

[0147] In other words, the transition period Tch = Top + Top + This is the result.

[0148] Also this Top + You may also consider the focal length. In that case, first determine the value for a focal length of 50mm, then use a value that is a proportional multiple of the focal length for focal lengths between 24mm and 200mm, use the same value for focal lengths less than 24mm as for 24mm, and use the same value for focal lengths greater than 200mm as for 200mm. For example, the value for a focal length of 100mm would be twice the value for 50mm, and the value for 25mm would be half the value for 50mm.

[0149] However, in this embodiment, unlike Embodiment 1, the speed of panning (tilting) is not reflected in the transition period, even if panning (tilting) is involved. After setting the transition period Tch in step S713, the same steps S416 to S418 as in Embodiment 1 are performed to transition from camera shake correction to subject blur correction, and then the process proceeds to step S719.

[0150] In step S719, similar to step S419, a correction amount is obtained by taking a weighted average of the hand shake amount detected in step S418 and the subject shake amount. The correction ratio is determined based on the set transition period Tch and the elapsed time since the start of the transition process. However, the correction ratio is determined so as to perform hand shake correction during the fine adjustment time at the end of the member operation.

[0151] For example, the correction ratio uses the transition period Tch obtained in step S713 and the time Top considering the fine adjustment operation + and the elapsed time Tpst since the start of the transition process, and is determined by the following formula. (When Tpst < Top + ) Correction ratio of hand shake amount = 1 Correction ratio of subject amount = 0 (When Top + ≦ Tpst ≦ Tch) Correction ratio of hand shake amount = 1 - (Tpst - Top + ) / (Tch - Top + ) Correction ratio of subject shake amount = (Tpst - Top + ) / (Tch - Top + ) The sum of the correction ratio of the hand shake amount and the correction ratio of the subject shake amount is 1.

[0152] Each shake correction amount is expressed by the following formula using the hand shake amount and the subject shake amount obtained in step S418, and the final shake correction amount is obtained by adding the hand shake correction amount and the subject shake correction amount. Hand shake correction amount = Correction ratio of hand shake amount × Hand shake amount Subject shake amount = Correction ratio of subject shake amount × Subject shake amount

[0153] Thereafter, shake correction is performed by performing the same process as step S409 in the first embodiment.

[0154] Thus, immediately after the photographer has finished changing the focal length, focus position, or exposure amount by a predetermined value or more, the system transitions from a control that corrects camera shake to a control that corrects subject blur occurring in the main subject of the captured image.

[0155] On the other hand, if the flag is not turned on, the process proceeds from step S711 to S421, and, as in Example 1, it is determined whether the period for transitioning from camera shake correction to subject blur correction has ended. If the transition period has ended, the process proceeds to step S422 to perform subject blur correction; otherwise, the process proceeds to step S416 to perform the transition from camera shake correction to subject blur correction. Once the processing in step S409 is completed, the process moves to the next frame, and the processing from step S701 is executed again.

[0156] As described above, in this embodiment, the CPU 15 plays the role of a determination means for detecting intentional changes in the main subject by the photographer, such as changes in the focal length during shooting, changes in the focus position by a certain amount or more, or changes in the exposure compensation amount by a certain amount or more. Furthermore, the CPU 15 plays the role of a means for calculating a predetermined direction for blur correction during changes in the main subject.

[0157] The CPU 15 is responsible for detecting changes in the AF area from the AF area detected by the main subject area detection means. The CPU 15 is also responsible for detecting changes in focus and exposure by the photographer by receiving the output of the operation switch 18. Furthermore, the CPU 15 is responsible for determining whether the photographer intentionally changes the main subject based on changes in focal length, focus position, exposure, or AF area. In addition, the CPU 15 is responsible for setting the correction direction, which determines a predetermined direction for performing blur correction during the main subject change operation.

[0158] By performing the processing described above, in this embodiment, even if the photographer changes the main subject during subject blur correction control, stable video recording is possible by performing image stabilization in a predetermined direction during the change. Furthermore, by reducing the camera shake that increases during the manipulation of components, it becomes possible to provide more stable video.

[0159] (modified version) Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0160] In Examples 1 and 2 described above, interchangeable-lens imaging devices were used as examples, but the present invention can also be applied to image stabilization for digital video cameras and digital cameras with integrated lenses. It can also be applied to cameras with an automatic shooting function that automatically takes pictures without user instructions. Furthermore, it can be applied to imaging devices mounted on a tripod head, or imaging devices equipped with a PT mechanism that can perform panning and tilting according to user instructions. In this case, the panning in Examples 1 and 2 may not be panning caused by user movement, but rather panning caused by the tripod head or PT mechanism being driven according to user instructions.

[0161] Furthermore, in the above-described examples 1 and 2, an electronic image stabilization means was used to electronically correct image blur, but an optical image stabilization means can also be used. As an optical image stabilization means, a correction means that corrects blur by moving the image stabilization lens 32 or a correction means that corrects blur by moving the image sensor 3 can be used.

[0162] Furthermore, the method for determining whether or not the main subject has been changed may be a combination of the methods described in Example 1 and Example 2 above.

[0163] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0164] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0165] 1. Imaging device 2 Lens Mounts 3 Image sensor 4. Imaging Circuit 13. Blur detection circuit 14. Motion detection sensor 15 CPU 18 Operating switches 26 Main subject detection circuit 27 Motion vector detection circuit 28 Image deformation and cropping circuit 31 interchangeable lenses

Claims

[Claim 1] A subject detection means for detecting the main subject, A first motion detection means for detecting the motion vector of the main subject, A second motion detection means for detecting the movement of the imaging device, A first blur correction amount acquisition means acquires a first blur correction amount to correct the image blur of the main subject based on the motion vector of the main subject detected by the first motion detection means, A second blur correction amount acquisition means acquires a second blur correction amount to correct the blur of the imaging device based on the movement of the imaging device detected by the second motion detection means, A shake correction control means controls the shake correction means based on at least one of the first shake correction amount and the second shake correction amount, The system includes a determination means for determining whether or not the main subject has changed between a first timing when the first image is captured and a second timing when the second image is captured. The aforementioned blur correction control means is When the main subject is detected by the subject detection means, blur correction is performed based on the first blur correction amount. A motion blur correction control device characterized in that, when the determination means determines that the main subject has been changed, motion blur correction is performed based on the first motion blur correction amount and the second motion blur correction amount with respect to the changed main subject.

Images