Image blur correction device, imaging device, and image blur correction device control method

Abstract

To provide an image blur correction device with which it is possible to obtain an excellent video image by switching blurs to be preferentially corrected.SOLUTION: The image blur correction device comprises: an imaging object detection unit 105 for detecting an imaging object from an inputted image; an imaging object motion information acquisition unit 106 for acquiring motion information indicating motion of the imaging object in the image; an imaging device motion information acquisition unit 108 for acquiring motion information indicating motion of an imaging device; a blur correction control unit 109 capable of controlling imaging object blur correction for correcting the motion of the imaging object in the image and imaging device blur correction for correcting an image blur due to the motion of the imaging device; and a determination unit 107 for determining, on the basis of the imaging object motion information, which of the imaging object blur correction and the imaging device blur correction should be prioritized. The blur correction control unit 109 controls the imaging object blur correction and the imaging device blur correction on the basis of a result of the determination of the determination unit 107.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to stabilization of a subject image using an image blur correction means. [Background technology]

[0002] In recent years, various blur correction functions have been proposed for correcting image blur caused by camera shake or the like applied to an imaging device such as a digital camera, and by installing these functions in an imaging device, it has become possible to obtain better captured images. In addition to camera shake caused by a user holding the imaging device body, it has also been proposed to correct (also called tracking) subject blur caused by a change in the position of a subject such as a person. Here, the user's camera shake can be detected based on the detection result of an angular velocity sensor attached to the imaging device or the amount of movement of a still area in a captured image. In contrast, subject blur can be detected based on the amount of movement of the subject's position by a subject recognition means or the like. Then, when correcting subject blur, it can be realized by controlling an image blur correction means based on the detected movement of the subject so that the subject is kept at a specific position such as the center of the image.

[0003] The imaging device described in Patent Document 1 performs pan and tilt driving of an imaging unit to track an object and correct image blur caused by shaking of the imaging device. This imaging device calculates a first driving amount for suppressing image blur caused by shaking of the imaging device and a second driving amount for tracking an object, and generates a driving signal based on a predetermined ratio of the first driving amount and the second driving amount. When the speed of the pan driving and the tilt driving are equal to or higher than a threshold, the predetermined ratio is set smaller than when the speed is less than the threshold, making it less likely for an oscillation phenomenon to occur. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-180341 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned Patent Document 1 does not provide any detailed description as to whether to give priority to correction of subject blur or blur applied to the image capture device, and does not fully consider how to switch between subject blur correction and blur correction applied to the image capture device.

[0006] The present invention provides an image stabilization device that can acquire good images by switching the type of shake that is given priority for correction. [Means for solving the problem]

[0007] An image blur correction device according to one aspect of the present invention includes a subject detection unit that detects a subject from an input image, a subject motion information acquisition unit that acquires motion information indicating motion of the detected subject in the image, an imaging device motion information acquisition unit that acquires motion information indicating motion of the imaging device, a blur correction control unit capable of controlling subject blur correction for correcting the motion of the subject in the image and imaging device blur correction for correcting image blur caused by the movement of the imaging device, and a determination unit that determines which of the subject blur correction or the imaging device blur correction is to be prioritized based on the subject motion information, and the blur correction control unit controls the subject blur correction and the imaging device blur correction based on a determination result of the determination unit.

[0008] Other aspects of the present invention are described in detail in the following detailed description of the invention. Effect of the Invention

[0009] According to the present invention, it is possible to provide an image stabilization device that is capable of acquiring a good image by switching the type of shake that is to be preferentially corrected. [Brief description of the drawings]

[0010] [Figure 1]FIG. 1 is a block diagram showing an example of the configuration of an imaging device according to an embodiment; [Diagram 2] Flowchart relating to subject tracking operation of the embodiment [Diagram 3] FIG. 1 is a schematic diagram illustrating switching of blur to be corrected; [Figure 4] FIG. 1 is a schematic diagram illustrating subject movement determination; [Diagram 5] Schematic diagram illustrating a method for switching the correction target DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0012] In this embodiment, an example of an imaging device having three image blur correction means, namely, lens type image blur correction, sensor type image blur correction, and electronic type image blur correction, will be described as an image blur correction device. Lens type image blur correction is a function that corrects image blur by driving (displacing) a correction lens. Sensor type image blur correction is a function that corrects image blur by driving (displacing) the image sensor 102. Electronic type image blur correction is a function that corrects image blur by performing image processing such as geometric transformation on an acquired image signal.

[0013] The imaging device of this embodiment has a subject blur correction function (also called subject tracking) that corrects the blur of the main subject image (hereinafter called subject blur) caused by the movement of the main subject in the image. If the movement of the subject in the image is large during subject tracking and it is determined that it is difficult to continue tracking the subject, the blur applied to the imaging device is corrected with priority. The determination of whether it is difficult to continue tracking the subject is made based on the movement information of the subject in the image.

[0014] In the present invention and this specification, the shake applied to the imaging device may be called camera shake, but this is not limited to shake transmitted from the user's hand, but also includes shake transmitted from a vehicle or a building when the imaging device is fixed to the vehicle or a building.

[0015] 1 is a block diagram showing the configuration of an image capture device 100 which is an embodiment of an image blur correction device according to the present invention. The image capture device 100 includes an optical system 101 which forms a subject image, an image capture element 102 such as a CCD sensor or a CMOS sensor which photoelectrically converts the subject image formed by the optical system 101, and a development processing unit 103 which forms an image signal from an electrical signal output from the image capture element 102.

[0016] The optical system 101 has a lens group. The lens group has a correction lens that is driven in a direction different from the optical axis of the optical system 101 to move a subject image on an imaging surface and correct image blur. This correction lens is driven based on a correction amount calculated by a correction amount calculation unit 109, which will be described later, to correct image blur. In addition, the image sensor 102 is configured to be able to correct image blur by being driven in a direction perpendicular to the optical axis.

[0017] The development processing unit 103 includes an A / D conversion circuit, an auto gain control circuit (AGC), and an auto white balance circuit (not shown), and forms a digital signal. The image sensor 102 and the development processing unit 103 form an imaging system that acquires an image. One frame or multiple frame images of the image signal formed by the development processing unit 103 are used as is for display or processing, and are also temporarily stored and held in a memory 104.

[0018] The subject detection unit 105 performs subject detection on the image signal output from the development processing unit 103, and detects the subject to be tracked (hereinafter, sometimes referred to as the main subject). The main subject is a moving subject, and examples of the main subject include a person (face), an animal, and a vehicle such as a train or an airplane. A known method may be used to detect the main subject, and examples of the method include a matching process in which the subject to be detected is used as a template, and a process in which the feature amount of the subject is stored as data and an area having the same feature amount is searched for. As another method, a method of configuring a classifier for subject recognition using a learning algorithm such as a neural network may be used.

[0019] Subject movement information acquisition section 106 acquires subject movement information indicating how the main subject detected by subject detection section 105 moves on the image over time. The subject movement information can be acquired by acquiring position information of the main subject included in the result of main subject detection by subject detection section 105 in chronological order.

[0020] Based on the subject motion information acquired by subject motion information acquisition unit 106, determination unit 107 estimates what kind of blur (movement) has occurred in the main subject in the image and whether or not blur correction (tracking) of the subject can be performed in the future. Then, based on the estimation result, it determines which of the subject motion (subject blur) and the motion applied to the imaging device (camera shake) is to be corrected with priority, that is, the ratio of subject blur correction to imaging device blur correction. When subject blur is to be corrected with priority, the ratio of subject blur correction is made larger than the ratio of imaging device blur correction, and when blur applied to the imaging device is to be corrected with priority, the ratio of subject blur correction is made smaller than the ratio of imaging device blur correction. The ratio of subject blur correction to imaging device blur correction is the amount of correction caused by each blur amount in the total amount of correction of subject blur correction and imaging device blur correction, and the ratio of subject blur correction is the amount of correction caused by the amount of subject blur in the total amount of correction. Even when it is determined that subject blur correction (or blur applied to the imaging device) is to be given priority, the blur applied to the imaging device (or subject blur) may also be corrected.

[0021] The imaging device motion information acquisition unit 108 acquires imaging device motion information indicating motion information such as camerawork motion and camera shake occurring in the imaging device. Here, the motion of the imaging device 100 in this embodiment refers to temporal changes in the position and orientation of the imaging device 100, and includes intentional motions such as panning and tilting, as well as motions unintended by the photographer, such as camera shake. There is no particular limit to the means for acquiring the imaging device motion information. For example, a detection result may be acquired from a gyro sensor or an acceleration sensor arranged in the imaging device. Also, the amount of motion (background vector) of the entire screen between frame images acquired successively in time may be calculated by image analysis using an image signal output from the development processing unit 103. Also, other methods may be used.

[0022] Correction amount calculation unit 109 calculates a correction amount based on the determination result in determination unit 107, motion information of the imaging device obtained from imaging device motion information acquisition unit 108, and motion information of the subject acquired from subject motion information acquisition unit 106. In this embodiment, this correction amount includes the drive amount of the correction lens and imaging element 102, and the amount of geometric deformation in geometric deformation unit 110. Each image blur correction means performs image blur correction based on this correction amount, so correction amount calculation unit 109 functions as a blur correction control unit by sending the calculated correction amount to each image blur correction means.

[0023] In this embodiment, the image capturing device 100 has three image blur correction mechanisms because the correction lens and image capturing element 102 of the optical system 101 are driven and the geometric transformation unit 110 performs geometric transformation processing. However, one or more image blur correction mechanisms are sufficient, and the image capturing device body does not have to have one. For example, image blur correction may be performed by an image processing device that receives an image signal captured by the image capturing device 100 via wired or wireless communication and performs geometric transformation processing on the received image signal.

[0024] A geometric transformation unit 110 performs a geometric transformation process to correct the blur, using the amount of geometric transformation calculated by the correction amount calculation unit 109. Then, the image whose blur has been corrected is displayed on a display device by a video output unit 111, or is stored and held in an image storage device (not shown).

[0025] The operation of subject blur correction by the imaging apparatus 100 configured as above will be described with reference to the flowchart shown in Fig. 2. Note that in this embodiment, subject blur correction is performed in real time by executing the following steps S201 to S207 for each frame.

[0026] In step S201, the subject image formed by the optical system 101 is output by the image sensor 102 as an analog signal according to the subject brightness, and an image signal is generated by processing the analog signal in the development processing unit 103. The development processing unit 103 converts the analog signal into, for example, a 14-bit digital signal by an A / D conversion unit (not shown). The digital image signal is further subjected to signal level correction and white level correction by AGC and AWB (not shown), and is transmitted to the motion vector detection unit 109 and the display unit 111, and is also stored and held in the memory 104. In the imaging device 100 of this embodiment, frame images are generated sequentially at a predetermined frame rate, and the transmitted and stored frame images are also updated sequentially.

[0027] In step S202, imaging device motion information is acquired by the imaging device motion information acquisition unit 108. The motion information acquired in this step is transmitted to the correction amount calculation unit 109.

[0028] In step S203, the subject detection unit 105 detects a main subject present in the frame image acquired in step S201. Even if there are multiple subjects that can be the main subject in the image, one subject is selected in this embodiment. One example of the selection method is a method in which the subject detection unit 105 selects one subject that is considered to be the most important and outputs that subject. For example, a subject that is larger than other subjects, is located closer to the center of the image, and has a higher detection reliability may be selected by estimating that it is more important. Another example of the selection method is to have the photographer select the main subject from multiple subject candidates.

[0029] This step acquires the position of the main subject, the size of the main subject, and the reliability of the main subject (if the main subject is a person, information indicating the likelihood of the subject being a person). The position information of the subject detected in this step is transmitted to subject movement information acquisition unit 106.

[0030] In step S204, subject movement information acquisition unit 106 acquires subject movement information indicating the amount of movement of the main subject based on the position information of the main subject detected and selected in step S203. Note that if a subject is detected for the first time in the process of S203 for this frame, this step is skipped, and subject movement information is acquired by performing this step from the next frame.

[0031] There is no particular limit to the method of acquiring the amount of movement of the main subject. For example, the position of the main subject is acquired using known template matching, using a partial image including the subject detected as the main subject in step S203 as a template. Then, by chronologically arranging the position of the main subject in the image up to the previous frame, the relationship between time and the amount of movement of the main subject can be acquired. Note that if the main subject is lost, the subject detection process (step S203) in subject detection unit 105 can be started again.

[0032] For example, by arranging the position information of the main subject output from subject detection section 105 in chronological order, subject movement information indicating the relationship between time and the amount of movement of the main subject can be obtained.

[0033] In step S205, determination unit 107 determines whether or not to prioritize subject blur correction over camera shake correction in the current frame, based on the movement of the main subject in the image acquired by subject movement information acquisition unit 106. Hereinafter, this determination may be referred to as subject movement determination.

[0034] Then, the state where it was determined that subject blur correction was prioritized in the previous frame is switched to a state where subject blur correction is not prioritized in the current frame, and where correction of subject blur is prioritized, to a state where correction of camera shake is prioritized. This switching will be explained in a step to be described later. As a result, although subject blur cannot be corrected, it is possible to generate an image that maintains stability overall.

[0035] Here, a method of determining whether or not to prioritize subject blur correction will be described. In this embodiment, based on the subject motion information up to the present acquired by the subject motion information acquisition unit 106, it is predicted whether or not there is a high possibility that subject blur will not be able to be corrected in the future. If it is predicted that there is a high possibility that subject blur will not be able to be corrected in the future, it is determined that subject blur correction will not be prioritized, and if it is predicted that there is a high possibility that subject blur will not be able to be corrected in the future, it is determined that subject blur correction will be prioritized. The high possibility that subject blur will not be able to be corrected refers to, for example, a high possibility that subject blur is large and the amount of image blur correction will be so large that it cannot be corrected by each image blur correction means.

[0036] The effect of shifting from camera shake compensation to subject blur compensation and the details of the subject movement determination method will be described with reference to FIGS.

[0037] Figures 3 (a and b) show graphs of changes over time in camera shake and subject blur. In Figure 3, the horizontal axis represents time, and the vertical axis represents position. Note that for the purposes of explanation in this specification, the graph shows movement in only one direction as blur, but in reality movement in an image is two-dimensional, and the same can be said for movement in other directions.

[0038] Fig. 3(a) shows a graph that shows a schematic representation of subject blur and camera shake that appear on an image when subject blur and camera shake occur. The position 301 of the main subject that is the subject of subject blur correction changes over time, indicating that the main subject is undergoing amplitude motion. On the other hand, the camera shake motion that appears on the image is the motion applied to the imaging device that appears on the image, and is therefore a motion that occurs over the entire image, unlike the motion of a localized area such as the main subject. Therefore, in Fig. 3(a), the position 302 of the background located at the center of the image is represented as the motion of the imaging device, that is, camera shake.

[0039] FIG. 3(b) is a graph that shows a schematic representation of the subject and camera shake that appear on an image when processing is performed to transition from subject blur correction to camera shake correction, as described above.

[0040] During the time period indicated by period 305, subject blur correction is performed to correct the blur of the subject and keep the subject at the target position. Therefore, the position 303 of the main subject on the image does not change during period 305, and the main subject image continues to stay at the target position. On the other hand, camera shake is not corrected, so the position 304 of the background located at the center of the image changes.

[0041] Then, in period 306, if the subject blur is so large that it is no longer possible to correct the subject blur, the image blur correction means is controlled to correct camera shake from the subsequent period 307. As a result, from period 307 onwards, although the position of the main subject has moved, the movement of the entire screen indicating the movement of camera shake can be reduced. On the other hand, if this switching is not performed, a period such as period 306 will continue in which neither subject blur nor camera shake can be reduced. In this way, by switching control from subject blur correction to camera shake correction based on the movement information of the subject on the image, it is possible to shorten the period such as period 306 in which neither subject blur nor camera shake can be reduced.

[0042] Incidentally, a situation in which subject blur correction becomes impossible occurs when the movement of the main subject is too fast or changes direction drastically, making it impossible for subject movement information acquisition unit 106 to acquire the amount of movement of the main subject, or when the movement exceeds the driving capacity of the image blur correction means.

[0043] Here, as shown in FIG. 3B, if it is determined whether or not to continue subject blur correction based on the motion information sent from subject motion information acquisition unit 106 after it becomes impossible to correct subject blur, a certain amount of time is required from when it becomes impossible to correct subject blur correction until it is determined not to continue subject blur correction. Therefore, a time period occurs in which neither the motion of the subject nor the motion of camera shake has stopped, as shown in period 306. Therefore, rather than determining whether or not to continue subject blur correction only after it becomes impossible to correct subject blur, it is preferable to predict whether or not it is likely to be possible to continue subject blur correction from the state in which subject blur correction has been possible (during period 305). This makes it possible to shorten (including eliminate) the period, such as period 306, in which neither subject blur nor camera shake can be reduced.

[0044] Fig. 4 shows a schematic diagram of an example of a method for determining subject blur. As in Fig. 3, the horizontal axis indicates time, and the vertical axis indicates position on the image.

[0045] FIG. 4(a) is a schematic diagram showing an example of a method for determining whether or not it is likely that object blur correction will become impossible based on the magnitude of object blur. The position 401 of the main object to be corrected for object blur moves in a period 402 such that object blur correction is possible, and an example is shown in which the amplitude of the movement amount becomes so large that object blur correction becomes impossible after period 403. In FIG. 3(b), it is determined that object blur correction is impossible after it becomes impossible to correct object blur, and a period 306 occurs in which both object blur and camera shake cannot be corrected completely and remain as image blur. However, in FIG. 4(a), object movement information is obtained from the stage of period 402, and a prediction is made as to whether or not it is likely that object blur correction will become impossible in the following time. In other words, a prediction is made as to whether or not object blur correction will be able to continue in the future in parallel with object blur correction.

[0046] As an example of a prediction method, there is a method of setting a threshold value 404 (here, th and th') and using it for judgment. In this method, if the movement of the subject exceeds the threshold value a predetermined number of times, it is judged that there is a high possibility that the movement of the subject will become large thereafter and that the subject blur correction will eventually become impossible. In the example of FIG. 4(a), the movement of the subject exceeds the threshold value a predetermined number of times during a period 405 within a period 402 in which the subject blur correction is possible, and it is judged that there is a high possibility that the subject blur correction will not be possible in the future at the timing of time 406. Here, the threshold value 404 may be a fixed value or a variable value. When it is a variable value, it may be set based on, for example, a search range of the main subject in the subject detection unit 105 (a range for searching for the main subject detected in S203) or a movable range of the optical system 101 and the image sensor 102, or based on a type or distance of the subject obtained separately.

[0047] In addition, since it is not known in which direction the subject will move, it is preferable to set a threshold value of the same magnitude for both directions of movement, for example, based on the subject blur correction control start position or the target position for stopping the subject, as shown in threshold value 404. The same applies to the method of setting the predetermined number of times, and a predetermined fixed value may be used, or the number of times may be set according to the movement of the subject. As a method of setting according to the movement of the subject, for example, a method of setting the number of times more for a subject that tends to move vigorously and less for a subject that does not move very much, according to the type of subject and the result of action determination obtained separately, can be mentioned.

[0048] In addition to the number of times the threshold is exceeded, the time the threshold is exceeded can also be used to predict whether subject blur correction will become impossible. If the subject continues to move in a manner that exceeds the threshold for a certain period of time, it can be determined that there is a high possibility that the subject movement will become larger thereafter, and that ultimately subject blur correction will become impossible.

[0049] Furthermore, when the subject motion includes an amplitude motion, there is a method of using the envelope of the subject motion (the envelope of the curve showing the change over time of the subject position 401) for the judgment. FIG. 4(b) shows an envelope 407 drawn for the subject motion. The inclination of this envelope 407 is calculated and successively observed from the point when the subject motion compensation is possible (the point of time of period 305 in FIG. 3(b)), and it is judged whether there is a high possibility that the subject motion compensation will not be possible in the future. FIG. 4(b) shows a tangent 408 of the envelope at a certain time. Then, when the inclination of the tangent 408 becomes larger than a predetermined threshold, it is judged that there is a high possibility that the subject motion will become large in the future and that the subject motion compensation will not be possible in the end. Also, if the inclination of the tangent 408, that is, the tendency of the subject motion to increase, is known, it is possible to calculate the magnitude of the subject motion at what time from the magnitude of the inclination, and therefore it is also possible to predict the timing when the subject motion compensation will not be possible.

[0050] In FIG. 4(a), one threshold is set for determining the motion of the subject, but by setting multiple thresholds of different magnitudes, it is possible to improve the accuracy of the determination. FIG. 4(c) shows a schematic diagram of a case where multiple thresholds are set to determine the motion of the subject. In FIG. 4(c), multiple thresholds 409 (three thresholds, th1, th2, and th3, in FIG. 4(c)) are set for determining the motion of the subject. As a method for determining the motion of the subject when multiple thresholds are set, for example, the time from the first exceedance of the smallest first threshold (th1 or th1′) to the second largest second threshold (th2 or th2′) and the time from the second threshold to the largest third threshold (th3 or th3′) can also be used. Instead of the time, the number of times the first threshold is exceeded from the first exceedance of the first threshold to the second threshold and the number of times the second threshold is exceeded from the second threshold to the third threshold may be used. Specifically, it is possible to predict the time until the movement of the subject cannot be corrected based on the difference between the length of time from exceeding the first threshold to reaching the second threshold and the length of time from exceeding the second threshold to reaching the third threshold. Similarly, it is possible to predict the number of amplitudes until the movement of the subject cannot be corrected based on the difference between the number of times the first threshold is exceeded from exceeding the first threshold to reaching the second threshold and the number of times the second threshold is exceeded from exceeding the second threshold to reaching the third threshold.

[0051] In this way, by using multiple thresholds, it is possible to predict and determine the future movement of the subject, that is, the timing when the subject blur cannot be corrected, more accurately than when a single threshold is used. As shown in FIG. 4(c), when the amount of movement of the subject gradually exceeds a larger threshold from a smaller threshold, it is determined that there is a high possibility that the movement of the subject will not be corrected in the future. Note that, although the number of thresholds is three in FIG. 4(c) is described, the number of thresholds is not particularly important, and more thresholds can be set when performing a more detailed determination. Furthermore, by using this determination method, it is also possible to change the ratio of the subject blur correction and the camera shake correction according to which stage of the threshold the subject movement exceeds. For example, when the smallest threshold is exceeded, the ratio of the subject blur correction is lowered and the ratio of the camera shake correction is increased compared to before the smallest threshold is exceeded. Then, when the second largest threshold is exceeded, the ratio of the subject blur correction is further lowered and the ratio of the camera shake correction is increased compared to after the smallest threshold is exceeded. Furthermore, when the largest threshold is exceeded, the ratio of the subject blur correction is further lowered and the ratio of the camera shake correction is increased compared to after the second largest threshold is exceeded.

[0052] Moreover, whether the movement of the subject can be corrected may be related to the movement of the imaging device. When the photographer moves the imaging device significantly by panning or other camera work, or when camera shake becomes significantly large due to changes in the shooting conditions, the apparent movement of the subject on the image increases relatively. When such a phenomenon occurs, even if the movement of the subject itself is small, the movement of the subject cannot be corrected completely, so it is better to switch from correcting the movement of the subject to correcting the camera shake to suppress the deterioration of the quality of the captured image. As an example of a method for determining the movement of the imaging device, the above-mentioned method for determining the movement of the subject can be applied directly to the movement of the imaging device. The information obtained by the imaging device movement information acquisition unit 108 may be used as the information on the movement of the imaging device.

[0053] In addition, the determination of whether the motion of the subject cannot be corrected due to the motion of the imaging device may not be based only on the motion information of the subject obtained by the subject motion information acquisition unit 106. For example, the determination may be made by adding the motion of the imaging device 100 to the motion information of the subject obtained by the subject motion information acquisition unit 106. The change in the position of the main subject in the image is caused by the motion of the main subject itself and the motion of the imaging device. The motion of the main subject itself may be extracted based on the motion information of the subject obtained by the subject motion information acquisition unit 106 and the motion of the imaging device 100, and the determination described in FIG. 4 may be made. Conversely, the motion information of the imaging device of a frequency (mainly high frequency) not included in the motion information of the subject may be obtained from a gyro or acceleration sensor, converted into a motion on the image, and the amount of motion may be added to the motion of the subject, to make the motion determination of the subject described in FIG. 4. This makes it possible to determine whether the motion of the subject can be tracked while taking into account the motion of the imaging device.

[0054] In addition, the amount of correction by the image blur correction means may be taken into consideration when determining whether or not the blur of the subject can be corrected. The correction lens, the image sensor, or the geometric deformation unit is driven to correct the blur of the subject, and the imaging range changes compared to when the correction lens, the image sensor, or the geometric deformation unit is not driven. The amount of change in the imaging range is obtained based on the amount of correction, and this amount of change is added to the movement of the subject. This makes it possible to obtain the amount of movement of the subject in the image when the blur correction of the subject is not performed, and therefore, based on this, it may be possible to determine whether or not the blur of the subject can be corrected, or to predict whether or not it will be possible to correct it in the future.

[0055] So far, we have used analytical methods to determine whether or not to continue subject blur correction based on the movement of the subject, but machine learning can also be used to determine whether or not to continue subject blur correction. By creating a neural network model in advance using the movement of the subject and the determination of whether or not subject blur correction can be performed as learning data, and then applying this to the subject movement information acquired during shooting, it is possible to predict whether or not subject blur correction can be performed in the future.

[0056] The above describes the methods for predicting whether or not the subject movement can be corrected in the future, and determining whether or not to continue subject blur correction. The results of subject movement determination obtained from one or more of these methods are transmitted to correction amount calculation unit 109.

[0057] Also, up to this point, the switching from object blur correction to camera shake correction has been described, but once the object motion has settled after switching from object blur correction to camera shake correction during the operation of object blur correction, the camera may switch back to object blur correction. The determination of whether or not to switch back to object blur correction can be performed in the same manner as the determination method described above. That is, based on the amount of movement of the object, it is predicted whether or not object blur correction is likely to be performed in the future, and if it is likely to be performed, it is determined that object blur correction is to be prioritized, and if it is not likely to be performed, it is determined that camera shake correction is to be prioritized (camera shake correction is to be continued). Steps S201 to S207 are performed for each of a number of frames acquired in succession, and this cycle is repeated even after switching to camera shake correction once, so that the camera can return to object blur correction again. However, once switching to camera shake correction, the camera may be configured not to return to object blur correction unless a user's operation such as a restart instruction is received. In addition, the threshold value, the predetermined number of times, the predetermined time, etc. used when determining whether to return to object blur correction again may be set to values ​​different from the threshold value, the predetermined number of times, and the predetermined time used when determining whether to switch from object blur correction to camera shake correction. In order to prevent frequent switching between object blur correction and camera shake correction, it is preferable to set conditions that make it more difficult to switch from camera shake correction to object blur correction than when switching from object blur correction to camera shake correction. For example, the threshold value used when determining whether to return to object blur correction again may be set to be smaller than the threshold value used when determining whether to switch from object blur correction to camera shake correction. Similarly, the predetermined number of times may be set to be larger than the predetermined number used when determining whether to switch from object blur correction to camera shake correction, and it may be determined that the object blur correction is to be returned to when the state in which the amplitude is equal to or smaller than the threshold exceeds the predetermined number of times. In addition, the predetermined time may be set to be longer than the predetermined time used when determining whether to switch from object blur correction to camera shake correction, and it may be determined that the object blur correction is to be returned to when the state in which the amplitude is equal to or smaller than the threshold continues for a predetermined time or more.Furthermore, when determining whether or not to return to object blur compensation based on the slope of tangent 408 to envelope 407, it may be determined to return to object blur compensation if the slope is opposite to the slope shown in FIG. 4(b) (i.e., the direction in which object blur becomes smaller) and if the slope is greater than a threshold value.

[0058] In step S206, the correction amount calculation unit 109 calculates the amount of correction for image blur based on the result of the subject motion determination, the subject motion information, and the image capture device motion information.

[0059] According to the result of the subject motion determination transmitted from the determination unit 107, the correction amount calculation unit 109 calculates a third correction amount by adding a first correction amount based on the subject movement amount indicated by the subject motion information and a second correction amount based on the movement amount of the imaging device indicated by the imaging device motion information. That is, when the result of the subject motion determination outputted from the determination unit 107 indicates that the subject blur correction is to be continued, the correction amount outputting unit 109 weights the first correction amount more heavily than the second correction amount and adds it, thereby making the ratio of the subject blur correction higher than the ratio of the camera shake correction. This allows the subject blur to be corrected with priority. On the other hand, when the result of the subject motion determination outputted from the determination unit 107 indicates that the subject blur correction is not to be continued, the correction amount outputting unit 109 weights the second correction amount more heavily than the first correction amount and adds it, thereby making the ratio of the camera shake correction higher than the ratio of the subject blur correction. This allows the camera shake to be corrected with priority.

[0060] As described above, giving priority to correcting subject blur (or camera shake) includes not correcting camera shake (or subject blur), and the weight of the correction amount that is not heavily weighted during weighting addition may be 0. In this way, it is possible to perform control for switching the blur to be preferentially corrected between subject blur and imaging device blur, that is, control for switching the control of the image blur correction means between control for subject blur correction and control for camera shake correction. In this embodiment, the image blur correction means refers to any one or more of the mechanism for displacing the blur correction lens of the optical system 101, the mechanism for displacing the imaging element 102, and the geometric deformation unit 110. When image blur correction is performed by a plurality of image blur correction means, the third correction amount is divided into control values ​​for the respective image blur correction means and transmitted to the respective image blur correction means as control values. It should be noted that the first correction amount can also be acquired based on the position information of the subject acquired from the subject detection unit 105, rather than the motion information of the subject acquired from the subject motion information acquisition unit 106. By obtaining a first correction amount based on the difference between the position of the main subject obtained from subject detection unit 105 and a target position of the main subject set in the image (e.g., the center of the image), subject blur correction can be performed even if the position of the main subject in the image is brought closer to the target position.

[0061] If the control of the image blur correction means is switched instantly, the movement that appears on the screen will suddenly switch from a state where the subject motion is stationary to a state where camera shake is stationary, or vice versa, resulting in a discontinuous movement on the screen and an unnatural image. One method for preventing this phenomenon is to gradually switch to camera shake correction control from the time it is determined that there is a high possibility that subject blur correction will not be possible in the future until subject blur correction actually becomes impossible. In other words, in FIG. 4(a), the control is gradually switched to camera shake correction control from time 406 to the end of period 402. This makes it possible to reduce the unnatural feeling that accompanies the transition from subject blur correction to camera shake correction.

[0062] When gradually switching from object tracking to camera shake compensation, the simplest method is to simply change the ratio between object shake compensation and camera shake compensation at a predetermined constant rate. However, such a switching method may result in the camera shake compensation becoming unable to compensate for object shake before the camera shake compensation is completely switched to, or the camera shake compensation may be switched to before the object shake can be sufficiently compensated. Therefore, in order to perform a more natural switching, it is preferable to use information obtained from the determination unit 107 about the time when it is predicted that the object will no longer be able to be tracked. A schematic diagram of the switching method is shown in FIG. 5.

[0063] The horizontal axis of FIG. 5 indicates time, and the vertical axis indicates the ratio of the control of the correction of object blur and the control of the camera shake correction. Furthermore, the time when the determination unit 107 predicts that the correction of object blur will not be possible in the future (time 401 in FIG. 4(a)) is indicated as time 501, and the time when the tracking of the object is predicted to be impossible (the predicted end time of the period 402 in FIG. 4(a)) is indicated as time 502. Then, the temporal variation of the control ratio 503 for correcting object blur and the control ratio 504 for correcting camera shake is indicated. As shown in FIG. 5, the correction of object blur is possible up to the time 501, so that only the control of the object blur correction is being performed. However, after it is predicted at the time 501 that the tracking of the object will not be possible in the future, the ratio of the control of the camera shake correction is gradually increased so that only the control of the camera shake correction is being performed by the time 502 when the tracking of the object is predicted to be impossible. At this time, it is preferable to determine the speed (slope) of changing the ratio based on the length of time from time 501 to time 502. This can reduce the possibility that the subject blur cannot be corrected before the camera shake correction is completely switched to, or that the camera shake correction is switched to when the subject blur can still be sufficiently corrected. Regarding the method of predicting the time when the subject cannot be tracked, for example, when using the determination method described in FIG. 4(a) or FIG. 4(c), it is possible to predict the tendency of the subject's movement thereafter from the number of times and the time when the subject's movement exceeds a threshold. Similarly, the method described in FIG. 4(b) can predict the time until the subject's movement reaches the upper limit of the magnitude that can be corrected from the slope of the envelope.

[0064] In Fig. 5, a method of linearly changing the ratio of control of object blur correction and camera shake correction has been described, but the present invention is not limited to this, and it is possible to change the ratio to draw a gentle curve to place emphasis on one control over the other. Furthermore, in cases where the subject movement changes significantly, there is a possibility that an error will occur in the predicted time when object blur correction will no longer be possible, or that object blur correction will suddenly become impossible. In such cases, the possibility of unintentional image distortion can be reduced by completing the switching process at an earlier time, rather than completely switching to camera shake correction control at the time when it is predicted that object blur correction will no longer be possible.

[0065] In step S207, the vibration isolation members are driven or image processing is performed for either or both of the purposes of subject blur correction and camera shake correction based on the control values ​​for each image blur correction means calculated in the correction amount calculation unit 109. The frame image thus obtained, whose blur has been corrected, is transmitted to the video output unit 111. The video output unit 111 displays the blur-corrected frame image obtained from the geometric transformation unit 110 on a monitor (not shown) or records and holds it in an image storage device.

[0066] As described above, in this embodiment, it is determined whether the movement of the subject can be tracked, and if it is determined that tracking will become impossible, the drive control of the image blur correction means is switched from tracking the subject to correcting camera shake. This makes it difficult for a situation to occur in which neither the movement of the subject nor image blur can be corrected, or even if it does occur, the time required can be shortened. In addition, by predicting whether or not it is likely that object blur correction will become impossible in the future when object blur correction is possible, it is possible to eliminate the time lag that occurs when switching, or to shorten the time lag if it does occur. This makes it less likely for a situation to occur in which neither object tracking nor camera shake correction can be performed, or to shorten the time required if it occurs.

[0067] In the above embodiment, the subject movement has been described as one-dimensional, but the subject blur and camera shake on the actual image occur two-dimensionally. In order to deal with such two-dimensional blur, the above-mentioned determination method and switching method may be applied for each direction of movement to be determined, such as the vertical direction and the horizontal direction. In this case, it is not necessary to switch the control at the same ratio in the vertical direction and the horizontal direction. For example, when it is determined that the subject blur correction cannot be performed for the vertical movement of the subject, the control may be switched to camera shake correction only for the vertical direction. Then, it is possible to continue tracking the subject as much as possible by continuing the subject blur correction for the horizontal movement as long as the subject blur correction is possible. After that, it is possible to switch to camera shake correction for the horizontal direction only after it is determined that the subject blur correction cannot be performed for the horizontal movement as well.

[0068] In addition, in the above-described embodiment, the imaging device 100 has been taken as an example of an image blur correction device, but as long as the image blur correction device can receive an image input from the imaging device, the image blur correction device may be separate from the imaging device 100. For example, an image processing device capable of performing the above-described electronic image blur correction may receive an image input from the imaging device, perform the above-described subject detection and subject movement determination on the input image, and perform electronic image blur correction based on the determination result.

[0069] In addition, in the above embodiment, a description has been given of video shooting, but the present invention is not limited to video shooting as long as multiple consecutive images are acquired; for example, the present invention can also be applied to continuous shooting of still images.

[0070] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0071] 105 Subject detection section 106 Subject movement information acquisition unit 107 Judgment section 108 Imaging device movement information acquisition unit 109 Correction amount calculation section

Claims

1. A subject detection unit that detects a subject from the input image, A subject motion information acquisition unit that acquires motion information indicating the movement of the detected subject in the aforementioned image, An imaging device motion information acquisition unit acquires motion information indicating the movement of the imaging device, A blur correction control unit capable of controlling subject blur correction to correct the movement of the subject in the aforementioned image, and image device blur correction to correct image blur caused by the movement of the imaging device, The system includes a determination unit that determines whether to prioritize the subject blur correction or the imaging device blur correction based on the motion information of the subject, The image blur correction device is characterized in that the blur correction control unit controls the subject blur correction and the imaging device blur correction based on the determination result of the determination unit.

2. The image blur correction device according to claim 1, characterized in that the determination unit performs the determination based on the motion information of the subject while performing subject blur correction.

3. The image blur correction device according to claim 1, characterized in that the subject blur correction brings the position of the subject in the image closer to a target position set within the image.

4. The image blur correction device according to claim 1, characterized in that the correction control unit calculates the amount of correction to be corrected by the image blur correction means based on the determination result by the determination unit, the movement of the subject, and the movement of the imaging device.

5. The image blur correction device according to claim 4, characterized in that the correction control unit changes the weighting of a first correction amount based on the movement of the subject and a second correction amount based on the movement of the imaging device, based on the determination result by the determination unit.

6. The image blur correction device according to claim 5, characterized in that the first correction amount is based on the position of the subject in the image and a target position set within the image.

7. The image blur correction device according to claim 5, characterized in that the first correction amount is based on the amount of movement of the subject.

8. The image blur correction device according to claim 1, characterized in that the determination unit determines, based on the motion information of the subject, that if the amount of movement of the subject exceeds a threshold a predetermined number of times, the image blur correction device should be prioritized.

9. The image blur correction device according to claim 1, characterized in that the determination unit determines, based on the motion information of the subject, that if the amount of movement of the subject exceeds a threshold for a predetermined time, the image blur correction of the imaging device should be prioritized.

10. The image blur correction device according to claim 1, characterized in that the determination unit determines, based on the motion information of the subject, that if the slope of the envelope of the amount of movement of the subject exceeds a threshold, the image blur correction of the imaging device should be prioritized.

11. The image blur correction device according to claim 1, characterized in that the determination unit has a plurality of thresholds of different sizes set, and determines, based on the motion information of the subject, to prioritize the image blur correction of the imaging device if the amount of movement of the subject exceeds the thresholds in stages.

12. The image blur correction device according to claim 1, characterized in that the determination unit performs the determination based on the motion information of the imaging device, and determines that if the temporal variation in the amount of movement of the imaging device is increasing, it will prioritize correcting the blur of the imaging device.

13. The image blur correction device according to claim 1, characterized in that the determination unit acquires the amount of change in the imaging range due to the driving of the image blur correction means that performs the subject blur correction and the image device blur correction that corrects the image blur, and makes the determination based on the amount of movement obtained by adding the amount of change and the amount of movement of the subject based on the motion information of the subject.

14. The image blur correction device according to claim 1, characterized in that the determination unit performs the determination using machine learning.

15. The image blur correction device according to claim 1, characterized in that, if the determination unit determines to prioritize image device blur correction while the image blur correction control unit is prioritizing the correction of the subject blur, it switches to image device blur correction at a certain rate.

16. If the determination unit determines that subject blur correction will become impossible while prioritizing subject blur correction, it estimates the time until subject blur correction becomes impossible. The image blur correction device according to claim 1, characterized in that the blur correction control unit gradually switches to the image imaging device blur correction during the period up to the time estimated by the determination unit.

17. The image blur correction device according to claim 1, characterized in that, when the determination unit determines that the image blur correction of the imaging device should be prioritized, the blur correction control unit switches to the image blur correction of the imaging device as soon as the amount of movement of the subject is large.

18. The image blur correction device according to claim 1, characterized in that, when the determination unit determines that image blur correction should be prioritized, the blur correction control unit switches to image blur correction in the horizontal and vertical directions at different speeds.

19. Image blur correction device according to any one of claims 1 to 18, An image sensor for capturing the aforementioned image, An imaging apparatus characterized by comprising: an image blur correction means controlled by the aforementioned blur correction control unit.

20. A subject detection step that detects a subject from the input image, A subject motion information acquisition step, which acquires motion information indicating the movement of the detected subject in the aforementioned image, An imaging device motion information acquisition process that acquires motion information indicating the movement of the imaging device, A blur correction control step controls subject blur correction to correct the movement of the subject in the aforementioned image, and image device blur correction to correct image blur caused by the movement of the imaging device, The system includes a determination step of determining whether to prioritize the subject blur correction or the imaging device blur correction based on the motion information of the subject, A control method for an image blur correction device, characterized in that the blur correction control step controls the subject blur correction and the imaging device blur correction based on the determination result of the determination step.

Images