Image blur correction control device, method for controlling the same, and imaging device

Abstract

To more properly perform subject blur correction.SOLUTION: An image blur correction control device 110 includes subject detection means 104 for detecting a subject from an input image, subject blur correction amount calculation means 106 for calculating a subject blur correction amount on the basis of the position in the input image of the detected subject and the target position set in the image, background information detection means 107 for detecting information included in the background from the input image as background information, and subject blur correction amount adjustment means 109 for setting a first direction where subject blur correction is performed and adjusting the subject blur correction amount so as to more restrict the subject blur correction amount in a second direction than the subject blur correction amount in the first direction. The first direction is set on the basis of the background information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image stabilization control device, a control method therefor, and an imaging device. [Background technology]

[0002] 2. Description of the Related Art In imaging devices such as digital cameras, correction is performed for blurring caused by the movement of a subject such as a person, as well as blurring caused by blurring of the imaging device itself, such as a hand shake caused by a user holding the imaging device.

[0003] When subject blur correction is used, constantly correcting both horizontal and vertical subject blur can result in blurring of the background, resulting in an unnatural image with a loss of quality. For example, in a scene where a person is kicking the ground and jumping, if you try to correct subject blur in the vertical direction (the direction of gravity), the vertical movement of the subject caused by the jump will be canceled out, and instead the background will vibrate vertically, creating an unnatural effect.

[0004] Patent Document 1 discloses a technology that tracks a subject in a predetermined direction while suppressing tracking of the subject in another direction, thereby leaving vertical movement of the subject due to jumps in a captured image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-100239 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the inventors of the present application have found through their investigation that the technology described in Patent Document 1 may not be able to deal with the above-mentioned sense of incongruity.

[0007] SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and has as its object to more appropriately correct subject blur. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, one aspect of the present invention provides an image blur correction control device comprising: subject detection means for detecting a subject from an input image and outputting detected subject information; subject blur correction amount calculation means for calculating an amount of subject blur correction based on the position of the detected subject in the input image and a target position set within the image; background information detection means for detecting information included in a background from the input image as background information and outputting the background information; first setting means for setting a first direction in which subject blur correction is to be performed from among a plurality of directions; subject blur correction amount adjustment means for adjusting the amount of subject blur correction in a second direction different from the first direction so that it is more limited than the amount of subject blur correction in the first direction; and image blur correction control means for controlling image blur correction so as to move the position of the subject closer to the target position, based on the amount of subject blur correction adjusted by the subject blur correction amount adjustment means,

[0009] Other aspects of the present invention will be described in the following embodiments. [Effects of the Invention]

[0010] According to the present invention, it is possible to more appropriately correct subject blur. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing the configuration of an imaging apparatus according to a first embodiment. [Figure 2] 4 is a flowchart illustrating processing performed in the first embodiment. [Figure 3] 5A and 5B are diagrams for explaining a method for calculating a subject blur correction amount. [Figure 4]5A and 5B are diagrams illustrating a method for adjusting the amount of subject blur correction. [Figure 5] FIG. 10 is a block diagram showing the configuration of an imaging apparatus according to a second embodiment. [Figure 6] 10 is a flowchart illustrating processing performed in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed in the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] [Embodiment 1] In this embodiment, we will describe an imaging device equipped with an image stabilization control device that sets the direction in which to correct subject blur and the direction in which to weaken the correction of subject blur based on the panning direction of the imaging device and background information included in an image captured by the imaging device. Weakening subject blur correction includes not performing subject blur correction, or in other words, setting the amount of subject blur correction to 0. Weakening subject blur correction also means limiting the amount of subject blur correction in a given direction to less than the amount of subject blur correction in a direction in which subject blur correction is performed.

[0014] (Overall configuration of the imaging device) Fig. 1 is a block diagram showing an example of the configuration of an image capturing apparatus according to this embodiment. In Fig. 1, the image capturing apparatus 100 includes a lens, but is not limited to this and may be of an interchangeable lens type. Each component of the image capturing apparatus 100 will be described below.

[0015] The imaging device 100 includes an optical system 101, an imaging unit 102, a development unit 103, and an image blur correction control device 110. The optical system 101 has optical elements such as lenses. In this embodiment, the optical system 101 is a correction lens (not shown) that corrects image blur by moving in a direction perpendicular to the optical axis.

[0016] The subject image formed by the optical system 101 is converted into an image signal by the imaging means 102. The imaging means 102 has an imaging element using a CMOS sensor (complementary metal oxide semiconductor) or the like, and the subject image is converted from an optical signal into an electrical signal by the imaging means 102. The imaging element in this embodiment is disposed in a movable unit so that image blur can be corrected by moving in a direction perpendicular to the optical axis of the optical system 101, but is not limited to this configuration.

[0017] The development processing means 103 performs development processing such as white balance processing, color (luminance and color difference signal) conversion, and gamma correction on the imaging signal read out from the imaging element provided in the imaging means 102, and outputs the result as an image.

[0018] The image blur correction control device 110 includes a subject detection unit 104 , a subject tracking unit 105 , a subject blur correction amount calculation unit 106 , a background information detection unit 107 , a panning direction acquisition unit 108 , and a subject blur correction amount adjustment unit 109 .

[0019] The subject detection means 104 performs subject detection processing to detect a specific subject on the input image from the development processing means 103, and outputs information about the detected subject.

[0020] The subject tracking means 105 tracks the position of a subject that is to be subjected to subject blur correction, among the subjects detected by the subject detection means 104. Here, tracking refers to tracking the position of the subject in the input image.

[0021] Subject blur compensation amount calculation means 106 calculates the subject blur compensation amount based on the position in the input image of the subject tracked by subject tracking means 105 and a target position set within the image. The subject blur compensation amount calculated here is a provisional subject blur compensation amount, as it may be adjusted by subject blur compensation amount adjustment means 109.

[0022] The background information detection means 107 performs background detection processing to detect the background of the input image from the development processing means 103, and detects information about the detected background as background information. Specifically, the background information includes at least one of information about boundary lines included in the background, information about the contrast of the background, and the type of background.

[0023] Panning direction detection means 108 detects the direction in which the user moves the imaging device to track the subject, that is, the panning direction, based on the amount of shake applied to imaging device 100. The panning direction can be inferred to be the direction in which the user moves the imaging device to capture the subject, and therefore can be inferred to be the direction in which the user wants to correct subject shake.

[0024] The subject blur compensation amount adjustment means 109 adjusts the subject blur compensation amount calculated by the subject blur compensation amount calculation means 106 based on the outputs of the background information detection means 107 and the panning direction detection means 108. The subject blur compensation amount adjustment means 109 sets the direction in which subject blur compensation is to be performed (hereinafter sometimes referred to as the first direction) based on the background information output from the background information detection means 107 and the panning direction output from the panning direction detection means 108. The subject blur compensation amount adjustment means 109 adjusts the subject blur compensation amount so that the subject blur compensation amount in a second direction different from the first direction is limited to be less than the subject blur compensation amount in the first direction. Here, "limitation" means that the subject blur compensation amount after adjustment by the subject blur compensation amount adjustment means 109 is smaller than the subject blur compensation amount before adjustment. There is no particular restriction as to the specific method for limiting the amount, and methods such as narrowing the frequency band included in the subject blur compensation amount, reducing the upper limit of the subject blur compensation amount, or reducing the gain can be used.

[0025] In scenes where birds or airplanes are flying and the horizon is visible in the background, correcting vertical subject blur can cause the horizon to vibrate vertically, creating an unnatural appearance.

[0026] This kind of unnaturalness is particularly noticeable in scenes where the background includes a boundary line, such as a horizon, and the boundary line vibrates in a direction other than the direction in which the imaging device is moved to track the subject. On the other hand, even in scenes where birds or airplanes are flying but the background is a completely blue sky, there is no boundary line, and correcting vertical subject blur does not create an unnatural feeling. In fact, the subject remains stable within the frame, resulting in a more pleasing image. For this reason, in this embodiment, the amount of subject blur correction is adjusted based on background information.

[0027] The compensation lens and / or image sensor in optical system 101, which serves as optical blur compensation means, is moved in a direction perpendicular to the optical axis of optical system 101 based on the amount of subject blur compensation adjusted by subject blur compensation amount adjustment means 109, adjusting the relative position between the image sensor and the subject image. This allows for subject blur compensation that brings the position of the subject in the image captured by imaging device 100 closer to a target position. Instead of performing optical blur compensation by moving the compensation lens or image sensor, electronic blur compensation may be performed, or subject blur compensation may be performed by combining optical blur compensation and electronic blur compensation. Subject blur compensation amount adjustment means 109 can control subject blur compensation by outputting the adjusted amount of subject blur compensation to at least one of the optical blur compensation means and the electronic blur compensation means.

[0028] (Subject shake correction processing in imaging device 100) Next, the subject blur correction control process in the imaging device 100 configured as described above will be described with reference to the flowchart shown in Fig. 2. Fig. 2 is a flowchart showing an example of the overall processing of the subject blur correction function, and unless otherwise explicitly stated, the subject tracking function described below is performed by the image blur correction control device 110. The flowchart in Fig. 2 starts when an instruction to start the subject blur correction function is given by operating the subject blur correction function start button, menu, etc.

[0029] When the subject blur correction function is started, first, in step S201, the subject detection means 104 performs detection processing to detect a specific subject in the input image from the development processing means 103, and outputs information about the detected subject as subject information. The specific subject is a subject that may move itself, typically a human face. To detect a face, a publicly known face detection method may be used. Publicly known face detection techniques include methods that utilize facial knowledge (skin color information, features such as eyes, nose, and mouth) and methods that use learning algorithms, such as neural networks, to construct a classifier for face detection. These methods may be combined to improve detection accuracy. In addition to human faces, human bodies, animals, vehicles, and the like may be detected as specific subjects, and multiple types of subjects may be detected as specific subjects. The output subject information includes at least one of the position, size, type, and number of detected subjects.

[0030] In step S202, the object tracking means 105 acquires the position in the input image of one of the objects detected by the object detection means 104 in step S201, and tracks the object. The object can be tracked using known template matching, using a partial image including the object to be subjected to object blur correction as a template. If multiple objects are detected in step S201, the object to be subjected to object blur correction may be selected automatically by the object tracking means 105 or may be selected based on a user operation. In the case of automatic selection, it is sufficient to select the object that is considered to be the most important, and for example, priority may be given to objects that are large in size, located closer to the center of the image, and have a higher reliability for subject detection.

[0031] From the second frame onwards, the subject tracked in the previous frame is tracked. If the subject cannot be tracked, a target for subject blur correction can be selected from the subjects detected in step S201 and tracked again. Furthermore, the detection process of step S201 to detect a specific subject may be performed for each frame.

[0032] In step S203, subject blur correction amount calculation means 106 calculates a provisional subject blur correction amount based on the position in the input image of the subject tracked by subject tracking means 105 in step S202 and a target position set within the image. The target position may be a preset position within the image, may be set automatically by blur correction control device 110, or may be set to any position by the user.

[0033] A method for calculating the object blur correction amount will be described using FIG. 3. FIG. 3 shows an example in which a target position 301 is set at the center of the image, and a person's face region 302 is the specific object detected in step S202. In this embodiment, the object blur correction amount calculation means 106 calculates the difference between the coordinates of the center of gravity 303 of the face region 302 and the coordinates of the target position 301 as the object blur correction amount. Note that the detected object position generally contains errors due to the influence of image noise and the accuracy of object detection. For this reason, it is desirable to apply a low-pass filter (hereinafter, LPF) to the detected position of the specific object (here, the coordinates of the center of gravity 303) or the object blur correction amount, and calculate the low-frequency components as a provisional object blur correction amount.

[0034] In step S204, the background information detection means 107 performs background detection processing to detect the background of the input image from the development processing means 103, and outputs information on the detected background as background information. Here, the background is a subject that does not move, and the background information includes at least one of a boundary line included in the background, background contrast, and background type.

[0035] The background boundary lines can be detected using a trained model or a known edge detection method such as the Canny algorithm, etc. In this embodiment, horizontal and vertical boundary lines are detected.

[0036] Background contrast can be calculated from the maximum and minimum brightness values ​​of the background. For example, it can be detected by taking brightness profiles in both the horizontal and vertical directions along any line in the image where no specific subject is present. If the difference between the maximum and minimum brightness values ​​in the brightness profile is equal to or greater than a predetermined threshold, it can be determined that a boundary line exists; if it is less than the predetermined threshold, it can be determined that no boundary line exists. Note that vertical boundaries can be detected from the horizontal brightness profile, and horizontal boundaries can be detected from the vertical brightness profile.

[0037] The type of background is determined by using a trained model to distinguish between types such as sky, ground, and water surface for each region of the image. If the image contains multiple types of background, it is determined that there is a high possibility of a boundary line being present. For example, if the image contains both sky and ground, it is determined that there is a horizon, and if the image contains both sky and water, it is determined that there is a horizon. The orientation of the horizon and other boundaries can also be determined from the relative positions of each region.

[0038] In step S205, the panning direction detection means 108 detects the panning direction based on the amount of shake of the imaging device 100. The amount of shake applied to the imaging device 100 can be determined using the detection results of an inertial sensor, such as a gyro sensor or angular velocity sensor, disposed in the imaging device. In this embodiment, the shake angle obtained by integrating the angular velocity detected by the gyro sensor is used as the amount of shake, and the shake angle in the horizontal direction and the shake angle in the vertical direction are compared with thresholds. As a result of the comparison, the direction in which the acquired shake angle is greater than the threshold is detected as the panning direction. If the shake angle in the horizontal direction is equal to or greater than the threshold, the panning direction is detected as the horizontal direction. If the shake angle in the vertical direction is equal to or greater than the threshold, the panning direction is detected as the vertical direction. If both the shake angles in the horizontal and vertical directions are equal to or greater than the threshold, the panning direction is detected as the horizontal and vertical directions. If both the shake angles in the horizontal and vertical directions are less than the threshold, the panning direction is detected as none. The horizontal and vertical thresholds may be the same or different. The threshold value can be set appropriately, but may also be set based on the angle of view. By setting the horizontal angle of view as the threshold value in the horizontal direction, it is possible to determine that panning has occurred when panning has occurred in the horizontal direction that is equal to or greater than the angle of view (when panning has occurred to the extent that the background in the horizontal direction changes before and after panning).

[0039] In step S206, the subject blur compensation amount adjustment means 109 adjusts the subject blur compensation amount calculated in step S203 based on the background information detected in step S204 and the panning direction detected in step S205.

[0040] An example of the adjustment method will be described using FIG. 4. FIG. 4(a) shows a case where a specific subject (face area) 401 is present and there is no boundary in the background. In this case, there is no boundary in the background that could cause an unnatural appearance, so the object blur correction amount calculated in step S203 can be used as is, regardless of the panning direction detected in step S205. As a result, the center of gravity of the specific subject 401 is controlled to approach the target position in both the horizontal and vertical directions. FIG. 4(b) shows a case where a specific subject 401 is present and there is a horizontal boundary 402 in the background. In this case, if the boundary 402 vibrates vertically when subject blur is corrected, an unnatural appearance will occur. Therefore, the horizontal object blur correction amount is not suppressed, but the vertical object blur correction amount is suppressed, thereby limiting the vertical object blur correction amount more than the horizontal object blur correction amount. However, if the panning direction is vertical, it is assumed that the photographer is intentionally moving the imaging device to correct vertical object blur, so the vertical object blur correction amount is not suppressed either. That is, when a horizontal boundary line exists in the background information and the panning direction is horizontal, the amount of subject blur compensation is adjusted so that the amount of subject blur compensation in the vertical direction is more limited than the amount of subject blur compensation in the horizontal direction.

[0041] FIG. 4(c) shows a case where a specific subject 401 is present and a vertical boundary line 403 is present in the background. In this case, if the boundary line 403 vibrates horizontally when subject blur is corrected, it will create an unnatural feeling. For this reason, the amount of subject blur correction in the horizontal direction is suppressed without suppressing the amount of subject blur correction in the vertical direction, thereby limiting the amount of subject blur correction in the horizontal direction more than the amount of subject blur correction in the vertical direction. However, if the panning direction is horizontal, it is assumed that the photographer is intentionally moving the imaging device to correct subject blur in the horizontal direction, so the amount of subject blur correction in the horizontal direction is not suppressed either. In other words, if a vertical boundary line is present in the background information and the panning direction is vertical, the amount of subject blur correction in the horizontal direction is adjusted so that the amount of subject blur correction in the vertical direction is more limited than the amount of subject blur correction in the vertical direction.

[0042] Figure 4(d) shows a table summarizing examples of the relationship between background information, panning direction, and subject blur correction. In this way, when the direction of a boundary line in the background matches the panning direction, that direction is set as the first direction for subject blur correction, and the direction perpendicular to that direction is set as the second direction for limiting subject blur correction. When no boundary line exists, regardless of the panning direction, both the horizontal and vertical directions are set as the first direction for subject blur correction, and neither direction is set as the second direction for limiting subject blur correction. In this case, the amount of subject blur correction for each direction is not adjusted based on the background information.

[0043] Generally, subject blur correction is more likely to cause discomfort in the vertical direction (gravity direction) than in the horizontal direction. Therefore, in Figure 4(d), if there is a horizontal boundary, even if there is no panning direction (no panning in either direction), the horizontal direction is set as the first direction, and the vertical direction is set as the second direction, limiting vertical subject blur correction. On the other hand, if there is a vertical boundary, both the horizontal and vertical directions are set as the first direction when there is no panning direction.

[0044] As mentioned above, the higher the contrast of the background, the more unnatural the boundary line movement will appear, so the higher the contrast, the more strongly the amount of subject blur correction in the second direction may be limited.

[0045] Furthermore, as mentioned above, when multiple types of backgrounds are included, it is highly likely that there will be boundaries, and the vibration of the boundaries will likely create a sense of incongruity. Therefore, the more types of backgrounds are included in the image, the more strongly the amount of subject blur correction in the second direction may be limited.

[0046] The method for limiting the amount of object blur compensation is not particularly limited. As long as the limit on the amount of object blur compensation in the second direction is stronger than the limit on the amount of object blur compensation in the first direction, the amount of object blur compensation in the first direction may also be limited. For example, if the amount of object blur compensation is large and cannot be fully compensated, the amount of object blur compensation in the first direction may also be limited. Specific examples of limiting the amount of object blur compensation include multiplying the gain so that the amount of object blur compensation is smaller, or lowering the cutoff frequency of the LPF for the amount of object blur compensation to narrow the frequency band included in the amount of object blur compensation. Another method is to lower the upper limit of the amount of object blur compensation between frames.

[0047] As described above, the first and second directions are set based on the background information of the shooting scene and the panning direction, and the amount of subject blur compensation is adjusted so that the amount of subject blur compensation in the second direction is more limited than the amount of subject blur compensation in the first direction. The subject blur compensation amount adjustment means 109 then outputs the adjusted amount of subject blur compensation to at least one of the optical blur compensation means and the electronic blur compensation means. The subject blur compensation amount adjustment means 109 then controls blur compensation so that the position of the subject (hereinafter sometimes referred to as the target subject) in the image that is the target of subject blur compensation approaches the target position. It is sufficient that the target subject approaches the target position in at least either the horizontal or vertical direction. In step S207, blur compensation is performed based on the amount of subject blur compensation adjusted by the subject blur compensation amount adjustment means 109 in step S206. As described above, the blur compensation means can be the optical blur compensation means, i.e., the compensation lens in the optical system 101, the image sensor 102, or an electronic blur compensation means (not shown).

[0048] In step S208, the imaging device 100 determines whether processing has been completed up to the final frame, and if processing has not been completed up to the final frame, the imaging device 100 proceeds to step S201. The next step may be step S202, and if the subject being tracked in the previous frame is lost, the imaging device 100 may return to step S201 and start over from subject detection.

[0049] By performing the above processing, the amount of subject blur correction in a specific direction is adjusted based on the background information of the shooting scene and the tracking direction, making it possible to correct subject blur without impairing the quality of the image.

[0050] It should be noted that steps S201 to S208 can all be processed in real time during shooting, but they do not necessarily have to be performed during shooting; they may be performed during image playback using an electronic image stabilization means. For example, the image during shooting and the detection result of the amount of shake of the imaging device (or the detection result of the panning direction) are recorded in a recording means (not shown), and steps S201 to S208 are performed on the recorded image, thereby enabling subject shake stabilization during image playback. Furthermore, subject information obtained by subject detection means 104 may be recorded in addition to the image during shooting, so that subject detection does not have to be performed during image playback.

[0051] [Embodiment 2] In the first embodiment, only subject shake is treated as the target of shake correction, but in this embodiment, shake correction is performed taking into account image capture device shake (so-called camera shake), which is the shake of the image capture device itself.

[0052] Fig. 5 is a block diagram showing an example of the configuration of an image capture device 500 according to this embodiment. In Fig. 5, parts that are common to the components shown in Fig. 1 are given the same reference numerals as in Fig. 1, and descriptions thereof will be omitted. In addition to the configuration shown in Fig. 1, the image capture device 500 according to this embodiment has a camera shake detection unit 501, a camera shake correction amount calculation unit 502, and a camera shake correction amount adjustment unit 503. In this embodiment, only parts that perform processing different from that in embodiment 1 will be described.

[0053] Camera shake detection means 501 detects shake applied to image capture device 500 and outputs the shake detection result to image shake control device 510. Specifically, an inertial sensor such as an angular velocity sensor or an acceleration sensor can be used. When the detection result of the inertial sensor is used for the panning determination described in the first embodiment, the detection result of camera shake detection means 501 can be used.

[0054] The camera shake correction amount calculation means 502 calculates the amount of camera shake correction for correcting camera shake based on the shake detection result detected by the camera shake detection means 501. A known method can be used to calculate the amount of camera shake correction.

[0055] The blur compensation amount adjustment means 503 calculates a final blur compensation amount by combining the object blur compensation amount output from the object blur compensation amount calculation means 106 and the camera shake compensation amount output from the camera shake compensation amount calculation means 502. At this time, by adjusting the combination ratio based on the background information detected by the background information detection means 107 and the panning direction detected by the panning direction detection means 108, it is possible to also fulfill the role of the object blur compensation amount adjustment means of the first embodiment.

[0056] Fig. 6 is a flowchart showing the image stabilization process in the second embodiment. In Fig. 6, steps common to those shown in Fig. 2 are given the same reference numerals as in Fig. 2, and description thereof will be omitted.

[0057] Steps S201 to S205 are the same processes as in embodiment 1. After the panning direction detection process in step S205 is completed, the process proceeds to step S601.

[0058] In step S601, the camera shake detection means 501 detects a shake applied to the imaging device 500. Note that step S601 can be executed at any timing as long as it is before step S602, and it may be executed before or after steps S201 to S205.

[0059] In step S602, the image stabilization amount calculation means 502 calculates the image stabilization amount for correcting the shake detected in step S601. For example, if the image stabilization amount calculation means 502 is a gyro sensor, the image stabilization amount is calculated by integrating the angular velocity signal detected by the gyro sensor and inverting the sign. The detection result of the gyro sensor may be subjected to appropriate filtering or the like.

[0060] In step S603, the shake compensation amount adjustment means 503 combines the object blur compensation amount obtained in step S203 with the camera shake compensation amount obtained in step S602 to calculate a final shake compensation amount. At this time, the object blur compensation amount is adjusted by adjusting the combination ratio (weight) of the object blur compensation amount and the camera shake compensation amount based on the background information detected in step S204 and the panning direction detected in step S205. In other words, the combination ratio of the object blur compensation amount in the first direction in which object blur compensation is performed is made smaller than the combination ratio of the object blur compensation amount in the second direction in which object blur compensation is suppressed. This limits the object blur compensation amount in the second direction, making it possible to reduce the sense of discomfort caused by performing object blur compensation. Note that it is preferable to dynamically adjust the combination ratio.

[0061] One method for combining signals is to separate them by frequency. For example, low-pass filter (LPF) processing is applied to the object blur compensation amount to extract low-frequency components. Meanwhile, high-pass filter (HPF) processing is applied to the camera shake compensation amount to extract high-frequency components. The low-frequency components of the object blur compensation amount obtained in this way and the high-frequency components of the camera shake compensation amount are combined by weighting and adding them based on background information and the tracking direction. The combining ratio can be controlled by roughly matching the cutoff frequencies of the LPF and HPF for the horizontal and vertical directions, and then changing the cutoff frequencies based on background information and the panning direction. Specifically, based on the relationship between background information and panning direction shown in Figure 4(d), the cutoff frequencies of the LPF and HPF are lowered in the second direction, in which the object blur compensation amount is suppressed, than in the first direction, in which the object blur compensation amount is not suppressed. This reduces the weight of the low-frequency components of the object blur compensation amount and increases the weight of the high-frequency components of the camera shake compensation amount, thereby strengthening the camera shake compensation effect by the amount of the object blur compensation amount being limited.

[0062] Furthermore, instead of dividing by frequency and combining, gain may be used for combining. In this case, the gain of the object blur compensation amount in the second direction may be set smaller than the gain of the object blur compensation amount in the first direction. By setting the gain of the image stabilization amount to (1 - gain of the object blur compensation amount), the effect of image stabilization can be strengthened by the amount of the object blur compensation amount being limited.

[0063] In step S604, shake correction is performed based on the final shake correction amount (referred to as the adjusted shake correction amount) obtained in step S603. As described above, shake correction can be performed using an optical shake correction means such as a correction lens in optical system 101 or image sensor 102, or an electronic shake correction means (not shown). As in the first embodiment, instead of performing processing in real time, shake correction can be performed using an electronic shake correction means during image playback.

[0064] Through the above processing, the amount of subject blur correction and the amount of camera shake correction are adjusted and combined based on the background information of the shooting scene and the tracking direction, making it possible to perform subject blur correction and camera shake correction without compromising the quality of the image.

[0065] [Variations] In the first and second embodiments described above, the panning direction detected by the panning direction detection unit 108 based on the amount of shake applied to the imaging device 100 is estimated as the direction in which the user wants to correct subject blur, and this direction is provisionally set as the direction in which subject blur correction will be performed. However, a configuration in which the direction in which subject blur correction will be performed can also be configured in which the user can set the direction in which subject blur correction will be performed based on input. In this case, the direction input by the user is provisionally set as the direction in which subject blur correction will be performed. If this direction matches the direction of a boundary line in the background, this direction is set as the first direction. If this direction does not match the direction of the boundary line, the provisionally set direction and the direction of the boundary line are set as the first direction. For example, if the user inputs that subject blur correction in the horizontal direction is to be performed, then if a horizontal boundary line exists, the horizontal direction is set as the first direction, and the vertical direction is set as the second direction. On the other hand, if a vertical boundary line exists, the horizontal and vertical directions are set as the first directions, and the second direction is not set. If there are no horizontal or vertical boundaries, either direction may be set as the first direction, as shown in Figure 4(d), or only the horizontal direction may be set as the first direction, giving priority to user input.

[0066] Furthermore, in the second embodiment, the blur of the imaging device itself has been described as camera shake, but the source of the blur is not particularly limited, and blur other than that transmitted from the user's hand may also be subject to correction. Furthermore, the present invention is not limited to imaging devices held by the user's hand. For example, the present invention can be applied to devices fixed to a vehicle that generates vibrations or a building that may also generate vibrations. The amount of blur applied to the imaging device due to these vibrations, camera shake, etc. is collectively referred to as the imaging device shake amount, the amount of correction for imaging device shake is referred to as the imaging device shake correction amount, and the means for calculating the imaging device shake correction amount is referred to as the imaging device shake correction amount calculation means.

[0067] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0068] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0069] 101 Optical system 102 Imaging means 103 Development processing means 104 Subject detection means 105 Subject tracking means 106 Subject blur correction amount calculation means 107 Background Information Detection Method 108 Panning direction detection means 109 Subject blur correction amount adjustment means 110 Image blur correction control device 501 Camera shake detection means 502 Image stabilization amount calculation means 503 Shake compensation amount adjustment means 510 Image blur correction control device

Claims

1. a subject detection means for detecting a subject from an input image and outputting information about the detected subject; a subject blur correction amount calculation means for calculating a subject blur correction amount based on a position of the detected subject in the input image and a target position set within the image; background information detection means for detecting information included in a background from the input image as background information and outputting the background information; a first setting means for setting a first direction in which subject blur correction is performed among a plurality of directions; a subject blur compensation amount adjustment means for adjusting the subject blur compensation amount in a second direction different from the first direction so that the subject blur compensation amount in the second direction is limited to be less than the subject blur compensation amount in the first direction; a blur correction control means for controlling blur correction so as to bring the position of the subject closer to the target position based on the amount of subject blur correction adjusted by the subject blur correction amount adjustment means; having The image blur correction control device according to claim 1, wherein the first setting means sets the first direction based on the background information.

2. 2. The image blur correction control device according to claim 1, wherein the background information includes at least one of a boundary line included in the background, a contrast of the background, and a type of the background.

3. a second setting unit configured to provisionally set a direction in which the subject blur correction is to be performed among the plurality of directions; 2. The image blur correction control device according to claim 1, wherein the first setting means sets the first direction based on a direction in which the subject blur correction is performed that is provisionally set by the second setting means and on the background information.

4. The second setting means is tentatively setting a direction in which the subject blur correction is to be performed based on the amount of blur in each of the plurality of directions of an imaging device that captures the input image; The first setting means is 4. The image blur correction control device according to claim 3, wherein the first direction is set based on a direction in which the subject blur correction is performed that is provisionally set by the second setting means and on the background information.

5. The second setting means is acquiring an amount of blur in each of the plurality of directions of the imaging device; Among the plurality of directions, a direction in which the amount of blur is greater than a threshold is provisionally set as a direction in which the object blur correction is to be performed; The first setting means is 5. The image blur correction control device according to claim 4, wherein the first direction is set based on a direction in which the amount of blur is greater than a threshold value and on the background information.

6. 4. The image blur correction control device according to claim 3, wherein the second setting means provisionally sets the direction in which the subject blur correction is to be performed based on a user operation.

7. 7. The image blur correction control device according to claim 3, wherein, when a direction in which the subject blur correction is performed provisionally set by the second setting means coincides with a direction of a boundary line included in the background information, the direction is set as a first direction.

8. When a horizontal boundary line exists in the background information and the direction in which the subject blur correction is performed that is provisionally set by the second setting means is a horizontal direction, the first setting means sets the horizontal direction as the first direction; The subject blur correction amount adjustment means 8. The image blur correction control device according to claim 7, wherein the amount of subject blur correction is adjusted so that the amount of subject blur correction in the vertical direction is more limited than the amount of subject blur correction in the horizontal direction.

9. When a vertical boundary line exists in the background information and the direction in which the subject blur correction is performed that is provisionally set by the second setting means is a vertical direction, the first setting means sets a vertical direction as a first direction; The subject blur correction amount adjustment means 8. The image blur correction control device according to claim 7, wherein the amount of subject blur correction is adjusted so that the amount of subject blur correction in the horizontal direction is more limited than the amount of subject blur correction in the vertical direction.

10. 7. The image blur correction control device according to claim 3, wherein, when a direction in which the subject blur correction is performed provisionally set by the second setting means does not coincide with a direction of a boundary line included in the background information, the provisionally set direction and the direction of the boundary line are set as the first direction.

11. When the direction in which the subject blur correction process is performed that is provisionally set by the second setting means is a horizontal direction and a vertical boundary line exists in the background information, 11. The image blur correction control device according to claim 10, wherein the first setting means sets the horizontal direction and the vertical direction as the first directions.

12. When the direction in which the subject blur correction is performed that is provisionally set by the second setting means is a vertical direction and a horizontal boundary line exists in the background information, 11. The image blur correction control device according to claim 10, wherein the first setting means sets the horizontal direction and the vertical direction as the first directions.

13. When one direction is provisionally set as the direction in which the subject blur correction is to be performed by the second setting means and a boundary line is not detected by the background information detection means, 11. The image blur correction control device according to claim 10, wherein the first setting means sets a plurality of directions as the first direction.

14. When a plurality of directions are set as the first direction by the first setting means, 7. The image blur correction control device according to claim 1, wherein the object blur correction amount adjustment means does not adjust the object blur correction amount, which differs for each direction.

15. 3. The image blur correction control device according to claim 2, wherein the subject blur correction amount adjustment means strengthens the restriction as the contrast of the background included in the background information becomes higher.

16. 3. The image blur correction control device according to claim 2, wherein the subject blur correction amount adjustment means strengthens the restriction as the number of types of the background included in the background information increases.

17. an image capture device shake correction amount calculation unit that acquires an image capture device shake amount, which is an amount of shake applied to an image capture device that captures the input image, and calculates an image capture device shake correction amount based on the image capture device shake amount; The subject blur correction amount adjustment means obtaining a shake compensation amount used by the shake compensation control means for control by combining the object shake compensation amount output from the object shake compensation amount calculation means and the imaging device shake compensation amount output from the imaging device shake compensation amount calculation means; The subject blur correction amount is adjusted by changing the composition ratio; 7. The image blur correction control device according to claim 1, wherein the image blur correction control means performs the image blur correction based on the image blur correction amount obtained by combining the object image blur correction amount and the image pickup device image blur correction amount.

18. The subject blur correction amount adjustment means 18. The image blur correction control device according to claim 17, wherein a combination ratio used in the combination of the object blur compensation amount in the second direction is set smaller than the combination ratio of the object blur compensation amount in the first direction, so that the object blur compensation amount in the second direction is limited to be smaller than the object blur compensation amount in the first direction.

19. The subject blur correction amount adjustment means 20. The image pickup apparatus according to claim 18, wherein the combination ratio of the image pickup apparatus blur compensation amount is increased by an amount corresponding to a reduction in the combination ratio of the object blur compensation amount.

20. The subject blur correction amount adjustment means 7. The image blur correction control device according to claim 1, wherein the amount of subject blur correction is adjusted by making a frequency band included in the amount of subject blur correction in the second direction narrower than a frequency band included in the amount of subject blur correction in the first direction.

21. The subject blur correction amount adjustment means 7. The image blur correction control device according to claim 1, wherein the amount of subject blur correction is adjusted by making an upper limit value of the amount of subject blur correction in the second direction smaller than an upper limit value of the amount of subject blur correction in the first direction.

22. The subject blur correction amount adjustment means 7. The image blur correction control device according to claim 1, wherein the amount of subject blur correction is adjusted by making an upper limit value of the amount of subject blur correction in the second direction smaller than an upper limit value of the amount of subject blur correction in the first direction.

23. The subject blur correction amount adjustment means 7. The image blur correction control device according to claim 1, wherein the amount of subject blur correction is adjusted by making a gain by which the amount of subject blur correction in the second direction is multiplied smaller than a gain by which the amount of subject blur correction in the first direction is multiplied.

24. The object blur correction amount calculation means 7. The image blur correction control device according to claim 1, wherein when the subject detection means detects a plurality of subjects, the amount of subject blur correction is calculated based on a position in the input image of a target subject for subject blur correction and the target position.

25. 2. An image pickup apparatus comprising: the image blur correction control device according to claim 1; and an image pickup means.

26. a subject detection step of detecting a subject from an input image and outputting information about the detected subject; a subject blur correction amount calculation step of calculating a subject blur correction amount based on a position of the detected subject in the input image and a target position set within the image; a background information detection step of detecting information included in a background from the input image as background information and outputting the background information; a first setting step of setting a first direction in which subject blur correction is performed among a plurality of directions; a subject blur compensation amount adjustment step of adjusting the subject blur compensation amount in a second direction different from the first direction so that the subject blur compensation amount in the second direction is more limited than the subject blur compensation amount in the first direction; a blur correction control step of controlling blur correction so as to bring the position of the subject closer to the target position based on the subject blur correction amount adjusted in the subject blur correction amount adjustment step; having 4. An image blur correction control device, comprising: a first setting step for setting the first direction based on the background information.

Images