Image blur correction apparatus, control method therefor, imaging apparatus, program, and storage medium

Abstract

To provide an image blur correction apparatus capable of performing object blur correction without impairing quality of an image even when parts of an object to be detected are switched.SOLUTION: Included are: object detection means configured to detect an object for each part from an input image and output a position of the part of the object detected; selection means configured to select a part of the object from the part of one or more objects detected by the object detection means on the basis of a predetermined priority; and object blur correction amount calculation means configured to calculate an object blur correction amount for correcting object blur on the basis of a difference between a position of the part of the object selected by the selection means and a predetermined target position in the image. The object blur correction amount calculation means, when the part of the object selected by the selection means is switched from a first part to a second part, calculates the object blur correction amount on the basis of a difference between the position of the selected part of the object and the target position, a position of the second part in the image, and a position of the second part in the image acquired before the switching.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image stabilization technique for an imaging device such as a digital camera or a digital video camera. [Background technology]

[0002] In imaging devices such as digital cameras, correction is performed to correct "image blur" caused by hand shake by a user holding the camera body, as well as "subject blur" caused by changes in the position of a subject such as a person.

[0003] "Image blur" caused by camera shake can be detected by using an angular velocity sensor attached to the imaging device or by acquiring the motion vector of a stationary object (background) in the captured image, while "subject blur" can be detected by detecting the subject and measuring the position of the subject.

[0004] Subject detection also makes it possible to detect each part of the subject (for example, head, face, eyes, nose, mouth, torso, arms, legs, etc.). Depending on the shooting conditions, it may be possible to detect some parts of the subject but not others. For example, if the subject is far away, the face may be too small to be detected, but the torso may be large enough to be detected. Conversely, there may be cases where the face can be detected from the skin color, but the torso cannot be detected because the color of the clothes blends in with the background. In this way, the parts of the subject that can be detected may change depending on the shooting conditions.

[0005] Therefore, for example, Patent Document 1 discloses a technique in which, when a face cannot be detected, the detection is switched to detecting a part other than the face, and the subject is tracked. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 5771461 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, the position of the subject is only tracked, and no consideration is given to subject blur correction that brings the position of the subject closer to a predetermined target position within the image.

[0008] In subject blur correction, the position of a subject's body part is controlled to remain stable near a predetermined position within the image. When the position of the detected subject's body part changes, the difference in the position of the body part before and after the change is reflected in the control, and the position of the subject may change within the screen even though the subject is not moving.

[0009] For example, when controlling the camera to keep a person's face in the center of an image, if the face becomes undetectable midway and the camera switches to detecting the torso, the position of the face and torso will change unnaturally, resulting in a loss of image quality.

[0010] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image stabilization device that can perform subject blur correction without impairing the quality of the image, even when the part of the subject being detected changes. [Means for solving the problem]

[0011] An image stabilization device according to the present invention comprises: subject detection means for detecting each part of a subject from an input image and outputting the positions of the detected subject parts; selection means for selecting a subject part from one or more subject parts detected by the subject detection means based on a predetermined priority; and subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the subject part selected by the selection means and a predetermined target position in the image, wherein, when the subject part selected by the selection means is switched from a first part to a second part, the subject blur correction amount calculation means calculates the subject blur correction amount based on the difference between the position of the selected subject part and the target position, the position of the second part in the image, and the position of the second part in an image acquired before the switch. [Effects of the Invention]

[0012] According to the present invention, even when the part of the subject to be detected changes, it is possible to perform subject blur correction without impairing the quality of the image. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing the configuration of an imaging apparatus according to a first embodiment. [Figure 2] 4 is a flowchart illustrating processing performed in the first embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a subject detection result. [Figure 4] 5A and 5B are diagrams for explaining a method for calculating a subject blur correction amount. [Figure 5] 10A and 10B are diagrams illustrating an example in which a part selected in subject detection is switched. [Figure 6] 6A to 6C are diagrams illustrating suppression processing of subject blur correction. [Figure 7] FIG. 10 is a block diagram showing the configuration of an imaging apparatus according to a second embodiment. [Figure 8] 10 is a flowchart illustrating processing performed in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0015] (First embodiment) <Block diagram explanation> FIG. 1 is a diagram showing the configuration of an image pickup apparatus 100 which is a first embodiment of an image stabilization apparatus according to the present invention.

[0016] 1, the imaging device 100 includes an imaging optical system 101, an imaging element 102, a development processing unit 103, a subject detection unit 104, a subject part selection unit 105, a subject blur correction amount calculation unit 106, a control unit 110, a ROM 112, and a RAM 114.

[0017] In this embodiment, the photographing optical system 101 includes a correction lens that corrects image blur by moving the subject image on the imaging surface by displacing the lens in a direction different from the optical axis of the photographing optical system 101. The image sensor 102 is also configured to be able to correct image blur by displacing the lens in a direction perpendicular to the optical axis.

[0018] Furthermore, although FIG. 1 shows the imaging device 100 as an imaging device in which the lens and the camera body are integrated, the present invention is also applicable to imaging devices with interchangeable lenses.

[0019] 1, a subject image formed by an imaging optical system 101 is converted into an analog image signal by an image sensor 102. The analog image signal is then A / D converted by a development processing unit 103, after which development processing such as white balance correction, color (luminance and color difference signal) conversion, and gamma correction is performed, and the signal is output as image data.

[0020] The subject detection unit 104 detects specific subjects by body part from the input image data from the development processing unit 103, and outputs the positions of the detected subject body parts. The body parts include, for example, the head, face, eyes, nose, mouth, torso, arms, and legs.

[0021] The subject region selection section 105 selects a subject region from one or more subject regions detected by the subject detection section 104 based on a predetermined priority.

[0022] The object blur correction amount calculation unit 106 calculates the object blur correction amount based on the difference between the position of the object part selected by the object part selection unit 105 and a predetermined target position within the image.

[0023] At least one of the correction lens in the photographing optical system 101, which is an optical image blur correction means, and the image sensor 102 is displaced based on the subject blur correction amount calculated by the subject blur correction amount calculation unit 106. In this way, optical image blur correction can be performed so that the position of a desired part of the subject approaches the target position.

[0024] A control unit 110 including a microcomputer or the like controls the entire imaging device 100 by loading a program stored in a ROM 112 into a RAM 114 and executing the program.

[0025] The present invention is not limited to optical image stabilization, and electronic image stabilization may be performed based on the amount of subject blur correction. Electronic image stabilization is an operation for correcting image blur by changing the position at which a smaller image area for display is cropped from the entire image area based on the amount of subject blur correction.

[0026] <Explanation of the flowchart> Next, the image stabilization operation in the imaging device 100 configured as described above will be described with reference to the flowchart shown in Fig. 2. The operation of the flowchart in Fig. 2 is realized by the control unit 110 loading a program stored in the ROM 112 into the RAM 114 and executing the program.

[0027] 2, in step S201, the control unit 110 uses the subject detection unit 104 to detect specific subjects by body part from the input image from the development processing unit 103. A typical subject body part is a person's face.

[0028] In this case, a known face detection method may be used as the detection method. Known face detection techniques include a method that uses knowledge about faces (skin color information, eyes, nose, mouth, and other features) and a method that uses a learning algorithm, such as a neural network, to configure a classifier for face detection. To improve detection accuracy, it is common to combine these methods to perform face detection. Similar methods can also be used to detect objects other than human faces.

[0029] 3 is a diagram showing an example of the subject detection result when the subject is a person. Rectangles 301 to 310 indicate detected body parts. Specifically, rectangle 301 represents the face or head, rectangle 302 represents the right eye, rectangle 303 represents the left eye, rectangle 304 represents the nose, rectangle 305 represents the mouth, rectangle 306 represents the right arm, rectangle 307 represents the left arm, rectangle 308 represents the torso, rectangle 309 represents the right leg, and rectangle 310 represents the left leg.

[0030] In the following, the representative point included in each rectangle is treated as the position of the subject's part. For example, the barycentric coordinates, which are the coordinates of the center of gravity of each rectangle, are used as the representative point. Here, the detected parts are represented by rectangles for convenience, but they are not limited to rectangles and may be represented by any shape.

[0031] The type of subject is not limited to people as in the example of Figure 3, but may be animals such as dogs, cats, birds, horses, etc. The method can also be applied to general objects other than animals; for example, in the case of a vehicle, the only difference is that the parts to be detected are tires, doors, hood, trunk, etc. The level of detail of the detected parts is also not limited to the example of Figure 3, but may be finer or coarser.

[0032] When multiple subjects exist in an image, the most important subject is selected and the part of that subject is output. One possible selection method is to prioritize subjects that are larger in size, closer to the center of the image, and more reliable in detection. Alternatively, the user may be allowed to select from multiple candidates. In this way, the position of the subject's part can be obtained.

[0033] In step S202, the control unit 110 uses the subject part selection unit 105 to select a subject part from the one or more subject parts detected in step S201 based on a predetermined priority.

[0034] The predetermined priority is, for example, when subject detection results are used for autofocus (AF), generally giving higher priority to small organs such as eyes and pupils than to the face.

[0035] On the other hand, when using subject detection results for subject blur correction in this embodiment, it is necessary to set priorities taking into consideration aspects such as detection stability, shape stability, and position stability.

[0036] The stability of detection indicates how stably the body part can be detected. For example, the eyes are smaller and harder to detect than the head and torso, so the stability of detection is low.

[0037] Shape stability indicates how stable the shape of a part is. For example, human arms and legs, and bird wings change shape by expanding and contracting, so they have low shape stability. Also, in the case of eyes, although the changes are not as great as those of arms and legs, their shape changes depending on how open they are. It can be said that the shape of the head is more stable than that of arms, legs, and eyes.

[0038] Positional stability indicates how stable the position of a subject's parts is when the subject changes direction. For example, in the case of the eyes, the position of the subject's parts changes when the subject's direction changes from the front, side, or back, so the positional stability is low. On the other hand, in the case of the head, the position of the subject's parts does not change significantly even when the subject's direction changes from the front, side, or back, so it can be said that the positional stability is high.

[0039] In order to perform subject blur correction stably, it is desirable that these factors be highly stable. Because the priority perspective differs from that in the case of AF, it is desirable to be able to select the part of the subject to be used for subject blur correction independently of the part of the subject to be used for AF.

[0040] Specifically, it is preferable to select the face or head as the subject's part with priority over facial organs such as the eyes, pupils, nose, mouth, etc. Furthermore, from the viewpoint of video, generally, attention is focused on the face or head rather than the torso, so it is preferable to correct blurring of the face or head as the subject's part with priority over the torso.

[0041] In step S203, control unit 110 uses object blur correction amount calculation unit 106 to calculate the object blur correction amount based on the difference between the position of the object part selected in step S202 and a predetermined target position within the image. The method for calculating the object blur correction amount will be described with reference to FIG. 4.

[0042] A black square 401 indicates a predetermined target position within the image, and can be set to any position. In the example of Fig. 4, it is set to the center of the image. A rectangle 402 indicated by a dotted line indicates the head selected in step S202, and a black circle 403 indicates its center of gravity. An arrow 404 indicates the difference between coordinates 401 and 403, and this difference is the amount of subject blur correction.

[0043] Note that the positions of detected subject parts generally contain errors due to image noise and subject detection accuracy. For this reason, it is desirable to apply a low-pass filter (LPF) to the positions of detected subject parts or the subject blur correction amount, and use the low-frequency components as the subject blur correction amount.

[0044] In step S204, control unit 110 determines whether the part of the subject selected in step S202 is the same as the part selected in the immediately preceding frame. If the part is different from the part selected in the immediately preceding frame, control unit 110 proceeds to step S205; if the part is the same, control unit 110 proceeds to step S206.

[0045] FIG. 5 shows an example in which the selected part is different from that in the previous frame. FIG. 5(a) shows the previous frame, FIG. 5(b) shows the current frame, and FIG. 5(c) shows the next frame. Compared to FIG. 5(a), FIGS. 5(b) and 5(c) show the subject moving away and becoming smaller. A black square 501 indicates the target position of the subject, which in this example is the center of the image.

[0046] In Figure 5(a), the head 502 of the subject is selected, and in Figures 5(b) and 5(c), the body 503 is selected. As explained in step S202, if the head can be detected in Figures 5(b) and 5(c), the head is selected with priority. However, in Figures 5(b) and 5(c), the head cannot be detected because it is small, and only the body is detected, so the body is selected.

[0047] In FIG. 5( a ), subject blur correction is performed so that the coordinates of the center of gravity of the head 502 overlap with the target position 501 .

[0048] In Figure 5(b), torso 502 is selected as the body part. At the time Figure 5(b) is acquired, subject blur correction is still being performed so that the coordinates of the center of gravity of head 501 overlap with target position 501. Black circle 504 indicates the coordinates of the center of gravity of the torso, and arrow 505 indicates the amount of subject blur correction required to move the coordinates of the center of gravity of the torso to target position 501.

[0049] 5(c) shows a frame obtained as a result of subject blur correction performed according to arrow 505 (subject blur correction amount) so that center of gravity coordinates 504 of the torso coincide with target position 501 (image center). In other words, as the detected body part of the subject changes, the state in which the subject's head is near target position 501, which is the center of the image (FIGS. 5(a) and 5(b)), changes to a state in which the subject's torso is at the center of the image. In other words, even though the position of the subject on the image remains almost the same between FIG. 5(a) and FIG. 5(b), the detected body part of the subject changes, and correcting subject blur using the center of gravity coordinates of the torso in FIG. 5(b) causes the position of the subject to move unnaturally by the amount of the change in center of gravity coordinates of the subject's body part.

[0050] Therefore, in order to prevent the position of the subject from moving unnaturally when the part of the subject is switched, when the part selected in step S202 is changed, processing for suppressing the amount of subject blur correction shown in the next step S205 is performed.

[0051] In step S205, the control unit 110 uses the object blur correction amount calculation unit 106 to suppress the object blur correction amount calculated in step S203 based on the amount of displacement of the part position when the part of the object selected by the object part selection unit 105 changes.

[0052] Specifically, the value obtained by subtracting the coordinates of the center of gravity of head 502 in Fig. 5(a) from the coordinates of the center of gravity of trunk 503 in Fig. 5(b) is used as an offset, and this value is subtracted from subject blur correction amount 505. In this way, the change in the coordinates of the center of gravity of the subject's parts is canceled out from the subject blur correction amount, and unnatural movement of the subject's position is eliminated.

[0053] The process of suppressing the amount of subject blur correction will be described in detail using Figure 6. The horizontal axis of Figure 6 represents time, and the vertical axis represents the vertical coordinate of the position of the subject's part selected in step S202. In this example, the head, which has the highest priority, is initially selected, and at time t1, the head is selected, followed by the torso, and then at time t2, the torso is selected again.

[0054] In Fig. 6, dashed line 601 represents time-series data of the positions of the subject's parts before suppression processing, and dotted line 602 represents time-series data of the positions of the subject's parts after suppression processing. Solid line 603 represents time-series data obtained by applying correction processing, described below, to dotted line 602. Here, in Fig. 5, the downward direction of the image is defined as the direction in which the vertical coordinate increases. Because the head is higher than the torso in Fig. 5, the vertical coordinate for the head is small and the vertical coordinate for the torso is large in Fig. 6.

[0055] Between times t1 and t2, the vertical coordinate is displaced significantly due to the change in body part. The amount of displacement of the position of the subject's body part at t1 is designated as 604, and the amount of displacement of the position of the subject's body part at t2 is designated as 605. If these amounts of displacement were reflected directly in the amount of subject blur correction, the position of the subject would change unnaturally simply because the selected body part had changed.

[0056] Therefore, from the position 601 of the subject's part before the suppression processing, the displacement amount 604 is subtracted as an offset from time t1 onwards, and further from time t2 onwards, the displacement amount 606, which is the sum of the displacement amounts 604 and 605, is subtracted as an offset, thereby obtaining the position 602 of the subject's part after the suppression processing.

[0057] In this way, the subject blur correction amount calculation unit 106 subtracts from the subject blur correction amount the amount of displacement of the part position when the part of the subject selected by the subject part selection unit 105 changes as an offset. Furthermore, the subject blur correction amount calculation unit 106 adds a further offset to the current offset each time the part of the subject selected by the subject part selection unit 105 changes, and subtracts the added offset from the subject blur correction amount. This makes it possible to prevent the position of the subject from moving unnaturally when the part of the subject changes.

[0058] Here, the head is selected again from time t2 onwards, so ideally the dotted line 602 after the suppression processing should match the dashed line 601 before the suppression processing, but they do not. This is because the offset at time t1 and the offset at time t2 do not cancel each other out, resulting in an offset error. Possible causes of the offset error include the movement of the subject when the subject's body part changes and errors in subject detection.

[0059] Therefore, after time t2 when the head is detected again, the added offset (displacement amount 604 + 605 = displacement amount 606) is gradually brought closer to zero, so that the dotted line 602 after the suppression process approaches the dashed line 601 before the suppression process, and the solid line 603 can be obtained.

[0060] That is, when the subject part selection unit 105 selects the part of the subject with the highest priority, the subject blur correction amount calculation unit 106 can correct the deviation in the subject blur correction amount due to the offset by gradually bringing the added offset closer to zero.

[0061] In step S206, the control unit 110 performs image blur correction based on the subject blur correction amount calculated in step S203 or step S205. As described above, the image blur correction means can be the correction lens in the photographing optical system 101, which is an optical image blur correction means, the image sensor 102, or an electronic image blur correction means (not shown).

[0062] In the final step S207, the control unit 110 determines whether or not the processing has been completed up to the final frame, and if the processing has not been completed up to the final frame, the control unit 110 returns to step S201. If the processing has been completed up to the final frame, the operation of this flow ends.

[0063] By the above processing, even when the part of the subject being detected changes, it is possible to perform subject blur correction without impairing the quality of the image.

[0064] It should be noted that although all of the processing in steps S201 to S207 can be performed in real time during shooting, it is not necessary to perform the processing during shooting, and it may be performed during image playback.

[0065] For example, by recording an image at the time of shooting in a recording unit (not shown) and performing the processes of steps S201 to S207 on the image recorded by electronic image stabilization, subject shake stabilization can be performed when the image is played back.

[0066] Furthermore, when capturing an image, information on the position of the subject's body part obtained by the subject detection unit 104 may be recorded in addition to the image, so that subject detection does not need to be performed when the image is played back.

[0067] (Modification of the first embodiment) In the process of suppressing the amount of subject blur correction in the above-described embodiment, the value obtained by subtracting the coordinates of the center of gravity of head 502 in Fig. 5(a) before the selected body part of the subject was changed from the coordinates of the center of gravity of trunk 503 in Fig. 5(b) after the selected body part of the subject was changed is used as an offset, and this value is subtracted from subject blur correction amount 505. However, the method of calculating the offset is not limited to this.

[0068] As the offset, it is preferable to use the difference between the center of gravity coordinate of the head after the selected part of the subject has been switched and the center of gravity coordinate of the torso after the selected part of the subject has been switched; however, the center of gravity coordinate of the head after the switch is unknown. Therefore, in the above-described embodiment, the change in the center of gravity coordinate of the head before and after the switch is assumed to be small, and the offset is calculated by subtracting the center of gravity coordinate of the head 502 before the switch from the center of gravity coordinate of the torso 503 after the switch. However, if the center of gravity coordinate of the head 502 before the switch is deviated from the target position, this deviation becomes the target of subject blur correction, and the center of gravity coordinate of the head 502 changes by the amount of subject blur correction even if the subject does not move. Therefore, the amount of subject blur correction corrected before and after the switch may be further added to the value (the above-described displacement amount 604) obtained by subtracting the center of gravity coordinate of the head 502 before the switch from the center of gravity coordinate of the torso 503 after the switch. However, since subject blur correction has been performed at the time of FIG. 5(a) before the switching, and the deviation between the target position and the coordinates of the center of gravity of the head 502 is considered to be small, the offset may be a value obtained by subtracting the coordinates of the center of gravity of the head 502 before the switching from the coordinates of the center of gravity of the body 503 after the switching, as described above.

[0069] Furthermore, if the torso 503 has been detected before the selected part of the subject is switched, the value obtained by subtracting the coordinates of the center of gravity of the head 502 from the coordinates of the center of gravity of the torso 503 in FIG. 5(a) before the selected part of the subject is switched can also be used as the offset. As described above, if there is a change in the size of the subject between FIG. 5(a) and FIG. 5(b), when the offset is calculated from the coordinates of the center of gravity of the head 502 and the coordinates of the center of gravity of the torso 503 before the switch, the offset will also contain an error corresponding to the change in size of the subject. However, the change in size of the subject that occurs in the short period before and after the switch is not large. Therefore, the offset may also be calculated from the coordinates of the center of gravity of the head 502 and the coordinates of the center of gravity of the torso 503 before the switch. Note that, to calculate the offset more accurately, the size of the torso before the switch and the size of the torso after the switch may be compared to obtain the change in size of the subject that occurs before and after the switch, and the offset may be calculated taking this into account.

[0070] (Second embodiment) In the first embodiment, only subject shake was treated as the target of image blur correction, but in this embodiment, image blur correction is performed taking into account camera shake as well.

[0071] Fig. 7 is a diagram showing the configuration of an image capture device 700 according to a second embodiment of the present invention. In Fig. 7, parts that are common to the components shown in Fig. 1 are assigned the same reference numerals as in Fig. 1, and descriptions thereof will be omitted. In addition to the configuration shown in Fig. 1, the image capture device of this embodiment has an image stabilization detection unit 701, an image stabilization amount calculation unit 702, and an image stabilization amount synthesis unit 703. In this embodiment, only parts that perform processing different from that of the first embodiment will be described.

[0072] The camera shake detection unit 701 detects camera shake applied to the imaging device 700. The camera shake correction amount calculation unit 702 calculates an image shake correction amount (camera shake correction amount) for correcting image shake caused by the shake detected by the camera shake detection unit 701.

[0073] The image blur correction amount synthesis unit 703 synthesizes the object blur correction amount output from the object blur correction amount calculation unit 106 and the image blur correction amount (image blur correction amount) output from the image blur correction amount calculation unit 702 to generate a final image blur correction amount (synthesized image blur correction amount).

[0074] Fig. 8 is a flowchart showing the image stabilization operation in the second embodiment. In Fig. 8, steps common to those shown in Fig. 2 are given the same reference numerals as in Fig. 2, and descriptions thereof will be omitted. The operation of the flowchart in Fig. 8 is realized by control unit 110 loading a program stored in ROM 112 into RAM 114 and executing it.

[0075] In step S204, control unit 110 determines whether the part of the subject selected in step S202 is the same as the part selected in the immediately preceding frame. If the part is different from the part selected in the immediately preceding frame, control unit 110 proceeds to step S205, and if the part is the same, control unit 110 proceeds to step S801. After completing step S205, control unit 110 proceeds to step S801.

[0076] In step S801, control unit 110 uses camera shake detection unit 701 to detect camera shake applied to imaging device 700. As camera shake detection unit 701, for example, a gyro sensor can be used. However, the detection method is not limited to this. Step S801 can be executed at any timing as long as it is before S802, and it may be executed before or after S201 to S205.

[0077] In step S802, the image stabilization amount calculation unit 702 calculates an image stabilization amount for correcting image blur caused by the image stabilization detected in step S801. For example, the image stabilization amount can be calculated by integrating the angular velocity signal of the image stabilization obtained by the gyro sensor and inverting the sign.

[0078] In step S803, the control unit 110 uses the image blur correction amount synthesis unit 703 to synthesize the subject blur correction amount obtained in step S204 or step S205 and the camera shake correction amount obtained in step S802 to calculate a final image blur correction amount.

[0079] One synthesis method is to separate the data by frequency. Specifically, a low-pass filter (LPF) is applied to the amount of object shake compensation to extract low-frequency components. On the other hand, a high-pass filter (HPF) is applied to the amount of image stabilization to extract high-frequency components. The low-frequency components of the amount of object shake compensation thus obtained are added to the high-frequency components of the amount of image stabilization and synthesized.

[0080] That is, the final image blur correction amount is an amount of image blur correction that corrects the subject blur component at low frequencies and corrects the camera shake component at high frequencies.

[0081] Here, it is preferable to roughly match the cutoff frequencies of the LPF and HPF and dynamically change them according to the reliability of the subject detection. For example, if the reliability of the subject detection is high, the cutoff frequency is increased to strengthen the subject shake correction, and if the reliability of the subject detection is low, the cutoff frequency is decreased to strengthen the image stabilization.

[0082] In step S804, the control unit 110 performs image blur correction based on the final image blur correction amount obtained in step S803. As described above, the image blur correction means can be the correction lens in the photographing optical system 101, which is an optical image blur correction means, the image sensor 102, or an electronic image blur correction means (not shown).

[0083] By the above processing, even when the part of the subject being detected changes, it is possible to perform correction of subject blur and image blur caused by camera shake without impairing the quality of the image.

[0084] (Third embodiment) In both the first and second embodiments, an offset is acquired when the subject part changes, and the acquired offset is subtracted from the subject shake correction amount. Similar effects can also be obtained by changing the target position when the subject part changes. In this embodiment, a description will be given of an embodiment in which the target position is changed when the subject part changes.

[0085] This embodiment differs from the first embodiment in the process of suppressing the amount of subject blur correction shown in step S205, but other configurations are the same as those of the first embodiment, so description thereof will be omitted. In this embodiment, the suppression process involves changing the target position. The target position is determined based on the position of the subject part selected after the switch. The subject part selected after the switch refers to the torso 503 in the case of FIGS. 5(a) and 5(b). For ease of explanation, the subject part selected before the switch (head 502 in the case of FIGS. 5(a) and 5(b)) will be referred to as the first part, and the subject part selected after the switch (torso 503 in the case of FIGS. 5(a) and 5(b)) will be referred to as the second part.

[0086] The target position after the change will now be described. In this embodiment, for example, the barycentric coordinates (position of the black circle 504) of the second body part (torso 503) in the frame after switching shown in FIG. 5B are used as the target position to calculate the amount of subject blur correction after switching. FIG. 5B shows an image of a frame obtained as a result of subject blur correction using the barycentric coordinates of the first body part (head 502), so the head 502 is located near the center of the image, which is the original target position 501. Therefore, by using the barycentric coordinates of the torso 503 in FIG. 5B as the target position and performing subject blur correction so that the barycentric coordinates of the torso 503 in subsequent frames are located at the target position (i.e., the barycentric coordinates of the torso 503 in FIG. 5B), subject blur correction control can be performed so that the head 502 is located near the center of the image, even if the head 502 is not selected. This reduces sudden changes in the subject position that occur when the subject part is switched.

[0087] The target position may be the coordinates of the center of gravity of the second part (torso 503) in the frame before switching shown in Figure 5(a). In this case, as in the modified example of the first embodiment, it is affected by the change in size of the subject that occurs between Figure 5(a) and Figure 5(b). However, since the change in size of the subject that occurs over a short period of time can be considered not to be large, there is little possibility that a large change in composition will occur even if the position of the center of gravity of the torso 503 in the frame before switching is set as the target position.

[0088] The target position may also be set based on the difference in the center of gravity coordinates between the first part (head 502) and the second part (body 503) in the frame before switching, as shown in FIG. 5(a). If the head 502 is displaced from the target position in the frame before switching, and the position of the body 503 in the frame before switching is set as the target position, subject blur correction is performed so that the displaced state is maintained. By setting the target position to the position obtained by adding the difference between the head 502 and body 503 in the frame before switching to the target position before switching (the center of the image in FIG. 5(a)), the position of the body 503 when the head 502 is located at the center of the image can be set as the target position. However, as in the case where the center of gravity coordinates of the body 503 in the frame before switching are set as the target position, this is affected by changes in the size of the subject that occur between FIG. 5(a) and FIG. 5(b).

[0089] Furthermore, whether the position of the second part (torso 503) before or after the change is used, various adjustments may be made in addition to using the center of gravity coordinates of the torso as the target position. For example, if the subject moves suddenly simultaneously with the change of the subject part, the center of gravity coordinates of the selected subject part may be located at the edge of the image. In this case, if the center of gravity coordinates themselves are set as the target position, subject blur correction control is performed so that the subject is located at the edge of the image. Therefore, a position closer to the center of the image than the center of gravity coordinates may be set as the target position by, for example, setting an upper limit on the distance from the center of the image to the target position or restricting the coordinates that can be set as the target position. Furthermore, if the size of the second part before and after the change can be detected, the center of gravity coordinates of the second part before the change may be adjusted to take into account the change in size and set as the target position. For example, if the subject part changes from the face to the torso due to a decrease in size, the center of gravity coordinates of the torso before the change may be moved closer to the center by the amount of the change in size.

[0090] The disclosure of this specification includes the following image stabilization device, control method thereof, imaging device, program, and storage medium.

[0091] (Item 1) a subject detection means for detecting each part of a subject from an input image and outputting the position of the detected part of the subject; a selection means for selecting a subject part from one or more subject parts detected by the subject detection means based on a predetermined priority; a subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the part of the subject selected by the selection means and a predetermined target position within the image, The subject blur correction amount calculation means an image stabilization device, when the part of the subject selected by the selection means is switched from a first part to a second part, calculating the amount of subject blur correction based on a difference between a position of the selected part of the subject and the target position, the position of the second part in the image, and the position of the second part in the image acquired before the switching.

[0092] (Item 2) Item 1. The image stabilization device according to item 1, wherein, when the part of the subject selected by the selection part is switched from a first part to a second part, the subject blur correction amount calculation part calculates the subject blur correction amount based on a difference between the position of the selected part of the subject and the target position, and the position of the first part in the image acquired before the switch and the position of the second part after the switch.

[0093] (Item 3) Item 2. The image stabilization device according to item 2, wherein, when the part of the subject selected by the selection part is switched from a first part to a second part, the subject blur correction amount calculation part calculates the subject blur correction amount based on a difference between the position of the selected part of the subject and the target position, and a difference between the position of the first part in an image acquired before the switching and the position of the second part in an image acquired after the switching.

[0094] (Item 4) Item 1. The image stabilization device according to item 1, wherein, when the part of the subject selected by the selection part is switched from a first part to a second part, the subject blur correction amount calculation part calculates the subject blur correction amount based on a difference between the position of the selected part of the subject and the target position, the position of the first part in the image acquired before the switching, and the position of the second part in the image acquired before the switching.

[0095] (Item 5) Item 5. The image stabilization device according to item 4, wherein, when the part of the subject selected by the selection part is switched from a first part to a second part, the subject blur correction amount calculation part calculates the subject blur correction amount based on a difference between the position of the selected part of the subject and the target position, and a difference between the position of the first part in an image acquired before the switching and the position of the second part in an image acquired before the switching.

[0096] (Item 6) 6. The image stabilization device according to item 3 or 5, wherein the object blur correction amount calculation means calculates the object blur correction amount by subtracting, as an offset, a difference between the position of the first part and the position of the second part from a tentative object blur correction amount based on a difference between the position of the second part and the target position.

[0097] (Item 7) 7. The image stabilization device according to item 6, wherein the object blur correction amount calculation means adds the offset each time the selected object part is switched, and subtracts the added offset from the provisional object blur correction amount.

[0098] (Item 8) 8. The image stabilization device according to item 7, wherein the object blur correction amount calculation means gradually brings the added offset closer to zero when a part of the object with the highest priority is selected.

[0099] (Item 9) Item 9. The image stabilization device according to item 8, wherein when the part of the subject selected by the selection means is switched from a first part to a second part, the target position is changed based on the position of the second part in the image before the switch and the position of the first part in the image before the switch.

[0100] (Item 10) a subject detection means for detecting each part of a subject from an input image and outputting the position of the detected part of the subject; a selection means for selecting a subject part from one or more subject parts detected by the subject detection means based on a predetermined priority; a subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the part of the subject selected by the selection means and a predetermined target position within the image, an image stabilization device, wherein when the part of the subject selected by the selection means is switched from a first part to a second part, the target position is changed based on the position of the second part.

[0101] (Item 11) Item 11. The image stabilization device according to item 10, wherein when the part of the subject selected by the selection means is switched from a first part to a second part, the target position is changed based on the position of the second part in an image acquired after the switching.

[0102] (Item 12) Item 11. The image stabilization device according to item 10, wherein when the part of the subject selected by the selection means is switched from a first part to a second part, the target position is changed based on the position of the second part in the image acquired before the switching.

[0103] (Item 13) 13. The image stabilization device according to any one of items 1 to 12, further comprising: an image stabilization amount calculation unit that calculates an image stabilization amount for correcting image blur caused by a shake applied to the imaging device; and a combination unit that generates a combined image stabilization amount by combining the subject blur compensation amount and the image stabilization amount calculated by the image stabilization amount calculation unit.

[0104] (Item 14) Item 14. The image stabilization device according to item 13, wherein the synthesizing means synthesizes by adding together the low frequency component of the subject shake compensation amount and the high frequency component of the hand shake compensation amount.

[0105] (Item 15) 15. The image stabilization device according to any one of items 1 to 14, wherein the selection means can select a part of the subject to be used for correcting subject shake independently of a part of the subject to be used for autofocusing.

[0106] (Item 16) 16. The image stabilization device according to any one of items 1 to 15, wherein the selection means selects the face or head as the part of the subject with priority over organs included in the face.

[0107] (Item 17) 17. The image stabilization device according to any one of items 1 to 16, wherein the selection means selects the face or head as the part of the subject with priority over the torso.

[0108] (Item 18) 18. The image stabilization device according to any one of items 1 to 17, further comprising an image stabilization unit that corrects image shake based on the subject shake correction amount.

[0109] (Item 19) Item 19. The image stabilization device according to item 18, wherein the image stabilization means has an image sensor that corrects image shake by displacing in a direction perpendicular to the optical axis of the photographing optical system.

[0110] (Item 20) Item 19. The image stabilization device according to item 18, wherein the image stabilization means has a correction lens that corrects image shake by being displaced in a direction different from the optical axis of the photographing optical system.

[0111] (Item 21) an imaging means for imaging a subject; An image stabilization device according to any one of items 1 to 20; An imaging device comprising:

[0112] (Item 22) a subject detection step of detecting a subject by part from an input image and outputting the position of the detected subject part; a selection step of selecting a subject part from one or more subject parts detected in the subject detection step based on a predetermined priority; a subject blur correction amount calculation step of calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the part of the subject selected in the selection step and a predetermined target position in the image, When the part of the subject selected in the selection step is switched from a first part to a second part, In the subject blur correction amount calculation step, a control method for an image stabilization device, the control method comprising: calculating the amount of object blur correction based on a difference between the position of the selected object part and the target position, the position of the second object part in the image, and the position of the first object part in the image acquired before switching.

[0113] (Item 23) a subject detection step of detecting a subject by part from an input image and outputting the position of the detected subject part; a selection step of selecting a subject part from one or more subject parts detected in the subject detection step based on a predetermined priority; a subject blur correction amount calculation step of calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the part of the subject selected in the selection step and a predetermined target position in the image, a control method for an image stabilization device, wherein when the part of the subject selected in the selection step is switched from a first part to a second part, the target position is changed based on the position of the second part.

[0114] (Item 24) 24. A program for causing a computer to execute each step of the control method according to item 22 or 23.

[0115] (Item 25) 24. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to item 22 or 23.

[0116] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0117] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0118] 100: imaging device, 101: optical system, 102: imaging element, 103: development processing unit, 104: subject detection unit, 105: subject part selection unit, 106: subject blur correction amount calculation unit, 701: camera shake detection unit, 702: camera shake correction amount calculation unit, 703: image blur correction amount synthesis unit

Claims

1. a subject detection means for detecting each part of a subject from an input image and outputting the position of the detected part of the subject; a selection means for selecting a subject part from one or more subject parts detected by the subject detection means based on a predetermined priority; a subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the part of the subject selected by the selection means and a predetermined target position within the image, The subject blur correction amount calculation means an image stabilization device, when the part of the subject selected by the selection means is switched from a first part to a second part, calculates the amount of subject blur correction based on a difference between a position of the selected part of the subject and the target position, the position of the second part in the image, and the position of the second part in an image acquired before the switching.

2. 2. The image blur correction device according to claim 1, wherein, when the part of the subject selected by the selection part is switched from a first part to a second part, the subject blur correction amount calculation part calculates the subject blur correction amount based on a difference between the position of the selected part of the subject and the target position, and on the position of the first part in the image acquired before the switching and the position of the second part after the switching.

3. 3. The image blur correction device according to claim 2, wherein, when the part of the subject selected by the selection part is switched from a first part to a second part, the subject blur correction amount calculation part calculates the subject blur correction amount based on a difference between the position of the selected part of the subject and the target position, and a difference between the position of the first part in an image acquired before the switching and the position of the second part in an image acquired after the switching.

4. 2. The image blur correction device according to claim 1, wherein, when the part of the subject selected by the selection part is switched from a first part to a second part, the subject blur correction amount calculation part calculates the subject blur correction amount based on a difference between the position of the selected part of the subject and the target position, the position of the first part in the image acquired before the switching, and the position of the second part in the image acquired before the switching.

5. 5. The image blur correction device according to claim 4, wherein, when the part of the subject selected by the selection part is switched from a first part to a second part, the subject blur correction amount calculation part calculates the subject blur correction amount based on a difference between a position of the selected part of the subject and the target position, and a difference between a position of the first part in an image acquired before the switching and a position of the second part in an image acquired before the switching.

6. 4. The image blur correction device according to claim 3, wherein the subject blur correction amount calculation means calculates the subject blur correction amount by subtracting, as an offset, a difference between the position of the first part and the position of the second part from a tentative subject blur correction amount based on a difference between the position of the second part and the target position.

7. 7. The image stabilization device according to claim 6, wherein the object blur correction amount calculation means adds the offset each time the selected object part is switched, and subtracts the added offset from the provisional object blur correction amount.

8. 8. The image stabilization device according to claim 7, wherein the object shake correction amount calculation means gradually brings the added offset closer to zero when a part of the object with the highest priority is selected.

9. 9. The image stabilization device according to claim 8, wherein when the part of the subject selected by the selection means is switched from a first part to a second part, the target position is changed based on the position of the second part in the image before the switch and the position of the first part in the image before the switch.

10. a subject detection means for detecting each part of a subject from an input image and outputting the position of the detected part of the subject; a selection means for selecting a subject part from one or more subject parts detected by the subject detection means based on a predetermined priority; a subject blur correction amount calculation means for calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the part of the subject selected by the selection means and a predetermined target position within the image, an image stabilization device, wherein when the part of the subject selected by the selection means is switched from a first part to a second part, the target position is changed based on the position of the second part;

11. 11. The image stabilization device according to claim 10, wherein when the part of the subject selected by the selection means is switched from a first part to a second part, the target position is changed based on the position of the second part in an image acquired after the switching.

12. 11. The image stabilization device according to claim 10, wherein when the part of the subject selected by the selection means is switched from a first part to a second part, the target position is changed based on the position of the second part in the image acquired before the switching.

13. 2. The image stabilization device according to claim 1, further comprising: an image stabilization amount calculation unit that calculates an image stabilization amount for correcting image blur caused by a shake applied to the imaging device; and a combination unit that generates a combined image stabilization amount by combining the subject blur compensation amount and the image stabilization amount calculated by the image stabilization amount calculation unit.

14. 14. The image stabilization device according to claim 13, wherein the synthesizing means synthesizes the low-frequency component of the subject shake compensation amount and the high-frequency component of the camera shake compensation amount by adding them together.

15. 2. The image stabilization device according to claim 1, wherein said selection means is capable of selecting a portion of the subject to be used for subject shake correction independently of a portion of the subject to be used for autofocusing.

16. 2. The image stabilization device according to claim 1, wherein the selection means selects the face or head as the object's part with priority over organs contained in the face.

17. 2. The image stabilization device according to claim 1, wherein the selection means selects the face or head as the object part with priority over the torso.

18. 2. The image stabilization device according to claim 1, further comprising an image stabilization unit that corrects image shake based on the subject shake correction amount.

19. 19. The image stabilization device according to claim 18, wherein the image stabilization means has an image pickup element that corrects image shake by displacing in a direction perpendicular to the optical axis of the photographing optical system.

20. 19. The image stabilization device according to claim 18, wherein the image stabilization means has a correction lens that corrects image stabilization by being displaced in a direction different from the optical axis of the photographing optical system.

21. A subject detection means for detecting one or more parts of a subject from an input image and outputting the positions of the detected parts of the subject; a control unit that performs control to correct subject blur of the subject, the control means controls the first portion to approach a target position in the image in a first state in which the subject detection means has detected the first portion; an image stabilization device, characterized in that, in a second state in which the subject detection means does not detect the first portion but detects a second portion, the image stabilization device controls the first portion to approach the target position more closely than the second portion.

22. A selection means for selecting one subject part from one or more subject parts detected by the subject detection means, the control means calculates an amount of object blur correction based on a difference between the position of the object part selected by the selection means and the target position; the selection means selects the first region in the first state and selects the second region in the second state; 22. The image blur correction device according to claim 21, wherein, when the part of the subject selected by the selection part is switched from the first part to the second part, the control part calculates the amount of subject blur correction based on a difference between the position of the first part and the position of the second part and a difference between the position of the second part and the target position.

23. an imaging means for imaging a subject; an image stabilization device according to any one of claims 1 to 22; An imaging device comprising:

24. a subject detection step of detecting a subject by part from an input image and outputting the position of the detected subject part; a selection step of selecting a subject part from one or more subject parts detected in the subject detection step based on a predetermined priority; a subject blur correction amount calculation step of calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the part of the subject selected in the selection step and a predetermined target position in the image, When the part of the subject selected in the selection step is switched from the first part to the second part, In the subject blur correction amount calculation step, a control method for an image stabilization device, the control method comprising: calculating the amount of object blur correction based on a difference between a position of the selected part of the object and the target position, the position of the second part in the image, and the position of the first part in the image acquired before switching.

25. a subject detection step of detecting a subject by part from an input image and outputting the position of the detected subject part; a selection step of selecting a subject part from one or more subject parts detected in the subject detection step based on a predetermined priority; a subject blur correction amount calculation step of calculating a subject blur correction amount for correcting subject blur based on a difference between the position of the part of the subject selected in the selection step and a predetermined target position in the image, a control method for an image stabilization device, wherein when the part of the subject selected in the selection step is switched from a first part to a second part, the target position is changed based on the position of the second part.

26. A subject detection step of detecting one or more parts of a subject from an input image and outputting the positions of the detected parts of the subject; a control step of performing control to correct subject blur of the subject, In the control step, in a first state in which a first portion is detected in the subject detection step, control is performed so that the first portion approaches a target position in the image; a control method for an image stabilization device, characterized in that, in a second state in which the first portion is not detected but a second portion is detected in the subject detection step, the first portion is controlled to approach the target position more closely than the second portion.

27. A program for causing a computer to execute each step of the control method according to any one of claims 24 to 26.

28. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to any one of claims 24 to 26.

Images