Image blur correction device, its control method, and program

Abstract

To provide an image shake correction device that can obtain an image shake correction effect desired by a user.SOLUTION: An image shake correction device comprises: a subject detection unit that detects a specific subject in an input image and outputs subject information; a camera information acquisition unit that acquires camera information necessary for estimating a photographing state; an estimation unit that estimates an object of attention in the image by using the subject information and the camera information; a motion detection unit that detects a motion of the background and a motion of the subject in the input image; a conversion unit that converts the motion of the background and the motion of the subject detected by the motion detection unit into a first shake correction amount for correcting shake of the background and a second shake correction amount for correcting shake of the subject, respectively; and a correction amount calculation unit that, based on the object of attention estimated by the estimation unit, combines the first shake correction amount and the second shake correction amount to generate a final shake correction amount.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for correcting image blur in an imaging device.

Background Art

[0002] In an imaging device such as a digital camera, image blur may occur due to "camera shake" where the user holds the camera body, or due to "subject movement" where a subject such as a person moves and the subject position changes.

[0003] As a method for detecting "camera shake", there are a method using an angular velocity sensor attached to the imaging device and a method using the motion vector of a stationary object (background) in the captured image. On the other hand, as a method for detecting "subject movement", there is a method of detecting a subject in the captured image and using the motion vector of the subject.

[0004] As a method for correcting image blur caused by "camera shake" or "subject movement", there are optical blur correction and electronic blur correction. In optical blur correction, a correction lens or an imaging element in the optical system is moved according to the shake to move the image formed on the light receiving surface of the imaging element so as to cancel out the shake, thereby correcting the image blur. In electronic blur correction, the blur is pseudo-corrected by image processing on the captured image.

[0005] Since the amount of shake (the direction and magnitude of the shake) generally differs between "camera shake" and "subject movement", it is impossible to completely correct both. Therefore, for example, Patent Document 1 discloses a technique for correcting "subject movement" when the face of a subject is detected and correcting "camera shake" when the face of the subject is not detected.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] It is believed that whether "camera shake" or "subject shake" should be corrected depends on the user's intention (target of attention) in the shooting scene. For example, if the user focuses on the background, it is desirable that "camera shake", which is the shaking of the entire screen, be corrected. On the other hand, if the user focuses on the main subject, it is desirable that "subject shake" be corrected. Therefore, it is necessary to appropriately control the target of shake correction according to the user's intention, which changes along with the shooting scene.

[0008] However, in the technology disclosed in Patent Document 1, the shake correction target is switched depending on the presence or absence of face detection regardless of the user's intention, so there is a problem that the image shake correction effect suitable for the user's intention cannot be obtained depending on the scene. Furthermore, when the technology in Patent Document 1 is applied to a video, the shake correction target is switched in a binary manner depending on the presence or absence of face detection, so there is a problem that an unnatural shift in the image occurs when switching, even if it is very small.

[0009] The present invention has been made in view of the above-mentioned problems, and has an object to provide an image stabilization device that can obtain an image stabilization effect desired by a user. [Means for solving the problem]

[0010] The image blur correction device according to the present invention includes: a subject detection means for detecting a specific subject in an input image and outputting subject information; a camera information acquisition means for acquiring camera information necessary for estimating a shooting situation; an estimation means for estimating a target of interest in the image using the subject information and the camera information; a first motion detection means for detecting the movement of the background and the movement of the subject in the input image; a conversion means for converting the movement of the background and the movement of the subject detected by the first motion detection means into a first blur correction amount for correcting the blur of the background and a second blur correction amount for correcting the blur of the subject, respectively; and a correction amount calculation means for generating a final blur correction amount by synthesizing the first blur correction amount and the second blur correction amount based on the target of interest estimated by the estimation means. , the camera information includes at least any one of information indicating the shutter speed, information indicating the AF area, information indicating the focal length, information indicating user operations, information indicating the detection result of the inertial sensor, information indicating the distance to the subject, and information indicating the detection result of the user's line of sight. It is characterized by this.

Effects of the Invention

[0011] According to the present invention, it is possible to provide an image blur correction device capable of obtaining an image blur correction effect desired by a user.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] (First Embodiment) FIG. 1 is a diagram showing the configuration of an imaging apparatus according to a first embodiment of the present invention.

[0015] In FIG. 1, the subject image formed by the optical system 101 is converted into an image signal by the imaging device 102, and development processing such as white balance processing, color (luminance / chrominance signal) conversion processing, and γ correction processing is performed by the development processing unit 103, and output as image data (input image data). The image memory 104 temporarily stores and holds the image data developed by the development processing unit 103 for one frame or a plurality of frames.

[0016] The motion vector detection unit 105 detects a motion vector from the image data input from the development processing unit 103 and the image memory 104. The motion separation unit 106 separates the motion vector detected by the motion vector detection unit 105 into a first motion vector representing the motion of the background and a second motion vector representing the motion of the subject. The blur correction amount conversion unit 107 converts the first motion vector and the second motion vector obtained from the motion separation unit 106 into a first blur correction amount and a second blur correction amount, respectively.

[0017] The subject detection unit 108 detects a specific subject from the image data input from the development processing unit 103 and outputs subject information. The subject information is, for example, a subject area (position and size of the subject), the humanity of the subject, the motion of the subject, and the like.

[0018] The camera information acquisition unit 109 acquires camera information necessary for estimating the shooting situation. The camera information is, for example, a shooting mode, a shutter speed, AF area information, a focal length, user instruction information, inertial sensor information, depth information, gaze information, and the like.

[0019] The target of interest estimation unit 110 estimates the target of interest in the image using the subject information obtained by the subject detection unit 108 and the camera information obtained by the camera information acquisition unit 109.

[0020] Based on the information of the target of interest obtained by the target of interest estimation unit 110, the shake correction amount calculation unit 111 synthesizes the first shake correction amount and the second shake correction amount obtained by the shake correction amount conversion unit 107 to generate a final shake correction amount.

[0021] The generated shake correction amount is output to a shift mechanism that shifts the correction lens and / or the imaging element 102 in the optical system 101, which is an optical shake correction means, in a direction perpendicular to the optical axis, and optical shake correction is performed. Note that the present invention is not limited to optical shake correction, and the shake correction amount may be output to an electronic shake correction means (not shown) to perform electronic shake correction.

[0022] Next, the image blur correction operation in the imaging device 100 configured as described above will be described using the flowchart shown in FIG. 2.

[0023] In step S201, the subject detection unit 108 detects a specific subject from the image data input from the development processing unit 103 and outputs subject information. As the subject information, subject area information (the position and size of the subject), information on the humanity of the subject, and movement information of the subject are used. As a specific subject, a human face is representative. As the detection method, a known face detection method may be used. Known techniques for face detection include methods that utilize knowledge about faces (skin color information, parts such as eyes, nose, and mouth), and methods that configure a discriminator for face detection using a learning algorithm typified by a neural network. And, in order to improve the detection accuracy, it is common to perform face detection by combining these.

[0024] When there are multiple human faces in the image, one face considered to be the most important is selected and output. As the selection method, for example, a face with a larger size, a face whose position is closer to the center of the image, and a face with a higher face detection reliability may be preferentially selected. Alternatively, the user may be allowed to select from multiple candidates.

[0025] By the above method, as subject information, a subject area and the humanity of the subject can be obtained. Also, the movement of the subject can be obtained, for example, by determining the displacement amount of the center of gravity of the subject area between consecutive frame images.

[0026] In step S202, the camera information acquisition unit 109 acquires camera information necessary for estimating the shooting situation. As the camera information, shooting mode information, shutter speed information, AF area information, focal length information, user instruction information, inertial sensor information, depth information, and gaze information shall be used.

[0027] Regarding the shooting mode information, shutter speed information, AF area information, focal length information, and user instruction information, the values set by the user for the camera can be acquired by reading them. The user instruction information is information indicating the degree of attention to the background or subject described later, and can be directly set by the user on the camera.

[0028] The inertial sensor information is obtained by an angular velocity sensor or an acceleration sensor installed in the camera, and is information representing the position and orientation of the camera. The depth information is detected, for example, by using a distance measuring sensor or by using known SfM (Structure from Motion) from the captured image data. The gaze information is obtained by a method such as a known corneal reflection method, and is information indicating the area where the gaze in the image is located.

[0029] In step S203, the target-of-interest estimation unit 110 estimates whether the photographer is focusing on the background or the subject as the target of interest based on the subject information detected in step S201 and the camera information acquired in step S202.

[0030] A method for estimating a target of interest based on subject information and camera information will be described. Here, first, for each piece of information, a background degree representing the degree to which the target of interest is the background and a subject degree representing the degree to which the target of interest is a subject are calculated. Here, the background degree and the subject degree are expressed as numerical values that sum to 1. Note that it is also possible to calculate only one of the background degree and the subject degree.

[0031] First, the subject information will be described. Regarding the position of the subject, since there is a higher possibility of shooting with focus on the subject as the subject is closer to the center of the screen, the subject degree is set high (for example, 0.8) and the background degree is set low (for example, 0.2).

[0032] Regarding the size of the subject, since there is a higher possibility of shooting with focus on the subject as the subject is larger, the subject degree is set high (for example, 0.8) and the background degree is set low (for example, 0.2).

[0033] Regarding the humanity of the subject, since there is a higher possibility of shooting with focus on the subject as the subject is more human-like, the subject degree is set high (for example, 0.7) and the background degree is set low (for example, 0.3).

[0034] Regarding the movement of the subject, since it is assumed that the camera is set up to capture the subject as the movement of the subject is smaller, and there is a higher possibility of shooting with focus on the subject, the subject degree is set high (for example, 0.6) and the background degree is set low (for example, 0.4).

[0035] Next, the camera information will be described. Regarding the shooting mode, for example, in the case of the portrait mode, since it is highly likely that the shooting is focused on a person (= subject), the subject degree is set high (e.g., 0.9), and the background degree is set low (e.g., 0.1). On the other hand, in the case of the landscape mode, since it is highly likely that the shooting is focused on the landscape, the subject degree is set low (e.g., 0.1), and the background degree is set high (e.g., 0.9). Thus, in the shooting mode, by assuming the likely shooting situation according to the mode, the background degree and the subject degree can be determined.

[0036] Regarding the shutter speed, the faster the shutter speed, the higher the likelihood of shooting while focusing on a subject moving at high speed. Therefore, the subject degree is set high (e.g., 0.7), and the background degree is set low (e.g., 0.3).

[0037] Regarding the AF area information, the larger the AF area, the higher the likelihood of shooting while focusing on the background. Therefore, the subject degree is set low (e.g., 0.3), and the background degree is set high (e.g., 0.7). Also, when face recognition AF or eye AF is set, since it is highly likely that the shooting is focused on a subject such as a person or an animal, the subject degree is set high (e.g., 0.9), and the background degree is set low (e.g., 0.1).

[0038] Regarding the gaze information, in combination with the subject information, for example, if the area where the gaze is located in the image is the subject area, it is highly likely that the shooting is focused on the subject. Therefore, the subject degree is set high (e.g., 0.9), and the background degree is set low (e.g., 0.1).

[0039] Regarding the focal length and depth information, it is difficult to infer the photographer's intention from each alone. Therefore, an example of a method for estimating the target of attention by combining the two will be described.

[0040] When the focal length f [mm] and the distance to the subject (depth information) d [mm] are given, if the size of the subject on the imaging surface is X [mm], the actual size Y [mm] of the subject can be calculated by the following formula (1).

[0041] Y = (d / f)·X (1) If the size of the actual subject is known, the intention of the photographer can be inferred from the relationship between the size of the subject on the image plane and the focal length. For example, if the size of the actual subject is small, but the size of the subject on the image plane is large and the focal length is long, it means that the photographer is paying great attention to the subject. Therefore, the smaller the size of the actual subject, the larger the size of the subject on the image plane, and the longer the focal length, the higher the degree of the subject and the lower the degree of the background.

[0042] In this way, in this embodiment, by combining a plurality of camera information, the intention of the photographer that cannot be inferred from a single camera information can be grasped.

[0043] It is also difficult to infer the intention of the photographer from the inertial sensor information alone. Therefore, an example of a method for estimating the target of interest by combining the inertial sensor information and the movement information of the subject will be described. When the target of interest of the photographer is the subject, the camera is moved to keep the subject at a fixed position on the screen, so the movement of the subject is relatively smaller than the movement of the camera.

[0044] Therefore, the smaller the movement amount of the subject compared to the movement amount of the camera between the frame images obtained from the inertial sensor information, the more it is assumed that the camera is set up to capture the subject. Therefore, since there is a high possibility that the subject is being photographed with attention, the degree of the subject is increased and the degree of the background is decreased.

[0045] This concept can be directly applied to panning shooting. In the case of panning shooting, since the photographer swings the camera greatly to keep the moving subject in the center, the movement of the background is large and the movement of the subject is small. Therefore, during panning shooting, based on the inertial sensor information and the movement information of the subject, the degree of the subject automatically becomes high.

[0046] Regarding the relationships among the subject information, camera information, and the object of interest described above, a part of them is summarized in the table of FIG. 3. The numerical values in parentheses indicate an example of the aforementioned background degree / subject degree.

[0047] When there are multiple pieces of information that can be used for estimating the object of interest, weighted addition may be performed for the background degree and subject degree obtained for each piece of information, respectively, to calculate the final background degree and subject degree. The weights may be set, for example, based on the reliability of each information source.

[0048] For example, assuming that the information sources include shooting mode, likeness of the subject to a person, size of the subject, movement of the subject, and shutter speed, let the background degrees obtained from each be Da, Db, Dc, Dd, De, and the weights be Ga, Gb, Gc, Gd, Ge. The final background degree D can be calculated by the following formula (2).

[0049] D = Ga × Da + Gb × Db + Gc × Dc + Gd × Dd + Ge × De (2) For example, let the weights be Ga = 0.3, Gb = 0.2, Gc = 0.2, Gd = 0.1, Ge = 0.2. Here, the weights are normalized so that their sum is 1. Now, if the shooting mode is the landscape mode (Da = 0.9), the likeness of the subject to a person is high (Db = 0.3), the size of the subject is small (Dc = 0.8), the movement of the subject is small (Dd = 0.4), and the shutter speed is slow (De = 0.7), then from formula (2), D = 0.6. The final subject degree can be calculated in the same way. When the reliability of each information source is the same or unknown, all the weights may be made the same.

[0050] When using user instruction information as camera information, for example, by having the user indicate the degree of attention to the background and the subject, the degree of background and the degree of subject can be determined without estimation. The degree of attention may be indicated directly by a numerical value, or may be indicated qualitatively such as strong or weak. When there are multiple subjects, any one of them may be selectable. Also, the degree of background and the degree of subject may be set independently in the horizontal and vertical directions.

[0051] In step S204, the motion vector detection unit 105 detects a motion vector from the input image from the development processing unit 103 and the image memory 104 by template matching.

[0052] FIG. 4 is a diagram showing an overview of template matching. FIG. 4(a) shows a reference image which is one of two vector detection images, and FIG. 4(b) shows a reference image which is the other. Here, by using the frame image held in the image memory 104 as the reference image and the image data directly input from the development processing unit 103 as the reference image, the motion vector from the past frame image to the current frame image is calculated. Note that the reference image and the reference image may be swapped, in which case it means calculating the motion vector from the current frame image to the past frame image.

[0053] The motion vector detection unit 105 arranges a template area 301 in the reference image and a search area 302 in the reference image, and calculates the correlation value between the template area 301 and the search area 302.

[0054] Here, the arrangement of the template area 301 is arbitrary. It may be arranged around a plurality of fixed coordinates defined within the screen, or it may be arranged around the coordinates of feature points detected by a known method. Also, using the subject information obtained in step S201, the coordinates for arrangement may be determined such that the number of template areas arranged in the subject area and the other areas becomes equal. Regarding the search area, it may be arranged with a predetermined size so as to evenly include the template area vertically, horizontally, and diagonally.

[0055] In this embodiment, as a method for calculating the correlation value, the sum of absolute differences (hereinafter abbreviated as SAD) is used. The calculation formula for SAD is shown in Equation (3).

[0056]

Equation

[0057] In Equation (3), f(i, j) represents the luminance value at the coordinates (i, j) within the template area 301. Also, g(i, j) represents the luminance value at each coordinate within the area to be the object of correlation value calculation (hereinafter referred to as the correlation value calculation area) 303 within the search area 302. In SAD, the absolute value of the difference between the luminance values f(i, j) and g(i, j) in both areas 302 and 303 is calculated, and the sum thereof is obtained to obtain the correlation value S_SAD. The smaller the value of the correlation value S_SAD, the higher the similarity of the texture between the template area 301 and the correlation value calculation area 303.

[0058] Note that methods other than SAD may be used for calculating the correlation value. For example, the sum of squared differences (SSD) or the normalized cross-correlation (NCC) may be used.

[0059] The motion vector detection unit 105 calculates the correlation value by moving the correlation value calculation area 303 throughout the search area 302. Thereby, a correlation value map as shown in FIG. 5 is created for the search area 302.

[0060] FIG. 5(a) shows a correlation value map calculated in the coordinate system of the search area 302, where the X-axis and Y-axis represent the correlation value map coordinates, and the Z-axis represents the magnitude of the correlation value at each coordinate. FIG. 5(b) is a diagram showing the contour lines of FIG. 5(a).

[0061] In FIGS. 5(a) and 5(b), the correlation value is the smallest at the minimum value 401. It can be determined that there is a texture very similar to the template area 301 in the area where the minimum value 401 is calculated within the search area 302. 402 represents the second minimum value, and 403 represents the third minimum value, which means that there are textures similar to the minimum value 401 following it.

[0062] In this way, the motion vector detection unit 105 calculates the correlation value between the template area 301 and the search area 302, and determines the position of the correlation value calculation area 303 where the value is the smallest. Thereby, the movement destination of the template area 301 on the reference image on the reference image can be specified. Then, a motion vector can be detected with the direction and amount of movement to the movement destination on the reference image based on the position of the template area on the reference image as the direction and magnitude.

[0063] In step S205, the motion separation unit 106 separates the motion vector detected in step S204 into a first motion vector (hereinafter referred to as a background vector) representing the motion of the background and a second motion vector (hereinafter referred to as a subject vector) representing the motion of the subject.

[0064] An example of the separation method will be described. First, using the subject information obtained in step S201, a first separation process is performed with the motion vector detected at a point belonging to the outside of the subject area as the background vector and the motion vector detected at a point belonging to the inside of the subject area as the subject vector.

[0065] In the case where the subject is not detected, for example, separation based on the depth information obtained in step S202 can be considered. In that case, for example, the motion vectors detected at points belonging to a region farther than a predetermined depth in the image are used as background vectors, and the motion vectors detected at points belonging to a region closer than the predetermined depth are used as the subject region, and the first separation process may be performed. When the subject is not detected and there is no depth difference within the screen, the separation process is terminated with all being background vectors.

[0066] The separation process may be completed only with the first separation process. However, in such "separation based on the subject region", the classification of the motion vectors detected at points near the boundary between the background and the subject may be incorrect. Also, when the subject region is incorrect due to the problem of the accuracy of subject detection, the classification of the motion vectors may be incorrect. Therefore, it is preferable to further perform "separation based on the amount of motion vectors" as the second separation process. Here, for example, a known k-means method can be used.

[0067] In the k-means method, it is necessary to determine in advance the number k of clusters to be classified and the initial value Vk of the centroid of each cluster. Regarding the number of clusters, in this case, since it is separated into a background cluster and a subject cluster, k = 2. When there are a plurality of subjects, the number of clusters may be changed according to the number of subjects. Also, a cluster for separating a group of motion vectors that do not belong to either the background or the subject may be provided.

[0068] Regarding the initial value of the centroid of each cluster, the result of the first separation process is utilized. Specifically, for the background vectors obtained in the first separation process, histograms are generated respectively with the amount of movement in the X direction and the amount of movement in the Y direction, and the mode values V1x and V1y of each histogram are obtained. Since this mode value is a representative value of the background vectors, this is set as the initial value V1 = (V1x, V1y) of the centroid of the background cluster. In the same way, the representative values V2x and V2y of the subject vectors obtained in the first separation process are obtained, and the initial value V2 = (V2x, V2y) of the centroid of the subject cluster is set.

[0069] Regarding the initial value of the background cluster, instead of the background vector, the inertial sensor information obtained in step S202 may be used. In that case, the angular velocities in the yaw direction and pitch direction obtained from the angular velocity sensor are integrated to be converted into angles θy and θp, respectively, and then converted into the displacement amounts on the imaging surface. If the focal length is f, then V1x = ftan(θy) and V1y = ftan(θp) are calculated.

[0070] Regarding the initial value of the subject cluster, instead of the subject vector, the displacement amount of the center of gravity between frames of the subject region obtained in step S201 may be used. In this case, it is premised that the same subject can be detected between frames.

[0071] As described above, since the number of clusters and the initial values of the centers of gravity of each cluster are determined, next, the distances between each motion vector data and the centers of gravity of each cluster are calculated. Then, each motion vector data is re-assigned to belong to the cluster with the closest distance.

[0072] In the above processing, when the assignment of clusters for all vector data does not change, or when the amount of change is less than a preset fixed threshold value, it is determined that the processing has converged, and the second separation process is terminated. Otherwise, after recalculating the center of gravity from the newly assigned clusters, the distances between each motion vector data and the centers of gravity of each cluster are calculated, and the above-described processing of re-assigning the clusters is repeated.

[0073] An example of the result of separating the motion vectors as described above is shown in FIG. 6.

[0074] FIG. 6(a) is a diagram in which the displacement amount in the X direction of the motion vector is plotted on the horizontal axis and the displacement amount in the Y direction is plotted on the vertical axis. □ represents the initial value of the center of gravity of the background cluster, ☆ represents the initial value of the center of gravity of the subject cluster, those separated as the background vector are indicated by ○, and those separated as the subject vector are indicated by △.

[0075] Further, FIG. 6(b) plots the starting point positions (feature point positions) of the background vector and the subject vector in FIG. 6(a) on the image. It can be seen from FIG. 6(b) that the background and the main subject can be correctly separated.

[0076] In step S206, the blur correction amount conversion unit 107 converts the background vector and the subject vector obtained in step S205 into a first blur correction amount (hereinafter referred to as a hand shake correction amount) and a second blur correction amount (hereinafter referred to as a subject shake correction amount), respectively.

[0077] As a method for converting the motion vector into a blur correction amount, for example, there is a method using a histogram as described above. Histograms are generated for the amount of movement in the X direction and the amount of movement in the Y direction of the motion vector, respectively, and the mode value of each histogram is calculated. Since this mode value is a representative value of the blur occurring between frames, a blur correction amount that cancels out the blur can be obtained by taking the inverse sign of this mode value.

[0078] Also, as another conversion method, there is a method that uses the correspondence relationship of feature points between frames obtained from the motion vector. A projective transformation matrix (or an affine transformation matrix) is calculated by a known method from the correspondence relationship of feature points between frames. Since the calculated projective transformation matrix represents the blur occurring between frames, a blur correction amount that cancels out the blur can be obtained by calculating the inverse matrix of this matrix.

[0079] In the case of the method using a histogram, only the translational component of the blur can be corrected, but it has the characteristic that a correction amount can be obtained relatively stably even when the number of motion vectors is small. On the other hand, in the case of the method using the correspondence relationship of feature points, it is possible to correct the rotational component and the sway component in addition to the translational component of the blur, but it has the characteristic that the correction amount cannot be correctly obtained when the number of motion vectors is small. Therefore, it may be possible to change which method to use according to the number of motion vectors.

[0080] Here, the background vector is used to obtain the first blur correction amount. However, when an angular velocity sensor is available, it is not always necessary to use the background vector. As described in the section on the method for calculating the initial value of the background cluster, the angular velocity information obtained from the angular velocity sensor may be converted into the displacement amount on the imaging plane and used instead.

[0081] In step S207, the blur correction amount calculation unit 111 generates the final blur correction amount by synthesizing the shake correction amount and the subject shake correction amount obtained in step S206 based on the target of interest obtained in step S203.

[0082] An example of the method for synthesizing the shake correction amount and the subject shake correction amount will be described. Let the background degree be A and the subject degree be 1 - A. If the shake correction amount and the subject shake correction amount calculated in step S206 are H0 and H1 respectively, the final synthesized blur correction amount H is obtained by the following formula.

[0083] H=(A×H0)+{(1 - A)×H1} (4) According to formula (4), when the photographer is focusing 100% on the background, A = 1, so H = H0 and the shake is 100% corrected. Conversely, when the photographer is focusing 100% on the subject, A = 0, so H = H1 and the subject shake is 100% corrected. And when the photographer is focusing 50% on the background and 50% on the subject, A = 0.5, so the shake is 50% corrected and the subject shake is 50% corrected. In this way, the correction ratios of the shake and the subject shake are controlled according to the background degree and the subject degree. Note that, similar to the background degree and the subject degree, the correction ratios of the shake and the subject shake may be made independent in the horizontal and vertical directions.

[0084] In the last step S208, the imaging device 100 determines whether the processing has been completed up to the final frame. If the processing has been completed up to the final frame, the process ends; if not, it returns to step S201.

[0085] As described above, in this embodiment, by controlling the correction ratio of camera shake and subject shake according to the subject of interest of the photographer, the blur correction effect desired by the photographer can be obtained.

[0086] Note that the technology described in this embodiment can be applied to both moving images and still images. In the case of moving images, camera shake correction and subject shake correction can be continuously switched according to the scene, and the discomfort such as minute image shift due to the switching can be suppressed. In the case of still images, an image in which camera shake and subject shake are suppressed with an appropriate balance can be generated. Also, as described above, it can be applied to special still image shooting such as panning.

[0087] (Second Embodiment) FIG. 7 is a diagram showing the configuration of an imaging device 600 according to a second embodiment of the present invention. In FIG. 7, for parts common to the components shown in FIG. 1, the same reference numerals as in FIG. 1 are given and the description is omitted. The imaging device of this embodiment has a reliability calculation unit 601 and a separation degree calculation unit 602 in addition to the configuration shown in FIG. 1. In this embodiment, only the parts that perform different processing from the first embodiment will be described.

[0088] In the first embodiment, the blur correction amount was calculated according to the degree of attention to the background and the main subject. However, depending on the shooting scene, even when the degree of attention is high, it may be difficult to detect motion or separate motions. For example, it is difficult to detect motion in the sky with low contrast or in soft bodies or fluids with drastic shape changes between frames. Also, if the amount of motion is different within the same subject or there is a difference in depth, it is difficult to separate the motions of the background and the main subject. In such cases, if the blur correction amount is calculated simply based on the degree of attention, it is considered that the blur correction will not be performed correctly and will become unstable.

[0089] Therefore, in this embodiment, a method for performing blur correction more stably will be described by taking into account the "reliability of motion detection" and the "degree of motion separation" in addition to the "degree of attention to the background and the main subject".

[0090] The difference between this embodiment and the first embodiment is that the output result of the reliability calculation unit 601 is used in the motion separation unit 106a, and the output results of the reliability calculation unit 601 and the separation degree calculation unit 602 are used in the blur correction amount calculation unit 111a.

[0091] The reliability calculation unit 601 calculates the reliability for the motion vectors input from the motion vector detection unit 105. The separation degree calculation unit 602 calculates the separation degree between the background vector and the subject vector input from the motion separation unit 106a.

[0092] FIG. 8 is a flowchart showing the image blur correction operation in the second embodiment. In FIG. 8, for the steps common to the steps shown in FIG. 2, the same reference numerals as those in FIG. 2 are given and the description is omitted.

[0093] In step S701, the reliability calculation unit 601 calculates the reliability of the motion vectors obtained in step S204.

[0094] To calculate the reliability of the motion vectors, a two-dimensional correlation value map is used. In the two-dimensional correlation value map of FIG. 5(b), the correlation values are arranged in raster order as indicated by the arrow 404, and the one-dimensional representation is shown in FIG. 9. The vertical axis in FIG. 9 is the correlation value, and the horizontal axis is the pixel address uniquely determined by the X coordinate and Y coordinate of the correlation value map. Hereinafter, this representation in FIG. 9 will be used to calculate the reliability of the motion vectors. Note that the point 801 indicates the position corresponding to the minimum value in FIG. 5.

[0095] FIG. 10 is a diagram showing an example of an index of the correlation value representing the reliability of the motion vectors. The horizontal axis in FIG. 10 is the pixel address, and the vertical axis is the correlation value.

[0096] In FIG. 10(a), the difference Da between the minimum value and the maximum value of the correlation value is used as an index. Da represents the range of the correlation value map. When Da is small, it is considered that the contrast of the texture is low, indicating low reliability.

[0097] In FIG. 10(b), the ratio Db (= B / A) of the difference A between the minimum value and the maximum value of the correlation value and the difference B between the minimum value and the average value is used as an index. Db represents the sharpness of the correlation value peak. When Db is small, it is considered that the similarity between the template region and the search region is low, indicating low reliability.

[0098] In FIG. 10(c), the difference Dc between the minimum value and the second minimum value of the correlation value is used as an index. Here, points 901, 902, and 903 respectively correspond to the correlation values 401, 402, and 403 in FIG. 5. Therefore, FIG. 10(c) means checking whether there is a minimum value similar to the minimum of the correlation value in the contour line of FIG. 5(b).

[0099] Dc represents the periodicity of the correlation value map. When Dc is small, it is considered that the texture is a repeating pattern, an edge, etc., indicating low reliability. Here, the minimum value and the second minimum value are selected, but other minimum values may be selected as long as the periodicity of the correlation value map can be determined.

[0100] In FIG. 10(d), the minimum value Dd of the correlation value is used as an index. When Dd is large, it is considered that the similarity between the template region and the search region is low, indicating low reliability. Since Dd and the reliability are in an inverse proportional relationship, the reciprocal (1 / Dd) of Dd is used as an index.

[0101] The index of the correlation value described above can be directly used as the confidence level. However, for example, as shown in FIG. 11, the correlation value index and the confidence level may be associated with each other. The horizontal axis in FIG. 11 represents the correlation value index (any one of the above-mentioned Da, Db, Dc, 1 / Dd), and the vertical axis represents the confidence level. In this example, two threshold values T1 and T2 are set. If it is less than or equal to the threshold value T1, the confidence level is 0, and if it is greater than or equal to the threshold value T2, the confidence level is 1. The threshold value may be changed for each correlation value index. Also, for the interval between the threshold value T1 and the threshold value T2, the correlation value index and the confidence level may be linearly associated or non-linearly associated. In the following description, the confidence levels obtained from each correlation value index are expressed as Ra, Rb, Rc, and Rd. Here, there is a relationship of Ra = f(Da), Rb = f(Db), Rc = f(Dc), Rd = f(Dd).

[0102] The confidence level R of the final motion vector may be calculated by combining these Ra, Rb, Rc, and Rd. Here, a combination method by weighted addition and logical operation will be described.

[0103] In the combination by weighted addition, if the weights of Ra, Rb, Rc, and Rd are Wa, Wb, Wc, and Wd respectively, the confidence level R is calculated as shown in Equation (5).

[0104] R = Wa×Ra + Wb×Rb + Wc×Rc + Wd×Rd (5) For example, let the weights be Wa = 0.4, Wb = 0.3, Wc = 0.2, and Wd = 0.1. When all the confidence levels are sufficiently high and Ra = Rb = Rc = Rd = 1, from Equation (5), R = 1.0. Also, when Ra = 0.6, Rb = 0.5, Rc = 0.7, and Rd = 0.7, from Equation (5), R = 0.6.

[0105] In the combination by logical operation, if the threshold values for Ra, Rb, Rc, and Rd are Ta, Tb, Tc, and Td respectively, the confidence level R is calculated as shown in Equation (6) using, for example, the logical product.

[0106] R = (Ra ≥ Ta) ∧ (Rb ≥ Tb) ∧ (Rc ≥ Tc) ∧ (Rd ≥ Td) (6) Here, ∧ is a symbol representing the logical product. When Ra≥Ta, Rb≥Tb, Rc≥Tc, and Rd≥Td all hold, R = 1 (high reliability); otherwise, R = 0 (low reliability).

[0107] Alternatively, it may be calculated as in Equation (7) using the logical sum.

[0108] R=(Ra<Ta)↓(Rb<Tb)↓(Rc<Tc)↓(Rd<Td) (7) Here, ↓ is a symbol representing the exclusive NOR. When none of Ra<Ta, Rb<Tb, Rc<Tc, and Rd<Td hold, R = 1 (high reliability); otherwise, R = 0 (low reliability).

[0109] In step S702, the motion separation unit 106a separates the motion vector detected in step S204 into a background vector and a subject vector using the reliability calculated in step S701.

[0110] The difference from step S205 is that first, among all the motion vectors, those with a reliability lower than a predetermined threshold are removed. Motion vectors with low reliability have a high possibility of false detection and increase the possibility of incorrect motion separation, so they are not used. After removing the motion vectors with low reliability, the motion separation process is performed in exactly the same way as in step S205.

[0111] In step S703, the separation degree calculation unit 602 calculates a separation degree U representing the degree of separation between the two vectors using the background vector and the subject vector separated in step S702. The aforementioned Figure 6 shows an example of a state with a high separation degree. On the other hand, Figure 12 shows an example of a state with a low separation degree.

[0112] As an example of a method for calculating the separation degree, for example, the variance S of the coordinates of the starting point (= feature point) of the subject vector can be used. When the separated subject vectors capture a subject with a specific movement, the variance S is known to be constant with respect to time changes. Therefore, for example, as shown in FIG. 13, the separation degree U may be calculated such that the larger the displacement amount δS between frames of the variance S, the lower the separation degree U.

[0113] The horizontal axis in FIG. 13 indicates the displacement amount δS between frames of the variance S, and the vertical axis indicates the separation degree U. In this example, two threshold values T1 and T2 are provided. When δS is small and equal to or less than the threshold value T1, as described above, it is considered that the movement of a specific subject can be separated from the background, and the separation degree U is set to 1. Conversely, when δS is large and equal to or greater than the threshold value T2, it is considered that the movement of a specific subject cannot be separated from the background, and the separation degree U is set to 0. In the interval between the threshold values T1 and T2, δS and the separation degree U may be linearly associated or non-linearly associated.

[0114] In step S704, the blur correction amount calculation unit 111a generates a final blur correction amount by synthesizing the camera shake correction amount and the subject shake correction amount obtained in step S206 based on the target of interest obtained in step S203, the reliability of the motion vector detection obtained in step S701, and the separation degree of the motion obtained in step S703.

[0115] The difference from step S207 is that the reliability of the motion detection and the separation degree of the motion are reflected as the synthesis ratio of the blur correction amount. Let the total values of the reliability of the background vector and the subject vector be RA and RB, respectively, and the separation degree be U. As an example of the reflection method, for example, a method of calculating the blur correction amount as shown in the following formula (8) can be considered. H = A·RA·(1 - U)·H0 / {A·RA·(1 - U)+(1 - A)·RB·U} +(1 - A)·RB·U·H1 / {A·RA·(1 - U)+(1 - A)·RB·U} (8) According to formula (8), the higher the reliability RA of the background vector is compared to the reliability RB of the object vector, the higher the ratio of background blur correction. Also, the higher the degree of separation U, the higher the ratio of object blur correction.

[0116] As described above, in this embodiment, by taking into account the "reliability of motion vector detection" and the "separability of motion" in addition to the "degree of attention to the background and main subject", it is possible to obtain the effect of performing blur correction more stably than in the first embodiment.

[0117] (Third embodiment) Fig. 14 is a diagram showing the configuration of an image capture device 1300 according to a third embodiment of the present invention. In Fig. 14, parts common to the components shown in Fig. 1 are given the same reference numerals as in Fig. 1 and description thereof will be omitted. In addition to the configuration shown in Fig. 1, the image capture device of this embodiment has a shake correction effect determination unit 1301 and a display unit 1302. In this embodiment, only parts that perform processing different from the first embodiment will be described.

[0118] In the first embodiment, the correction ratio of camera shake and subject shake is controlled according to the object of the photographer's attention. However, when the amount of blur caused by camera shake and subject shake is roughly equal, the photographer cannot know which type of blur correction is working. Cameras in recent years are equipped with a camera shake correction function as standard, but many do not yet have a subject shake correction function. Therefore, there are many photographers who are familiar with camera shake correction but not with subject shake correction. It is very important to accurately inform such photographers of the conditions under which subject shake correction is working.

[0119] Therefore, in this embodiment, a method of determining a situation in which the effect of subject shake correction is prominent and notifying the photographer of that situation will be described.

[0120] The difference between this embodiment and the first embodiment is that the shake correction effect determination unit 1301 determines the effect of subject shake correction using the outputs of the target-of-interest estimation unit 110 and the shake correction amount conversion unit 107, and the display unit 1302 notifies the photographer of the determination result.

[0121] The shake correction effect determination unit 1301 determines the effect of subject shake correction based on the information of the target of interest obtained by the target-of-interest estimation unit 110 and the hand shake correction amount and subject shake correction amount obtained from the shake correction amount conversion unit 107. The display unit 1302 displays the determination result of the shake correction effect input from the shake correction effect determination unit 1301 together with the image data obtained from the development processing unit 103.

[0122] FIG. 15 is a flowchart showing the processing operation of the third embodiment. In FIG. 15, steps common to the steps shown in FIG. 2 are denoted by the same reference numerals as in FIG. 2, and the description thereof is omitted.

[0123] In step S1401, the shake correction effect determination unit 1301 determines the effect of subject shake correction using the target of interest obtained in step S203 and / or the hand shake correction amount and subject shake correction amount obtained in step S206.

[0124] As a method for determining the effect of subject shake correction, for example, the higher the degree of subject, which indicates the degree to which the target of interest is the subject, and the larger the absolute value of the difference between the hand shake correction amount and the subject shake correction amount, the higher the effect of subject shake correction is determined. Note that the degree of subject and the absolute value of the difference do not necessarily have a linear relationship with the degree of height of the correction effect. When the degree of subject is equal to or greater than a predetermined value, it may be determined that the effect of subject shake correction is higher than when it is less than the predetermined value, or when the absolute value of the difference between the hand shake correction amount and the subject shake correction amount is equal to or greater than a predetermined value, it may be determined that the effect of subject shake correction is high.

[0125] In addition, as information for determining the effect of subject shake correction, the reliability of subject information, the reliability of the subject vector, the separation degree between the background vector and the subject vector, and the upper limit of the shake correction amount may be taken into account.

[0126] The reliability of the subject information is the reliability of the subject information obtained from the subject detection unit 108. When a specific subject is a person's face, the reliability of face detection may be used. If the reliability of the subject information is low, subject shake correction may not operate correctly. Therefore, it is determined that the higher the reliability of the subject information, the higher the effect of subject shake correction compared to when it is low.

[0127] The reliability of the subject vector can be obtained from the reliability calculation unit 601 as described in the second embodiment. If the reliability of the subject vector is low, subject shake correction may not operate correctly. Therefore, it is determined that the higher the reliability of the subject vector, the higher the effect of subject shake correction compared to when it is low.

[0128] The separation degree between the background vector and the subject vector can be obtained from the separation degree calculation unit 602 as described in the second embodiment. If the separation degree between the background vector and the subject vector is low, subject shake correction may not operate correctly. Therefore, it is determined that the higher the separation degree between the background vector and the subject vector, the higher the effect of subject shake correction compared to when it is low.

[0129] The upper limit of the blur correction amount is the upper limit of the blur amount that can be corrected by the above-described optical blur correction means or electronic blur correction means. In the case of the optical blur correction means, the upper limit is determined by the movable range of the correction lens or the imaging element within the optical system. In the case of electronic blur correction, the upper limit is determined by the size of the surplus pixel area provided around the output area of the image. If the subject shake correction amount exceeds the upper limit of the blur correction amount, subject shake correction may not operate correctly. Therefore, it is determined that the larger the value obtained by subtracting the subject shake correction amount from the upper limit of the blur correction amount, the higher the effect of subject shake correction compared to when it is small.

[0130] Based on the multiple pieces of information shown above, the effect of subject shake correction is determined. The method of determination from the multiple pieces of information is the same as the method of estimating the target of interest by the aforementioned formula (2), so the description is omitted. Finally, when the value indicating the effect of subject shake correction is greater than a predetermined threshold, it is determined that there is an effect of subject shake correction, and when it is less than or equal to the predetermined threshold, it is determined that there is no effect of subject shake correction. At that time, in order to prevent the determination result from frequently switching, it is preferable to switch the determination result when the state of being greater than or less than the predetermined threshold continues for a predetermined time or more.

[0131] Next, in step S1402, the display unit 1302 displays the determination result of the blur correction effect obtained in step S1401 together with the image data obtained from the development processing unit 103. What is to be displayed is a display item such as text or an icon, and there is no particular limitation.

[0132] FIG. 16 shows an example of a display item corresponding to the determination result of the blur correction effect. FIG. 16(a) shows an icon indicating that the hand shake correction is effective (hereinafter referred to as the hand shake correction icon), and FIG. 16(b) shows an icon indicating that the subject shake correction is effective (hereinafter referred to as the subject shake correction icon).

[0133] For example, when it is determined in step S1401 that there is no effect of subject shake correction, the display unit 1402 displays the hand shake correction icon of FIG. 16(a) at a specific location within the screen. On the other hand, when it is determined in step S1402 that there is an effect of subject shake correction, the display unit 1402 displays the subject shake correction icon of FIG. 16(b) at a specific location within the screen. By switching the display item in this way, the photographer can accurately distinguish the situation where the hand shake correction is effective from the situation where the subject shake correction is effective.

[0134] The following describes the display method of display items. Regarding shake correction, since it corrects the blur of the entire background, the shake correction icon is generally displayed at a fixed position within the screen. However, subject shake correction corrects the blur of a specific subject, and there may be more than one subject within the screen. Therefore, a display method different from that of the normal shake correction icon is required for the subject shake correction icon.

[0135] Fig. 17 shows an example of the display method of the subject shake correction icon. In Fig. 17, three persons (1601, 1602, 1603) exist as subjects within the screen. Now, consider the case where the subject of subject shake correction (the target of blur correction) is the person 1602 at the center of the screen.

[0136] In Fig. 17(a), the subject shake correction icon 1604 is displayed in the vicinity of the person 1602 who is the target of subject shake correction. By dynamically changing the display position of the subject shake correction icon in this way, it is possible to convey to the photographer that the subject shake of the person 1602 has been corrected.

[0137] In Fig. 17(b), the subject shake correction icon 1604 is displayed at a fixed position in the upper left of the screen. The detection frame 1605 of the person 1602 is displayed on the person 1602 who is the target of subject shake correction. By combining the subject shake correction icon and the detection frame of the subject in this way, it is possible to convey to the photographer that the subject shake of the person 1602 has been corrected. Note that the detection frame of the subject may surround some parts such as the face of the subject, or may surround the entire subject.

[0138] In FIG. 17(c), the detection frame for person 1601 is indicated by 1606, the detection frame for person 1602 by 1607, and the detection frame for person 1603 by 1608. In this case, only the detection frame 1607 for person 1602, who is the target of subject shake correction, is displayed with a design different from the other detection frames 1606 and 1608. In this example, the detection frame 1607 is distinguished from the other detection frames by drawing a double line. Alternatively, only the detection frame 1607 may be distinguished from the other detection frames by changing its color, shape, or size. By changing the design of the subject detection frames in this way, it is possible to inform the photographer that the subject shake of person 1602 has been corrected.

[0139] As described above, in this embodiment, compared to the first embodiment, an advantage is obtained in that the photographer can be accurately notified of a situation in which the effect of subject shake correction is more noticeable.

[0140] (Fourth embodiment) In this embodiment, an embodiment in which both effective use of the correction stroke and blur correction performance are improved will be described.

[0141] 18 is a diagram showing the configuration of an image pickup apparatus according to the fourth embodiment of the present invention. Description of the configuration common to the image pickup apparatus according to the first embodiment shown in FIG.

[0142] In FIG. 18, the inertial sensor 112 is a sensor that detects the shake of the imaging device 100, and an angular velocity sensor or the like is used. The output processing unit 113 converts the output signal of the inertial sensor (hereinafter referred to as the angular velocity sensor) 112 into a third blur correction amount. The blur correction amount calculation unit 111 uses the information of the target of interest obtained by the target of interest estimation unit 110, the first and second blur correction amounts obtained by the blur correction amount conversion unit 107, and the third blur correction amount obtained by the output processing unit 113 to generate an optical blur correction amount and an electronic blur correction amount. The generated optical blur correction amount is output to a shift mechanism that shifts the correction lens and / or the imaging element 102 in the optical system 101, which is an optical blur correction means, in a direction perpendicular to the optical axis, and optical blur correction is performed. The generated electronic blur correction amount is output to the electronic blur correction unit 114, and electronic blur correction for controlling the position of the subject near a predetermined position is performed.

[0143] Next, the image blur correction operation in the imaging device 100 configured as described above will be described using the flowchart shown in FIG. 19. Steps S201 to S205 and step S208 are the same as those in the first embodiment, so only the flow of these steps will be described.

[0144] In step S201, the subject detection unit 108 detects a specific subject from the image data input from the development processing unit 103 and outputs subject information.

[0145] In step S202, the camera information acquisition unit 109 acquires camera information necessary for estimating the shooting situation.

[0146] In step S203, the target of interest estimation unit 110 estimates whether the photographer is focusing on the background or the subject as the target of interest based on the subject information detected in step S201 and the camera information acquired in step S202.

[0147] In step S204, the motion vector detection unit 105 detects a motion vector for the image input from the development processing unit 103 and the image memory 104 by template matching.

[0148] In step S205, the motion separation unit 106 separates the motion vector detected in step S204 into a background vector and a subject vector.

[0149] In step S1906, the blur correction amount conversion unit 107 converts the background vector into a first blur correction amount (hereinafter referred to as a hand shake correction amount) and the subject vector into a second blur correction amount (hereinafter referred to as a subject shake correction amount) in the same manner as in step S206 of FIG. 2. At the same time, the output processing unit 113 converts the output signal of the angular velocity sensor 112 into a third blur correction amount. In the output processing unit 113, for the output signal of the angular velocity sensor 112, after removing its offset component, integration processing is performed to convert the angular velocity information into a displacement amount on the imaging surface. Then, by taking the inverse sign thereof, it is converted into a third blur correction amount that cancels the blur.

[0150] In step S1907, the blur correction amount calculation unit 111 obtains a hand shake correction amount and a subject shake correction amount from the first, second, and third blur correction amounts obtained in step S1906 based on the target of interest obtained in step S203. Further, a blur correction amount is generated by synthesizing the hand shake correction amount and the subject shake correction amount.

[0151] Note that the subject shake correction amount directly uses the second blur correction amount calculated from the subject vector. The hand shake correction amount is obtained using the first blur correction amount calculated from the background vector and the third blur correction amount calculated from the information of the angular velocity sensor 112. This method generally uses frequency division.

[0152] The third blur correction amount is advantageous for detecting high-frequency blur with a short sample period, but detection errors due to the offset error and drift of the angular velocity sensor occur in the low-frequency region. This error becomes larger at high temperatures. The first blur correction amount is disadvantageous for detecting high-frequency blur with a long sample period, but detection errors due to offset error and drift do not occur. However, in the dark, the error is likely to occur because the SNR of the imaging device 102 decreases.

[0153] Therefore, filter processing is performed using a low-pass filter and a high-pass filter with a cut-off frequency of about 1 Hz. The addition result of the output obtained by applying the low-pass filter to the first blur correction amount and the output obtained by applying the high-pass filter to the third blur correction amount is used as the shake correction amount. Also, at high temperatures, the output obtained by applying the low-pass filter to the first blur correction amount may be used as the shake correction amount, and in the dark, the output obtained by applying the high-pass filter to the third blur correction amount may be used as the shake correction amount.

[0154] The method of synthesizing the shake correction amount and the subject shake correction amount in step S1907 is the same as that in step S207 of FIG. 2, so a detailed description thereof will be omitted. For example, the final synthesized blur correction amount H can be obtained using the above-described formula (4).

[0155] In step S1909, the synthesized blur correction amount H calculated in step S1907 is allocated to the correction lens, the shift mechanism of the imaging device, and image cropping, and blur correction control for correcting the blur occurring in the image is performed. This blur correction control is performed by outputting a blur correction control signal to the correction lens and the shift mechanism of the imaging device to drive them. At the same time, a control signal is output to the electronic blur correction unit 114, and electronic blur correction is performed by executing control of the image cropping position (electronic blur correction control) for cutting out a part of the image signal output from the development processing unit 103 to generate a new image signal. Then, the output image from the imaging device 102 is sent to an image signal recording means or an image display means (not shown).

[0156] In the last step S208, the imaging device 100 determines whether the processing has been completed up to the final frame. If the processing has been completed up to the final frame, the processing is terminated. If not, the process returns to step S201.

[0157] Here, the shake correction control in step S1909 will be further described.

[0158] Even when blurring caused by normal hand shake or subject shake can be corrected, when the panning of the photographer is delayed with respect to the high-speed movement of the subject, or when the main subject (correction target) is temporarily hidden by another subject, etc., the main subject may move significantly on the screen. In such a case, if an attempt is made to keep the correction target (main subject) at a predetermined position on the screen, the correction stroke will be insufficient and it will be impossible to correct the blur generated in the correction target.

[0159] Therefore, in the present embodiment, in order to enable both the elimination of the shortage of the correction stroke and the subject shake correction, simply applying a correction limit when the correction stroke reaches the end is not done. Instead, when the main subject (correction target) moves significantly, its movement is tolerated to some extent, and according to the remaining amount of the correction stroke, a filter that waits for a high-pass effect is applied to the correction amount to create a correction signal that emphasizes high-frequency shake correction. Also, the output gain is made variable.

[0160] Hereinafter, a method for generating a shake correction control signal will be described. In the following description, for the sake of easy understanding, the case of performing shake correction for only one axis will be described as an example. In reality, there are a plurality of correction axes, and shake correction is performed by performing the same control for each axis. It is also possible to allocate each correction means for each axis. As an example, all the shake corrections in the roll direction may be performed by the shift mechanism of the imaging element, and the shake corrections in the pitch direction and the yaw direction may be performed by the correction lens.

[0161] Using FIG. 20, a method for generating a shake correction control signal will be described.

[0162] In step S2001, it is determined whether or not a predetermined time has elapsed since the coefficients Fc and the gain β of the high-pass filter for the shake correction amount were obtained. This is done by checking whether the difference between the time when the coefficients Fc and the gain β were set in step S2012 or S2013 (to be described later) and the current time is equal to or more than the predetermined time. If the predetermined time has elapsed, the process proceeds to step S2002; if less than the predetermined time, the process proceeds to step S2005.

[0163] In step S2002, after resetting the time at which the coefficients Fc and the gain β were set, the shake correction amount to be corrected by the correction lens, the shift mechanism of the imaging device 102, and the electronic shake correction unit 114 is calculated (the allocation process of the shake correction amount H is performed).

[0164] In calculating the correction amount shared by each of these mechanisms, settings are made such that a situation where the correction cannot be performed due to insufficient stroke for shake correction does not occur even if the correction amount increases. Specifically, the coefficient Fcc and the gain βc of the high-pass filter for the correction amount by controlling the image cutout position, and the coefficient Fcls of the high-pass filter for the third shake correction amount calculated from the information of the angular velocity sensor 112 and the gains βl and βs of the correction lens and the shift mechanism are set.

[0165] When the shake correction amount H is small (case 1), the image cutout position is fixed at the center, and shake correction is performed by driving the correction lens and the shift mechanism. This is because the optical performance near the center of the screen is superior to that of the periphery and there is almost no shading, so an image with higher resolution and no left-right luminance difference can be obtained.

[0166] When the shake correction amount H increases and approaches the maximum amount that can be corrected by the correction lens and the shift mechanism (case 2), by controlling the image cutout position, the correction amount by the correction lens and the shift mechanism is reduced to avoid a shortage of the shake correction stroke. This enables good shake correction.

[0167] When the shake correction amount H further increases (Case 3), the coefficient Fcc and the gain βc of the high-pass filter with respect to the correction amount by controlling the image cutout position are set. Also, the coefficient Fcls and the gains βl and βs of the high-pass filter for the third shake correction amount calculated from the information of the angular velocity sensor 112 are set. By doing so, the sensitivity of the shake correction for low-frequency shakes is reduced, the stroke shortage of the shake correction is avoided, and good shake correction is enabled.

[0168] Here, in the present embodiment, the coefficient Fcc and the gain βc of the high-pass filter with respect to the correction amount by controlling the image cutout position, and the coefficient Fcls and the gains βl and βs of the high-pass filter for the third shake correction amount are switched according to each other's signals. However, in step S2002, they are first set to the optimum values individually. Below, the method of setting the coefficients Fcc and Fcls and the gains βc, βl, and βs of the high-pass filter to the optimum values individually in step S2002 will be described.

[0169] The correction amount by controlling the image cutout position and the third shake correction amount calculated from the information of the angular velocity sensor 112 are passed through a high-pass filter, and by changing the coefficients Fcc and Fcls of this high-pass filter, the effect can be changed. Also, by multiplying the output by the gains βc, βl, and βs with a maximum value of 100%, it is possible to provide a limit on the control level.

[0170] The correction amount Hl by controlling the correction lens, the correction amount Hs by controlling the shift mechanism of the imaging element 102, and the correction amount Hc by controlling the image cutout position of the electronic shake correction unit 114 are calculated from the correction amount H as follows.

[0171] (Case 1) When H ≦ (Hlmax + Hsmax) * αls Hl = H × Hlmax / (Hlmax + Hsmax) Hs = H × Hsmax / (Hlmax + Hsmax) (Case 2) When H > (Hlmax + Hsmax) * αls and When H ≦ (Hlmax + Hsmax) * αls + Hcmax * αc Hc = H - (Hlmax + Hsmax) * αls Hl = Hlmax * αls Hs = Hsmax * αls (Case 3) When H > (Hlmax + Hsmax) * αls + Hcmax * αc Hc = Hcmax * αc + Hover * Hcmax / Hmax Hl = Hlmax * αls / 2 + Hover * Hlmax / Hmax Hs = Hsmax * αls / 2 + Hover * Hsmax / Hmax However Hover = H - {(Hlmax + Hsmax) * αls + Hcmax * αc} Hmax = Hlmax + Hsmax + Hcmax Note that Hlmax, Hsmax, and Hcmax are the maximum values of the correction amount Hl by the control of the correction lens, the correction amount Hs by the control of the shift mechanism of the imaging device, and the correction amount Hc by the control of the image cutout position, that is, the correction stroke. αls and αc are ratios to the correction stroke and are predetermined values used for dividing the final shake correction amount and setting the correction values.

[0172] And in (Case 3), according to the correction amount Hl, the correction amount Hs, and the correction amount Hc, the coefficient Fcc of the high - pass filter of the correction amount Hc by the control of the image cutout position, the coefficient Fcls of the high - pass filter of the third shake correction amount, the gain βl of the correction amount Hl, the gain βs of the correction amount Hs, and the gain βc of the correction amount Hc are set. Specifically, they are set as follows.

[0173] When Hover is less than Pe1(%) of [Hmax - {(Hlmax + Hsmax) * αls + Hcmax * αc}] Fcc = Fcc1(Hz) > Fcc0(Hz), βc = βc1(%) < 100% Fcls = Fcls0(Hz), βl = βs = 100% When Hover is equal to or greater than [Hmax - {(Hlmax + Hsmax) * αls + Hcmax * αc}] Pe1 (%) and less than Pe2 (%). Fcc = Fcc2 (Hz) > Fcc1 (Hz), βc = βc2 (%) < βc1 (%) Fcls = Fcls2 (Hz) > Fcls1 (Hz), βl = βs = βls2 (%) < 100% When Hover is equal to or greater than Pe2 (%) of [Hmax - {(Hlmax + Hsmax) * αls + Hcmax * αc}]. Fcc = Fcc3 (Hz) > Fcc2 (Hz), βc = βc3 (%) < βc2 (%) Fcls = Fcls3 (Hz) > Fcls1 (Hz), βl = βs = βls3 (%) < βls2 (%) However, Fcc is the coefficient of the high - pass filter for the correction amount by controlling the image cut - out position, βc is the gain of the correction amount Hc, Fcls is the coefficient of the high - pass filter for the third blur correction amount, and βl, βs are the gains of the correction amounts Hl, Hs. Note that Fcc0 (Hz) is the minimum value of the coefficient Fcc of the high - pass filter, and Fcls0 (Hz) is the minimum value of the coefficient Fcls of the high - pass filter.

[0174] In addition, for the cases of (Case 1) and (Case 2), the coefficients and gains of the high - pass filter are Fcc = Fcc0 Hz (minimum value), βc = 100%, Fcls = Fcls0 (Hz), βl = βs = 100%.

[0175] Next, proceed to step S2003 to determine whether the imaging device 100 is in a panning state. If a large blur occurring at the start or in the first half of panning is detected from the output of the angular velocity sensor 112 or the vector detection result of the motion vector detection unit 105, it is determined that the device is in a panning state and proceed to step S2004. If panning is not detected, proceed to step S2006.

[0176] In step S2004, increase the value of the coefficient Fcc of the high-pass filter for the correction amount by controlling the image cutting position so that the ratio of the correction amount of higher frequencies becomes larger. For example, if the value before the change is Fcc0 (Hz), change it to Fcc1 (Hz), if it is Fcc1 (Hz), change it to Fcc2 (Hz), if it is Fcc2 (Hz), change it to Fcc3 (Hz), and do not change it when it is Fcc3 (Hz).

[0177] At the same time, decrease the value of the gain βc of the correction amount by controlling the image cutting position. If the value before the change is βc = 100%, change it to βc1 (%), if it is βc1 (%), change it to βc2 (%), if it is βc2 (%), change it to βc3 (%), and do not change it when it is βc3 (%).

[0178] This can reduce the influence of a sudden increase in the signal of the angular velocity sensor 112 due to panning or the like.

[0179] Also, since the signal of the angular velocity sensor 112 reflects the movement of the camera such as panning, it is delayed compared to the motion vector that reflects the movement of the subject. In order to reduce the influence of this delay in high-speed camera movement, based on the values of the coefficient Fcc and the gain βc for the correction amount by controlling the image cutting position set at that time, the coefficient Fcls for the third correction amount, the gain βl of the correction amount Hl by the correction lens, and the gain βs of the correction amount Hs by the shift mechanism are reset according to FIG. 21.

[0180] If the coefficient Fcc of the correction amount by controlling the image cutting position is Fcc1 (Hz) (the gain in this case is βc1), for the coefficient Fcls of the high-pass filter of the third blur correction amount, select the higher cut-off frequency between Fcls0 and the set value at that time. For the gain βl of the correction amount Hl by controlling the correction lens and the gain βs of the correction amount Hs by controlling the shift mechanism, select the lower value between 100% and the set value at that time. The same applies to other cases.

[0181] Thereafter, it proceeds to step S2005, and according to the set value, it outputs a correction control signal to the correction lens of the optical system 101, the shift mechanism of the imaging element 102, and the electronic shake correction unit 114 to perform shake correction.

[0182] On the other hand, when proceeding from step S2003 to S2006, it determines whether it is near the end of panning. If it is near the end, it proceeds to step S2007; if not, it proceeds to step S2010. The determination of whether it is near the end of panning is performed by observing the output of the angular velocity sensor 112 and the vector detection result in time series.

[0183] When a large shake occurring at the start or in the first half of panning is detected from the output of the angular velocity sensor 112 and the vector detection result, and then the absolute value of the signal indicating panning decreases to a predetermined value or less and its slope becomes a predetermined slope or less, it is determined that it is near the end of panning. Conversely, when it is above the predetermined value, it is determined that it is not likely to be near the end of panning. The predetermined value of the absolute value and the predetermined slope value may be predetermined values, or may be calculated each time from the signal detecting panning. The predetermined value of the absolute value may be a predetermined multiple (less than 1) of the maximum value of the signal during panning, and the predetermined slope value may be a predetermined multiple (less than 1) of the average slope to the maximum value of the signal during panning.

[0184] In step S2007, the coefficient Fcls of the third correction amount is decreased to perform correction of lower-frequency shake. For example, if the value of Fcls before the change is Fcls3 (Hz), it is changed to Fcls2 (Hz); if it is Fcls2 (Hz), it is changed to Fcls1 (Hz); if it is Fcls0 (Hz), it is not changed.

[0185] At the same time, the gain βl of the correction amount Hl by the control of the correction lens and the gain βs of the correction amount Hs by the control of the shift mechanism are increased. If the values of βl and βs before the change are βl3 (%) and βs3 (%), they are changed to βl2 (%) and βs2 (%); if they are βl2 (%) and βs2 (%), βl = βs = 100%; if βl and βs are 100%, they are not changed. Thereby, it is possible to perform correction of lower-frequency shake even near the end of panning.

[0186] In addition, since detecting a motion vector requires images of one or more frames, it takes time for detection. In order to reduce this influence during low-speed camera movement such as tracking a subject with small movement, based on the value of Fcls for the third correction amount set at that time, the gain βl of the correction amount Hl by controlling the correction lens, and the gain βs of the correction amount Hs by controlling the shift mechanism, the coefficient Fcc of the correction amount and the gain βc by controlling the image cut-out position are reset according to FIG. 22.

[0187] If the coefficient Fcls of the high-pass filter for the third shake correction amount is Fcls2 (Hz) (in this case, the gains βl and βs of the correction lens and the shift mechanism are βls2), Fcc of the correction amount by controlling the image cut-out position selects the lower cut-off frequency between Fcc2 and the set value at that time. In that case, the gain βc selects the higher value between βc2 and the set value at that time. The same applies to other cases.

[0188] On the other hand, if it is determined in step S2006 that panning is not near the end, in step S2010, integration of the vector detection results is performed. Note that resetting of the integration result of the vectors is performed in step S2004.

[0189] Thereafter, it proceeds to step S2011, and determines whether the absolute value of the vector integration value (electronic shake correction amount) is decreasing and the absolute value is smaller than a predetermined value. If the condition is satisfied, it proceeds to step S2012, and if not, it proceeds to step S2013. However, if the predetermined time has not elapsed since the start of vector integration, it proceeds to step S2005 without performing the following processing. This is because the integration value for which the predetermined time has not elapsed since the start of integration has low reliability.

[0190] In step S2012, based on the coefficient Fcls of the high-pass filter for the third correction amount, the coefficient Fcc of the high-pass filter for the correction amount by controlling the image cut-out position and the setting of the gain βc are performed. At that time, the coefficient Fcls of the third correction amount is made lower than the coefficient Fcc of the correction amount by controlling the image cut-out position.

[0191] If the absolute value of this vector integration value (electronic blur correction amount) is small and decreasing, it is considered that the subject movement can be followed, so a setting enabling blur correction in a wide frequency band is performed.

[0192] First, compare the coefficient Fcls and the coefficient Fcc. If the coefficient Fcls is not lower than the coefficient Fcc, perform the following changes to the coefficient Fcc to make it lower. However, no change is made when the coefficient Fcls is already the lowest frequency Fcls0 (Hz).

[0193] When Fcls ≧ Fcc, If Fcls = Fcls0, then Fcc = Fcc1 If Fcls = Fcls1, then Fcc = Fcc2 If Fcls = Fcls2, then Fcc = Fcc3 If Fcls = Fcls3, then Fcc = Fcc3 Make the change as follows.

[0194] Then, the gain βc is changed according to the changed coefficient Fcc. If Fcc = Fcc1, then βc = βc1 If Fcc = Fcc2, then βc = βc2 If Fcc = Fcc3, then βc = βc3 Make the change as follows.

[0195] Then, in step S2013, it is determined whether the absolute value of the vector integration value (electronic blur correction amount) is increasing and whether the absolute value is greater than a predetermined value. If the condition is satisfied, proceed to step S2014; if not, proceed to step S2005.

[0196] In step S2014, based on the coefficient Fcc of the correction amount by controlling the image cutout position, the coefficient Fcls of the third correction amount and the gains βl, βs of the correction lens and the shift mechanism are set. At this time, the coefficient Fcls of the third correction amount is made higher than Fcc of the correction amount by controlling the image cutout position.

[0197] When the absolute value of this vector integrated value (electronic blur correction amount) increases significantly, it is considered that the subject's movement cannot be followed, so the ratio of the correction amount of higher frequencies is increased.

[0198] First, compare the coefficient Fcls and the coefficient Fcc. If the coefficient Fcls is not higher than the coefficient Fcc, make the following changes to increase it. However, no changes are made when Fcls is already the lowest frequency Fcls0 (Hz).

[0199] When Fcls ≤ Fcc, If Fcc = Fcc0 or Fcc = Fcc1, then Fcls = Fcls2 If Fcc = Fcc2, then Fcls = Fcls3 If Fcc = Fcc3, then Fcls = Fcls3 Make the change as follows.

[0200] And the gains βl and βs follow the changed Fcls, If Fcls = Fcls2, then βl = βs = βls2 If Fcls = Fcls3, then βl = βs = βls3 Make the change as follows.

[0201] Also, according to the gain βc of the correction amount by controlling the image cut-out position, the viewfinder display may be changed to warn the photographer. An LED or the like in the viewfinder that warns that the margin of the correction stroke is decreasing may be lit or blinked, or after making the periphery of the viewfinder semi-transparent, the semi-transparent part may be darkened according to the decrease in the margin of the correction stroke.

[0202] In addition, by observing in advance the phase difference between the output of the angular velocity sensor 112 and the vector detection result, it is possible to improve the blur correction performance. Since the detection of the vector requires one or more frames, a detection delay occurs. Therefore, calculate the delay amount in advance. Relate the calculated value and the measured value of the hand shake and the vector delay amount from the output of the angular velocity sensor. As a result, it becomes possible to calculate the vector integration value (electronic blur correction amount) considering the detection delay. Therefore, it is possible to more accurately set the coefficient Fcc and gain of the correction amount, the coefficient Fcls of the third correction amount, and the gains βl and βs of the correction lens and the shift mechanism by controlling the image cutout position using the vector integration value (electronic blur correction amount).

[0203] The signals with the coefficients Fc and gain β thus obtained are output to the respective correction mechanisms, and blur correction is performed.

[0204] As described above, in the present embodiment, by controlling the correction ratio of the hand shake and the subject shake according to the subject of interest of the photographer, it is possible to obtain the blur correction effect desired by the photographer.

[0205] Furthermore, the coefficients and gains of the high-pass filters of the correction amount by controlling the image cutout position and the third blur correction amount are switched according to each other's signals. As a result, it is possible to avoid the lack of stroke of the blur correction and achieve both the prevention of frame out of the main subject and the blur correction that has a great influence on image degradation.

[0206] Note that the technology described in the present embodiment can be applied to both moving images and still images. In the case of a moving image, the hand shake correction and the subject shake correction can be continuously switched according to the scene, and the discomfort such as a minute image shift due to the switching can be suppressed. In the case of a still image, an image in which the hand shake and the subject shake are suppressed with an appropriate balance can be generated. It can also be applied to a special still image shooting called panning.

[0207] (Other Embodiments) Furthermore, the present invention can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that implements one or more functions.

[0208] The invention is not limited to the above-described embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, the claims are appended to disclose the scope of the invention.

Explanation of Reference Numerals

[0209] 100: Imaging device, 101: Optical system, 102: Image sensor, 103: Development processing unit, 104: Image memory, 105: Motion vector detection unit, 106: Motion separation unit, 107: Shake correction amount conversion unit, 108: Subject detection unit, 109: Camera information acquisition unit, 110: Target-of-interest estimation unit, 111: Shake correction amount calculation unit, 601: Reliability calculation unit, 602: Separation degree calculation unit, 1301: Shake correction effect determination unit, 1302: Display unit

Claims

1. Subject detection means for detecting a specific subject with respect to an input image and outputting subject information; Camera information acquisition means for acquiring camera information necessary for estimating a shooting situation; Estimation means for estimating a target of interest in the image using the subject information and the camera information; First motion detection means for detecting the motion of the background and the motion of the subject in the input image; Conversion means for converting the motion of the background and the motion of the subject detected by the first motion detection means into a first blur correction amount for correcting the blur of the background and a second blur correction amount for correcting the blur of the subject, respectively; Correction amount calculation means for synthesizing the first blur correction amount and the second blur correction amount based on the target of interest estimated by the estimation means to generate a final blur correction amount; Comprising: The camera information includes at least any one of information indicating a shutter speed, information indicating an AF area, information indicating a focal length, information indicating a user operation, a detection result of an inertial sensor, information indicating a distance to a subject, and a detection result of a user's line of sight. An image blur correction device characterized by this.

2. The first motion detection means detects a motion vector for the input image and separates the motion vector into a first motion vector representing the motion of the background and a second motion vector representing the motion of the subject. The conversion means converts the first motion vector and the second motion vector into the first blur correction amount and the second blur correction amount, respectively. The image blur correction device according to claim 1, characterized by this.

3. Further comprising acquisition means for acquiring a signal from an angular velocity sensor that detects shake of the imaging device. The first motion detection means detects a motion vector for the input image and separates the motion vector into a first motion vector representing the motion of the background and a second motion vector representing the motion of the subject. The conversion means converts the shake detected by the angular velocity sensor and the second motion vector into the first blur correction amount and the second blur correction amount, respectively. The image blur correction device according to claim 1, characterized by this.

4. Further comprising reliability calculation means for calculating the reliability of the motion vector and separation degree calculation means for calculating the separation degree between the first motion vector and the second motion vector; The correction amount calculation means generates a final blur correction amount by synthesizing the first blur correction amount and the second blur correction amount based on the target of interest estimated by the estimation means, the reliability of the motion vector calculated by the reliability calculation means, and the separation degree between the first motion vector and the second motion vector calculated by the separation degree calculation means. The image blur correction apparatus according to claim 2 or 3.

5. The subject information includes at least any one of the position and size of the subject, the human-likeness of the subject, and the motion of the subject. The image blur correction apparatus according to any one of claims 1 to 4.

6. The estimation means calculates at least either a background degree indicating the degree to which the target of interest is a background or a subject degree indicating the degree to which the target of interest is a subject based on at least one of the subject information and the camera information. The image blur correction apparatus according to any one of claims 1 to 5.

7. The correction amount calculation means generates a final blur correction amount by weighted addition of the first blur correction amount and the second blur correction amount based on at least either the background degree or the subject degree. The image blur correction apparatus according to claim 6.

8. The correction amount calculation means increases the weight of the first blur correction amount as the background degree increases. The image blur correction apparatus according to claim 7.

9. The correction amount calculation means increases the weight of the second blur correction amount as the subject degree increases. The image blur correction apparatus according to claim 7 or 8.

10. The correction amount calculation means generates a final blur correction amount by weighted addition of the first blur correction amount and the second blur correction amount based on the reliability of the motion vector. The image blur correction apparatus according to claim 4.

11. The correction amount calculation means increases the weight of the first blur correction amount as the reliability of the first motion vector becomes higher than the reliability of the second motion vector. The image blur correction apparatus according to claim 10.

12. The correction amount calculation means generates a final shake correction amount by weighted addition of the first shake correction amount and the second shake correction amount based on the degree of separation between the first motion vector and the second motion vector. The image shake correction device according to claim 4, characterized in that.

13. The correction amount calculation means increases the weight of the second shake correction amount as the degree of separation between the first motion vector and the second motion vector increases. The image shake correction device according to claim 12, characterized in that.

14. The first motion detection means separates the motion vector into a first motion vector and a second motion vector based on the subject information and the amount of the motion vector. The image shake correction device according to any one of claims 2 to 4, characterized in that.

15. Determination means for determining the effect of shake correction based on the second shake correction amount; Display means for displaying a display item corresponding to the determination result by the determination means together with the input image, further comprising: The determination means determines the effect of shake correction based on the second shake correction amount based on at least any one of the target of interest, the first shake correction amount and the second shake correction amount, the reliability of the subject information, the reliability of the second motion vector, the degree of separation between the first motion vector and the second motion vector, and the upper limit of the shake correction amount. The image shake correction device according to claim 2, characterized in that.

16. When the degree to which the target of interest is a subject is a first value, the determination means determines that the effect of shake correction based on the second shake correction amount is higher than when the degree is a second value lower than the first value. The image shake correction device according to claim 15, characterized in that.

17. When the absolute value of the difference between the first shake correction amount and the second shake correction amount is a third value, the determination means determines that the effect of shake correction based on the second shake correction amount is higher than when the absolute value of the difference is a fourth value smaller than the third value. The image shake correction device according to claim 15, characterized in that.

18. The determination means determines that the effect of shake correction based on the second shake correction amount is higher when the reliability of the subject information is high than when the reliability is low. The image shake correction device according to claim 15, characterized in that.

19. The determination means determines that the effect of blur correction based on the second blur correction amount is higher when the reliability of the second motion vector is high than when the reliability is low. The image blur correction apparatus according to claim 15.

20. The determination means determines that the effect of blur correction based on the second blur correction amount is higher when the degree of separation between the first motion vector and the second motion vector is high than when the degree of separation is low. The image blur correction apparatus according to claim 15.

21. The determination means determines that the effect of blur correction based on the second blur correction amount is higher when the value obtained by subtracting the second blur correction amount from the upper limit of the blur correction amount is a fifth value than when it is a sixth value smaller than the fifth value. The image blur correction apparatus according to claim 15.

22. The determination means determines that there is an effect of blur correction based on the second blur correction amount when the state where the effect of blur correction based on the second blur correction amount is higher than a predetermined threshold continues for a predetermined time or more. The image blur correction apparatus according to claim 15.

23. When the determination means determines that there is no effect of blur correction based on the second blur correction amount, the display means displays a first display item corresponding to the blur correction based on the first blur correction amount, and when it is determined that there is an effect of blur correction based on the second blur correction amount, a second display item corresponding to the blur correction based on the second blur correction amount is displayed. The image blur correction apparatus according to claim 15.

24. The display means displays the second display item in the vicinity of the subject area to be subjected to blur correction. The image blur correction apparatus according to claim 23.

25. The display means notifies the photographer of the subject to be subjected to blur correction by the second display item and a frame surrounding the subject. The image blur correction apparatus according to claim 23.

26. The display means surrounds the subject with a frame and changes at least any one of the color, shape, and size of the frame of the subject to be subjected to blur correction so as to be different from other subjects, thereby notifying the photographer of the subject to be subjected to blur correction. The image blur correction apparatus according to claim 23.

27. The image blur correction apparatus further includes second motion detection means for detecting the motion of the image blur correction apparatus. The conversion means further converts the movement of the image blur correction device detected by the second movement detection means into a third blur correction amount for correcting blur. Based on the target of interest estimated by the estimation means, the correction amount calculation means calculates the final blur correction amount from the first blur correction amount, the second blur correction amount, and the third blur correction amount. The image blur correction device according to any one of claims 1 to 26, wherein filter processing is performed on the outputs of the first movement detection means and the second movement detection means, and the characteristics or output gain of the filter are set according to the outputs of each other.

28. When it is determined by the second movement detection means that the speed of movement of the image blur correction device is equal to or higher than a first predetermined value, the correction amount calculation means, based on the output of the first movement detection means, performs at least one of the processes of increasing the cut-off frequency of the filter for the output of the second movement detection means or decreasing the output gain. The image blur correction device according to claim 27.

29. When it is determined by the second movement detection means that the speed of movement of the image blur correction device is equal to or lower than a second predetermined value, the correction amount calculation means, based on the output of the second movement detection means, performs at least one of the processes of decreasing the cut-off frequency of the filter for the output of the first movement detection means or increasing the output gain. The image blur correction device according to claim 27.

30. When the absolute value of the output of the first movement detection means increases and the value is equal to or greater than a predetermined value, the correction amount calculation means makes the cut-off frequency of the filter for the output of the second movement detection means higher than the cut-off frequency of the filter for the output of the first movement detection means. The image blur correction device according to claim 27.

31. When the absolute value of the output of the first movement detection means decreases and the value is equal to or less than a predetermined value, the correction amount calculation means makes the cut-off frequency of the filter for the output of the second movement detection means lower than the cut-off frequency of the filter for the output of the first movement detection means. The image blur correction device according to claim 27.

32. Subject detection means for detecting a specific subject from an input image and outputting subject information. Camera information acquisition means for acquiring camera information necessary for estimating the shooting situation. first motion detection means for detecting the movement of the background and the movement of the subject in the input image; conversion means for converting the movement of the background and the movement of the subject detected by the first motion detection means into a first blur correction amount for correcting the blur of the background and a second blur correction amount for correcting the blur of the subject, respectively; correction amount calculation means for synthesizing the first blur correction amount and the second blur correction amount to generate a final blur correction amount; display control means for controlling to display, on the display means together with the input image, a display item according to the correction effect of the subject blur and the correction effect of the background blur based on at least any one of the difference between the first blur correction amount and the second blur correction amount, the difference between the upper limit of the blur correction amount and the second blur correction amount, the reliability of the subject information, the reliability of the second motion vector representing the movement of the specific subject, and the degree of separation between the first motion vector representing the movement of the background and the second motion vector. An image blur correction apparatus characterized by comprising:

33. The image blur correction apparatus according to any one of claims 1 to 32, further comprising imaging means for imaging a subject image to generate an input image.

34. a subject detection step of detecting a specific subject from an input image and outputting subject information; a camera information acquisition step of acquiring camera information necessary for estimating a shooting situation; an estimation step of estimating an object of interest in the image using the subject information and the camera information; a motion detection step of detecting the movement of the background and the movement of the subject in the input image; a conversion step of converting the movement of the background and the movement of the subject detected in the motion detection step into a first blur correction amount for correcting the blur of the background and a second blur correction amount for correcting the blur of the subject, respectively; a correction amount calculation step of synthesizing the first blur correction amount and the second blur correction amount based on the object of interest estimated in the estimation step to generate a final blur correction amount; having The control method of an image blur correction apparatus, wherein the camera information includes at least any one of information indicating a shutter speed, information indicating an AF area, information indicating a focal length, information indicating a user operation, a detection result of an inertial sensor, information indicating a distance to a subject, and a detection result of a user's line of sight.

35. a subject detection step of detecting a specific subject from an input image and outputting subject information; A camera information acquisition step of acquiring camera information necessary for estimating a shooting situation; A motion detection step of detecting the movement of the background and the movement of the subject in the input image; A conversion step of converting the movement of the background and the movement of the subject detected in the motion detection step into a first blur correction amount for correcting the blur of the background and a second blur correction amount for correcting the blur of the subject, respectively; A correction amount calculation step of synthesizing the first blur correction amount and the second blur correction amount to generate a final blur correction amount; A display control step of controlling to display a display item according to the correction effect of the blur of the subject and the correction effect of the blur of the background based on at least any one of the difference between the first blur correction amount and the second blur correction amount, the difference between the upper limit of the blur correction amount and the second blur correction amount, the reliability of the subject information, the reliability of the second motion vector representing the movement of the specific subject, and the separation degree between the first motion vector representing the movement of the background and the second motion vector, together with the input image on a display means. A control method of an image blur correction device, characterized by comprising: [

36. ] A program for causing a computer to function as each means of the image blur correction device according to claim 1 or 32.

Images