Subject motion blur correction control device and method, and image capture apparatus

Abstract

To provide a subject motion blur correction control device and method for improving accuracy of subject motion blur correction by detecting motion of a subject using a plurality of methods and appropriately combining and utilizing detection results.SOLUTION: A subject motion blur correction control device calculates a correction amount for correcting subject motion blur caused by motion of a subject. The device calculates a first subject motion blur amount based on a position of a subject region in a captured image and a second subject motion blur amount based on a subject vector representing the motion of the subject between captured images. The device calculates a subject motion blur correction amount by adding the first subject motion blur amount and the second subject motion blur amount with use of weights corresponding to relative reliabilities of the first subject motion blur amount and the second subject motion blur amount.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an apparatus and method for controlling subject blur correction, and an imaging apparatus. [Background technology]

[0002] 2. Description of the Related Art In imaging devices such as digital cameras, a function for correcting image blur (camera shake) caused by camera movement and a function for correcting subject blur caused by subject movement during exposure are known.

[0003] Camera shake can be estimated by detecting the movement of the imaging device, and subject shake can be estimated by detecting the movement of the subject. Subject shake can be corrected (reduced) by moving the imaging range so as to follow the estimated movement of the subject (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6727791 Summary of the Invention [Problem to be solved by the invention]

[0005] There are various methods for detecting the movement of a subject, but each method has its own advantages and disadvantages, making it difficult to detect the movement of a subject with high accuracy using a single method.

[0006] In one aspect, the present invention provides an apparatus and method for controlling subject blur correction that improves the accuracy of subject blur correction by detecting the movement of a subject using a plurality of methods and using an appropriate combination of the detection results. [Means for solving the problem]

[0007] In one aspect, the present invention provides a subject blur correction control device that calculates a correction amount for correcting subject blur caused by subject movement, the subject blur correction control device comprising: first blur amount calculation means that calculates a first subject blur amount based on a position of a subject area in a captured image; second blur amount calculation means that calculates a second subject blur amount based on a subject vector that represents the movement of the subject between the captured images; and correction amount calculation means that calculates a subject blur correction amount by adding the first subject blur amount and the second subject blur amount with a weight corresponding to the relative reliability of the first subject blur amount and the second subject blur amount. Effect of the Invention

[0008] According to the present invention, it is possible to provide an apparatus and method for controlling subject blur correction that improves the accuracy of subject blur correction by detecting the movement of a subject using a plurality of methods and using an appropriate combination of the detection results. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an image capture apparatus incorporating a subject blur correction control device according to a first embodiment; [Diagram 2] Flowchart for the object blur correction operation in the first embodiment [Diagram 3] Schematic diagram of template matching that can be implemented in the embodiment. [Figure 4] FIG. 13 is a diagram showing an example of a correlation value map that can be generated in an embodiment. [Diagram 5] Diagram of vector separation in embodiment [Figure 6] FIG. 13 is a diagram showing another example of an expression of a correlation value map in the embodiment; [Figure 7] FIG. 1 shows an example of a motion vector reliability measure that can be used in an embodiment. [Figure 8] FIG. 1 is a diagram illustrating an example of a method for calculating the reliability of a motion vector according to an embodiment. [Figure 9] FIG. 13 is a diagram showing an example of the relationship between subject information and reliability that can be used in the embodiment; [Figure 10] FIG. 13 is a diagram showing an example of the relationship between reliability and cutoff frequency that can be used in the embodiment; [Figure 11] FIG. 11 is a block diagram showing an example of the functional configuration of an imaging apparatus incorporating a subject blur correction control device according to a second embodiment; [Figure 12] Flowchart for the subject blur correction operation in the second embodiment [Figure 13] FIG. 13 is a diagram showing an example of the relationship between reliability and gain that can be used in the embodiment. [Figure 14] FIG. 13 is a block diagram showing an example of the functional arrangement of an imaging device according to a third embodiment. [Figure 15] Flowchart for the object blur correction operation in the third embodiment [Figure 16] FIG. 13 is a block diagram showing an example of the functional configuration of an imaging apparatus incorporating a subject blur correction control device according to a fourth embodiment. [Figure 17] Flowchart for the object blur correction operation in the fourth embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0011] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in a digital camera. However, the present invention can also be implemented in any electronic device having an imaging function. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive shake recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] ●(First embodiment) Fig. 1 is a block diagram showing an example of the functional configuration of an image capture device 100 that implements an object blur correction control device according to a first embodiment of the present invention. Note that in Fig. 1, among the components of the image capture device 100, components related to object blur correction are shown as functional blocks.

[0013] 1 may be realized by software, hardware, or a combination thereof, except for parts that can be realized only by hardware, such as a lens and an image sensor. For example, the functional blocks may be realized by dedicated hardware, such as an ASIC. Also, the functional blocks may be realized by a processor included in a control unit 120 that controls the operation of the imaging device 100 executing a program stored in advance in a non-volatile memory (ROM 121).

[0014] A plurality of functional blocks may be realized by a common configuration (for example, one ASIC). Furthermore, hardware that realizes a part of the functions of a certain functional block may be included in hardware that realizes another functional block. For example, the development processing unit 103, the subject detection unit 104, and the subject vector detection unit 107 may be realized as functions of an image processing circuit.

[0015] In Fig. 1, functional blocks 104 to 114 provided after development processing unit 103 constitute an object blur correction control device. The object blur correction control device calculates the amount of correction for each of the horizontal (X) direction and the vertical (Y) direction in an image. The amount of object blur correction is used to reduce (correct) blur in the object image caused by the movement of the object during the exposure period of the image sensor.

[0016] The optical system 101 has multiple lenses and forms an optical image of a subject on the imaging surface of an image sensor. The multiple lenses include movable lenses, which include a focus lens and a blur correction lens. The focus lens is movable in the optical axis direction, and the blur correction lens is movable in directions perpendicular to the optical axis (X direction and Y direction). The blur correction lens is used for subject blur correction and camera shake correction. The optical system 101 has a drive mechanism (motor, actuator, etc.) for the movable lenses. The operation of the drive mechanism is controlled by the control unit 120. The optical system 101 may include other components, such as an aperture and a shutter.

[0017] The imaging unit 102 includes an imaging element and a driving circuit for the imaging element. The imaging element may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out a signal from each pixel. In this embodiment, the imaging element is movable in the horizontal and vertical directions within a plane perpendicular to the optical axis. Each pixel accumulates electric charges according to the amount of incident light through photoelectric conversion.

[0018] A signal having a voltage corresponding to the amount of charge accumulated during the exposure period is read from each pixel through a peripheral circuit, thereby obtaining a pixel signal group (analog image signal) representing the subject image formed on the imaging surface. Here, the imaging unit 102 applies noise reduction processing and A / D conversion processing to the analog image signal to generate a digital image signal (image data). The imaging unit 102 outputs the image data to the development processing unit 103.

[0019] The control unit 120 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing a program. The control unit 50 controls the operation of the imaging device 100, including the subject blur correction control device, by loading a program stored in the ROM 121 into the RAM 122 and executing it. Although omitted in Fig. 1, the control unit 120 is connected to each functional block. The control unit 120 also executes operations related to autofocus detection (AF) and automatic exposure control (AE).

[0020] The ROM 121 is a rewritable non-volatile memory that stores programs executed by the control unit 120, various setting values ​​of the imaging device 100, graphical user interface (GUI) data, etc. The RAM 122 is a main memory used when the control unit 120 executes programs. The RAM 122 can also be used to store image data output by the imaging unit 102, and to store intermediate data and processed data by the development processing unit 103.

[0021] The development processing unit 103 applies predetermined image processing such as color interpolation processing, white balance processing, gradation correction processing, and RGB-YUV conversion processing to the image data output by the imaging unit 102, and generates image data to be used for display or recording. Color interpolation processing is processing for generating values ​​of color components not included in the individual pixel data constituting the image data, and is also called demosaic processing. The development processing unit 103 may apply other processing such as correction of image degradation caused by optical aberration of the optical system 101, correction of the effect of peripheral light falloff of the optical system 101, and color correction. The image data generated by the development processing unit 103 is output to the subject detection unit 104 and the subject vector detection unit 107. The development processing unit 103 may store the image data in the RAM 122, and the subject detection unit 104 and the subject vector detection unit 107 may read the image data from the RAM 122.

[0022] The operation of each functional block constituting the subject blur correction control device, which will be described below, is executed on image data output by the development processing unit 103 at a predetermined frame rate during video shooting. However, the process for calculating the subject blur correction amount does not have to be executed for each frame, and the process does not have to be completed within one frame period.

[0023] The subject detection unit 104 applies subject detection processing to one frame's worth of captured image data output by the development processing unit 103. The subject detection processing is processing for detecting an area (subject area) in which a predetermined type of subject is captured, and outputting the detection result. The detection result may be information for each detected subject area (e.g., type of subject, position, size, and detection reliability) of the subject area. The position may be, for example, the center of gravity coordinates (x, y) of the subject area expressed in an image coordinate system. The size may be, for example, the number of pixels included in the subject area or the area (in pixels) of the smallest rectangular area inscribed by the subject area. The position and size may also be in other formats.

[0024] Here, it is assumed that the subject detection unit 104 applies a subject detection process using a machine learning model (trained model) that has been trained in advance to detect a subject area of ​​a specific type of subject from input two-dimensional image data. The trained model may be, for example, a convolutional neural network (CNN). The subject detection unit 104 may detect subject areas of multiple types of subjects using multiple trained models. The trained model can be implemented in the subject detection unit 104 in advance. In addition, the type of subject to be detected may be automatically determined according to the settings of the imaging device 100 (e.g., shooting mode), or may be designated by the user.

[0025] For each subject region, the subject detection unit 104 outputs the position of the subject region to the first subject blur amount calculation unit 105, the type of subject to the soft body subject determination unit 110, and the type of subject and detection reliability to the subject information acquisition unit 111.

[0026] The first object blur amount calculation section 105 calculates the difference between the detection position obtained from the object detection section 104 and a target position of the object set in advance as image coordinates as a first object blur amount for each X direction component and Y direction component. The first object blur amount calculation section 105 outputs the first object blur amount to the first filter section 106.

[0027] The first filter unit 106 is a high-frequency cutoff filter (low-pass filter). The cutoff frequency is set by a filter control unit 113, which will be described later. The first object blur amount is susceptible to high-frequency noise because it uses the object detection position without integrating it, but integral drift does not occur. Therefore, a low-pass filter is applied to the first object blur amount, and low-frequency components are extracted and used to calculate the object blur correction amount. In this way, in this embodiment, for object blur amounts not based on integral values, low-frequency components are used to calculate the object blur correction amount.

[0028] The first filter unit 106 applies known low-pass filter processing to, for example, time-series data for each directional component of the first object blur amount obtained from the first object blur amount calculation unit 105. The first filter unit 106 outputs the low-frequency components of the first object blur amount obtained by the low-pass filter processing to the object blur correction amount calculation unit 114.

[0029] The subject vector detection unit 107 detects a vector (subject vector) representing the movement of the subject between frames (between captured images) and its reliability from two frames of image data output by the development processing unit 103. The detection of the subject vector can be performed by any known method. For example, a moving area (subject area) and a still area (background area) can be separated based on the difference between frames, and a vector representing the amount of movement and direction of movement of the moving area can be detected as the subject vector. Note that the detection accuracy of the subject vector may be improved by not including the movement of the imaging device detected for camera shake correction.

[0030] (Subject vector detection method) Here, a method of detecting a subject vector using template matching will be described as an example of a method of detecting a subject vector that can be implemented by subject vector detection unit 107. Fig. 3(a) shows, in a schematic manner, an image (base image) captured earlier, of two frame images used for detection, and Fig. 3(b) shows an image (reference image) captured later.

[0031] The subject vector detection unit 107 sets a template region 301 in the base image and a search region 302 in the reference image. The subject vector detection unit 107 searches within the search region 302 for a region having the highest correlation with the image of the template region 301.

[0032] The template region 301 may be, for example, a square region of a predetermined size. The subject vector detection unit 107 may set the template region 301 around a predetermined coordinate, or around the coordinates of a feature point in an image detected by a known method. The subject vector detection unit 107 may also determine the center coordinates of the template region 301 based on the position of the subject region detected by the subject detection unit 104 so as to set the same number of template regions 301 in the subject region and outside the subject region. The subject vector detection unit 107 may set, for example, in the reference image, a region having the same center coordinates as the template region 301 and having a size obtained by equally expanding the template region 301 in the vertical and horizontal directions as the search region 302.

[0033] In this embodiment, the sum of absolute differences (SAD) of the luminance values ​​of pixels whose positions correspond to each other is calculated as the correlation value S-SAD. The calculation formula for SAD is shown in Equation (1).

number

[0034] The subject vector detection section 107 may calculate a value other than the SAD as the correlation value, such as the sum of squared differences (SSD) or normalized cross correlation (NCC).

[0035] The subject vector detection unit 107 moves the position of the correlation value calculation region 303 in the horizontal direction and in the horizontal direction, for example, by one pixel within the search region 302, and calculates the correlation value at each position. The subject vector detection unit 107 creates a correlation value map for the search region 302 using the calculated correlation values.

[0036] Fig. 4(a) shows an example of a correlation value map calculated in the coordinate system of search region 302, with the X-axis and Y-axis representing correlation value map coordinates and the Z-axis representing the magnitude of the correlation value calculated for correlation value calculation region 303 set at coordinates (X, Y). Fig. 4(b) is a two-dimensional correlation value map seen from the Z-axis direction of Fig. 4(a), with coordinates having predetermined correlation values ​​connected by lines.

[0037] 4(a) and 4(b), the correlation value calculation region 303 when the smallest correlation value 401 is calculated has a texture most similar to that of the template region 301. The correlation value calculation regions 303 corresponding to the second smallest correlation value 402 and the third smallest correlation value 403 also have textures similar to that of the template region 301.

[0038] In this way, subject vector detection section 107 searches for the area within search area 302 that has the smallest correlation value with template area 301, and specifies the searched area as the destination of template area 301. Then, subject vector detection section 107 detects, as a motion vector, a vector that has its starting point at the center coordinates of template area 301 and its end point at the center coordinates of the area specified as the destination, in the image coordinate system.

[0039] An example of a method for separating motion vectors detected for each template region 301 set in the reference image into background vectors and subject vectors will be described below. Based on the position of the subject region detected by subject detection section 104, subject vector detection section 107 regards motion vectors originating from coordinates within the subject region as subject vectors, and motion vectors originating from coordinates outside the subject region as background vectors (first separation process).

[0040] If no subject is detected, the first separation process may be performed using depth information (distance information) obtained by a known method based on the captured image. For example, a motion vector whose subject distance corresponding to the image coordinates of the starting point is equal to or greater than a first threshold value can be regarded as a background vector, and a motion vector whose subject distance corresponding to the image coordinates of the starting point is less than a second threshold value can be regarded as a subject vector. Here, the first threshold value is equal to or greater than the second threshold value.

[0041] The separation of the background vector and the object vector may be completed in the first separation process. However, the separation process based on the detected object region as described here may erroneously classify the motion vector originating from the coordinates near the boundary between the background and the object. In addition, if the object detection accuracy is low, the accuracy of the separation process also decreases.

[0042] Therefore, the accuracy of the separation process can be improved by further executing a second separation process that does not depend on the object region. For example, as an example of the separation process based on the amount of motion vectors, the second separation process based on the well-known k-means method can be executed.

[0043] In the k-means method, it is necessary to determine in advance the number k of clusters to be classified and the initial value Vk of the center of gravity of each cluster. In order to classify the images into background clusters and subject clusters, the number of clusters k is set to 2. If there are multiple subjects, the number of clusters may be changed according to the number of subjects. Also, a cluster may be provided to classify the motion vectors that do not belong to either the background or the subject.

[0044] The initial value Vk of the center of gravity of each cluster can be set using the results of the first separation process. The object vector detection unit 107 generates a histogram of the amount of movement in the X direction and a histogram of the amount of movement in the Y direction for the motion vectors classified as background vectors in the first separation process. Since the most frequent values ​​V1x and V1y of each histogram are representative values ​​of the background vectors, the object vector detection unit 107 sets the initial value V1=(V1x, V1y) of the center of gravity of the background cluster. Similarly, the object vector detection unit 107 obtains representative values ​​V2x and V2y of the motion vectors classified as object vectors in the first separation process, and sets the initial value V2=(V2x, V2y) of the center of gravity of the object cluster.

[0045] The initial value V1 of the center of gravity of the background cluster may be set using the motion of the imaging device 100 detected for camera shake correction. For example, when the motion of the imaging device 100 is detected by an angular velocity sensor, the subject vector detection unit 107 integrates the angular velocities in the Yaw direction and Pitch direction, respectively, to convert them into angles θy and θp, and further converts the angles θy and θp into displacement amounts V1x and V1y on the imaging surface. The displacement amounts Vlx and Vly can be calculated as V1x=ftan(θy) and V1y=ftan(θp), where f is the focal length of the optical system 101 [mm].

[0046] The initial value V2 of the center of gravity of the object cluster may be set by determining a representative value of the amount of displacement of the detection position of the object region obtained in time series by object detection section 104 in the same way as the representative value of the object vector.

[0047] After setting the number of clusters and the initial value of the center of gravity of each cluster in this way, the subject vector detection unit 107 calculates the coordinates (movement amount in the X direction, movement amount in the Y direction) for each motion vector and the distance between the center of gravity of each cluster. Then, the subject vector detection unit 107 classifies the motion vector into the cluster with the smallest calculated distance.

[0048] If the cluster classification has not changed for all motion vectors, or if the ratio of motion vectors whose classification has changed falls below a certain threshold set in advance, subject vector detection unit 107 determines that the classification process has converged and ends the second separation process. If it is not determined that the classification process has converged, subject vector detection unit 107 recalculates the center of gravity for each cluster based on the classification result, and repeats the above-mentioned distance calculation and classification.

[0049] An example of the results of the second classification process is shown in Figure 5. Figure 5(a) is a diagram showing the classification results in which the X-direction displacement amount of the motion vector is plotted on the horizontal axis and the Y-direction displacement amount is plotted on the vertical axis. □ indicates the initial value V1 of the center of gravity of the background cluster, and ☆ indicates the initial value V2 of the center of gravity of the subject cluster. Motion vectors classified as background vectors are indicated by ○, and motion vectors classified as subject vectors are indicated by △.

[0050] Figure 5(b) is a plot of the reference image of the starting points of the motion vectors corresponding to the classification results in Figure 5(a), with circles for the starting points of the motion vectors classified as background vectors and stars for the starting points of the motion vectors classified as subject vectors. It can be seen that the classification is accurate when the subject is a car and the other parts are the background.

[0051] (Method of calculating the reliability of subject vectors) Next, an example of a method for calculating the reliability of an object vector will be described. Object vector detection unit 107 calculates the reliability of a motion vector using the two-dimensional correlation value map shown in Fig. 4(b). Fig. 6 is a diagram showing the correlation values ​​plotted on the two-dimensional correlation value map shown in Fig. 4(b) as one-dimensional data arranged in the raster order indicated by arrow 404. The vertical axis of Fig. 6 is the correlation value, and the horizontal axis is the pixel address uniquely determined by the X coordinate and Y coordinate of the two-dimensional correlation value map. 601 corresponds to the minimum correlation value 401 shown in Fig. 4.

[0052] Fig. 7 shows an example of an index of the reliability of a motion vector based on a correlation value. As in Fig. 6, the horizontal axis of Fig. 7(a) to Fig. 7(d) is the pixel address, and the vertical axis is the correlation value. Fig. 7(a) shows the difference Da between the minimum and maximum correlation values ​​as an example of an index of the reliability of a motion vector. The difference Da corresponds to the range of a two-dimensional correlation value map, and if Da is small, it is assumed that the texture contrast is low, and therefore the reliability of the motion vector is also low.

[0053] 7(b) shows, as another example of an index of reliability, the ratio Db (=B / A) of the difference A between the minimum and maximum correlation values ​​to the difference B between the minimum and average correlation values. Db represents the steepness of the correlation value peak, and when Db is small, it is assumed that the similarity between the template region and the search region is low, and the reliability of the motion vector is also low.

[0054] Fig. 7(c) shows the difference Dc between the minimum and second smallest correlation values ​​as another example of an index of reliability. 701 to 703 correspond to the correlation values ​​401, 402, and 403 in Fig. 4, respectively. Dc indicates the periodicity of the correlation value map, and when Dc is small, it is assumed that the texture contains repeated patterns or edges, and the reliability of the motion vector is also considered to be low. Here, the minimum and second smallest values ​​are used, but other minimum values ​​may be used as long as the periodicity of the correlation value map can be determined.

[0055] Figure 7(d) shows the minimum correlation value Dd as another example of the reliability index. When Dd is large, it is assumed that the similarity between the template region and the search region is low, and the reliability of the motion vector is also low. Since Dd and reliability are inversely proportional, the reciprocal of Dd (1 / Dd) is actually used as the reliability index.

[0056] When determining the level of reliability based on these indices, the indices are compared with thresholds that are predefined for each indices, and when an index is below a first threshold, the reliability can be determined to be low. Similarly, when an index is equal to or greater than a second threshold, the reliability can be determined to be high. The first threshold is less than or equal to the second threshold.

[0057] The index explained using Fig. 7 can be used as the reliability as it is. However, as shown in Fig. 8, for example, a specific correspondence may be established between the index value and the reliability. The horizontal axis of Fig. 8 is the index (any of Da, Db, Dc, or 1 / Dd mentioned above), and the vertical axis is the reliability. 0 is the lowest reliability and 1 is the highest reliability.

[0058] In the example shown in FIG. 8, two thresholds T1 and T2 are set for the index, and the reliability is set to 0 if the index is below the threshold T1, and 1 if the index is above the threshold T2. At least one of the thresholds T1 and T2 may be changed for each index. In addition, in the section from exceeding the threshold T1 to reaching T2, the reliability is increased linearly with respect to the increase in the index. The reliability may be increased non-linearly with respect to the increase in the index. In the following explanation, the reliability obtained from each of the above-mentioned indexes is expressed as Ra, Rb, Rc, and Rd. Here, the relationship is Ra=f(Da), Rb=f(Db), Rc=f(Dc), and Rd=f(Dd). f() indicates a function that converts the index into reliability.

[0059] The subject vector detection unit 107 can calculate the reliability of the motion vector based on one or more of the reliability Ra, Rb, Rc, and Rd. When calculating the final reliability R by combining two or more reliability, weighted addition and logical operations can be applied.

[0060] For example, if the weights of the four reliabilities Ra, Rb, Rc, and Rd are Wa, Wb, Wc, and Wd, respectively, the subject vector detection unit 107 can calculate the final reliability R according to the following equation (2).

number

Number

[0061] Also, the final reliability R may be calculated as shown in Equation (4) using the logical sum.

Number

[0062] The subject vector detection unit 107 outputs the detected subject vector to the second subject blur amount calculation unit 108 and the software subject determination unit 110. Also, the subject vector detection unit 107 outputs the reliability of the detected subject vector to the subject information acquisition unit 111.

[0063] The second subject blur amount calculation unit 108 calculates, as the second subject blur amount, the integral value for each of the X - direction component and the Y - direction component of the subject vector obtained from the subject vector detection unit 107. The second subject blur amount calculation unit 108 can integrate a predetermined number of subject vectors or the subject vectors detected in a recent predetermined period. The second subject blur amount calculation unit 108 outputs the second subject blur amount to the second filter unit 109.

[0064] The second filter unit 109 is a low-frequency cutoff filter (high-pass filter). The cutoff frequency is set by a filter control unit 113, which will be described later. Since the second object blur amount is obtained by integrating the object vector, the high-frequency cutoff effect makes it less susceptible to high-frequency noise, but integral drift occurs. Therefore, a high-pass filter is applied to the second object blur amount to extract high-frequency components, which are used to calculate the object blur correction amount. In this way, in this embodiment, for the object blur amount based on the integral value, high-frequency components are used to calculate the object blur correction amount.

[0065] In this embodiment, as an example, the cutoff frequencies of the first filter section 106 and the second filter section 109 are the same. Therefore, controlling the cutoff frequency is equivalent to controlling the relative proportion (weight) of the first object blur amount and the second object blur amount in the frequency band of the object blur amount. Note that the cutoff frequencies do not have to be completely the same.

[0066] The second filter unit 109 applies known high-pass filter processing to, for example, time-series data for each directional component of the second object blur amount obtained from the second object blur amount calculation unit 108. Note that the second filter unit 109 may be configured to realize a high-pass filter by subtracting data that has been subjected to low-pass filter processing from data before filtering. The second filter unit 109 outputs high-frequency components of the second object blur amount obtained by the high-pass filter processing to the object blur correction amount calculation unit 114.

[0067] The soft-body subject determination unit 110 determines whether or not the detected subject is a soft-body subject based on at least one of the type of subject obtained from the subject detection unit 104 and the subject vector obtained from the subject vector detection unit 107. Here, a soft-body subject is a subject whose shape of the subject area can change significantly in a short period of time (for example, between frames). Living subjects such as humans and animals are typical soft-body subjects. Note that the shape of the subject area of ​​a non-living moving subject such as a vehicle or airplane can also change over time, but it is unlikely that the shape of the subject area will change significantly in a short period of time. Therefore, these subjects are not determined to be soft-body subjects.

[0068] For example, among the types of subjects detected by subject detection unit 104, those that correspond to soft body subjects can be registered in advance in soft body subject determination unit 110. Then, if the type of subject obtained from subject detection unit 104 corresponds to a soft body subject registered in advance, soft body subject determination unit 110 can determine that the detected subject is a soft body subject.

[0069] Furthermore, when multiple object vectors are detected for the same type of object in one object vector detection process, it is possible to determine whether the detected object is a soft-body object based on the variation of the object vectors (for example, the variance of at least one of the magnitude and direction). When the variation of the object vectors for the same type of object is large (above a preset threshold), the soft-body object determination unit 110 can determine that the object is a soft-body object.

[0070] For example, in the example shown in Figure 5(a), if the variance of the motion vectors classified as object vectors (△) is greater than a threshold, the corresponding object can be determined to be a soft-body object. For objects that do not deform significantly between frames, such as cars or airplanes, many of the corresponding object vectors indicate movement between frames, so the difference in direction and magnitude is small, and the variance is small. On the other hand, for soft-body objects that can deform significantly between frames, such as animals, in addition to the motion vectors indicating movement between frames, motion vectors indicating the movement of parts of the object, such as the flapping of wings in the case of birds, are also detected. This results in a large variance of the object vectors.

[0071] The soft-body subject determination section 110 outputs the determination result for each detected subject region to the subject information acquisition section 111 .

[0072] The subject information acquisition unit 111 acquires the type of subject and detection reliability from the subject detection unit 104, the reliability of the subject vector from the subject vector detection unit 107, and the judgment result from the soft-body subject judgment unit 110. This information is called subject information. The subject information acquisition unit 111 outputs the acquired subject information to the relative reliability judgment unit 112.

[0073] The relative reliability determination unit 112 determines the relative reliability of the object detection position and the object vector based on the object information obtained from the object information acquisition unit 111. This relative reliability is also the relative reliability of the first object blur amount calculated based on the object detection position and the second object blur amount calculated based on the object vector. The relative reliability determination unit 112 outputs the determined relative reliability to the filter control unit 113.

[0074] Here, an example of a method for determining the relative reliability based on the object information, which is executed by the relative reliability determination unit 112, will be described with reference to Fig. 9. Fig. 9 shows the relationship between conditions for each type of object information and the relative reliability of the first and second object blur amounts when the conditions are satisfied.

[0075] When the subject is determined to be a soft-body subject, the corresponding subject vector is likely to include a vector representing the movement of the subject's parts in addition to a vector representing the overall movement of the subject. Therefore, the relative reliability determination unit 112 determines that the reliability of the second subject blur amount calculated based on the subject vector is relatively lower than the reliability of the first subject blur amount calculated based on the subject detection position.

[0076] When the reliability of the object detection is lower than a predetermined threshold, the reliability of the object detection position is also considered to be low, and therefore the relative reliability determination unit 112 determines that the reliability of the first object blur amount calculated from the object detection position is relatively lower than the reliability of the second object blur amount calculated from the object vector.

[0077] If the reliability of the subject vector is lower than a predetermined threshold, the relative reliability determination unit 112 determines that the reliability of the second subject blur amount calculated from the subject vector is relatively lower than the reliability of the first subject blur amount calculated from the subject detection position.

[0078] The relative reliability determination unit 112 may use the determination result for any one of the above-mentioned items of the object information as the final relative reliability. For example, the relative reliability determination unit 112 can use the determination result for an item with the highest predetermined priority among the items of the object information as the final relative reliability.

[0079] Alternatively, the relative reliability determination unit 112 may determine the final relative reliability based on the determination results for two or more items of the object information described above. When determining the final relative reliability using the determination results for a plurality of items, the relative reliability determination unit 112 determines the final reliability by taking a weighted average of the reliability of the first object blur amount and the reliability of the second object blur amount obtained for each object information item.

[0080] For example, the relative reliability determination unit 112 converts the relatively high reliability and the relatively low reliability into numerical values ​​that sum to 1. Here, it is assumed that the relatively high reliability is converted to 0.7, and the relatively low reliability is converted to 0.3. Then, the relative reliability determination unit 112 determines a weight (summing up to 1) for each item.

[0081] For example, the weight of the item "soft-body subject" is W1, the weight of the item "reliability of subject detection" is W2, and the weight of the item "reliability of subject vector" is W3. The values ​​to be assigned to W1 to W3 are determined in advance. The relative reliability determination unit 112 calculates the reliability of the subject blur amount by weighting and adding the reliability of each item depending on whether the condition for each item shown in FIG. 9 is satisfied.

[0082] For example, the reliability R1 of the first amount of subject blurring can be obtained by the following formula (5), where the reliability of each item is R11, R12, and R13. The reliability R2 of the second amount of subject blurring is relative to the reliability R1, and is therefore obtained by formula (6).

number

[0083] Specifically, when a soft-body subject is determined, the reliability of the subject detection degree is not low, and the reliability of the subject vector is low, the reliability R1 is: R1 = W1 x 0.7 + W2 x 0.7 + W3 x 0.3 It is calculated as:

[0084] The relative reliability determination unit 112 may vary at least one of the weights W1 to W3 for each item of object information and the numerical values ​​assigned to relatively high and relatively low reliability depending on the type of object obtained from the object detection unit 104. The relative reliability determination unit 112 outputs, for example, the reliability R1 of the first object blur amount or the reliability R2 of the second object blur amount to the filter control unit 113 as the relative reliability.

[0085] The filter control unit 113 determines the cutoff frequencies of the first filter unit 106 and the second filter unit 109 based on the relative reliability of the amount of subject blurring obtained from the relative reliability determination unit 112. The filter control unit 113 notifies the corresponding filter unit of the determined cutoff frequencies.

[0086] The filter control unit 113 can determine the cutoff frequency based on, for example, a relationship between the relative reliability and the cutoff frequency, which is registered in advance. FIG. 10 shows an example of the relationship between the reliability R1 of the first object blur amount and the cutoff frequency. In FIG. 10, the horizontal axis represents the cutoff frequency of the first filter unit 106, and the vertical axis represents the reliability R1. In this example, a lower limit Fc1 and an upper limit Fc2 are set for the cutoff frequency, and when the reliability R1 is at its minimum value 0, the cutoff frequency is Fc1, and when the reliability R1 is at its maximum value 1, the cutoff frequency is Fc2. Here, the cutoff frequency is set to increase linearly with an increase in the reliability, but the cutoff frequency may be set to increase nonlinearly or discontinuously with respect to an increase in the reliability. The lower limit Fc1 and upper limit Fc2 of the cutoff frequency are set to satisfy the condition Fc2>Fc1 within a range that can be set in the filter unit. The upper limit Fc2 is set within a range not exceeding the sampling frequency / 2 according to the sampling theorem.

[0087] By setting the cutoff frequency of the first filter unit 106 (low-pass filter) higher as the reliability R1 of the first object blur amount is higher, the frequency band of the first object blur amount included in the object blur correction amount can be made wider. In this embodiment, the filter control unit 113 determines the same or approximately the same cutoff frequency for the first filter unit 106 and the second filter unit 109. The filter control unit 113 sets the cutoff frequency determined based on, for example, the reliability R1 of the first object blur amount to both the first filter unit 106 and the second filter unit 109. In this way, the output frequency bands of the first filter unit 106 and the second filter unit 109 are substantially continuous frequency bands.

[0088] The object blur compensation amount calculation unit 114 calculates an object blur compensation amount by adding, for each directional component, the low frequency components of the first object blur amount output by the first filter unit 106 and the high frequency components of the second object blur amount output by the second filter unit 109. The object blur compensation amount calculation unit 114 outputs the object blur compensation amount to the control unit 120.

[0089] Control unit 120 performs at least one of optical subject blur correction and electronic subject blur correction based on the amount of subject blur correction calculated by subject blur correction amount calculation unit 114. When performing optical subject blur correction, control unit 120 drives at least one of a blur correction lens included in optical system 101 and an image sensor included in imaging unit 102 so that the position of the subject approaches a target position within the shooting range. When performing electronic subject blur correction, control unit 120 controls the crop position of the captured image so that the position of the subject approaches a target position within the shooting range.

[0090] (Subject shake correction during video recording) Next, the subject blur correction operation in imaging device 100 will be described with reference to the flowchart shown in Fig. 2. The operation described below is executed during video shooting when subject blur correction is enabled in imaging device 100. Video shooting may be for the purpose of live view display, which is executed during shooting standby, for example, or for the purpose of video recording.

[0091] In S201, the subject detection unit 104 executes subject detection processing using a trained model for one frame of image data output by the development processing unit 103. For each detected subject region, the subject detection unit 104 outputs the position of the subject region to the first subject blur amount calculation unit 105, the type of subject to the soft-body subject determination unit 110, and the type of subject and detection reliability to the subject information acquisition unit 111. Position information of the detected subject is output.

[0092] In S202, the first object blur amount calculation unit 105 calculates the difference between the detection position obtained from the object detection unit 104 and a target position of the object set in advance as image coordinates as a first object blur amount. The first object blur amount calculation unit 105 outputs the first object blur amount to the first filter unit 106.

[0093] In S203, the subject vector detection unit 107 detects, from two frames of image data output by the development processing unit 103, a vector (subject vector) that indicates the movement of the subject between frames and its reliability.

[0094] In S204, second subject blur amount calculation section 108 determines a second amount of subject blur based on the multiple subject vectors obtained in S203 from subject vector detection section 107. Second subject blur amount calculation section 108 can determine the second amount of subject blur, for example, by using a histogram of the magnitude of the subject vector in each direction, as described above.

[0095] In S205, soft body subject determination unit 110 determines whether or not the subject is a soft body subject based on at least one of the type of subject obtained from subject detection unit 104 and the subject vector obtained from subject vector detection unit 107. A soft body subject is a subject whose shape of the subject region can change significantly between frames.

[0096] In S206 , subject information acquisition unit 111 acquires subject information from subject detection unit 104 , subject vector detection unit 107 , and soft-body subject determination unit 110 .

[0097] In S207, the relative reliability determination unit 112 determines the relative reliability between the subject detection position and the subject vector, that is, the relative reliability between the first subject blur amount and the second subject blur amount, based on the subject information obtained from the subject information acquisition unit 111.

[0098] In S208, the filter control unit 113 determines the cutoff frequency of the first filter unit 106 and the cutoff frequency of the second filter unit 109 based on the reliability of the first amount of object blurring and the reliability of the second amount of object blurring obtained from the relative reliability determination unit 112. Then, the filter control unit 113 sets the determined cutoff frequencies in each of the first filter unit 106 and the second filter unit 109.

[0099] In S209, the first filter unit 106 applies low-pass filtering having a cutoff frequency notified by the filter control unit 113 to the first object blur amount obtained from the first object blur amount calculation unit 105. The first filter unit 106 outputs the extracted low-frequency component of the first object blur amount (third object blur amount) to the object blur correction amount calculation unit 114.

[0100] In S210, the second filter unit 109 applies high-pass filtering having a cutoff frequency notified by the filter control unit 113 to the second object blur amount obtained from the second object blur amount calculation unit 108. The second filter unit 109 outputs the extracted high-frequency components of the second object blur amount (fourth object blur amount) to the object blur correction amount calculation unit 114.

[0101] In S211, the object blur correction amount calculation unit 114 calculates an object blur correction amount by adding the third object blur amount obtained from the first filter unit 106 and the fourth object blur amount obtained from the second filter unit 109. The object blur correction amount calculation unit 114 outputs the object blur correction amount to the control unit 120.

[0102] In S212, control unit 120 performs subject blur correction so that the position of the subject in the imaging range approaches the target position during the exposure period of the image sensor, based on the subject blur correction amount obtained from subject blur correction amount calculation unit 114. Control unit 120 can perform at least one of optical correction, which drives the correction lens and / or the image sensor to move the imaging range, and electronic correction, which moves the range to be cut out from the captured image.

[0103] In S213, control unit 120 determines whether or not video shooting has ended. If control unit 120 determines that video shooting has ended, control unit 120 controls the subject blur correction control device to end the subject blur correction operation, and if control unit 120 does not determine that video shooting has ended, control unit 120 controls the subject blur correction control device to repeatedly execute the process from S201.

[0104] In the first embodiment, the cutoff frequency for extracting frequency components used in calculating the object blur compensation amount is determined so that frequency components of the object blur amount with a relatively high reliability are reflected more in the object blur compensation amount. In this embodiment, the object motion (amount of blur) detected by a plurality of methods is used at an appropriate ratio (weight) according to the relative reliability, thereby making it possible to improve the accuracy of object blur compensation.

[0105] ●(Second embodiment) Next, a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in the method of calculating the amount of object blur correction from the first amount of object blur and the second amount of object blur. Therefore, the following description will focus on the differences from the first embodiment.

[0106] Fig. 11 is a block diagram showing an example of the functional configuration of an image capture device 1100 that implements an object blur correction control device according to a second embodiment of the present invention. As in Fig. 1, Fig. 11 shows, as functional blocks, components related to object blur correction among the components of the image capture device 1100. Furthermore, components similar to those of the image capture device 100 described in the first embodiment are given the same reference numerals as in Fig. 1, and descriptions thereof will be omitted.

[0107] The imaging device 1100 has a configuration in which the first filter section 106, the second filter section 109, and the filter control section 113 in the imaging device 100 are replaced with a first gain processing section 1102, a second gain processing section 1103, and a gain control section 1101.

[0108] The gain control unit 1101 determines the gains to be applied by the first gain processing unit 1102 and the second gain processing unit 1103 based on the reliability of the first subject blur amount and the relative reliability of the second subject blur amount obtained from the relative reliability determination unit 112. The gain control unit 1101 also sets the determined gains to the first gain processing unit 1102 and the second gain processing unit 1103, respectively.

[0109] The gain control unit 1101 determines a first gain G1 to be applied to the first subject blur amount and a second gain G2 to be applied to the second subject blur amount. The first gain G1 and the second gain G2 have values between 0 and 1, and the sum of G1 and G2 is 1. In this way, the gain control unit 1101 determines the relative weights of the first subject blur amount and the second subject blur amount used for calculating the subject blur correction amount.

[0110] The gain control unit 1101 can determine the gain based on, for example, a pre-registered relationship between the relative reliability and the gain. FIG. 13 shows an example of the relationship between the reliability R1 of the first subject blur amount and the gain. In FIG. 13, the horizontal axis represents the first gain G1 applied by the first gain processing unit 1102, and the vertical axis represents the reliability R1. In this example, a lower limit value Ga and an upper limit value Gb are set for the gain. When the reliability R1 is the minimum value 0, the first gain G1 is Ga, and when the reliability R1 is the maximum value 1, the first gain G1 is Gb. Here, the gain increases linearly as the reliability increases, but the gain may increase non-linearly or discontinuously with respect to the increase in reliability. Values of 0 ≦ Ga, Gb ≦ 1 can be taken within a range that satisfies Ga < Gb and Ga + Gb = 1.

[0111] The gain control unit 1101 determines the first gain G1 so that the higher the reliability R1 of the first object blur amount, the larger the value becomes. Note that the gain control unit 1101 can determine the second gain G2 as a value obtained by subtracting G1 from 1 (G2=1-G1). By determining the first gain G1 and the second gain G2 in this manner, the proportion (weight) of the first or second object blur amount, whichever has a higher relative reliability, in the object blur correction amount becomes larger than the proportion (weight) of the other object blur correction amount.

[0112] The first gain processing unit 1102 is, for example, a multiplier that multiplies an input signal by a gain. The first gain processing unit 1102 applies a first gain G1 set by the gain control unit 1101 to the first amount of object blur. The first gain processing unit 1102 outputs the first amount of object blur to which the first gain G1 has been applied to the object blur correction amount calculation unit 114.

[0113] The second gain processing unit 1103 is, for example, a multiplier that multiplies an input signal by a gain. The second gain processing unit 1103 applies the second gain G2 set by the gain control unit 1101 to the second amount of object blur. The second gain processing unit 1103 outputs the second amount of object blur to which the second gain G2 has been applied to the object blur correction amount calculation unit 114.

[0114] The object blur correction amount calculation unit 114 calculates the object blur correction amount by adding the first object blur amount to which the first gain G1 has been applied and the second object blur amount to which the second gain G2 has been applied. In this way, the object blur correction amount calculated in this embodiment corresponds to a weighted average value of the first object blur amount and the second object blur amount. A larger weight is given to an object blur amount with a higher relative reliability than to an object blur amount with a lower relative reliability, so that the accuracy of the object blur correction amount can be improved.

[0115] (Subject shake correction during video recording) Next, the object blur compensation operation in the imaging device 1100 will be described with reference to the flowchart shown in Fig. 12. The conditions for performing the object blur compensation operation may be the same as those in the first embodiment. In Fig. 12, the steps for performing the same operations as those in the first embodiment are given the same reference numerals as those in Fig. 2, and the description thereof will be omitted.

[0116] The operations of S201 to S206 are the same as those in the first embodiment. In S207, the relative reliability determination unit 112 outputs the relative reliability determined as described above to the gain control unit 1101.

[0117] In S1201, the gain control unit 1101 determines the first gain G1 and the second gain G2 as described above based on the reliability of the first amount of object blurring and the reliability of the second amount of object blurring obtained from the relative reliability determination unit 112. The gain control unit 1101 sets the first gain G1 in the first gain processing unit 1102, and sets the second gain G2 in the second gain processing unit 1103.

[0118] In S1202, the first gain processing unit 1102 applies the first gain G1 to the first amount of object blurring (performs first gain processing). The first gain processing unit 1102 outputs the first amount of object blurring to which the first gain G1 has been applied, to the object blur correction amount calculation unit 114.

[0119] In S1203, the second gain processing unit 1103 applies the second gain G2 to the second amount of object blur (performs second gain processing). The second gain processing unit 1103 outputs the second amount of object blur to which the second gain G2 has been applied to the object blur correction amount calculation unit 114. The operations from S211 onwards are the same as those in the first embodiment.

[0120] In the second embodiment, the gain to be applied to each amount of subject blur is determined so that the amount of subject blur with a relatively high reliability is reflected more in the amount of subject blur compensation. This embodiment also uses the subject movements (amounts of blur) detected by a plurality of methods at appropriate ratios (weights) according to the relative reliability, thereby improving the accuracy of subject blur compensation.

[0121] ●(Third embodiment) Next, a third embodiment of the present invention will be described. This embodiment differs from the first and second embodiments in that image stabilization is performed in addition to subject stabilization.

[0122] Fig. 14 is a block diagram showing an example of the functional configuration of an image capture device 1400 according to a third embodiment of the present invention. In Fig. 14, components related to correction of camera shake and subject blur among components of the image capture device 1400 are shown as functional blocks. Components similar to those of the image capture device 100 described in the first embodiment are given the same reference numerals as in Fig. 1, and descriptions thereof will be omitted.

[0123] The imaging device 1400 has a camera shake detection unit 1401, a camera shake correction amount calculation unit 1402, and a blur correction amount synthesis unit 1403 in addition to the configuration of the imaging device 100. The configuration and operation of the image blur correction control device are the same as those of the first embodiment, except that the subject blur correction amount is output to the blur correction amount synthesis unit 1403 instead of the control unit 120. The configuration of the image blur correction control device may be changed as in the second embodiment.

[0124] The camera shake detection unit 1401 is, for example, a gyro sensor, and outputs a signal (here, an angular velocity signal about each axis of a Cartesian coordinate system) representing the movement of the image capture device 1400 to the camera shake correction amount calculation unit 1402. Note that the movement of the image capture device 1400 may be detected using other known methods. For example, the background vector separated by the subject vector detection unit 107 may be used as the movement of the image capture device 1400.

[0125] The camera shake compensation amount calculation unit 1402 converts the signal obtained from the camera shake detection unit 1401 into the movement of the image capture device 1400. The camera shake compensation amount calculation unit 1402 calculates the camera shake compensation amount for each of the X direction and the Y direction for driving the shake compensation lens so as to cancel the movement of the image capture device 1400. The camera shake compensation amount calculation unit 1402 outputs the calculated camera shake compensation amount to the camera shake compensation amount synthesis unit 1403.

[0126] The camera shake correction amount calculation unit 1402 integrates the angular velocities in the Yaw and Pitch directions obtained from the camera shake detection unit 1401, respectively, and converts them into angles θy and θp. The camera shake correction amount calculation unit 1402 converts the angles θy and θp into displacement amounts V1x and V1y on the imaging surface, which are set as the camera shake correction amount in the X direction and the Y direction. The displacement amounts Vlx and Vly can be calculated as V1x=ftan(θy) and V1y=ftan(θp), where f is the focal length of the optical system 101 [mm].

[0127] The blur correction amount combination unit 1403 calculates a final blur correction amount by combining the object blur correction amount calculated by the object blur correction amount calculation unit 114 and the camera shake correction amount calculated by the camera shake correction amount calculation unit 1402. The blur correction amount combination unit 1403 outputs the calculated blur correction amount to the control unit 120.

[0128] The blur correction amount synthesis unit 1403 weights and synthesizes the object blur correction amount and the camera shake correction amount. There is no particular limit to the method of synthesizing the object blur correction amount and the camera shake correction amount, and any known method can be used. As an example, different frequency bands can be extracted and synthesized. Specifically, the blur correction amount synthesis unit 1403 can synthesize the correction amounts by weighting and adding the low-frequency component of the object blur correction amount and the high-frequency component of the camera shake correction amount based on the reliability of object tracking.

[0129] The blur correction amount synthesis unit 1403 can extract low-frequency components by applying low-pass filter processing to the subject blur correction amount, and extract high-frequency components by applying high-pass filter processing to the camera shake correction amount. Like the filter control unit 113, the blur correction amount synthesis unit 1403 can control the synthesis ratio (weight) of the correction amounts by controlling the cutoff frequency common to the low-pass filter and the high-pass filter.

[0130] Specifically, the blur correction amount synthesis unit 1403 can lower the cutoff frequency as the reliability of the object detection becomes lower. As a result, the weight of the low-frequency components of the object blur correction amount becomes smaller and the weight of the high-frequency components of the camera shake correction amount becomes larger as the reliability of the object detection becomes lower. Alternatively, the cutoff frequency may be lowered as the reliability of the object vector becomes lower.

[0131] (Subject shake correction during video recording) Next, the blur correction operation in the imaging device 1400 will be described with reference to the flowchart shown in Fig. 15. The execution conditions for the subject blur correction operation may be the same as those in the first embodiment. It is also assumed that the camera shake correction is set to be valid. In Fig. 15, the steps that execute the same operations as those in the first embodiment are given the same reference numerals as those in Fig. 2, and the description thereof will be omitted.

[0132] The operations of S201 to S211 are the same as those in the first embodiment. In S211, the object blur correction amount calculation unit 114 outputs the calculated object blur correction amount to the blur correction amount combination unit 1403.

[0133] In S1501, the camera shake detection unit 1401 detects the movement of the image capturing device 1400.

[0134] In S1502, the camera shake compensation amount calculation unit 1402 calculates the amount of camera shake compensation for compensating for the movement detected by the camera shake detection unit 1401. The camera shake compensation amount calculation unit 1402 outputs the calculated amount of camera shake compensation to the camera shake compensation amount combination unit 1403.

[0135] Note that S1501 to S1502 may be executed before S201 to S211 or in parallel with S201 to S211.

[0136] In S1503, the blur correction amount combination unit 1403 combines the object blur correction amount calculated in S211 and the camera shake correction amount calculated in S1502 to calculate a final blur correction amount. The blur correction amount combination unit 1403 outputs the calculated blur correction amount to the control unit 120.

[0137] In S1504, the control unit 120 executes optical or electronic blur correction using the blur correction amount obtained from the blur correction amount synthesis unit 1403. Except for the fact that the correction amount used is different, the operation of the control unit 120 may be the same as in S212.

[0138] In the third embodiment, a blur compensation amount obtained by combining an image stabilization amount with the object blur compensation amount obtained in the first or second embodiment is used. According to this embodiment, it is possible to achieve the effects of the first or second embodiment while performing image stabilization.

[0139] ●(Fourth embodiment) Next, a fourth embodiment of the present invention will be described. In this embodiment, the amount of object blur correction is calculated by combining the first and second embodiments.

[0140] Fig. 16 is a block diagram showing an example of the functional configuration of an image capturing device 1600 according to a fourth embodiment of the present invention. In Fig. 16, components related to correction of subject blurring among components of the image capturing device 1400 are shown as functional blocks. Components similar to those of the image capturing devices 100 and 1100 described in the first and second embodiments are given the same reference numerals as those in Figs. 1 and 11, and descriptions thereof will be omitted.

[0141] The imaging device 1600 has a configuration in which a first gain processing unit 1102, a second gain processing unit 1103, and a gain control unit 1101 are added to the configuration of the imaging device 100. However, the first gain processing unit 1102 applies a first gain G1 to the output of the first filter unit 106. In addition, the second gain processing unit 1103 applies a second gain G2 to the output of the second filter unit 109.

[0142] Note that the order of the first filter unit 106 and the first gain processing unit 1102 may be reversed so that the first amount of object blur is input to the first gain processing unit 1102 and the output of the first gain processing unit 1102 is input to the first filter unit 106. Similarly, the order of the second filter unit 109 and the second gain processing unit 1103 may be reversed so that the second amount of object blur is input to the second gain processing unit 1103 and the output of the second gain processing unit 1103 is input to the second filter unit 109.

[0143] (Subject shake correction during video recording) The flowchart shown in Fig. 17 shows the subject blur correction operation in the imaging device 1600, and S1201 to S1203 shown in Fig. 12 are added between S210 and S211 of the flowchart shown in Fig. 2. The operation in each step is as described in relation to the first or second embodiment.

[0144] In the fourth embodiment, the cutoff frequency of the filter to be applied to the amount of object blur and the gain to be applied to the amount of object blur are determined so that the amount of object blur with a relatively high reliability is reflected more in the amount of object blur compensation. This embodiment also uses the subject movements (amounts of blur) detected by a plurality of methods at appropriate ratios (weights) according to the relative reliability, thereby improving the accuracy of object blur compensation.

[0145] (Other embodiments) As in the third embodiment, a configuration related to camera shake correction may be added to the imaging device 1600 of the fourth embodiment, and camera shake correction may be performed based on a composite camera shake correction amount obtained by combining the subject shake correction amount and the camera shake correction amount.

[0146] In the above-described embodiment, a configuration has been described in which the amount of object blur is calculated before the application of filter processing or gain processing. However, filter processing or gain processing may be applied before the amount of object blur is calculated. For example, the positions of the first filter unit 106 and the first object blur amount calculation unit 105 may be interchanged, or the positions of the first gain processing unit 1102 and the first object blur amount calculation unit 105 may be interchanged. Similarly, the positions of the second filter unit 109 and the second object blur amount calculation unit 108 may be interchanged, or the positions of the second gain processing unit 1103 and the second object blur amount calculation unit 108 may be interchanged.

[0147] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0148] The disclosure of the present embodiment includes the following subject blur correction control device, subject blur correction control method, imaging device, and program. (Item 1) 1. A subject blur correction control device that calculates a correction amount for correcting subject blur caused by a movement of a subject, a first blur amount calculation means for calculating a first object blur amount based on the position of the object area in the captured image; a second blur amount calculation means for calculating a second amount of subject blur based on a subject vector representing a motion of the subject between captured images; and a correction amount calculation means for calculating an amount of object blur correction by adding the first amount of object blur and the second amount of object blur with a weight corresponding to a relative reliability between the first amount of object blur and the second amount of object blur. (Item 2) 2. The object blur correction control device according to item 1, further comprising a determination means for determining the relative reliability based on one or more items of the detection reliability of the object region, the reliability of the object vector, and the type of the object. (Item 3) 3. The object blur correction control device according to item 2, characterized in that, when the object is a type of object that is predetermined as an object whose shape of an object area can change significantly between captured images, the determination means determines the reliability of the first object blur amount to be relatively higher than the reliability of the second object blur amount. (Item 4) 4. The subject blur correction control device according to item 3, wherein when the subject area is an area of ​​a specific subject, and the specific subject is the subject of the predetermined type, the determination means determines the reliability of the first subject blur amount to be relatively higher than the reliability of the second subject blur amount. (Item 5) 4. The object blur correction control device according to item 3, characterized in that, when the variance of the object vectors is equal to or greater than a threshold value, the determination means determines the reliability of the first object blur amount to be relatively higher than the reliability of the second object blur amount. (Item 6) 3. The object blur correction control device according to item 2, characterized in that, when the detection reliability of the object region is less than a threshold value, the determination means determines the reliability of the first object blur amount to be relatively lower than the reliability of the second object blur amount. (Item 7) 3. The object blur correction control device according to item 2, characterized in that, when the reliability of the object vector is less than a threshold value, the determination means determines the reliability of the first object blur amount to be relatively higher than the reliability of the second object blur amount. (Item 8) 8. The object blur correction control device according to any one of items 2 to 7, wherein when the determination means determines the relative reliability based on two or more items of the detection reliability of the object region, the reliability of the object vector, and the type of object, the determination means determines a final relative reliability by taking a weighted average of the relative reliability of the first object blur amount and the second object blur amount determined for each item. (Item 9) 8. The object blur correction control device according to any one of items 2 to 7, wherein when the determination means determines the relative reliability based on two or more items of the detection reliability of the object region, the reliability of the object vector, and the object type, the determination means determines the relative reliability of the first object blur amount and the second object blur amount determined for the item with the highest predetermined priority as a final relative reliability. (Item 10) the correction amount calculation means calculates the object blur correction amount by adding together a low-frequency component of the first object blur amount and a high-frequency component of the second object blur amount; 10. The subject blur correction control device according to any one of items 1 to 9, further comprising filter control means for controlling cutoff frequencies of a low-pass filter means for extracting the low-frequency components and a high-pass filter means for extracting the high-frequency components in accordance with the relative reliability. (Item 11) the correction amount calculation means calculates the object blur correction amount by adding together the first object blur amount to which a first gain has been applied and the second object blur amount to which a second gain has been applied; 10. The subject blur correction control device according to any one of items 1 to 9, further comprising a gain control means for controlling the first gain and the second gain in accordance with the relative reliability. (Item 12) the correction amount calculation means calculates an amount of object blur correction by adding together a low-frequency component of the first amount of object blur to which a first gain has been applied and a high-frequency component of the second amount of object blur to which a second gain has been applied; a filter control means for controlling cutoff frequencies of a low-pass filter means for extracting the low-frequency components and a high-pass filter means for extracting the high-frequency components in accordance with the relative reliability; 10. The subject blur correction control device according to any one of items 1 to 9, further comprising a gain control means for controlling the first gain and the second gain in accordance with the relative reliability. (Item 13) An object blur correction control device according to any one of items 1 to 12, and a blur correction control means for executing optical or electronic blur correction using the subject blur correction amount. (Item 14) a camera shake correction amount calculation means for calculating a camera shake correction amount based on the movement of the imaging device; a combining unit that combines the object blur compensation amount and the camera shake compensation amount to obtain a blur compensation amount, 14. The imaging apparatus according to item 13, wherein the blur correction control means executes the optical or electronic blur correction using the blur correction amount determined by the synthesis means. (Item 15) 1. A subject blur correction control method for determining a correction amount for correcting subject blur caused by a movement of a subject, the method being executed by a subject blur correction control device, the method comprising: Calculating a first amount of object blur based on a position of a subject area in a captured image; calculating a second amount of object blur based on an object vector that represents a movement of the object between captured images; and determining an amount of object blur compensation by adding the first amount of object blur and the second amount of object blur with a weight corresponding to a relative reliability between the first amount of object blur and the second amount of object blur. (Item 16) 13. A program for causing a computer to function as each of the means included in the subject blur correction control device according to any one of items 1 to 12.

[0149] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0150] 100, 1100, 1400, 1600...imaging device, 101...optical system, 102...imaging section, 104...subject detection section, 105...first subject blur amount calculation section, 106...first filter section, 107...subject vector detection section, 108...second subject blur amount calculation section, 109...second filter section, 111...subject information acquisition section, 112...relative reliability determination section, 113...filter control section, 114...subject blur correction amount calculation section, 120...control section

Claims

1. 1. A subject blur correction control device that calculates a correction amount for correcting subject blur caused by a movement of a subject, a first blur amount calculation means for calculating a first object blur amount based on a position of an object area in a captured image; a second blur amount calculation means for calculating a second object blur amount based on an object vector representing a movement of the object between captured images; and a correction amount calculation means for calculating an amount of object blur correction by adding the first amount of object blur and the second amount of object blur with a weight corresponding to a relative reliability between the first amount of object blur and the second amount of object blur.

2. 2. An object blur correction control device according to claim 1, further comprising a determination means for determining the relative reliability based on one or more items of the detection reliability of the object region, the reliability of the object vector, and the type of the object.

3. 3. The subject blur correction control device according to claim 2, wherein, when the subject is a type of subject that is predetermined as a subject whose shape of a subject area can change significantly between captured images, the determination means determines the reliability of the first subject blur amount to be relatively higher than the reliability of the second subject blur amount.

4. 4. The subject blur correction control device according to claim 3, wherein when the subject area is an area of ​​a specific subject, and the specific subject is the subject of the predetermined type, the determination means determines the reliability of the first subject blur amount to be relatively higher than the reliability of the second subject blur amount.

5. 4. An object blur correction control device according to claim 3, wherein, when the variance of the object vectors is equal to or greater than a threshold value, the determination means determines the reliability of the first object blur amount to be relatively higher than the reliability of the second object blur amount.

6. 3. The subject blur correction control device according to claim 2, wherein, when the detection reliability of the subject area is less than a threshold value, the determination means determines the reliability of the first subject blur amount to be relatively lower than the reliability of the second subject blur amount.

7. 3. The subject blur correction control device according to claim 2, wherein, when the reliability of the subject vector is less than a threshold value, the determination means determines the reliability of the first subject blur amount to be relatively higher than the reliability of the second subject blur amount.

8. 3. The subject blur correction control device according to claim 2, wherein when the determination means determines the relative reliability based on two or more items of the detection reliability of the subject area, the reliability of the subject vector, and the type of subject, the determination means determines a final relative reliability by taking a weighted average of the relative reliability of the first subject blur amount and the second subject blur amount determined for each item.

9. 3. The subject blur correction control device according to claim 2, wherein when said determination means determines the relative reliability based on two or more items of the detection reliability of the subject area, the reliability of the subject vector, and the type of subject, said determination means determines the relative reliability of said first subject blur amount and said second subject blur amount determined for the item with the highest predetermined priority as a final relative reliability.

10. the correction amount calculation means calculates the object blur correction amount by adding together a low-frequency component of the first object blur amount and a high-frequency component of the second object blur amount; 2. The subject blur correction control device according to claim 1, further comprising filter control means for controlling cutoff frequencies of the low-pass filter means for extracting the low-frequency components and the high-pass filter means for extracting the high-frequency components in accordance with the relative reliability.

11. the correction amount calculation means calculates the object blur correction amount by adding together the first object blur amount to which a first gain has been applied and the second object blur amount to which a second gain has been applied; 2. An object blur correction control device according to claim 1, further comprising a gain control means for controlling the first gain and the second gain in accordance with the relative reliability.

12. the correction amount calculation means calculates an amount of object blur correction by adding together a low-frequency component of the first object blur amount to which a first gain has been applied and a high-frequency component of the second object blur amount to which a second gain has been applied; a filter control means for controlling cutoff frequencies of a low-pass filter means for extracting the low-frequency components and a high-pass filter means for extracting the high-frequency components in accordance with the relative reliability; 2. An object blur correction control device according to claim 1, further comprising a gain control means for controlling the first gain and the second gain in accordance with the relative reliability.

13. An object blur correction control device according to any one of claims 1 to 12, and a blur correction control means for executing optical or electronic blur correction using the subject blur correction amount.

14. a camera shake correction amount calculation means for calculating a camera shake correction amount based on the movement of the imaging device; a combining unit that combines the object blur compensation amount and the camera shake compensation amount to obtain a blur compensation amount, 14. The image pickup apparatus according to claim 13, wherein the blur correction control means executes the optical or electronic blur correction using the blur correction amount determined by the combining means.

15. 1. A subject blur correction control method for determining a correction amount for correcting subject blur caused by a movement of a subject, the method being executed by a subject blur correction control device, the method comprising: Calculating a first amount of object blur based on a position of a subject area in a captured image; calculating a second amount of subject blur based on a subject vector that represents a motion of the subject between captured images; and determining an amount of object blur compensation by adding the first amount of object blur and the second amount of object blur with a weight corresponding to a relative reliability between the first amount of object blur and the second amount of object blur.

16. A program for causing a computer to function as each of the means included in the subject blur correction control device according to any one of claims 1 to 12.

Images