Anti-vibration control device and method, imaging device, imaging system, program, and storage medium

Abstract

To accurately determine erroneous detection or transition of a subject and perform subject blur correction without degrading image quality.SOLUTION: An anti-vibration control device includes detection means for detecting a subject from an input image using a trained model and outputting subject information of the detected subject, tracking means for tracking one subject among the detected subjects in a plurality of continuously input images, reliability calculating means for calculating tracking reliability on the basis of the subject information, and correction quantity calculation means that calculates the amount of blur correction used for blur correction by adjusting the amount of provisional blur correction based on the difference between the tracked subject's position and a predetermined target position in the image on the basis of the reliability.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a vibration control device and method, an imaging device, an imaging system, a program, and a storage medium, and particularly to a control technique for subject blur correction.

Background Art

[0002] Conventionally, in an imaging device such as a digital camera, in addition to "camera shake" of a user holding the camera body, "subject blur" caused by a change in the position of a subject such as a person is corrected.

[0003] "Camera shake" can be detected by using an angular velocity sensor attached to the imaging device or the motion vector of a stationary object (e.g., background) between captured images. On the other hand, "subject blur" can be detected by detecting a subject from a captured image and measuring the position of the detected subject in the image.

[0004] In this subject detection, there is a problem of false detection in which a subject different from the intended subject (hereinafter referred to as "target subject") is detected as the target subject. When a non-target subject is falsely detected as the target subject from the state where the target subject is being detected, a phenomenon occurs in which the subject to be subjected to subject blur shifts (hereinafter referred to as "shift").

[0005] Therefore, Patent Document 1 discloses a technique for preventing subject shift by performing template matching between consecutive frames using a partial image showing a subject as a template and obtaining the reliability of subject detection based on the matching result.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

[0007] Patent Document 1 determines the reliability of subject detection based on the distribution of evaluation values ​​(SAD values) obtained by template matching and the distribution of feature colors in the subject area. However, relying solely on SAD values ​​and feature color distributions makes it difficult to accurately determine the presence of non-target subjects similar to the target subject, and thus it is difficult to sufficiently suppress false detections and transfers of the target subject.

[0008] Furthermore, the technology described in Patent Document 1 determines false detection of a subject based on the correlation of brightness and color information between past frames and the current frame, and therefore cannot address cases where a non-target subject is falsely detected as the target subject in the first frame.

[0009] On the other hand, subject blur correction controls the image so that the target subject remains stable near a predetermined position within the image. However, if a non-target subject is mistakenly detected as the target subject, the target subject's position cannot be stabilized. Furthermore, if a subject is transferred to another subject, the difference in the target subject's position before and after the transfer is reflected in the control, resulting in a change in the target subject's position in the image even though the target subject has not moved, thus degrading image quality.

[0010] This invention was made in view of the above-mentioned problems, and aims to accurately detect false detections and transfers of subjects, and to perform subject blur correction without compromising image quality. [Means for solving the problem]

[0011] To achieve the above objective, the vibration isolation control device of the present invention includes: a first detection means that uses a trained model to detect a subject from an input image and outputs subject information of the detected subject; a tracking means that tracks one subject from among the subjects detected by the first detection means in a plurality of continuously input images; a reliability calculation means that calculates the reliability of the tracking by the tracking means based on the subject information; and a first correction amount calculation means that calculates a first blur correction amount to be used for blur correction by adjusting a provisional blur correction amount based on the reliability, which is based on the difference between the position of the subject tracked by the tracking means and a predetermined target position in the image. [Effects of the Invention]

[0012] According to the present invention, it is possible to accurately detect false detections and transfers of subjects, and to perform subject blur correction without compromising image quality. [Brief explanation of the drawing]

[0013] [Figure 1] A block diagram showing the configuration of an imaging device according to the first embodiment of the present invention. [Figure 2] A flowchart illustrating the subject blur correction process in the first embodiment. [Figure 3] A diagram illustrating the method for calculating the reliability of subject tracking in the first embodiment. [Figure 4] A diagram illustrating the method for calculating the subject blur correction amount in the first embodiment. [Figure 5] A block diagram showing the configuration of the imaging device according to the second embodiment. [Figure 6] A flowchart illustrating the image blur correction process in the second embodiment. [Modes for carrying out the invention]

[0014] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0015] <First Embodiment> Figure 1 is a block diagram showing the configuration of an imaging device 100 having a vibration damping control device according to a first embodiment of the present invention. The optical image of a subject formed by the optical system 101, which includes multiple lenses such as a focusing lens, a magnification lens, and a correction lens for image stabilization, as well as an aperture, is converted into an image signal by the imaging unit 102, which includes an image sensor such as a CMOS sensor or a CCD sensor, and output. The optical system 101 may be configured to be detachable from the imaging device 100 as a lens unit. In that case, the imaging device 100 and the lens unit constitute an imaging system. The image signal output from the imaging unit 102 is subjected to development processing such as white balance processing, color (luminance / chrominance signal) conversion, and gamma correction by the development processing unit 103, and image data is output.

[0016] The subject detection unit 104 detects one or more subjects in the image data input from the development processing unit 103 using a trained model, and generates and outputs subject information for the detected subjects. The subject information shall include at least one of the following: the position of each subject, the size of each subject, the type of each subject, and the number of subjects detected. The subject tracking unit 105 tracks any one of the subjects detected by the subject detection unit 104. The tracking confidence calculation unit 106 calculates the confidence level of subject tracking by the subject tracking unit 105 based on the subject information obtained from the subject detection unit 104.

[0017] The subject blur correction amount calculation unit 107 calculates the subject blur correction amount using the position of the subject being tracked by the subject tracking unit 105 and the target position in the image.

[0018] Then, based on the subject blur correction amount calculated by the subject blur correction amount calculation unit 107, blur correction is performed so that the position of the subject approaches the target position. Note that a known method can be used for the blur correction. For example, an optical blur correction in which a correction lens (not shown) included in the optical system 101 or an imaging element included in the imaging unit 102 is moved on a plane perpendicular to the optical axis can be used. Also, an electronic blur correction in which the pixel positions of the image data output from the imaging unit 102 are shifted and cut out based on the subject blur correction amount may be used, or further, an optical blur correction and an electronic blur correction may be used in combination.

[0019] Next, the subject blur correction process in the imaging device 100 having the configuration shown in FIG. 1 will be described using the flowchart shown in FIG. 2. This process is started, for example, when subject blur correction is instructed by an operation member (not shown) during live view display before still image shooting, or when video shooting is instructed. In S201, the subject detection unit 104 detects one or more subjects from the image of the image data input from the development processing unit 103 using a learned model, and generates and outputs subject information of the detected subjects.

[0020] In S202, the subject tracking unit 105 tracks any one of the subjects detected by the subject detection unit 104 in S201. The tracking of the subject can be performed using known template matching with a partial image including the subject to be tracked as a template.

[0021] For the second and subsequent images, the subject that was the tracking target in the previous image is tracked. However, in the case of the first image, the subject to be tracked is selected. At that time, if a plurality of subjects are detected in S201, the most dominant subject may be selected. For example, those with a larger subject size, a position closer to the center of the image, and a higher reliability of subject detection are preferentially selected. Or, the user may be allowed to select. Furthermore, if the subject to be tracked cannot be tracked in the second and subsequent images, S201 will select one of the subjects detected again and attempt to track it.

[0022] Next, in S203, the tracking reliability calculation unit 106 calculates the reliability of subject tracking by the subject tracking unit 105 in S202 based on the subject information output from the subject detection unit 104 in S201.

[0023] Here, an example of a method for calculating the reliability of subject tracking based on subject information will be explained with reference to Figure 3. As mentioned above, subject information includes at least one of the following: the position of each subject, the size of each subject, the type of each subject, and the number of subjects detected. When multiple subjects are detected, the position information of each subject can be used after being converted into the distance between subjects. For example, the shorter (closer) the distance between the subject being tracked and other subjects, the higher the probability of a transfer to another subject occurring. Therefore, the tracking reliability calculation unit 106 lowers the reliability of subject tracking when the distance between subjects detected by the subject detection unit 104 is short, and increases the reliability of subject tracking when the distance is long (far).

[0024] Furthermore, the smaller the size of the subject, the less information about the subject is included in the template during tracking using template matching, which increases the likelihood of the subject switching to another subject. Therefore, the tracking reliability calculation unit 106 calculates the reliability of subject tracking based on the size information of the subject, lowering it as the size of the subject being tracked decreases and increasing it as the size of the subject increases.

[0025] Information on the type of subject can be used, for example, after being converted into a similarity score between subjects. The higher the similarity score between the subject being tracked and other subjects, the higher the likelihood of a transfer to another subject. For example, transfers are more likely to occur between subjects of the same type, such as people, animals, or vehicles. Also, people and animals have a higher similarity score than people and vehicles, making transfers more likely. Therefore, the tracking reliability calculation unit 106 lowers the reliability of subject tracking the higher the similarity score between subjects detected by the subject detection unit 104, and increases the reliability of subject tracking the lower the similarity score.

[0026] The similarity score can be predetermined and stored depending on the type of subject detected. For example, a high similarity score is set when the detected subjects are both people, a low similarity score is set when the subjects are both people and both vehicles, and a lower similarity score is set when the subjects are both people and both animals, but a higher similarity score is set when the subjects are both people and both vehicles. If the type of animal can be identified, for example, a high similarity score is set when the subjects are both cats, a low similarity score is set when the subjects are both cats and both people, and an even lower similarity score is set when the subjects are both cats and both vehicles or other non-living things. Also, a lower similarity score is set when the subjects are both cats and both dogs, but a higher similarity score is set when the subjects are both cats and both people. The same applies to other animals.

[0027] Furthermore, if three or more subjects are detected, the one with the highest similarity will be used. For example, if one person, one dog, and one train are detected, the similarity between the person and the dog will be used.

[0028] Furthermore, it is believed that the more subjects detected, the higher the probability of subject tracking switching to other subjects. Therefore, based on the information about the number of subjects, the subject tracking reliability calculation unit 106 lowers the reliability of subject tracking when the number of subjects detected by the subject detection unit 104 is large, and increases the reliability of subject tracking when the number is small.

[0029] As described above, the reliability of subject tracking can be obtained based on each type of subject information. For the final subject tracking reliability, thresholds may be set for each type of subject information, the reliability may be quantified into discrete values, and a weighted average may be taken; or the reliability with the highest priority may be adopted.

[0030] Furthermore, the above method can be combined with known methods for calculating the reliability of subject tracking from the distribution of evaluation values ​​(SAD values) obtained by template matching or from the distribution of feature colors in the subject area. In that case, the final subject tracking reliability can be determined by taking a weighted average of the subject tracking reliability values ​​calculated by each method, or by adopting the one with the higher priority.

[0031] In S204, the subject blur correction amount calculation unit 107 first calculates the subject blur correction amount using the position of the subject tracked by the subject tracking unit 105 in S202 and the target position in the image. Then, in S203, it adjusts the subject blur correction amount based on the reliability of subject tracking calculated by the tracking reliability calculation unit 106. Here, the method for calculating the subject blur correction amount will be explained using Figure 4.

[0032] The black square 401 indicates the target position within the image and can be set to any position. In the example in Figure 4, it is set to the center of the imaging screen. The dotted rectangle 402 indicates the area of ​​the subject detected by S201, and the black circle 403 indicates its centroid. The arrow 404 is the difference between coordinates 401 and 403, and this difference is used as the provisional subject blur correction amount.

[0033] It should be noted that, due to image noise and the accuracy of subject detection, the detected subject's position generally has errors. Therefore, it is desirable to apply an LPF (low-pass filter) to the detected subject's position or the amount of subject blur correction, and use the low-frequency component as a provisional amount of subject blur correction.

[0034] Next, the provisional subject blur correction amount is adjusted, taking into account the reliability of subject tracking, to calculate the final subject blur correction amount. If the reliability of subject tracking is low, false detection or transfer of the subject is possible. Therefore, the subject blur correction amount calculation unit 107 adjusts so that the lower the reliability of subject tracking, the smaller the subject blur correction amount becomes. Note that the adjustment of the subject blur correction amount may be performed in steps by setting a threshold for determining reliability. Several methods can be considered to reduce the amount of motion blur correction. For example, one could multiply the image by a gain such that the amount of motion blur correction decreases as the reliability of subject tracking decreases, or by lowering the cutoff frequency of the LPF relative to the amount of motion blur correction, or by reducing the upper limit of the amount of motion blur correction between images. Based on the reliability of subject tracking as described above, the amount of subject blur correction is calculated.

[0035] In S205, motion blur correction is performed based on the motion blur correction amount calculated in S204 by the motion blur correction amount calculation unit 107. Then, in S206, it is determined whether to terminate motion blur correction. Here, for example, it is determined whether an instruction to terminate motion blur correction or an instruction to terminate video recording has been given by an operating member (not shown). If motion blur correction is to continue, the process returns to S201 and the above process is repeated for the image data that is next input, and if it is to terminate, motion blur correction is terminated.

[0036] Through the above processing, it is possible to accurately detect false detections and transfers of subjects in multiple images captured in succession, and to correct subject blur without compromising the image quality.

[0037] The subject blur correction process shown in Figure 2 can be performed in real time on the imaging device 100 during shooting, but if video is shot, it can also be performed during video playback using electronic image stabilization. For example, by recording the captured video on a recording means (not shown) and performing the S201-S206 processes on each frame of the recorded video, motion blur correction can be performed during video playback. Alternatively, during video recording, subject information generated by the subject detection unit 104 can be recorded in addition to the captured video, eliminating the need for subject detection during video playback.

[0038] <Second Embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment, only subject blur was treated as the target of blur correction, but in this embodiment, camera shake is also taken into account when performing blur correction.

[0039] Figure 5 is a block diagram showing the configuration of an imaging device 500 having a vibration isolation control device according to a second embodiment of the present invention. In Figure 5, parts common to the components shown in Figure 1 are denoted by the same reference numerals as in Figure 1 and their descriptions are omitted. In addition to the configuration shown in Figure 1, the imaging device 500 of this embodiment includes a camera shake detection unit 501, a camera shake correction amount calculation unit 502, and a camera shake correction amount synthesis unit 503.

[0040] The image stabilization unit 501 detects camera shake applied to the imaging device 500. The image stabilization unit 501 can, for example, use a gyro sensor, but is not limited to this. The image stabilization amount calculation unit 502 calculates an image stabilization amount to correct the camera shake detected by the image stabilization unit 501. The image stabilization amount synthesis unit 503 synthesizes the subject image stabilization amount output from the subject image stabilization amount calculation unit 107 and the image stabilization amount output from the image stabilization amount calculation unit 502 to calculate the final image stabilization amount (composite image stabilization amount).

[0041] Figure 6 is a flowchart showing the image blur correction operation in the second embodiment. In Figure 6, steps that are common to the steps shown in Figure 2 are denoted by the same reference numerals as in Figure 2 and their descriptions are omitted.

[0042] In S204, when the subject blur correction amount calculation unit 107 calculates the subject blur correction amount, the process proceeds to S601.

[0043] In S601, the image stabilization unit 501 detects camera shake applied to the imaging device 500. Note that the timing of executing S601 does not need to be before S602, and it may be performed before, after, or in parallel with S201 to S204.

[0044] In S602, the image stabilization amount calculation unit 502 calculates the amount of image stabilization to correct the camera shake detected in S601. For example, if the image stabilization detection unit 501 is a gyro sensor, the image stabilization amount can be calculated by integrating the obtained angular velocity signal of the camera shake and inverting its sign.

[0045] In S603, the image stabilization amount synthesis unit 503 synthesizes the subject image stabilization amount obtained in S204 and the camera shake stabilization amount obtained in S602 to calculate the final image stabilization amount (composite image stabilization amount). At this time, it is preferable to dynamically change the synthesis ratio (weight) of the subject image stabilization amount and the camera shake stabilization amount according to the subject tracking reliability calculated by the tracking reliability calculation unit 106 in S203.

[0046] One method of synthesis is to separate and synthesize the signals by frequency. Specifically, an LPF (low-pass filter) is applied to the subject blur correction amount to extract the low-frequency components. On the other hand, an HPF (high-pass filter) is applied to the camera shake correction amount to extract the high-frequency components. The low-frequency components of the subject blur correction amount and the high-frequency components of the camera shake correction amount obtained in this way are then combined by weighting and adding them together based on the reliability of subject tracking.

[0047] Here, weight control can be achieved by roughly matching the cutoff frequencies of the LPF and HPF, and then changing the cutoff frequencies according to the reliability of subject tracking. Specifically, the lower the reliability of subject tracking, the lower the cutoff frequencies of the LPF and HPF, thereby reducing the weight of the low-frequency components of the subject blur correction and increasing the weight of the high-frequency components of the image stabilization.

[0048] In S604, image stabilization is performed based on the final image stabilization amount obtained in S603. Here, as in the first embodiment, optical image stabilization, electronic image stabilization, or a combination of both are possible methods for image stabilization.

[0049] As described above, according to the second embodiment, false detection and transfer of subjects can be accurately determined, and subject blur correction and camera shake correction can be performed without compromising the quality of the image.

[0050] <Other Embodiments> Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0051] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0052] 101: Optical system, 102: Imaging unit, 103: Development processing unit, 104: Subject detection unit, 105: Subject tracking unit, 106: Tracking reliability calculation unit, 107: Subject blur correction amount calculation unit, 501: Camera shake detection unit, 502: Camera shake correction amount calculation unit, 503: Blur correction amount synthesis unit

Claims

1. A first detection means for detecting a subject from an input image using a trained model and outputting subject information of the detected subject; a tracking means for tracking one of the subjects detected by the first detection means in a plurality of images that are successively input; a reliability calculation means for calculating a reliability of the tracking by the tracking means based on the subject information; a first correction amount calculation means for calculating a first blur correction amount to be used for blur correction by adjusting a provisional blur correction amount based on a difference between a position of the subject tracked by the tracking means and a predetermined target position in the image based on the reliability; An anti-vibration control device comprising:

2. 2. The image stabilization control device according to claim 1, wherein said tracking means tracks the subject by template matching.

3. 3. The image stabilization control device according to claim 1, wherein the subject information includes at least one of the position, size, and type of each subject detected by the first detection means, and the number of subjects detected by the first detection means.

4. 4. The image stabilization control device according to claim 3, wherein, when the first detection means detects a plurality of subjects, the reliability calculation means sets the reliability lower when the distance between the plurality of subjects is a first distance than when the distance is a second distance longer than the first distance.

5. 5. The image stabilization control device according to claim 3, wherein the reliability calculation means sets the reliability lower when the size of the subject tracked by the tracking means is a first size than when the size of the subject is a second size larger than the first size.

6. 6. The image stabilization control device according to claim 3, wherein, when the first detection means detects a plurality of subjects, the reliability calculation means lowers the reliability as the similarity between the type of the subject tracked by the tracking means and the types of other subjects increases.

7. The image stabilization control device according to any one of claims 3 to 6, characterized in that the reliability calculation means sets the reliability lower when the number of subjects detected by the first detection means is a first number than when the number is a second number that is smaller than the first number.

8. 8. The image stabilization control device according to claim 1, wherein the first correction amount calculation means adjusts the provisional image stabilization amount so that the provisional image stabilization amount is smaller when the reliability is a first reliability than when the reliability is a second reliability higher than the first reliability.

9. 9. The image stabilization control device according to claim 8, wherein the first correction amount calculation means applies a gain such that, when the reliability is the first reliability, the provisional image stabilization amount is smaller than when the reliability is the second reliability.

10. 9. The image stabilization control device according to claim 8, wherein the first correction amount calculation means lowers a cutoff frequency of an LPF for the provisional image stabilization amount when the reliability is the first reliability, compared to when the reliability is the second reliability.

11. 9. The image stabilization control device according to claim 8, wherein the first correction amount calculation means sets an upper limit of the first image stabilization amount to a value smaller when the reliability is the first reliability than when the reliability is the second reliability.

12. 12. The image stabilization control device according to claim 1, further comprising an electronic blur correction unit that performs electronic blur correction so as to bring the position of the subject closer to the target position based on the first blur correction amount.

13. a second detection means for detecting a camera shake applied to the imaging means that captured the input image; a second correction amount calculation means for calculating a second blur correction amount for correcting the camera shake detected by the second detection means; a combining means for combining the first motion compensation amount and the second motion compensation amount to calculate a combined motion compensation amount to be used for motion compensation; 13. The vibration isolation control device according to claim 1, further comprising:

14. 14. The image stabilization control device according to claim 13, wherein the synthesis means weights and synthesizes the first and second image stabilization amounts in accordance with the reliability.

15. 15. The image stabilization control device according to claim 14, wherein the synthesizing means adds a low-frequency component of the first image stabilization amount and a high-frequency component of the second image stabilization amount with weighting based on the reliability.

16. 16. The image stabilization control device according to claim 15, wherein the synthesis means reduces a weighting of a low-frequency component of the first image stabilization amount and increases a weighting of a high-frequency component of the second image stabilization amount as the reliability becomes lower.

17. 17. The image stabilization control device according to claim 13, further comprising an electronic blur correction unit that performs electronic blur correction based on the composite blur correction amount so as to bring the position of the subject closer to the target position.

18. An anti-vibration control device according to any one of claims 1 to 17; An imaging means; an optical blur correction means for performing optical blur correction; An imaging device comprising:

19. An imaging device including the vibration isolation control device according to any one of claims 1 to 17 and an imaging means; a lens unit detachable from the imaging device; an optical blur correction means for performing optical blur correction; An imaging system comprising:

20. A first detection step in which a first detection means detects a subject from an input image using a trained model and outputs subject information of the detected subject; a tracking step of tracking one of the subjects detected in the first detection step in a plurality of images successively input by a tracking means; a reliability calculation step of calculating a reliability of the tracking in the tracking step based on the subject information by a reliability calculation means; a first correction amount calculation step in which a first correction amount calculation means calculates a first blur correction amount to be used for blur correction by adjusting a provisional blur correction amount based on a difference between a position of the subject tracked in the tracking step and a predetermined target position in the image based on the reliability; 13. A vibration isolation control method comprising:

21. a second detection step in which a second detection means detects camera shake applied to an imaging means that captures the input image; a second correction amount calculation step in which a second correction amount calculation means calculates a second blur correction amount for correcting the camera shake detected in the second detection step; a combining step in which a combining unit combines the first blur correction amount and the second blur correction amount to calculate a combined blur correction amount to be used for blur correction; 21. The vibration isolation control method according to claim 20, further comprising:

22. A program for causing a computer to function as each of the means of the vibration isolation control device according to any one of claims 1 to 17.

23. A computer-readable storage medium storing the program according to claim 22.

Images