Vibration isolation control device and method, imaging device, imaging system, program and storage medium

Abstract

To accurately determine erroneous detection or transition of a subject and perform subject blur correction without degrading image quality.SOLUTION: An anti-vibration control device includes detection means for detecting a subject from an input image using a trained model and outputting subject information of the detected subject, tracking means for tracking one subject among the detected subjects in a plurality of continuously input images, reliability calculating means for calculating tracking reliability on the basis of the subject information, and correction quantity calculation means that calculates the amount of blur correction used for blur correction by adjusting the amount of provisional blur correction based on the difference between the tracked subject's position and a predetermined target position in the image on the basis of the reliability.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a vibration control device and method, an imaging device, an imaging system, a program, and a storage medium, and particularly relates to a control technique for subject blur correction.

Background Art

[0002] Conventionally, in an imaging device such as a digital camera, in addition to "camera shake" of a user holding the camera body, "subject blur" caused by a change in the position of a subject such as a person is corrected.

[0003] "Camera shake" can be detected by using an angular velocity sensor attached to the imaging device or the motion vector of a stationary object (for example, the background) between captured images. On the other hand, "subject blur" can be detected by detecting a subject from the captured image and measuring the position of the detected subject in the image.

[0004] In this subject detection, there is a problem of false detection in which a subject different from the intended subject (hereinafter referred to as "target subject") is detected as the target subject (hereinafter referred to as "non-target subject"). When a non-target subject is falsely detected as the target subject from the state where the target subject is detected, a phenomenon occurs in which the subject to be subjected to subject blur shifts (hereinafter referred to as "shift").

[0005] Therefore, Patent Document 1 discloses a technique for preventing the shift of a subject by performing template matching between consecutive frames using a partial image showing the subject as a template and obtaining the reliability of subject detection based on the matching result.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

[0007] Patent Document 1 determines the reliability of subject detection based on the distribution of evaluation values ​​(SAD values) obtained by template matching and the distribution of feature colors in the subject area. However, relying solely on SAD values ​​and feature color distributions makes it difficult to accurately determine the presence of non-target subjects similar to the target subject, and thus it is difficult to sufficiently suppress false detections and transfers of the target subject.

[0008] Furthermore, the technology described in Patent Document 1 determines false detection of a subject based on the correlation of brightness and color information between past frames and the current frame, and therefore cannot address cases where a non-target subject is falsely detected as the target subject in the first frame.

[0009] On the other hand, subject blur correction controls the image so that the target subject remains stable near a predetermined position within the image. However, if a non-target subject is mistakenly detected as the target subject, the target subject's position cannot be stabilized. Furthermore, if a subject is transferred to another subject, the difference in the target subject's position before and after the transfer is reflected in the control, resulting in a change in the target subject's position in the image even though the target subject has not moved, thus degrading image quality.

[0010] This invention was made in view of the above-mentioned problems, and aims to accurately detect false detections and transfers of subjects, and to perform subject blur correction without compromising image quality. [Means for solving the problem]

[0011] To achieve the above objective, the vibration isolation control device of the present invention includes: a first detection means that uses a trained model to detect a subject from an input image and outputs subject information of the detected subject; a tracking means that tracks one subject from among the subjects detected by the first detection means in a plurality of continuously input images; and a reliability calculation means that calculates the reliability of the tracking by the tracking means based on the subject information.The motion blur caused by a change in the position of one of the subjects in the image. A first correction amount calculation means for calculating a first image correction amount used for image correction, and The first correction amount calculation means calculates the first blur correction amount based on a provisional subject blur correction amount, which is the difference between the position of the subject tracked by the tracking means and a predetermined target position in the image, and the reliability. do. [Effects of the Invention]

[0012] According to the present invention, it is possible to accurately detect false detections and transfers of subjects, and to perform subject blur correction without compromising image quality. [Brief explanation of the drawing]

[0013] [Figure 1] A block diagram showing the configuration of an imaging device according to the first embodiment of the present invention. [Figure 2] A flowchart illustrating the subject blur correction process in the first embodiment. [Figure 3] A diagram illustrating the method for calculating the reliability of subject tracking in the first embodiment. [Figure 4] A diagram illustrating the method for calculating the subject blur correction amount in the first embodiment. [Figure 5] A block diagram showing the configuration of the imaging device according to the second embodiment. [Figure 6] A flowchart illustrating the image blur correction process in the second embodiment. [Modes for carrying out the invention]

[0014] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0015] <First Embodiment> FIG. 1 is a block diagram showing the configuration of an imaging device 100 having a vibration control device according to a first embodiment of the present invention. An optical image of a subject formed by an optical system 101 including a plurality of lenses such as a focus lens, a zoom lens, a correction lens for shake correction, and an aperture is converted into an image signal by an imaging unit 102 including an imaging element such as a CMOS sensor or a CCD sensor and output. Note that the optical system 101 may be configured to be detachable from the imaging device 100 as a lens unit. In that case, the imaging device 100 and the lens unit constitute an imaging system. The image signal output from the imaging unit 102 is subjected to development processing such as white balance processing, color (luminance / chrominance signal) conversion, and γ correction in a development processing unit 103, and image data is output.

[0016] A subject detection unit 104 detects one or more subjects from the image data input from the development processing unit 103 using a learned model, and generates and outputs subject information of the detected subjects. The subject information shall include at least any one of the position of each subject, the size of each subject, the type of each subject, and the number of detected subjects. A subject tracking unit 105 tracks any one of the subjects detected by the subject detection unit 104. A tracking reliability calculation unit 106 calculates the reliability of the subject tracking by the subject tracking unit 105 based on the subject information obtained from the subject detection unit 104.

[0017] A subject blur correction amount calculation unit 107 calculates a subject blur correction amount using the position of the subject being tracked by the subject tracking unit 105 and a target position in the image.

[0018] Then, based on the subject blur correction amount calculated by the subject blur correction amount calculation unit 107, blur correction is performed to bring the position of the subject closer to the target position. Note that a known method can be used for the blur correction. For example, an optical blur correction can be used in which a correction lens (not shown) included in the optical system 101 or an imaging element included in the imaging unit 102 is moved on a plane perpendicular to the optical axis. Also, an electronic blur correction in which the pixel positions of the image data output from the imaging unit 102 are shifted and cut based on the subject blur correction amount may be used, or further, an optical blur correction and an electronic blur correction may be used in combination.

[0019] Next, the subject blur correction process in the imaging device 100 having the configuration shown in FIG. 1 will be described using the flowchart shown in FIG. 2. Note that this process is started, for example, when subject blur correction is instructed by an operation member (not shown) during live view display before still image shooting, or when video shooting is instructed. In S201, the subject detection unit 104 detects one or more subjects from the image of the image data input from the development processing unit 103 using a learned model, and generates and outputs subject information of the detected subjects.

[0020] In S202, the subject tracking unit 105 tracks any one of the subjects detected by the subject detection unit 104 in S201. The tracking of the subject can be performed using known template matching with a partial image including the subject to be tracked as a template.

[0021] For the second and subsequent images, the subject that was the tracking target in the immediately previous image is tracked. However, in the case of the first image, the subject to be tracked is selected. At that time, if a plurality of subjects are detected in S201, the most main subject may be selected. For example, the larger the size of the subject, the closer the position of the subject is to the center of the image, and the higher the reliability of subject detection, the more preferentially it is selected. Alternatively, the user may be allowed to select. Furthermore, if the subject to be tracked cannot be tracked in the second and subsequent images, S201 will select one of the subjects detected again and attempt to track it.

[0022] Next, in S203, the tracking reliability calculation unit 106 calculates the reliability of subject tracking by the subject tracking unit 105 in S202 based on the subject information output from the subject detection unit 104 in S201.

[0023] Here, an example of a method for calculating the reliability of subject tracking based on subject information will be explained with reference to Figure 3. As mentioned above, subject information includes at least one of the following: the position of each subject, the size of each subject, the type of each subject, and the number of subjects detected. When multiple subjects are detected, the position information of each subject can be used after being converted into the distance between subjects. For example, the shorter (closer) the distance between the subject being tracked and other subjects, the higher the probability of a transfer to another subject occurring. Therefore, the tracking reliability calculation unit 106 lowers the reliability of subject tracking when the distance between subjects detected by the subject detection unit 104 is short, and increases the reliability of subject tracking when the distance is long (far).

[0024] Furthermore, the smaller the size of the subject, the less information about the subject is included in the template during tracking using template matching, which increases the likelihood of the subject switching to another subject. Therefore, the tracking reliability calculation unit 106 calculates the reliability of subject tracking based on the size information of the subject, lowering it as the size of the subject being tracked decreases and increasing it as the size of the subject increases.

[0025] Information on the type of subject can be used, for example, after being converted into a similarity score between subjects. The higher the similarity score between the subject being tracked and other subjects, the higher the likelihood of a transfer to another subject. For example, transfers are more likely to occur between subjects of the same type, such as people, animals, or vehicles. Also, people and animals have a higher similarity score than people and vehicles, making transfers more likely. Therefore, the tracking reliability calculation unit 106 lowers the reliability of subject tracking the higher the similarity score between subjects detected by the subject detection unit 104, and increases the reliability of subject tracking the lower the similarity score.

[0026] The similarity score can be predetermined and stored depending on the type of subject detected. For example, a high similarity score is set when the detected subjects are both people, a low similarity score is set when the subjects are both people and both vehicles, and a lower similarity score is set when the subjects are both people and both animals, but a higher similarity score is set when the subjects are both people and both vehicles. If the type of animal can be identified, for example, a high similarity score is set when the subjects are both cats, a low similarity score is set when the subjects are both cats and both people, and an even lower similarity score is set when the subjects are both cats and both vehicles or other non-living things. Also, a lower similarity score is set when the subjects are both cats and both dogs, but a higher similarity score is set when the subjects are both cats and both people. The same applies to other animals.

[0027] Furthermore, if three or more subjects are detected, the one with the highest similarity will be used. For example, if one person, one dog, and one train are detected, the similarity between the person and the dog will be used.

[0028] Furthermore, it is believed that the more subjects detected, the higher the probability of subject tracking switching to other subjects. Therefore, based on the information about the number of subjects, the subject tracking reliability calculation unit 106 lowers the reliability of subject tracking when the number of subjects detected by the subject detection unit 104 is large, and increases the reliability of subject tracking when the number is small.

[0029] As described above, the reliability of subject tracking can be obtained based on each type of subject information. For the final subject tracking reliability, thresholds may be set for each type of subject information, the reliability may be quantified into discrete values, and a weighted average may be taken; or the reliability with the highest priority may be adopted.

[0030] Furthermore, the above method can be combined with known methods for calculating the reliability of subject tracking from the distribution of evaluation values ​​(SAD values) obtained by template matching or from the distribution of feature colors in the subject area. In that case, the final subject tracking reliability can be determined by taking a weighted average of the subject tracking reliability values ​​calculated by each method, or by adopting the one with the higher priority.

[0031] In S204, the subject blur correction amount calculation unit 107 first calculates the subject blur correction amount using the position of the subject tracked by the subject tracking unit 105 in S202 and the target position in the image. Then, in S203, it adjusts the subject blur correction amount based on the reliability of subject tracking calculated by the tracking reliability calculation unit 106. Here, the method for calculating the subject blur correction amount will be explained using Figure 4.

[0032] The black square 401 indicates the target position within the image and can be set to any position. In the example in Figure 4, it is set to the center of the imaging screen. The dotted rectangle 402 indicates the area of ​​the subject detected by S201, and the black circle 403 indicates its centroid. The arrow 404 is the difference between coordinates 401 and 403, and this difference is used as the provisional subject blur correction amount.

[0033] It should be noted that, due to image noise and the accuracy of subject detection, the detected subject's position generally has some error. Therefore, it is desirable to apply an LPF (low-pass filter) to the detected subject's position or the amount of subject blur correction, and use the low-frequency component as a provisional amount of subject blur correction.

[0034] Next, the provisional subject blur correction amount is adjusted, taking into account the reliability of subject tracking, to calculate the final subject blur correction amount. If the reliability of subject tracking is low, false detection or transfer of the subject is possible. Therefore, the subject blur correction amount calculation unit 107 adjusts so that the lower the reliability of subject tracking, the smaller the subject blur correction amount becomes. Note that the adjustment of the subject blur correction amount may be performed in steps by setting a threshold for determining reliability. Several methods can be considered to reduce the amount of motion blur correction. For example, one could multiply the image by a gain such that the amount of motion blur correction decreases as the reliability of subject tracking decreases, or by lowering the cutoff frequency of the LPF relative to the amount of motion blur correction, or by reducing the upper limit of the amount of motion blur correction between images. Based on the reliability of subject tracking as described above, the amount of subject blur correction is calculated.

[0035] In S205, motion blur correction is performed based on the motion blur correction amount calculated in S204 by the motion blur correction amount calculation unit 107. Then, in S206, it is determined whether to terminate motion blur correction. Here, for example, it is determined whether an instruction to terminate motion blur correction or an instruction to terminate video recording has been given by an operating member (not shown). If motion blur correction is to continue, the process returns to S201 and the above process is repeated for the image data that is next input, and if it is to terminate, motion blur correction is terminated.

[0036] Through the above processing, it is possible to accurately detect false detections and transfers of subjects in multiple images captured in succession, and to correct subject blur without compromising the image quality.

[0037] The subject blur correction process shown in Figure 2 can be performed in real time on the imaging device 100 during shooting, but if video is shot, it can also be performed during video playback using electronic image stabilization. For example, by recording the captured video on a recording means (not shown) and performing the S201-S206 processes on each frame of the recorded video, motion blur correction can be performed during video playback. Alternatively, during video recording, subject information generated by the subject detection unit 104 can be recorded in addition to the captured video, eliminating the need for subject detection during video playback.

[0038] <Second Embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment, only subject blur was treated as the target of blur correction, but in this embodiment, camera shake is also taken into account when performing blur correction.

[0039] Figure 5 is a block diagram showing the configuration of an imaging device 500 having a vibration isolation control device according to a second embodiment of the present invention. In Figure 5, parts common to the components shown in Figure 1 are denoted by the same reference numerals as in Figure 1 and their descriptions are omitted. In addition to the configuration shown in Figure 1, the imaging device 500 of this embodiment includes a camera shake detection unit 501, a camera shake correction amount calculation unit 502, and a camera shake correction amount synthesis unit 503.

[0040] The image stabilization unit 501 detects camera shake applied to the imaging device 500. The image stabilization unit 501 can, for example, use a gyro sensor, but is not limited to this. The image stabilization amount calculation unit 502 calculates an image stabilization amount to correct the camera shake detected by the image stabilization unit 501. The image stabilization amount synthesis unit 503 synthesizes the subject image stabilization amount output from the subject image stabilization amount calculation unit 107 and the image stabilization amount output from the image stabilization amount calculation unit 502 to calculate the final image stabilization amount (composite image stabilization amount).

[0041] Figure 6 is a flowchart showing the image blur correction operation in the second embodiment. In Figure 6, steps that are common to the steps shown in Figure 2 are denoted by the same reference numerals as in Figure 2 and their descriptions are omitted.

[0042] In S204, when the subject blur correction amount calculation unit 107 calculates the subject blur correction amount, the process proceeds to S601.

[0043] In S601, the image stabilization unit 501 detects camera shake applied to the imaging device 500. Note that the timing of executing S601 does not need to be before S602, and it may be performed before, after, or in parallel with S201 to S204.

[0044] In S602, the image stabilization amount calculation unit 502 calculates the amount of image stabilization to correct the camera shake detected in S601. For example, if the image stabilization detection unit 501 is a gyro sensor, the image stabilization amount can be calculated by integrating the obtained angular velocity signal of the camera shake and inverting its sign.

[0045] In S603, the image stabilization amount synthesis unit 503 synthesizes the subject image stabilization amount obtained in S204 and the camera shake stabilization amount obtained in S602 to calculate the final image stabilization amount (composite image stabilization amount). At this time, it is preferable to dynamically change the synthesis ratio (weight) of the subject image stabilization amount and the camera shake stabilization amount according to the subject tracking reliability calculated by the tracking reliability calculation unit 106 in S203.

[0046] One method of synthesis is to separate and synthesize the signals by frequency. Specifically, an LPF (low-pass filter) is applied to the subject blur correction amount to extract the low-frequency components. On the other hand, an HPF (high-pass filter) is applied to the camera shake correction amount to extract the high-frequency components. The low-frequency components of the subject blur correction amount and the high-frequency components of the camera shake correction amount obtained in this way are then combined by weighting and adding them together based on the reliability of subject tracking.

[0047] Here, weight control can be achieved by roughly matching the cutoff frequencies of the LPF and HPF, and then changing the cutoff frequencies according to the reliability of subject tracking. Specifically, the lower the reliability of subject tracking, the lower the cutoff frequencies of the LPF and HPF, thereby reducing the weight of the low-frequency components of the subject blur correction and increasing the weight of the high-frequency components of the image stabilization.

[0048] In S604, image stabilization is performed based on the final image stabilization amount obtained in S603. Here, as in the first embodiment, optical image stabilization, electronic image stabilization, or a combination of both are possible methods for image stabilization.

[0049] As described above, according to the second embodiment, false detection and transfer of subjects can be accurately determined, and subject blur correction and camera shake correction can be performed without compromising the quality of the image.

[0050] <Other Embodiments> Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0051] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0052] 101: Optical system, 102: Imaging unit, 103: Development processing unit, 104: Subject detection unit, 105: Subject tracking unit, 106: Tracking reliability calculation unit, 107: Subject blur correction amount calculation unit, 501: Camera shake detection unit, 502: Camera shake correction amount calculation unit, 503: Blur correction amount synthesis unit

Claims

1. A first detection means that uses a trained model to detect a subject from an input image and outputs subject information of the detected subject, A tracking means that tracks one subject among the subjects detected by the first detection means in a series of images that are input consecutively, A reliability calculation means that calculates the reliability of the tracking by the tracking means based on the subject information, A first correction amount calculation means for calculating a first blur correction amount used to correct blur caused by a change in the position of one subject in the image. It has, The vibration damping control device is characterized in that the first correction amount calculation means calculates the first blur correction amount based on a provisional subject blur correction amount based on the difference between the position of the subject tracked by the tracking means and a predetermined target position in the image, and the reliability.

2. The vibration isolation control device according to claim 1, characterized in that the tracking means tracks the subject by template matching.

3. The vibration isolation control device according to claim 1 or 2, characterized in that the subject information includes at least one of the position, size, type, and number of subjects detected by the first detection means.

4. The vibration isolation control device according to claim 3, characterized in that when the first detection means detects a plurality of subjects, the reliability is lower when the distance between the plurality of subjects is a first distance than when the distance is a second distance which is longer than the first distance.

5. The vibration isolation control device according to claim 3 or 4, characterized in that the reliability calculation means lowers the reliability when the size of the subject being tracked by the tracking means is a first size compared to when the size is a second size which is larger than the first size.

6. The vibration isolation control device according to any one of claims 3 to 5, characterized in that when the first detection means detects multiple subjects, the reliability calculation means lowers the reliability as the similarity between the type of subject being tracked by the tracking means and the types of other subjects increases.

7. The vibration isolation control device according to any one of claims 3 to 6, characterized in that the reliability calculation means lowers the reliability when the number of subjects detected by the first detection means is a first number compared to when the number is a second number which is less than the first number.

8. The vibration isolation control device according to any one of claims 1 to 7, characterized in that the first correction amount calculation means calculates the first vibration correction amount such that when the reliability is a first reliability, the provisional vibration correction amount is smaller than when the reliability is a second reliability which is higher than the first reliability.

9. The vibration isolation control device according to claim 8, characterized in that the first correction amount calculation means calculates the first vibration correction amount by multiplying it by a gain that makes the provisional vibration correction amount smaller when the reliability is the first reliability than when the reliability is the second reliability.

10. The vibration isolation control device according to claim 8, characterized in that the first correction amount calculation means lowers the cutoff frequency of the LPF for the provisional vibration correction amount when the reliability is the first reliability compared to when the reliability is the second reliability.

11. The vibration isolation control device according to claim 8, characterized in that the first correction amount calculation means reduces the upper limit of the first shake correction amount when the reliability is the first reliability compared to when the reliability is the second reliability.

12. The vibration damping control device according to any one of claims 1 to 11, further comprising an electronic image stabilization means that performs electronic image stabilization to bring the position of the subject closer to the target position based on the first image stabilization amount.

13. A second detection means for detecting camera shake applied to the imaging means that captured the input image, A second correction amount calculation means for calculating a second shake correction amount to correct the camera shake detected by the second detection means, A combining means for calculating a combined image stabilization amount to be used for image stabilization by combining the first image stabilization amount and the second image stabilization amount. The vibration isolation control device according to any one of claims 1 to 12, further comprising the above.

14. The vibration control device according to claim 13, characterized in that the combining means weights and combines the first vibration correction amount and the second vibration correction amount according to the reliability.

15. The vibration isolation control device according to claim 14, characterized in that the combining means weights and adds the low-frequency component of the first vibration correction amount and the high-frequency component of the second vibration correction amount based on the reliability.

16. The vibration isolation control device according to claim 15, characterized in that the combining means reduces the weight of the low-frequency component of the first vibration correction amount and increases the weight of the high-frequency component of the second vibration correction amount as the reliability decreases.

17. The vibration damping control device according to any one of claims 13 to 16, further comprising electronic image stabilization means that performs electronic image stabilization to bring the position of the subject closer to the target position based on the composite image stabilization amount.

18. The vibration isolation control device according to claim 1, characterized in that the reliability represents the likelihood that the tracking means can continuously track the same subject across the plurality of images.

19. A vibration control device according to any one of claims 1 to 18, Imaging means, Optical image stabilization means that performs optical image stabilization and An imaging device characterized by having the following features.

20. An imaging device comprising a vibration isolation control device according to any one of claims 1 to 18, and an imaging means, A lens unit that can be attached to and detached from the imaging device, Optical image stabilization means that performs optical image stabilization and An imaging system characterized by having the following features.

21. The first detection means performs a first detection step in which it detects a subject from an input image using a trained model and outputs subject information of the detected subject, The tracking means performs a tracking step in which it tracks one subject from among the subjects detected in the first detection step in a plurality of images that are input sequentially, The reliability calculation means includes a reliability calculation step that calculates the reliability of the tracking in the tracking step based on the subject information, A first correction amount calculation step for calculating a first blur correction amount used to correct subject blur caused by a change in the position of one subject in the image. It has, The vibration damping control method is characterized in that, in the first correction amount calculation step, the first blur correction amount is calculated based on a provisional subject blur correction amount based on the difference between the position of the subject tracked in the tracking step and a predetermined target position in the image, and the reliability.

22. The second detection means includes a second detection step of detecting camera shake applied to the imaging means that captured the input image, The second correction amount calculation means includes a second correction amount calculation step for calculating a second shake correction amount to correct the camera shake detected in the second detection step, The combining means includes a combining step of combining the first blur correction amount and the second blur correction amount to calculate a combined blur correction amount to be used for blur correction, and The vibration isolation control method according to claim 21, further comprising the above.

23. A program for causing a computer to function as one of the means of a vibration control device according to any one of claims 1 to 18.

24. A computer-readable storage medium storing the program described in claim 23.

Images