Imaging apparatus, control method thereof, and storage medium

Abstract

The present application provides a camera device, a control method thereof and a storage medium. The device comprises a detection unit configured to detect a subject from an image acquired by an image acquisition unit; and a determination unit configured to determine whether the subject is the same as a main subject, wherein the determination unit comprises a first determination unit configured to determine whether a coordinate of the subject candidate and a coordinate of the main subject satisfy a distance condition; a second determination unit configured to determine whether a difference between the subject candidate and the main subject is within a predetermined range; and a third determination unit configured to determine whether the first determination unit and the second determination unit have determined positively for a predetermined number of times in succession, and the determination unit determines whether a shift of the main subject has occurred based on determination results of the first to third determination units.

Description

Technical Field

[0001] Aspects of the embodiments relate to apparatus, methods, and non-transitory computer-readable storage media. Background Art

[0002] There are well-known image processing methods for automatically detecting a specific subject pattern (for example, an area of ​​a person's face) from an image. For example, an imaging device such as a digital camera detects a specific subject area from a captured image, and optimizes focus and exposure by using the detection result as a control object. The imaging device performs tracking processing so that the same subject can be continuously tracked regardless of how the position and shape of the subject change over time. There are two methods for such tracking processing. One is a method of comparing detection results of captured images in a time series and tracking detected subjects with a high degree of correlation. The other is a tracking method that uses template matching to extract feature values ​​of the captured image and tracks areas where the consistency of the feature values ​​of the captured image is high.

[0003] Recently, technology is developing to make it easier to photograph the subject the user is aiming at by expanding the range of subject types to be automatically detected, such as not only the facial area of ​​a person but also the whole body of a person and subjects other than people (e.g., animals).

[0004] However, in scenes where detecting a subject is difficult due to its small size or the subject not facing forward, for example, a tracking method using template matching can be used to track the subject after detecting a body part as an "object." However, this method suffers from poor tracking accuracy. Consequently, variations in the shape and color of the subject can cause the tracking position to become unstable, or other subjects with similar color and brightness can be mistakenly tracked. This results in poor tracking accuracy compared to tracking using face detection.

[0005] In this case, if the subject to be photographed becomes detectable during tracking using the template matching method, it is desirable to be able to continue tracking after switching to the subject to be photographed with higher tracking accuracy.

[0006] To address this situation, Japanese Patent Application Laid-Open No. 2013-228930 (hereinafter referred to as Document 1) discloses comparing the estimated positions of face detection areas detected by a face detection unit and other detection components (e.g., human body detection) to determine whether the subjects are identical. Document 1 proposes a method for selecting an estimated area from the results of face detection or human body detection as the subject detection area based on the result of the correlation determination. Therefore, even when a face cannot be detected, tracking accuracy can be improved by using the results of other detection methods, such as human body detection, together.

[0007] Japanese Patent Application Laid-Open No. 2019-8075 (hereinafter referred to as Document 2) proposes a method according to which, when identifying an area of ​​the same subject in multiple frames of images captured in time sequence, judgment is made taking into account that the area should have the same color as the subject in the previous frame and the defocus amount should be within a predetermined range.

[0008] However, according to document 1, when the face detection unit and other detection components perform correlation judgments regarding whether the subjects are the same, the correlation is calculated based on the position and size of the estimated areas respectively detected by the face detection unit and other detection components, and therefore there is a problem that other subjects that temporarily pass by the front or back side of the targeted subject or faces that are erroneously detected in the background, etc. are also erroneously detected as having a high correlation.

[0009] Furthermore, although subjects at different distances are not determined to be the same subject in Document 2, Document 2 does not disclose a method for resolving a situation where distance information cannot be obtained. Summary of the Invention

[0010] According to a first aspect of the embodiment, there is provided an imaging device having an imaging unit that captures images at a predetermined frame rate, the imaging device comprising: a detection unit for detecting a candidate subject as a main subject from an image in a current frame acquired by the imaging unit; and a subject judgment unit for performing processing for judging whether the detected subject is the same as the main subject, and determining the subject judged to be the same as the main subject as the main subject in the current frame, wherein the subject judgment unit comprises: a first judgment unit for judging whether the coordinates of the candidate subject in a two-dimensional space of the image in the current frame detected by the detection unit and the coordinates of the main subject satisfy a distance coefficient based on the corresponding type of the subject. distance condition; a second judgment unit for judging whether the difference in distance from the camera device between the subject candidate and the main subject in the current frame detected by the detection unit is within a predetermined range; and a third judgment unit for judging whether the first judgment unit and the second judgment unit have judged to be affirmative for a predetermined number of times in a row, and in a case where the first judgment unit and the second judgment unit have judged to be affirmative for a predetermined number of times in a row, the subject judgment unit judges that the transfer of the main subject has occurred, and in a case where any one of the first judgment unit, the second judgment unit and the third judgment unit judges to be negative, the subject judgment unit judges the subject with a predetermined priority in the image in the current frame as the main subject.

[0011] According to a second aspect of the embodiment, a control method for a camera device is provided, wherein the camera device has a camera unit, and the camera unit captures images at a predetermined frame rate, the control method comprising: detecting a candidate subject as a main subject from an image in a current frame acquired by the camera unit; and performing processing for determining whether the detected subject is the same as the main subject, and determining the subject determined to be the same as the main subject as the main subject in the current frame, wherein the processing comprises: a first judgment for determining whether the coordinates of the candidate subject in the two-dimensional space of the image in the current frame detected in the detection and the coordinates of the main subject are the same Satisfying a distance condition based on a corresponding type of subject; a second judgment, used to judge whether the difference in distance from the camera device between the subject candidate and the main subject in the current frame detected in the detection is within a predetermined range; a third judgment, used to judge whether the first judgment and the second judgment are judged to be affirmative for a predetermined number of consecutive times; in a case where the first judgment and the second judgment are judged to be affirmative for a predetermined number of consecutive times, it is judged that a transfer of the main subject has occurred; and in a case where any one of the first judgment, the second judgment and the third judgment is negative, the subject with a predetermined priority in the image in the current frame is judged to be the main subject.

[0012] According to a third aspect of the embodiment, there is provided a non-transitory computer-readable storage medium storing a program, wherein the program, when read and executed by a processor in a camera device, causes the camera device to execute a control method for the camera device, wherein the camera device has a camera unit for capturing images at a predetermined frame rate, the control method comprising: detecting a subject as a candidate for a main subject from an image in a current frame acquired by the camera unit; and performing processing for determining whether the detected subject is the same as the main subject, and determining the subject determined to be the same as the main subject as the main subject in the current frame, wherein the processing comprises: a first judgment for determining whether the two-dimensional image in the current frame detected in the detection whether the coordinates of the candidate subject in the space and the coordinates of the main subject satisfy the distance conditions based on the corresponding types of subjects; a second judgment is used to judge whether the difference in distance from the camera device between the candidate subject and the main subject in the current frame detected in the detection is within a predetermined range; a third judgment is used to judge whether the first judgment and the second judgment are judged to be affirmative for a predetermined number of times in a row; in a case where the first judgment and the second judgment are judged to be affirmative for a predetermined number of times in a row, it is judged that a transfer of the main subject has occurred; and in a case where any one of the first judgment, the second judgment and the third judgment is negative, the subject with a predetermined priority in the image in the current frame is judged to be the main subject.

[0013] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram showing a structure of an automatic focusing apparatus according to an embodiment.

[0015] Figure 2 is a flowchart illustrating an operation for main subject selection.

[0016] Figures 3A to 3E 3 is a conceptual diagram showing determination conditions for identical subject determination.

[0017] Figure 4 is a flowchart illustrating an operation for same-subject determination according to the first embodiment.

[0018] Figure 5A and 5B is a conceptual diagram illustrating an operation for same-subject determination.

[0019] Figure 6 2 is a diagram illustrating an operation for transfer judgment.

[0020] Figure 7A and 7B is a conceptual diagram illustrating an operation for transfer judgment.

[0021] Figure 8 is a flowchart illustrating an operation for same-subject determination according to the second embodiment.

[0022] Figure 9A and 9B is a diagram showing determination conditions for same-subject determination according to the second embodiment. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the present invention. A plurality of features are described in the embodiments, but this does not limit the invention to require all such features, and a plurality of such features may be appropriately combined. Furthermore, in the accompanying drawings, identical or similar structures are given the same reference numerals, and redundant descriptions thereof are omitted.

[0024] First embodiment

[0025] Figure 1 1 is a block diagram of the structure of an imaging apparatus, typically a digital camera, to which the exemplary embodiments are applied. The imaging apparatus includes an imaging lens 101, an aperture 102, a focus lens 104, an AF processing unit 105, an imaging device 108, an A / D conversion unit 109, an image processing unit 110, a WB (white balance) processing unit 111, a format conversion unit 112, a RAM 113, and an image recording unit 114. The electronic camera also includes a system control unit 115, a VRAM 116, an operation display unit 117, an operation unit 118, a switch 121, and a subject detection unit 123.

[0026] The system control unit 115 includes a CPU, a ROM storing programs to be executed by the CPU and various setting values, and a RAM to be used as a work area by the CPU. The CPU executes the programs stored in the ROM and controls Figure 1 Note that this can be achieved by using the CPU of the system control unit 115 (ie, by using software). Figure 1 One or more components of .

[0027] The imaging lens 101 has a zoom mechanism and forms an optical image of a subject on the imaging device 108. The aperture 102 controls the amount of light incident on the imaging lens 101 and adjusts the depth of field. The focus lens 104 focuses the optical image passed through the imaging lens 101 and the aperture 102 onto the imaging plane of the imaging device 108. The AF processing unit 105 drives the focus lens 104 based on a control signal from the system control unit 115.

[0028] The imaging device 108 functions as a light receiving element or a photoelectric conversion element, and converts the optical image formed on the imaging surface into an electrical signal. The A / D conversion unit 109 converts the signal from the imaging device 108 into a digital signal. The A / D conversion unit 109 includes a CDS (correlated double sampling) circuit for noise removal and a nonlinear amplifier circuit for processing before A / D conversion.

[0029] The image processing unit 110 performs various processes on the captured image data from the A / D conversion unit 109. The image processing unit 110 includes a brightness calculation unit that calculates the brightness of the subject in the captured image and an extraction unit that extracts a signal component in a specific frequency band from the output of the brightness calculation unit.

[0030] The WB processing unit 111 performs white balance processing on the image data passed through the image processing unit 110 under the control of the system control unit 115. The format conversion unit 112 converts the format of the image data passed through the image processing unit 110. This conversion also includes processing related to encoding and decoding. Therefore, the format conversion unit 112 also performs decoding processing on the encoded image data recorded on the recording medium. The RAM 113 is a writable memory capable of high-speed access, such as SRAM or DRAM. The image recording unit 114 serves as an interface with non-volatile recording media such as SD cards, performs recording processing on the recording medium, and reads out image data recorded thereon.

[0031] The VRAM 116 is a memory for storing image data to be displayed. The operation display unit 117 displays the image developed on the VRAM 116, performs a display for operation assistance, and also displays the status of the camera, the image pickup screen during image pickup, and the focus detection area.

[0032] The operation unit 118 includes various switches and buttons and also includes a touch panel and the like, and has a function of transmitting a user's operation to the system control unit 115 .

[0033] The switch 121 is a shooting standby switch (also referred to as SW1), which is a focus position confirmation indicator for performing shooting standby operations such as AF and AE. As a result of operating the switch 121 (SW1), the focus position of the image captured by the focus lens 104 is confirmed.

[0034] The subject detection unit 123 detects various types of subjects from the image data acquired through processing by the image processing unit 110, acquires one or more information (type, position, size, reliability, and distance) related to the detected subject, and also detects the main subject to be processed by the AF processing unit 105 and the WB processing unit 111.

[0035] The subject detection unit 123 also performs tracking processing for tracking the same subject as the main subject in consecutive images in a time series. Tracking processing is performed by first determining whether the subjects in the currently captured frame image include a subject that is highly correlated with the previous main subject based on subject information (type, position, size, reliability, and distance), and then tracking the highly correlated subject as the main subject. If a subject that is highly correlated with the previous main subject is not found among the subjects in the currently captured frame, the subject detection unit 123 extracts the area of ​​the previous main subject and the corresponding feature values ​​(such as color) of the currently captured image, and tracks the area with a high degree of consistency with the feature values ​​of the previously detected subject area as the main subject.

[0036] The RAM 113 is used as a high-speed buffer as a temporary image storage means or as a working memory for image compression / decompression. Examples of the operation unit 118 include the following: a menu switch for making various settings such as the setting of the image pickup function of the image pickup device and the setting of image playback; a zoom lever for instructing the image pickup lens to perform a zoom operation; and an operation mode switch for switching between an image pickup mode and a playback mode.

[0037] Next, refer to Figure 2 The flowchart in FIG. 1 describes the main subject selection process performed by the subject detection unit 123 according to the embodiment. Here, the main subject to be photographed is selected based on the information about the detected subject and the information about the subject to be tracked. In addition, the imaging device 108 captures images at a frame rate of, for example, 30 frames per second, and Figure 2 The processing performed when one frame of image data is acquired is shown.

[0038] In step S201, the subject detection unit 123 refers to the image data of the current frame processed by the image processing unit 110 to detect a subject. Examples of subjects to be detected include the face or entire body of a person, the face or entire body of an animal, and subjects with significant color or brightness. The subject detection unit 123 acquires information related to the subject to be detected, including the type, position, size, reliability, and distance of the subject to be detected. The detected subject serves as a candidate for the main subject.

[0039] In step S202, the subject detection unit 123 determines whether the main subject exists in the previous frame. If it is determined that the main subject exists in the previous frame, the subject detection unit 123 branches the process to step S203, and if it is determined that the main subject does not exist, the subject detection unit 123 branches the process to step S206.

[0040] In step S203 , the subject detection unit 123 performs tracking processing for determining a subject highly correlated with the main subject in the previous frame, and determines a main subject candidate in the image of the current frame. Thereafter, the subject detection unit 123 advances the processing to step S204 .

[0041] In step S204, the subject detection unit 123 performs the same subject determination process (see later). Figures 3A to 3E 、 Figure 4 、 Figure 5A and Figure 5B (described in detail), and the process proceeds to step S205. Here, if it is determined that the subject detected in step S201 is the same subject as the primary subject candidate determined in step S203, the subject detection unit 123 performs a transition based on the same subject determination and determines the detected subject as the primary subject candidate. If the transition based on the same subject determination is not performed, the primary subject candidate determined in step S203 will continue to be used as the primary subject candidate.

[0042] In step S205, the subject detection unit 123 determines whether a subject with a higher priority than the primary subject candidate set in step S203 exists, and the operation for selecting the primary subject ends. If a higher-priority subject exists, the subject detection unit 123 determines the higher-priority subject to be used as the primary subject. If no higher-priority subject exists, the subject detection unit 123 determines the primary subject candidate to be the current primary subject. For example, the subject detection unit 123 may determine that a subject has a higher priority for use as the primary subject if the subject is located closer to the center of the image, is larger, is more reliable, or is closer to the camera. Furthermore, the priority of the detected subject type, such as a person or animal, or the priority of the detected subject part, such as a face or the whole body, may be pre-set.

[0043] In this example of priority determination processing, if the user sets high priority for a human face, the human face is determined to have high priority when a human face, a human's entire body, or an animal's face is detected. Furthermore, if multiple human faces are detected, an evaluation value indicating the priority of the primary subject is calculated based on the subject detection information, and the priorities are compared. For example, a position evaluation value based on proximity to the center of the screen and a size evaluation value based on the size of the detected face can be calculated, and the priority evaluation value can be calculated by multiplying the position evaluation value and the size evaluation value.

[0044] In step S206, the subject detection unit 123 estimates the priority of the subject detected in step S201 as the main subject, as determined in step S205, and determines the subject with the highest priority as the main subject as the current main subject, and ends the operation for selecting the main subject.

[0045] Next, refer to Figures 3A to 3E and Figure 4 To describe Figure 2 Details of the same subject determination process in step S204 are described.

[0046] According to this embodiment, the following three conditions are used for the same subject determination:

[0047] (1) Conditions related to distance in the XY directions.

[0048] (2) Conditions related to the distance in the Z direction.

[0049] (3) A condition related to the number of times the conditions related to the distances in the XY direction and the Z direction are continuously satisfied.

[0050] First, "(1) Conditions Related to the Distance in the XY Directions" will be described. Note that X and Y mentioned here are represented by coordinates in a two-dimensional space in the horizontal and vertical directions of the image. "(1) Conditions Related to the Distance in the XY Directions" are conditions related to the overlap between the peripheral area of ​​the main subject candidate and the peripheral area of ​​the transfer destination in the same subject judgment in the coordinates of the two-dimensional space. In this regard, judgment is made using a threshold value of the distance corresponding to the combination of the type of the main subject candidate and the type of the subject at the transfer destination.

[0051] First, if Figure 3AAs shown, in the case where the detection at the transfer destination is face detection (3a-1) and the detection at the main subject candidate is the object detection area (3a-2), if the subject area (3a-3) estimated based on the result of face detection and the object area (3a-2) of the main subject candidate overlap, the condition is satisfied because the relationship between their respective types and distances is reasonable. Here, as an example of the estimated subject area (3a-3), an area of ​​3 frames × 10 frames in the case where the face detection size corresponds to one frame is estimated to be the area of ​​the whole body of a person. In the horizontal direction, the estimated area is set as the range covering the torso by arranging three frames centered on the face position. In the vertical direction, the area covering the whole body of a person is set as the estimated area by setting one frame on the face and eight frames below the face to cover the body to the bottom of the feet.

[0052] like Figure 3B As shown, when the detection at the transfer destination is face detection (3b-1) and the detection at the main subject candidate is full-body detection (3b-2), if the area (3b-3) obtained by multiplying the full-body detection area (3b-2), which is the main subject candidate area, by a predetermined number overlaps with the face detection area (3b-1), the condition is determined to be satisfied. When tracking is performed using the results of full-body detection as the main subject, tracking tends to become unstable due to changes in the shape of the whole body. Therefore, tracking can be stabilized by shifting detection to face detection.

[0053] Then, if Figure 3C As shown, in a case where the detection at the transfer destination is a full-body detection (3c-1) and the detection at the main subject candidate is an object detection area (3c-2), if the area (3c-3) obtained by multiplying the full-body detection area (3c-1) by a predetermined number overlaps with the main subject candidate area (3c-2), it is judged that the condition is met.

[0054] As a result of the above, for example, in a case where the “face” cannot be detected during tracking of the “face” as the main subject, and in a case where a part of a person other than the face (such as the “body”, etc.) is detected, and in a case where it is located within a distance corresponding to the type of the detected subject, it is judged that they are the same.

[0055] Next, "(2) Conditions related to the distance in the Z direction" will be described. Z mentioned here represents the distance from the imaging device. Figure 3DAs shown, when the subject candidate (3d-2) is captured by the imaging device (3d-1), the AF processing unit 105 obtains the difference between the distance to the subject candidate (3d-2) and the distance to the subject (3d-3) at the transfer destination. If the difference between the distances is within a predetermined range of the Z distance, it is determined that the condition is satisfied. On the other hand, if the difference is outside the Z distance range, there is a high possibility that the subject is another subject (3d-4) in the background or foreground, and it is determined that the condition related to the same subject is not satisfied.

[0056] Next, “(3) Condition related to the number of times the conditions related to the distances in the XY and Z directions are continuously satisfied” will be described. This condition is provided to prevent other subjects that happen to pass by from being erroneously determined to be the same subject.

[0057] Figure 3E The following shows the changes in the detected position of the subject and the tracking position of the main subject candidate in the time series from the captured image in the first frame to the captured image in the fifth frame at the start of the same-subject determination. The following describes the same-subject determination processing performed in this scenario when the number of consecutive times in "(3) Conditions Related to the Number of Times Conditions Related to the Distances in the X, Y, and Z Directions are Successively Satisfied" is "5."

[0058] exist Figure 3E In the captured image of the first frame in , for the main subject candidate (3e-1), there are a face detection area (3e-2) and a face detection area (3e-3) that satisfy "(1) the condition related to the distance in the XY direction" and "(2) the condition related to the distance in the Z direction". The face detection area (3e-2) is a face detection area for the same subject as the main subject candidate (3e-1). The face detection area (3e-3) indicates a subject different from the main subject candidate (3e-1), but exists directly behind the main subject candidate (3e-1) and the distance is very short. Therefore, it is not possible to use "(1) the condition related to the distance in the XY direction" and "the condition related to the distance in the Z direction" to determine whether the face detection area (3e-2) indicates another subject.

[0059] exist Figure 3EIn the captured image of the third frame, the position of the main subject candidate (3e-1) moves within the XY plane, and therefore, the difference in distance from the face detection area (3e-3) in the XY direction changes. At this time, "(1) the condition related to the distance in the XY direction" and "(2) the condition related to the distance in the Z direction" are satisfied for the face detection area (3e-2) and the face detection area (3e-3). However, the current frame is the third frame since the check of the conditions for identical subject judgment, and therefore "(3) the condition related to the number of times the conditions related to the distances in the XY direction and the Z direction are continuously satisfied" is not satisfied. Therefore, at this time, the subject is not judged to be the same subject.

[0060] exist Figure 3E In the captured image of the fifth frame, as a result of the significant movement of the tracking position of the main subject candidate (3e-1) within the XY plane, the face detection area (3e-3) does not satisfy "(1) the condition related to the distance in the XY directions," and therefore it can be determined that the subject is not the same as the main subject candidate (3e-1). In contrast, the face detection area (3e-2) satisfies "(1) the condition related to the distance in the XY directions" and "(2) the condition related to the distance in the Z direction" for five consecutive frames, and also satisfies "(3) the condition related to the number of times the conditions related to the distance in the XY and Z directions are continuously satisfied." Therefore, it can be correctly determined that the subject is the same.

[0061] As described above, in a scene where it is difficult to distinguish between different subjects based on "(1) conditions related to the distance in the XY directions" and "(2) conditions related to the distance in the Z direction", correct judgment can be made based on "(3) conditions related to the number of times the conditions related to the distance in the XY directions and the Z direction are continuously met".

[0062] Next, refer to Figure 4 describe Figure 2 Here, the conditions for the same subject judgment have been checked, and whether to perform the same subject judgment is determined based on the degree of necessity of transferring to the subject at the destination of the same subject judgment.

[0063] In step S401, the subject detection unit 123 checks whether Figure 2The subject detection unit 123 determines whether the main subject candidate determined in step S203 is a subject for which same-subject determination is to be performed. If it is determined that same-subject determination is to be performed on the subject, the subject detection unit 123 branches the process to step S402. If it is determined that same-subject determination is not to be performed on the subject, the subject detection unit 123 branches the process to step S408. For example, when a human face is selected as the target for the transfer destination based on same-subject determination, the detection of objects that may be the same subject as the face or the entire human body is set as the target of same-subject determination, and main subject candidates that may not be the same subject as the human face, such as human faces, animal faces, and entire animal bodies, are excluded. Furthermore, when the decision to perform same-subject determination or the transfer destination of same-subject determination is changed according to a menu setting, the decision to perform same-subject determination can be made based on the menu setting.

[0064] In step S402, the subject detection unit 123 determines whether there is a target object to be transferred to and which satisfies the reference condition at the transfer destination for the previous main subject. Figures 3A to 3E The subject detection unit 123 branches the process to step S403 if it determines that there is a subject to be transferred to, and branches the process to step S408 if it determines that there is no such subject.

[0065] In step S403, the subject detection unit 123 determines whether the AF processing unit 105 has successfully acquired the distance information about the subject at the transfer destination determined in step S402. If it is determined that the distance information has not been successfully acquired, the subject detection unit 123 branches the process to S404. If it is determined that the distance information has been successfully acquired, the subject detection unit 123 branches the process to S406.

[0066] In step S404, the subject detection unit 123 performs a transfer determination process (which will be referred to later). Figure 5A 、 5B and 6 for description), and advances the processing to step S405.

[0067] In step S405, the subject detection unit 123 determines whether the necessity of the transfer according to the transfer determination in step S404 is high or low. If it is determined that the necessity of the transfer is high, the subject detection unit 123 branches the process to step S409, and if it is determined that the necessity is low, the subject detection unit 123 branches the process to step S408.

[0068] In step S406, the subject detection unit 123 determines whether the subject at the transfer destination satisfies the reference Figures 3A to 3E The subject detection unit 123 branches the process to step S407 if it determines that the condition is satisfied, and branches the process to step S408 if it determines that the condition is not satisfied.

[0069] In step S407, the subject detection unit 123 determines whether the subject at the transfer destination satisfies the reference condition for at least a predetermined number of times in succession. Figures 3A to 3E The subject detection unit 123 branches the processing to step S409 in the event that it determines that the “conditions (1) concerning the distances in the XY directions” and “conditions (2) concerning the distances in the Z direction” have been satisfied consecutively for a predetermined number of times. In the event that it determines that these conditions have not been satisfied consecutively for a predetermined number of times, the subject detection unit 123 branches the processing to step S408.

[0070] In step S408, the subject detection unit 123 maintains the previous main subject as the main subject candidate, and Figure 5A As shown, in a case where the result of face detection at the transfer destination (5a) and the main subject candidate (5b) are not determined to be the same subject, the subject detection unit 123 maintains the subject candidate (5b) as the main subject candidate.

[0071] In step S409, the subject detection unit 123 determines the subject at the transfer destination as a main subject candidate, and Figure 5B As shown, when the result (5c) of face detection at the transfer destination and the main subject candidate (5d) are determined to be the same subject, the subject detection unit 123 transfers the main subject candidate to the result (5c) of face detection.

[0072] Next, refer to Figure 5A 、 Figure 5B and Figure 6 right Figure 4 The transfer determination processing of step S404 will be described.

[0073] The subject detection unit 123 determines the necessity of transfer based on a combination of the following three conditions.

[0074] (1) Movement amount of the main subject candidate

[0075] (2) Tracking reliability of main subject candidates

[0076] (3) AF reliability of main subject candidates

[0077] Regarding “(1) Movement amount of the main subject candidate”, whether or not there is movement is determined based on whether or not the amount of change in the main subject position in time series exceeds a predetermined threshold. For example, when Figure 2 When the main subject is determined in step S206, the difference from the previous subject position may be constantly calculated and stored, and the presence or absence of movement may be determined based on an average value of main subject determinations performed a predetermined number of times in the past.

[0078] Regarding “(2) Tracking reliability of main subject candidates”, when Figure 2 When the main subject is determined in step S206, the reliability of the subject acquired from the subject detection unit 123 is stored, and the reliability regarding whether the main subject is being correctly tracked is judged step by step based on whether the reliability exceeds a predetermined threshold.

[0079] Regarding “(3) AF reliability of the main subject candidate”, reliability regarding whether AF is being correctly performed on the main subject is determined step by step based on whether the reliability of AF acquired from the AF processing unit 105 exceeds a predetermined threshold.

[0080] like Figure 6 As shown, the degree of necessity of the transfer is determined based on a combination of the determination results according to (1) to (3) above.

[0081] For example, in Figure 7A In FIG. 7 , the main subject candidate shown by reference numeral 7 a is a still subject, the tracking reliability and AF reliability of which are high, and the subject is tracked satisfactorily. Figure 6 As shown in the range drawn by the dotted line denoted by reference numeral 601 in FIG. 8 , the necessity of transferring the face detection result to the transfer destination denoted by reference numeral 7 b is low.

[0082] Then, in Figure 7B In FIG. 7 , the main subject candidate shown by reference numeral 7 c is moving, its tracking reliability and AF reliability are low, and it is difficult to track the subject. Figure 6 As shown in the range drawn by the solid line shown by the reference numeral 602 in the figure, the necessity of transferring to the face detection at the transfer destination shown by the reference numeral 7d is high, and if there is a subject that satisfies the conditions for the same subject judgment except for (2) the condition related to the distance in the Z direction, the transfer to other main subjects is to be performed.

[0083] As described above, according to the first embodiment, even when information related to the distance to the subject cannot be acquired, the necessity of a transfer can be determined by the same subject determination. If the necessity of a transfer is determined to be high, the transfer can be performed to a subject that meets the conditions of the same subject determination, thereby continuously tracking the subject aimed at by the user to maintain the quality of the frame display.

[0084] Second embodiment

[0085] Will refer to Figure 8 describe Figure 2 The same subject determination according to the second embodiment is shown in step S204. Note that the device structure is the same as that of the first embodiment. Figure 1 The device structure is the same as in , and its description will be omitted.

[0086] In step S801, the subject detection unit 123 checks whether Figure 2 The subject detection unit 123 determines whether the main subject candidate determined in step S203 is the subject for which same-subject judgment is to be performed. If it is determined that same-subject judgment is to be performed on the subject, the subject detection unit 123 branches the processing to step S802; otherwise, the processing branches to step S809. For example, in the case of targeting a human face as the target of the transfer destination based on same-subject judgment, the subject detection unit 123 sets the detection of an object or a human body that may be the same subject as the face as the target of same-subject judgment, and excludes main subject candidates such as human faces, animal faces, and animal bodies that may not be the same subject as the human face. In addition, in the case of changing whether to perform same-subject judgment or changing the transfer destination of same-subject judgment according to menu settings, whether to perform same-subject judgment can be determined according to the menu settings.

[0087] In step S802, the subject detection unit 123 determines whether there is a candidate reference object that satisfies the requirements for the main subject. Figures 3A to 3E The subject detection unit 123 branches the process to step S803 if it determines that there is a subject that satisfies the “(1) condition related to the distance in the XY direction” described above, and branches the process to step S809 if it determines that there is no such subject.

[0088] In step S803, the subject detection unit 123 determines whether the distance information about the subject that satisfies "(1) the condition regarding the distance in the XY directions" in step S802 has been acquired by the AF processing unit 105. If it is determined that the AF processing unit 105 has acquired the distance information, the subject detection unit 123 branches the process to step S807. If it is determined that the distance information has not been acquired, the subject detection unit 123 branches the process to step S804.

[0089] In step S804, the subject detection unit 123 performs a reference Figure 5A 、 Figure 5B and Figure 6 Thereafter, the subject detection unit 123 advances the process to step S805. Here, reference will be made to Figure 9A and 9B Describes the conditions for identical subject judgment.

[0090] Figure 9A A specific example of "(1) Conditions Related to the Distance in the XY Directions" according to the second embodiment is shown. Regarding the case where distance information in the Z direction (depth direction of the imaging device) can be acquired, the range in which the same subject can exist is set as a condition with a margin, based on the subject of the transfer destination determined based on the same subject. However, regarding the case where distance information in the Z direction cannot be acquired, the range of the condition to be set as the distance in the XY directions is limited, taking into account the risk of the subject being transferred to other subjects due to the inability to distinguish the subject from other subjects adjacent to the subject at a certain distance in the XY directions.

[0091] Regarding the case where the necessity of transfer is low, the range to be set as the condition of the distance in the XY directions is further restricted to be narrower than the case where the necessity of transfer is high, thereby placing more emphasis on avoiding the risk of transfer to other subjects.

[0092] Figure 9B A specific example of "(3) the condition related to the number of times the conditions related to the distances in the XY and Z directions are continuously satisfied" is shown. However, if distance information can be acquired from part of the screen, or if the distance in the Z direction cannot be acquired due to poor AF accuracy, "(2) the condition related to the distance in the Z direction" is excluded from the conditions.

[0093] As reference Figure 9AAs described above, when distance information in the Z direction cannot be acquired, the value of the condition related to the number of times is increased, thereby temporarily preventing other subjects adjacent to the subject at a certain distance from the subject in the XY directions from being determined to be the same subject due to the inability to distinguish the subject from the other subjects adjacent to the subject at a certain distance. In cases where the necessity for transfer is low, the value of the condition related to the number of times is set to be even larger than when the necessity for transfer is high, thereby placing greater emphasis on avoiding the risk of transfer to other subjects.

[0094] Will further describe Figure 8 In step S806, the subject detection unit 123 determines whether the condition for same-subject determination is satisfied based on the condition for same-subject determination set in step S805. If it is determined that the condition for same-subject determination is satisfied, the subject detection unit 123 branches the process to step S810, and if it is determined that the condition for same-subject determination is not satisfied, the subject detection unit 123 branches the process to step S809.

[0095] In step S807, the subject detection unit 123 determines whether the subject at the transfer destination satisfies the reference Figures 3A to 3E “(2) Condition related to the distance in the Z direction” described above. If the subject at the transfer destination is determined to satisfy “(2) Condition related to the distance in the Z direction”, the subject detection unit 123 branches the process to step S808, and if the subject at the transfer destination is determined not to satisfy the condition, the subject detection unit 123 branches the process to step S809.

[0096] In step S808, the subject detection unit 123 determines whether "(1) the condition related to the distance in the XY direction" and "(2) the condition related to the distance in the Z direction" are satisfied at least a predetermined number of times in succession. If it is determined that the condition related to the number of times is satisfied, the subject detection unit 123 branches the process to step S810. If it is determined that the condition is not satisfied, the subject detection unit 123 branches the process to step S809.

[0097] In step S809 , the subject detection unit 123 continuously maintains the main subject candidate as the main subject candidate, and terminates the operation for the same subject judgment.

[0098] In step S810 , the subject detection unit 123 determines the subject at the transfer destination as a main subject candidate, and terminates the operation for the same subject judgment.

[0099] As described above, according to the second embodiment, even when distance information related to the subject cannot be acquired, same subject judgment can be performed based on the necessity of transfer through same subject judgment, thereby continuously tracking the subject aimed by the user to maintain the quality and reliability of the frame display.

[0100] Other embodiments

[0101] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0102] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An imaging device having an imaging unit configured to capture images at a predetermined frame rate, the imaging device comprising: a detection unit, configured to detect a candidate subject serving as a main subject from the image in the current frame acquired by the imaging unit; as well as a subject determination unit configured to determine whether the detected subject is the same as the main subject, and to determine the subject determined to be the same as the main subject as the main subject in the current frame; The subject determination unit includes: a first determination unit configured to determine whether the coordinates of the subject serving as the candidate for the main subject and the coordinates of the main subject in the two-dimensional space of the image in the current frame detected by the detection unit satisfy a distance condition corresponding to a combination of the type of the subject serving as the candidate for the main subject and the type of the main subject; a second determining unit configured to determine whether a difference between a distance between a candidate subject serving as the main subject in a current frame detected by the detecting unit and the imaging device and a distance between the main subject and the imaging device is within a predetermined range; and a third judging unit, configured to judge whether the first judging unit and the second judging unit have judged positively for a predetermined number of consecutive times, and When the first determination unit and the second determination unit make affirmative determinations a predetermined number of times in succession, the subject determination unit determines that the main subject has shifted, and When any one of the first determination unit, the second determination unit, and the third determination unit makes a negative determination, the subject determination unit determines a subject having a higher priority in the image in the current frame as a main subject.

2. The imaging device according to claim 1, wherein The detection unit detects faces and bodies of humans and animals, and objects.

3. The imaging device according to claim 1, in, The subject judgment unit also includes a fourth judgment unit, which is used to obtain the necessity of transfer based on the change amount of the time series in the two-dimensional space of the main subject, the reliability of the main subject and the AF reliability when the first judgment unit judges that the distance condition is affirmative and the second judgment unit cannot obtain the distance information used to obtain the distance from the camera device, and to judge whether to transfer the main subject according to the necessity.

4. The imaging device according to claim 1, in, The subject determination unit further includes a setting unit configured to set a distance condition for the first determination unit and a condition related to the predetermined number of times for the third determination unit when the second determination unit cannot obtain distance information related to the candidate.

5. A method for controlling an imaging device, the imaging device comprising an imaging unit configured to capture images at a predetermined frame rate, the method comprising: detecting a subject as a candidate for a main subject from an image in a current frame acquired by the imaging unit; as well as performing a process for determining whether the detected subject is the same as the main subject, and determining the subject determined to be the same as the main subject as the main subject in the current frame, Wherein, the performing includes: a first determination for determining whether the coordinates of the subject serving as the candidate for the main subject and the coordinates of the main subject in the two-dimensional space of the image in the current frame detected in the detection satisfy a distance condition corresponding to a combination of the type of the subject serving as the candidate for the main subject and the type of the main subject; a second determination for determining whether a difference between a distance between a candidate subject serving as the main subject in the current frame detected in the detection and the imaging device and a distance between the main subject and the imaging device is within a predetermined range; a third judgment, for judging whether the first judgment and the second judgment are affirmative for a predetermined number of consecutive times; If the first and second determinations are affirmative a predetermined number of times in succession, it is determined that the main subject has shifted; and If any one of the first determination, the second determination, and the third determination is negative, the subject with a higher priority in the image in the current frame is determined as the main subject.

6. The control method according to claim 5, wherein: The detection detects faces and bodies of humans and animals, as well as objects.

7. The control method according to claim 5, further comprising: determining the necessity of transfer based on a time-series change amount in a two-dimensional space of the main subject, reliability of the main subject, and AF reliability when the first judgment is affirmative regarding the distance condition and the second judgment cannot acquire distance information for obtaining the distance to the imaging device; as well as Whether to transfer the main subject is determined according to the necessity.

8. The control method according to claim 5, further comprising: In a case where the second judgment cannot acquire the distance information related to the candidate, a distance condition for the first judgment and a condition related to the predetermined number of times for the third judgment are set.

9. A non-transitory computer-readable storage medium storing a program, wherein when the program is read and executed by a processor in an imaging device, the imaging device executes a method for controlling the imaging device, the imaging device having an imaging unit that captures images at a predetermined frame rate, the method comprising: detecting a subject as a candidate for a main subject from an image in a current frame acquired by the imaging unit; as well as performing a process for determining whether the detected subject is the same as the main subject, and determining the subject determined to be the same as the main subject as the main subject in the current frame, Wherein, the performing includes: a first determination for determining whether the coordinates of the subject serving as the candidate for the main subject and the coordinates of the main subject in the two-dimensional space of the image in the current frame detected in the detection satisfy a distance condition corresponding to a combination of the type of the subject serving as the candidate for the main subject and the type of the main subject; a second determination for determining whether a difference between a distance between a candidate subject serving as the main subject in the current frame detected in the detection and the imaging device and a distance between the main subject and the imaging device is within a predetermined range; a third judgment, for judging whether the first judgment and the second judgment are affirmative for a predetermined number of consecutive times; If the first and second determinations are affirmative a predetermined number of times in succession, it is determined that the main subject has shifted; and If any one of the first determination, the second determination, and the third determination is negative, the subject with a higher priority in the image in the current frame is determined as the main subject.

10. The non-transitory computer-readable storage medium of claim 9, wherein: The detection detects faces and bodies of humans and animals, as well as objects.

11. The non-transitory computer-readable storage medium according to claim 9, wherein the control method further comprises: determining the necessity of transfer based on a time-series change amount in a two-dimensional space of the main subject, reliability of the main subject, and AF reliability when the first judgment is affirmative regarding the distance condition and the second judgment cannot acquire distance information for obtaining the distance to the imaging device; as well as Whether to transfer the main subject is determined according to the necessity.

12. The non-transitory computer-readable storage medium according to claim 9, wherein the control method further comprises: In a case where the second judgment cannot acquire the distance information related to the candidate, a distance condition for the first judgment and a condition related to the predetermined number of times for the third judgment are set.

Images