Image processing device, image processing method, and imaging device

Abstract

To provide an image processing device capable of improving subject tracking performance by appropriately using detection results of a plurality of types of subjects.SOLUTION: An image processing device detects a first type of subject and a second type of subject from an image. When subject tracking processing is executed on the basis of the detection result, and in a case in which the same subject in the image is detected as both the first type of subject and the second type of subject, the image processing device selects which of the detection results of the first type of subject and the second type of subject is to be used for tracking processing of the first type of subject.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and an imaging device, and in particular to a technique for detecting a subject in an image. [Background technology]

[0002] There is known an imaging device having a subject tracking function that detects a specific subject, such as a human face, from an image and continuously performs an operation of focusing on the detected subject. In addition, a technology for detecting human faces and animal faces from an image is also known (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-154438 A Summary of the Invention [Problem to be solved by the invention]

[0004] When detecting multiple types of subjects, areas detected as different subjects may overlap, for example, a person riding in a vehicle may be detected as part of a vehicle subject and also as a person subject.

[0005] In Patent Document 1, when both a human face and an animal face are detected within a predetermined range, one of them is determined to be the main subject depending on the size of the area. However, in the overlapping area, a certain subject may be detected as multiple types of subjects. In such a case, it may be preferable not to treat the subject as a certain type of subject.

[0006] For example, consider a case where a person riding in a vehicle is detected both as a part of the vehicle subject and as a human subject. In this case, if the detected subject is treated as a vehicle subject, when it can no longer be detected as a vehicle subject, it may become impossible to track the subject even if it can be detected as a human subject, or the tracking accuracy may decrease.

[0007] In consideration of the problems with the conventional techniques, one aspect of the present invention provides an image processing device and an image processing method capable of improving subject tracking performance by appropriately using detection results of a plurality of types of subjects. [Means for solving the problem]

[0008] The above-mentioned object can be achieved by an image processing device having a detection means for applying a process for detecting a first type of subject and a second type of subject to an image, and a control means for executing a process for tracking the subjects based on the detection result of the detection means, wherein when the same subject in the image is detected as both the first type of subject and the second type of subject, the control means selects whether to perform the process for tracking the first type of subject using the detection result for the first type of subject or the detection result for the second type of subject. Effect of the Invention

[0009] According to the present invention, it is possible to provide an image processing device and an image processing method capable of improving subject tracking performance by appropriately using the detection results of a plurality of types of subjects. [Brief description of the drawings]

[0010] [Figure 1] Configuration diagram of the first embodiment [Diagram 2] A simplified flowchart showing the operation of detection and tracking control [Diagram 3] Flowchart showing the operation of selecting a main subject [Figure 4] Flowchart showing the operation when multiple types of subjects are detected simultaneously [Diagram 5] Flowchart showing the operation when multiple types of subjects are not detected simultaneously [Figure 6] A diagram showing correlated head and bike detections [Figure 7] 1 is a flowchart showing the operation of a second embodiment of the present invention. [Figure 8] 11 is a flowchart showing the operation when multiple types of subjects are detected simultaneously in the second embodiment. [Figure 9] Diagram showing vehicle and head and organ detection DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0012] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in a digital camera. However, the imaging function is not essential to the present invention, and the present invention can be implemented in any electronic device. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0013] ●(First embodiment) 1 is a block diagram showing an example of the functional configuration of a digital camera according to a first embodiment of the present invention. The digital camera has a body 120 and a lens unit 100 that is detachable from the body 120.

[0014] The lens unit 100 has an imaging optical system 101 having a main optical system 102, an aperture 103, and a focus lens group 104. The focal length (angle of view) of the imaging optical system 101 may be variable. The lens unit 100 also has components for detecting the positions of the aperture 103 and movable lenses (such as the focus lens group 104, a zoom lens, and an anti-vibration lens) and for driving them.

[0015] The lens unit 100 also has a lens control unit 111 that controls the operation of the lens unit 100. The lens control unit 111 has, for example, a memory that stores a program and a processor capable of executing the program.

[0016] The lens control unit 111 executes a program using a processor to control the operation of the lens unit 100 and communicates with the main body 120. The aperture control unit 112 and the focus lens control unit 113 are functional blocks that represent functions that are realized by the processor of the lens control unit 111 executing a program.

[0017] The aperture control unit 112 controls the aperture amount (aperture value) of the aperture 103 under the control of the camera control unit 131. Furthermore, the aperture control unit 112 supplies the aperture value of the aperture 103 to the camera control unit 131 upon request.

[0018] The focus lens control unit 113 drives the focus lens group 104 in the optical axis direction of the imaging optical system 101 and controls the position according to the control of the camera control unit 131. In addition, the focus lens control unit 113 supplies position information of the focus lens group 104 to the camera control unit 131 upon request.

[0019] If the imaging optical system 101 has a zoom lens or an anti-vibration lens, the lens control unit 111 has a function of controlling the positions of these movable lenses.

[0020] The lens unit 100 and the body 120 have mount portions that fit together. The mount portion has mount contact portions 114 and 116 that are configured to come into contact with each other when the lens unit 100 is attached to the body 120. The lens unit 100 and the body 120 are electrically connected via the mount contact portions 114 and 161. Power required for the operation of the lens unit 100 is supplied from the body 120 via the mount contact portions 114 and 161. In addition, the lens control portion 111 and the camera control portion 131 can communicate with each other via the mount contact portions 114 and 161.

[0021] The imaging optical system 101 forms an optical image on the imaging surface of an imaging element 122 provided in a main body 120. The imaging element 122 may be, for example, a general CMOS color image sensor. An openable and closable shutter 121 is provided between the imaging optical system 101 and the imaging element 122. When photographing, the shutter 121 is opened to expose the imaging element 122.

[0022] The imaging element 122 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element 122 has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from the pixels. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A group of pixel signals (analog image signals) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0023] The analog image signal is input to an analog front end (AFE) 123. The AFE 123 applies analog signal processing such as correlated double sampling and gain adjustment to the analog image signal, and then outputs the signal to a signal processing circuit .

[0024] The camera control unit 131 has, for example, a memory that stores a program and a processor capable of executing the program. The camera control unit 131 controls the operation of the main body 120 and realizes various functions of the main body 120 by executing the program with the processor.

[0025] Furthermore, the camera control unit 131 communicates with the lens control unit 111 by executing a program in the processor. For example, a command for controlling the operation of the lens unit 100 or a command for requesting information about the lens unit 100 is transmitted from the camera control unit 131 to the lens control unit 111. The lens control unit 111 controls the operation of the focus lens group 104 and the aperture 103 in response to the received command, and transmits information about the lens unit 100 to the camera control unit 131. The information about the lens unit 100 transmitted to the camera control unit 131 includes, for example, product information about the lens unit 100, and information about the position of a movable lens and an aperture value.

[0026] In the figure, functional blocks 151 to 156 shown inside camera control unit 131 are functional blocks that represent functions that are realized by the processor of camera control unit 131 executing a program.

[0027] Operation unit 181 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to main body 120. The input devices constituting operation unit 181 have names according to the functions assigned to them. For example, operation unit 181 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, menu buttons, directional keys, an enter key, etc.

[0028] The release switch is a switch for recording still images, and the camera control unit 131 recognizes the half-pressed state of the release switch as an instruction to prepare for shooting, and the full-pressed state as an instruction to start shooting. The camera control unit 131 also recognizes the video recording switch pressed in the shooting standby state as an instruction to start recording a video, and recognizes the switch pressed during video recording as an instruction to stop recording. Note that the functions assigned to the same input device may be variable.

[0029] Angular velocity sensor 126 is, for example, a three-axis gyro sensor, and outputs a signal indicating the movement of main body 120 to camera control unit 131. Camera control unit 131 detects the movement of main body 120 based on the signal output from angular velocity sensor 126. Furthermore, camera control unit 131 executes predetermined control based on the detected movement of main body 120.

[0030] The display unit 171 is a display device (touch display) equipped with a touch panel 172. The display unit 171 functions as an electronic viewfinder (EVF) by continuously capturing moving images using the image sensor 122 and displaying the captured moving images on the display unit 171.

[0031] The display unit 171 can play back and display image data recorded on the memory card 125, display information related to the state and settings of the main body 120, and display a GUI (Graphical User Interface) such as a menu screen. The user can operate the displayed GUI or specify a focus detection area by performing a touch operation on the touch panel 172.

[0032] When the camera control unit 131 detects an operation on the operation unit 181 and the touch panel 172, it executes an operation according to the detected operation. For example, when an operation to instruct preparation for shooting a still image is detected, the camera control unit 131 executes AF processing, AE processing, etc. Also, when an operation to instruct shooting a still image is detected, the camera control unit 131 controls or executes shooting processing of the still image, generation processing of image data for recording by the signal processing circuit 124, processing of recording image data for recording on the memory card 125 (recording medium), etc.

[0033] The signal processing circuit 124 applies predetermined image processing to the analog image signal input from the AFE 123, generates signals and image data, and acquires and / or generates various information. The signal processing circuit 124 may be, for example, a dedicated hardware circuit such as an ASIC designed to realize a specific function, or may be configured to realize a specific function by a programmable processor such as a DSP executing software.

[0034] The image processing applied by the signal processing circuit 124 includes pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing includes signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation is a process that interpolates the values ​​of color components that cannot be obtained during shooting, and is also called demosaic processing. The correction processing includes white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system 101 (image restoration), correction of the effect of peripheral light falloff of the imaging optical system 101, color correction, and the like. The detection process includes detection of feature regions (for example, face regions and human body regions) and their movements, and person recognition processing. The data processing includes processes such as synthesis, scaling, encoding and decoding, and header information generation (data file generation). The evaluation value calculation process includes processes such as generation of signals and evaluation values ​​used in automatic focus detection (AF) and generation of evaluation values ​​used in automatic exposure control (AE). Special effects processing includes adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are merely examples of processes that the signal processing circuit 124 can apply, and do not limit the processes that the signal processing circuit 124 can apply.

[0035] Function blocks 141 to 144 shown in the signal processing circuit 124 in FIG. 1 are functional blocks that represent functions related to subject detection processing that the signal processing circuit 124 realizes by executing a program, for example.

[0036] The specific subject detection unit 141 applies a detection process for a plurality of types of subjects determined in advance to the image data, and detects a subject area for each type of subject. The specific subject detection unit 141 holds parameters for detecting a subject area for each type of subject as dictionary data. The specific subject detection unit 141 can detect subject areas for a plurality of types of subjects by switching the dictionary data used in the detection process.

[0037] The dictionary data can be generated in advance by a known method such as machine learning. There is no limit to the type of subject detected by the specific subject detection unit 141, but in this embodiment, it is assumed that the detection result is used for subject tracking. Therefore, the specific subject detection unit 141 detects one or more types of subjects from among moving subjects, for example, human bodies, vehicles (motorcycles, automobiles, trains, airplanes, ships, etc.), and animals (dogs, cats, birds, etc.). In particular, in this embodiment, a case in which two or more types of subjects including human bodies and vehicles are detected will be described later. For a human body (a second type of subject), one or more specific parts such as the head, torso, and pupils can be detected. For a vehicle (a first type of subject), the whole body and one or more predefined specific parts can be detected. For an animal, one or more specific parts such as the whole body, face, and pupils can be detected.

[0038] In this embodiment, the specific part detected for a vehicle is the head of a vehicle occupant. The head of the occupant is detected as a specific part of a vehicle subject, which differs from the head detected as a human subject.

[0039] The specific subject detection unit 141 generates a detection result for each subject to be detected. The detection result includes, but is not limited to, the total number of detected regions, and the position, size, and detection reliability of each region.

[0040] The face and organ detection unit 142 detects organ areas such as the face, eyes (pupils), nose, and mouth from, for example, a human body subject area among the subject areas detected by the specific subject detection unit 141. The face and organ detection unit 142 can detect the face and organs by a known method using feature parameters and templates. The detection of the organs described above can also be performed by the specific subject detection unit 141. In that case, the face and organ detection unit 142 can be removed from the configuration shown in FIG. 1.

[0041] The face and organ detection unit 142 generates a detection result for each detected face region or organ region. The detection result includes, but is not limited to, the total number of detected regions, and the position, size, and detection reliability of each region.

[0042] The distance information acquisition unit 143 generates a distribution (distance map) of the defocus amount or subject distance in the entire or part of the imaging range for the current state of the imaging optical system 101. The distance information acquisition unit 143 generates the distance map by determining the defocus amount or subject distance for each pixel or each pixel block. The distance map can be generated by a known method, so detailed description of the generation method will be omitted.

[0043] The vector detection unit 144 detects a motion vector for each pixel block obtained by dividing image data, for example, in the horizontal and vertical directions. A motion vector can be detected between two frames of images captured at different times. A motion vector can be detected by a known method, such as using a part of an earlier (older) frame as a template and searching for an area with high similarity in a later (newer) frame.

[0044] The information acquired by the specific subject detection section 141 , the face and organ detection section 142 , the distance information acquisition section 143 and the vector detection section 144 is supplied from the signal processing circuit 124 to the camera control section 131 .

[0045] In this specification, the processes performed by specific subject detection section 141, face and organ detection section 142, distance information acquisition section 143, and vector detection section 144 are collectively referred to as subject detection processing. Image data on which subject detection processing is performed may be that acquired by image sensor 122, or that read from memory card 125. Furthermore, subject detection processing is applicable to both still image data and video data.

[0046] In the camera control unit 131, the target subject setting unit 151 sets a target subject for tracking processing based on the results of the subject detection processing by the specific subject detection unit 141 and the face and organ detection unit 142.

[0047] The inter-subject correlation determination section 152 determines whether or not the detected multiple types of subjects constitute the same subject. Subject tracking control section 153 executes subject tracking processing using information on the subject set as the tracking target by target subject setting section 151, the distance map generated by distance information acquisition section 143, and the like. The subject peripheral information confirmation section 154 acquires defocus information in the peripheral area of ​​the subject to be tracked from the distance map generated by the distance information acquisition section 143. The subject lost determination unit 155 determines whether or not the subject to be tracked has been lost. The display frame control unit 156 displays an index such as a frame indicating the subject region of the tracking target on the display unit 171, for example, by superimposing it on a live view image.

[0048] The camera control unit 131 determines the subject to be tracked and controls the tracking process based on the detection result and distance information relating to a specific type of subject obtained from the signal processing circuit 124.

[0049] Fig. 2 is a flowchart of the subject tracking process executed by the camera control unit 131. Here, the camera control unit 131 executes the subject tracking process shown in the flowchart of Fig. 2 in parallel with the video shooting process by the image sensor 122. That is, the subject tracking process is executed substantially in real time on the video shot by the image sensor 122. Here, the camera control unit 131 executes the subject tracking process for each frame of the video. However, the frequency of execution of the process may be changed depending on the number of pixels of the frame, the frame rate, the processing capacity of the camera control unit 131, and the like.

[0050] In addition, the subject detection process in the signal processing circuit 124 is also executed in parallel with the video shooting process in the image sensor 122. Since the subject detection process imposes a large load, it may not be possible to execute it for each frame. Here, it is assumed that the subject tracking process is executed every other frame (once every two frames).

[0051] In S201, the camera control unit 131 checks whether or not a subject has been detected by the signal processing circuit 124 (specific subject detection unit 141), and if a subject has been detected, executes S202, and if not, executes S203. If no subject has been detected, this includes the case where no subject area has been detected and the case where subject detection processing has not been completed.

[0052] In S202, the camera control unit 131 determines a tracking part based on the result of the subject detection process. The tracking part is a type of subject area used in the tracking process. The process of S202 will be described in detail later.

[0053] In S203, the camera control unit 131 executes a tracking process using the detection result corresponding to the tracking part determined in S202, among the detection results of the specific subject detection unit 141 and the face and organ detection unit 142. The tracking process includes a search process for the tracking part in the current frame by a known method such as template matching, a display process of an index indicating the searched tracking part, a template update process, etc. In addition, if the frequency of execution of S203 directly from S201 is high, a process for suppressing a decrease in tracking accuracy (for example, initialization or re-determination of the tracking part, etc.) can also be executed.

[0054] Next, the tracking portion determination process in S202 will be described in detail with reference to FIGS. In S301, camera control unit 131 determines whether or not a tracking subject (main subject) has been determined in the processing for the previous frame, and if it is determined that it has been determined, executes S303, and if it is not determined that it has been determined, executes S302.

[0055] In S302, the camera control unit 131 determines the tracking subject based on the most recent subject detection processing result obtained from the signal processing circuit 124. There is no particular limitation on the method of determination here, and for example, a subject of a type of subject with the highest predetermined priority among the detected subject areas and a subject in an area whose size is equal to or greater than a predetermined threshold value can be determined as the tracking subject. Alternatively, the determination may be based on other conditions, such as determining a subject of an area whose size is equal to or greater than a predetermined threshold value and whose size is closest to the camera among the detected subject areas as the tracking subject. The camera control unit 131 executes S303 after storing information of the determined tracking subject in memory.

[0056] In S303, the camera control unit 131 determines whether the tracking subject is a vehicle subject. In the following, a motorcycle subject will be described as an example of a vehicle subject, but other vehicle subjects can be processed in the same manner. If the tracking subject is determined to be a motorcycle subject, the camera control unit 131 executes S304, and if not, executes S307.

[0057] In S304, the camera control unit 131 determines whether or not both a head as a human subject and a motorcycle subject have been detected in the subject detection process in the signal processing circuit 124. If it is determined that both types of subjects have been detected, the camera control unit 131 executes S305, and if not, it executes S306.

[0058] What is determined in S304 is a type of subject that can detect an area that is the same as or overlaps with the area of ​​the tracked subject, among the types of subjects different from the type of subject to be tracked. Combinations of types of subjects that can detect an area that is the same as or overlaps with the area of ​​the tracked subject are assumed to be registered in advance, for example, in the memory of the camera control unit 131. Therefore, when the type of subject to be tracked is identified, the types of other subjects determined in S304 are also identified.

[0059] In S305, the camera control unit 131 executes a selection process in the case where both a head (human subject) and a motorcycle subject are detected, as will be described in detail later. In S306, the camera control unit 131 executes a selection process in the case where at least one of a head (human subject) and a motorcycle subject is not detected, as will be described in detail later.

[0060] In S307, the camera control unit 131 determines a tracking target when the tracking subject is other than a vehicle subject. For example, the camera control unit 131 can change the tracking part of the tracking subject (switching between eyes ←→ face ←→ torso, or switching between the whole body ←→ a specific part).

[0061] When any one of S305, S306, and S307 is executed, the camera control unit 131 ends the process of determining the tracking part.

[0062] Next, the selection process executed in S305 when both a head (human subject) and a motorcycle subject are detected will be described in detail with reference to the flowchart shown in FIG.

[0063] In S401, the camera control unit 131 determines whether or not a specific part of the motorcycle subject (the rider's head) has been detected by the subject detection process, and if it is determined that it has been detected, executes S402, and if not, executes S410.

[0064] In S402, the camera control unit 131 checks the correlation between the detected head (human subject) and a specific part of the motorcycle (rider's head). For example, based on the detection results of the specific subject detection unit 141, the distance information acquisition unit 143, and the vector detection unit 144, the camera control unit 131 determines whether or not there is a positive correlation between the two regions in one or more of the positional relationship in the image, the size, the position, and the change over time in the subject distance.

[0065] Then, in S403, the camera control unit 131 determines whether or not the detected head (human subject) and the specific part of the motorcycle (rider's head) are the same subject based on the correlation checked in S402. If it is determined that they are the same subject, the camera control unit 131 executes S404, and if not, executes S406.

[0066] There is no particular limitation on the method of determination in S403. For example, the camera control unit 131 can determine that the detected areas are the same subject when the degree of overlap of the detected areas is equal to or greater than a threshold and / or when there is a positive correlation in one or more of the size, position, and change over time of the subject distance of the areas.

[0067] In S404, the camera control unit 131 determines whether to prioritize the head (human subject) or a specific part of the motorcycle (rider's head). For example, the camera control unit 131 can prioritize a part with a higher detection reliability. In addition to or instead of the detection reliability, one or more of the position of the area (the area closer to the image center is prioritized), the size of the area (the area larger is prioritized), and the movement of the main body 120 based on the output of the angular velocity sensor 126 or the vector detection unit 144 may be taken into consideration.

[0068] For example, if the area of ​​the specific part of the motorcycle is in the peripheral area away from the center of the image by a threshold or more, and the movement of the main body 120 is large (above the threshold), it is considered that there is a high possibility that the motorcycle subject will go out of the frame. When a part of the motorcycle subject goes out of the frame and the motorcycle subject cannot be detected, the specific part also cannot be detected at the same time. Therefore, when the conditions that make it considered that there is a high possibility that the motorcycle subject will go out of the frame are met, the head (human subject) may be prioritized even if the specific part of the motorcycle (rider's head) has a higher detection reliability. Also, when the area is small (below the threshold), taking into consideration the possibility that it will not be possible to detect it as a human subject, the specific part of the motorcycle (rider's head) may be prioritized even if the head (human subject) has a higher detection reliability. However, a lower limit is set for the detection reliability, and parts with detection reliability below the lower limit are not prioritized.

[0069] Next, in S405, the camera control unit 131 determines whether the subject determined to be prioritized in S404 is the same as the previous time, and if it is determined to be the same, executes S406, and if not, executes S408.

[0070] In S408, the camera control unit 131 determines whether the priority determination level of the part determined to be prioritized in S404 is high (above a threshold value) or not, and if it is determined that the priority determination level is high, executes S409, and if not, executes S406. The priority determination level can be determined mainly based on the detection reliability of the subject determined to be prioritized in S404. For example, if the detection reliability determined to be prioritized in S404 is above a threshold value, it may be determined that the priority determination level is high. The detection reliability taking into account other information, such as multiplying the detection reliability by a weight depending on the part, may be compared with the threshold value.

[0071] In S409, the camera control unit 131 changes the tracking part to the part determined to be prioritized in S404. The camera control unit 131 stores information about the tracking part after the change in memory, and then executes S407.

[0072] On the other hand, in S406, the camera control unit 131 determines to maintain the same tracking part as the previous time, and executes S407.

[0073] In S407, the camera control unit 131 sets the tracking subject to the motorcycle again. In this way, while the tracking subject is set to the motorcycle, if both a specific part of the motorcycle subject and the head of the human subject are detected, which part is used to perform the tracking process is appropriately changed. As a result, even if the motorcycle subject cannot be detected, if the head of the human subject is set as the tracking part, it is possible to continue tracking the motorcycle subject.

[0074] If it is not determined in S401 that a specific part of the motorcycle subject has been detected, the camera control unit 131 checks in S410 the correlation between the head (human subject) and the motorcycle subject (whole) in the same manner as in S402. However, since the size and position of the area of ​​the head (human subject) and the motorcycle subject (whole) are different, it is determined whether or not there is a positive correlation based on the change over time in size, position, and subject distance.

[0075] In S411, the camera control unit 131 determines whether the detected head (human subject) and the motorcycle subject (whole) are the same subject, as in S403. If they are determined to be the same subject, the camera control unit 131 executes S412, and if they are not, the camera control unit 131 executes S406. For example, the camera control unit 131 can determine that they are the same subject when there is a positive correlation in one or more of the size, position, and change over time of the subject distance of the area.

[0076] In S412, the camera control unit 131 determines whether to give priority to the head (human subject) or the subject (whole body) of the motorcycle, in the same manner as in S404.

[0077] Next, in S413, the camera control unit 131 executes S414 if it has been determined in S412 that the head (human subject) is to be prioritized, and executes S415 if not. In S414, the camera control unit 131 determines the head (human subject) as the tracking part, and executes S407. In S415, the camera control unit 131 determines the motorcycle subject (as a whole) as the tracking part, and executes S407.

[0078] FIG. 6 is a diagram showing an example of a condition for determining whether the subject is the same or not among the actions shown in FIG. 4. FIG. 6(a) and (a') show the detection result of the head (human subject), and FIG. 6(b) and (b') show the detection result of the motorcycle subject (whole) and a specific part of the motorcycle subject (rider's head). FIG. 6(a) and (b) are the same frame, and FIG. 6(a') and (b') are the same frame. In addition, the frames of FIG. 6(a') and (b') are frames that are later in time than the frames of FIG. 6(a) and (b).

[0079] In the example shown in Figure 6(a) and (b), the area detected as the head (human subject) and the area detected as the specific part of the motorcycle subject (rider's head) are almost at the same position in the image (frame). The subject distances obtained for each area based on the distance map are also almost the same. Both of these correspond to a positive correlation.

[0080] In addition, in the frames of Fig. 6(a') and (b') after a predetermined time, the change in size over time (e.g., magnification) of the area detected as the head (human subject) and the area detected as a specific part of the motorcycle subject (rider's head) is almost equal. Also, the subject distances obtained for each area based on the distance map are almost equal. Both of these correspond to a positive correlation.

[0081] Although the head (human subject) and the bike subject (whole subject) are detected at different positions in the image (frame), the subject distances obtained for each region based on the distance map are almost equal. Also, the change in size of the regions over time (e.g. magnification) between frames is also almost equal. Both of these correspond to a positive correlation.

[0082] Next, the process of determining the part to be tracked when either the head (human subject) or the motorbike subject is not detected, which is executed in S306, will be described in detail with reference to the flowchart shown in FIG.

[0083] In S501, the camera control unit 131 determines whether or not a motorcycle subject has been detected by the subject detection process, and if it is determined that a motorcycle subject has been detected, executes S502, and if not, executes S506.

[0084] S502 is executed when a motorcycle subject is detected but a head (human subject) is not detected. In this case, the camera control unit 131 determines whether a specific part of the motorcycle subject (rider's head) is detected in S502, and executes S503 if it is determined that it is detected, and executes S504 if it is not determined that it is detected.

[0085] In S503, the camera control unit 131 sets a specific part of the motorcycle subject as a tracking part, and executes S505. In S504, the camera control unit 131 determines the entire motorcycle subject as the tracking part, and executes S505. In S505, the camera control unit 131 again sets the subject to be tracked to the motorcycle subject.

[0086] On the other hand, if it is determined that the bike subject has not been detected, in S506 the camera control unit 131 determines whether or not a head (human subject) has been detected by the subject detection process, and if it is determined that a head has been detected, executes S507, and if not, executes S510.

[0087] In S507, the camera control unit 131 determines whether the previously tracked part is a head (human subject) or not, and if it is determined to be a head (human subject), executes S508, and if it is not determined to be a head (human subject), executes S509.

[0088] In S508, the camera control unit 131 resets the detected subject to "none" and executes S509. The operations of S507 to S509 are performed to avoid tracking other types of subjects when the state in which the motorcycle subject cannot be detected continues. By setting the detected subject to none in S508, it is possible to execute initialization or redetermining of the tracking portion in S203 of FIG. 2.

[0089] In S509, the camera control unit 131 maintains the same tracking part as the previous time, and then executes S505.

[0090] S510 is executed when neither the bike subject nor the head (human subject) can be detected. In this case, the camera control unit 131 executes processing when the subject is lost. Specifically, the same processing as S203 in Fig. 2 may be performed. If the intended type of subject cannot be detected and the state in which the tracking reliability is low continues for a predetermined time, the camera control unit 131 determines that the tracking subject has changed and can initialize or re-determine the tracking part.

[0091] As described above, according to this embodiment, when it is determined that the area of ​​the same subject is detected as an area of ​​a different type of subject, it is possible to select which detection result is used for the tracking process. Therefore, even if it becomes impossible to detect one type of subject, it is possible to increase the possibility of continuing tracking, and as a result, it is possible to improve tracking performance. In addition, when continuous focus adjustment is performed to focus on the tracking target, it is possible to improve the robustness of being able to continue focusing on the intended subject.

[0092] (Second embodiment) Next, a second embodiment of the present invention will be described. This embodiment is similar to the first embodiment except for the process of determining the tracking portion. Therefore, the process of determining the tracking portion according to this embodiment will be described below. In this embodiment, a priority is set for the type of subject detected by the specific subject detection unit 141, and here, it is set so that a vehicle subject is given priority over other types of subjects.

[0093] The process of determining a tracking portion in this embodiment will be described with reference to the flowchart shown in FIG. In S701, the camera control unit 131 determines whether or not a vehicle subject has been detected by the specific subject detection unit 141, and if it is determined that a vehicle subject has been detected, executes S702, and if not, ends the determination process.

[0094] In S702, the camera control unit 131 determines whether or not any of the head, face, or eyes of the human subject has been detected by the specific subject detection unit 141 and the face and organ detection unit 142, and if it is determined that any of the head, face, or eyes has been detected, executes S703, and if not, executes S704.

[0095] In S703, the camera control unit 131 executes a selection process in the case where both a head, a face, or an eye (a human subject) and a vehicle subject are detected. Details of the selection process in S703 will be described with reference to the flowchart shown in FIG.

[0096] Note that S401' to S406' and S408' to S415' in FIG. 8 are the same as S401 to S406 and S408 to S415 in FIG. 4 except that the type of subject is changed from a motorcycle to a vehicle, and therefore a description thereof will be omitted.

[0097] In S801, if the information on the previously determined tracking part has been backed up in, for example, an internal memory, the camera control unit 131 acquires the backed up data.

[0098] In S802, the camera control unit 131 determines whether the area of ​​the detected vehicle subject (whole) is larger than a predetermined size, and if it is determined that it is larger, executes S803, and if not, executes S401'. The predetermined size is determined in advance as the size of the vehicle area where the eyes of the occupant are likely to be detected with significant reliability.

[0099] In S803, the camera control unit 131 sets, for example, an eye selection "possible" flag allocated to a memory area (sets the value to 1). If the subject area is larger than a predetermined size, there is a high possibility that the eyes of the occupant will be detected with significant reliability, so in S802 and S803, preparations for selecting the eyes are made.

[0100] Then, the processing from S401' onwards is executed, and if it is not determined in S403' that the head (human subject) detected and the specific part of the vehicle (the head of the occupant) are the same subject, the camera control unit 131 executes S804 before executing S406'.

[0101] In S804, the camera control unit 131 clears the pupil selection "possible" flag (sets it to 0). This is because if the detected head (human subject) and the specific part of the vehicle (head of the occupant) are not the same subject, even if an eye (human subject) is detected, it is not the eye of the occupant of the vehicle. Note that the eyes of the occupant are not detected as a specific part of the vehicle subject.

[0102] Thereafter, when the tracking part is determined in any one of S409', S406', S414', and S415', the camera control unit 131 executes S805.

[0103] In S805, the camera control unit 131 determines whether or not the conditions that an eye (human subject) has been detected and the eye selection "allowed" flag is set are met, and if it is determined that the conditions are met, executes S806, and if not, executes S808.

[0104] In S806, the camera control unit 131 backs up (stores) in memory information about the tracking part determined in any one of S409', S406', S414', and S415' as a candidate for the tracking part other than the eye (human subject).

[0105] In S807, the camera control unit 131 determines the eye (human subject) as the final tracking part. In S808, the camera control unit 131 clears the backed up information on the tracking part.

[0106] S807 is executed when the eyes of the vehicle occupant are expected to be detected as a part of the human subject with significant reliability. However, it is not easy to detect the eyes of the occupant reliably, and it is quite possible that the eyes of the occupant will not be detected as a part of the human subject in the next subject detection process.

[0107] Therefore, in this embodiment, information on the tracking part other than the eyes, which was determined in any one of S409', S406', S414', and S415' in S806, is backed up in memory, and the backed up information is acquired the next time S801 is executed. As a result, if the eyes of the occupant cannot be detected as a part of the human subject in the next subject detection process, the tracking process can be performed using the tracking part determined in any one of S409', S406', S414', and S415'.

[0108] 7, if it is not determined in S702 that a part of the human subject (head, face, or eye) has been detected, the camera control unit 131 determines in S704 whether or not a specific part of the vehicle subject (head of a passenger) has been detected. If it is determined that a specific part of the vehicle subject has been detected, the camera control unit 131 executes S705, and if not, executes S706.

[0109] In S705, the camera control unit 131 sets a specific part of the vehicle subject (the head of the passenger) as the tracking part. In S706, the camera control unit 131 sets the vehicle subject (entirely) as the tracking portion.

[0110] FIG. 9 shows a schematic diagram of the tracking process implemented in this embodiment when a vehicle, which is a tracking subject, approaches the camera. If the subject is far away, only the entire subject can be detected. Then, as the subject approaches to a certain size, both a specific part of the vehicle subject and the head (human subject) are detected as the same subject. As the subject approaches even closer, the eyes of the passengers can be detected as pupils (human subjects). As the part being tracked changes, the frame display indicating the tracking area also changes as shown in the figure.

[0111] According to this embodiment, information on a part (here, the pupil) that is not detected as a tracking subject but may be detected as another type of subject is actively used in the tracking process. Therefore, in addition to the effect of the first embodiment, it is possible to provide a more useful tracking function to the user by using the detection result of another type of subject.

[0112] For example, when the camera is set to prioritize animal subjects, the use of the detection results of human subjects in combination may lead to erroneous tracking. Therefore, the use of the detection results of human subjects in combination may be limited to cases where the type of subject to be tracked is one that is expected to include a person, such as a vehicle subject.

[0113] (Other embodiments) The above-described embodiment does not need to be performed in parallel with shooting, and may be performed, for example, when playing back a recorded video, etc. In addition, displaying an indicator (frame) indicating a tracking part is not essential, and the use of the tracking result is not limited to use related to shooting, such as exposure control and focus control.

[0114] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0115] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0116] 100: interchangeable lens, 120: camera body, 122: image sensor, 124: signal processing circuit, 131: camera control unit, 141: specific subject detection unit, 142: face and organ detection unit, 143: distance information acquisition unit, 144: vector information detection unit

Claims

1. a detection means for applying a process for detecting a first type of subject and a second type of subject to an image; a control unit that executes a subject tracking process based on a detection result of the detection unit, When a same subject in the image is detected as both the first type of subject and the second type of subject, the control means 13. An image processing device comprising: a first object detection unit that detects an object to be tracked by a first object; a second object detection unit that detects an object to be tracked by a second object;

2. When a same subject in the image is detected as both the first type of subject and the second type of subject, the control means The image processing device according to claim 1, characterized in that, based on one or more of a detection reliability, a position in the detected image, and a movement of the image processing device, it is selected whether to perform tracking processing of the first type of subject using a detection result for the first type of subject or a detection result for the second type of subject.

3. 3. The image processing device according to claim 2, characterized in that the control means performs tracking processing of the first type of subject using a detection result having a higher detection reliability given by the detection means, out of the detection results for the first type of subject and the detection results for the second type of subject.

4. The image processing device according to claim 3, characterized in that when the first type of subject is detected in the peripheral part of the image and the movement is equal to or greater than a threshold, the control means performs tracking processing of the first type of subject using the detection result of the second type of subject, even if the detection reliability of the first type of subject is higher than the detection reliability of the second type of subject.

5. 3. The image processing device according to claim 2, characterized in that the control means performs tracking processing of the first type of subject using the detection result detected at a position closer to the center of the image, out of the detection results for the first type of subject and the detection results for the second type of subject.

6. 6. The image processing device according to claim 1, wherein the first type of subject is a vehicle subject and the second type of subject is a person subject, and the detection means detects the entire vehicle subject and the head of an occupant, and detects the head of the person subject.

7. 6. The image processing apparatus according to claim 1, wherein the second type of subject is a portion that is not detected as the first type of subject.

8. 8. The image processing device according to claim 7, wherein the first type of subject is a vehicle subject and the second type of subject is a person subject, and the detection means detects the entire vehicle subject and the head of an occupant, and detects a face and eyes as the person subject.

9. The image processing device described in claim 8, characterized in that when the same subject in the image is detected as both the vehicle subject and the human subject, the size of the overall detected area of ​​the vehicle subject is equal to or greater than a threshold, and an eye is detected as the human subject, the control means decides to use the eye detection result for tracking processing of the vehicle subject.

10. The image processing device described in claim 8 or 9, characterized in that when the control means decides to use the pupil detection result for tracking processing of the vehicle subject, it saves one of the detection results other than the pupil among the detection results for the first type of subject and the detection results for the second type of subject as a candidate for use in tracking processing.

11. An imaging element; The image processing device according to claim 1 , which uses an image obtained by the imaging element; and adjusting means for adjusting a focus of an imaging optical system based on a result of the tracking process by said image processing device.

12. An image processing method executed by an image processing device, comprising: applying a process to the image to detect a first type of object and a second type of object; and executing a subject tracking process based on a result of the process. Executing the tracking process includes: selecting whether to perform a tracking process for the first type of subject using a detection result for the first type of subject or a detection result for the second type of subject when a same subject in the image is detected as both the first type of subject and the second type of subject; 13. An image processing method comprising:

13. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 10.

Images