Object Tracking Device

Abstract

To provide a subject tracking device capable of reducing erroneous tracking of a subject. [Solution] The subject tracking device is characterized by having an image acquisition means for sequentially acquiring images, a tracking means for tracking a subject detected from the images acquired by the image acquisition means by comparing the images across multiple images acquired sequentially by the image acquisition means, and a switching means for switching the time for which tracking by the tracking means continues depending on the type of subject detected from the images.

Description

Technical Field

[0001] The present invention relates to a subject tracking device and the like capable of tracking a subject in a plurality of images.

Background Art

[0002] In recent years, many imaging devices such as digital cameras have been able to detect a subject in real time during live view and continue to track the subject even if it moves while distinguishably displaying the detected subject area on an operation screen. The tracked subject (area) often becomes the focus of autofocus (AF) processing as the main subject. In order to detect the subject position to be AF-targeted in real time, Patent Document 1 performs subject tracking by template matching. Further, a configuration is disclosed in which a tracking period is calculated based on a reliability representing the subject-likeness of the subject to be tracked, and subject tracking is performed based on the tracking period.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, false tracking is prevented by providing a tracking period based on the reliability representing the subject-likeness. However, in a configuration such as Patent Document 1, for example, even if the reliability representing the subject-likeness is high, it may be difficult to track because the feature amount of the subject itself is small, so there is a possibility that false tracking may occur frequently within the tracking period. In addition, in an animal (for example, a dalmatian) having a pattern with features similar to those of a pupil all over its body, false tracking of the pupil is likely to occur. Therefore, an object of the present invention is to provide a subject tracking device capable of reducing the erroneous tracking of a subject.

Means for Solving the Problems

[0005] The subject tracking device according to the present invention includes: image acquisition means for sequentially acquiring images; tracking means for tracking a subject detected from the images acquired by the image acquisition means over a plurality of images sequentially acquired by the image acquisition means; switching means for switching the time for continuing the tracking in the tracking means according to the type of the subject detected from the images, and the switching means further switches the time for continuing the tracking in the tracking means according to the part of the subject, wherein the time for continuing the tracking when the subject is a human face is made different from the time for continuing the tracking when the subject is one of a face or pupil of a dog, a cat, or a bird. Larger than This is the gist of the present invention.

Effects of the Invention

[0006] According to the present invention, it is possible to realize a subject tracking device capable of reducing the erroneous tracking of a subject.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0008] Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described using examples. In each figure, the same members or elements are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified. Also, in the examples, an example applied to a digital still camera as an imaging device will be described. However, the imaging device includes electronic devices having an imaging function such as a digital movie camera, a smartphone with a camera, a tablet computer with a camera, an in-vehicle camera, and a network camera.

Examples

[0009] Hereinafter, Example 1 of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of Example 1 of the present invention. In FIG. 1, 100 is an imaging device (subject tracking device), 10 is a photographic lens, and 12 is a mechanical shutter having an aperture function. 14 is an imaging element such as a CMOS sensor that converts an optical image into an electrical signal, and functions as an image acquisition means for sequentially acquiring an image of a subject. Also, 16 is an A / D converter that converts the analog signal output from the imaging element 14 into a digital signal.

[0010] 18 is a timing generation unit that supplies a clock signal and a control signal to the imaging element 14 and the A / D converter 16, and is controlled by the memory control circuit 22 and the system control circuit 50. By controlling the reset timing of the imaging element 14 by the timing generation unit 18, it is possible to control the charge accumulation time, and it can be used as an electronic shutter in video shooting and the like separately from the mechanical shutter 12.

[0011] Note that the system control circuit 50 incorporates a CPU as a computer and functions as control means for executing various operations of the entire apparatus based on a computer program stored in the non-volatile memory 31. 20 is an image processing circuit that performs pixel interpolation processing, color conversion processing, noise removal processing, edge enhancement processing, etc. for image enlargement / reduction on the data from the A / D converter 16 or the data from the memory control circuit 22. Further, the image processing circuit 20 functions as detection means for detecting a specific subject from an image.

[0012] In addition, the image processing circuit 20 has a face detection function for detecting a person, animal, or face area by image recognition, a function for detecting organs (parts) such as pupils, noses, and mouths in the face, a whole body detection function for detecting the entire body (entire object) of the subject, etc. And it performs calculation processing of the positions of the face, organs, and whole body from the results of face detection, organ detection, and whole body detection.

[0013] Face detection and whole body detection in the image processing circuit 20 save the shape of the contour part of the face or whole body as feature data inside the image processing circuit 20, and specify an image area in the detected image that matches the feature data by pattern matching processing. Face detection performs pattern matching processing to specify an image area in the area obtained by whole body detection that matches the feature data representing the shape of the face stored in advance inside the image processing circuit 20 within the area.

[0014] Also, it calculates the degree of coincidence with the feature data stored inside the image processing circuit 20, and sets an area with a coincidence degree of a predetermined value or more as the area of the face or whole body. In addition, in order to increase the detection opportunities for the face and whole body and improve the detection accuracy, pattern matching processing is performed using a plurality of feature data stored inside the image processing circuit 20. Note that pattern matching processing may be performed using only the feature data of a part of the shape of the face or whole body. Also, pattern matching processing may be performed by changing the size of the feature data in order to detect regardless of the size of the face or whole body.

[0015] Organ detection is performed by pattern matching processing to identify an image area that matches the feature data representing the shape of an organ stored in advance inside the image processing circuit 20 within the area obtained by face detection. As another detection method, detection by deep learning can also be performed. The image processing circuit 20 has a plurality of multiplier-accumulators and is also used as a processor that performs processing for deep learning.

[0016] The image processing circuit 20 applies subject detection processing to the image data using one learning model selected by the system control circuit 50 from among the plurality of learning models stored in the non-volatile memory 31 described later. Also, the image processing circuit 20 may perform multiple types of detection processing by switching the learning models stored in the non-volatile memory 31 for one piece of image data.

[0017] For example, the non-volatile memory 31 stores three learning models: a learning model capable of detecting parts such as the pupils, faces, and entire bodies of dogs and cats, a learning model capable of detecting parts such as the pupils, faces, and entire bodies of birds, and a learning model capable of detecting vehicles such as trains and cars. The system control circuit 50 performs image detection processing based on the learning model in the image processing circuit 20 using one of the three learning models.

[0018] By the system control circuit 50 performing detection three times using the three learning models for one image, it becomes possible to detect parts such as the pupils, faces, and entire bodies of dogs and cats, parts such as the pupils, faces, and entire bodies of birds, and vehicles such as trains and cars. Note that in this embodiment, parts such as the pupils, faces, and entire bodies of dogs and cats, parts such as the pupils, faces, and entire bodies of birds, and vehicles such as trains and cars are detected, but the subjects to be detected are not limited to these. Also, as long as the subject can be detected, another method different from the method shown in this embodiment may be used.

[0019] In the image processing circuit 20, tracking processing between images such as images during live view is also performed. When the image processing circuit 20 detects a subject, it temporarily stores an image of the detected subject area in the memory 30 as a template. Also, based on the template information temporarily stored in the memory 30, the image processing circuit 20 searches for an area that matches the template temporarily stored in the memory 30 from the images generated during live view, and performs subject tracking processing with the matched area as the subject area.

[0020] As a method for the image processing circuit 20 to search for an area that matches the template temporarily stored in the memory 30, there is a method of cutting out the image for each area, taking the absolute value of the difference from the template temporarily stored in the memory 30, and setting the area with a small difference as the subject area. There are also other methods such as obtaining from the degree of match between the template temporarily stored in the memory 30 and the histogram, color data, etc. However, as long as an area that matches the template temporarily stored in the memory 30 can be specified from the image, other methods can be used.

[0021] In this way, in this embodiment, the image processing circuit 20 functions as a tracking unit that tracks a specific subject (at least one part thereof) detected from an image by comparing between images over a plurality of images sequentially acquired by the image acquisition means. Note that the above tracking unit may be configured to include the system control circuit 50. Note that the tracking unit performs tracking based on the subject area corresponding to the specific subject (part) detected by the detection unit.

[0022] When a specific subject (part) is no longer detected by the detection unit, the tracking of the specific subject (part) is continued by referring to the subject area in the image in which the specific subject (part) was detected. This will be described later with reference to FIG. 4. When there are a plurality of detected subjects, the system control circuit 50 determines the main subject from the subject selected by the user or the size and position of the subject within the screen, and sets the area of the main subject as the focus detection area described later.

[0023] When the subject detected during live view becomes undetectable, the system control circuit 50 switches to subject tracking processing using the image processing circuit 20, and sets the subject area obtained by the subject tracking processing as the focus detection area. By performing subject tracking processing, it is possible to continue focusing on the subject by performing focus detection on the same subject part for a longer time.

[0024] If the system control circuit 50 can detect the part of the same subject within the tracking continuation time described later, it executes the focus detection process, and if tracking becomes impossible, it sets another part of the same subject or another type of subject as the main subject (focus detection area) according to the flow shown in FIG. 4. That is, when the time for continuously tracking the part of the same subject has passed the tracking continuation period shown in FIG. 3, the system control circuit 50 sets another part of the same subject or a different type of subject as the main subject and sets the focus detection area as shown in FIG. 4.

[0025] In addition, the image processing circuit 20 performs predetermined arithmetic processing using the captured image data in order to perform auto white balance (hereinafter referred to as AWB) processing. Further, the obtained arithmetic result is calculated as a white balance (hereinafter referred to as WB) evaluation value. Also, color conversion of the image data is performed based on the calculated WB evaluation value. Furthermore, the image processing circuit 20 performs predetermined arithmetic processing using the captured image data in order to calculate an AE evaluation value and an EF evaluation value for performing automatic exposure control (hereinafter referred to as AE) processing and strobe exposure control (hereinafter referred to as EF) processing.

[0026] Based on the obtained AE evaluation value and EF evaluation value, the system control circuit 50 controls the exposure control unit 40 and the flash 48 according to a predetermined algorithm. 22 is a memory control circuit that controls the A / D converter 16, the timing generation unit 18, the image processing circuit 20, the memory 30, the compression / expansion unit 32, etc.

[0027] The data of the A / D converter 16 is written into the memory 30 via the image processing circuit 20 and the memory control circuit 22, or the data of the A / D converter 16 is directly written into the memory 30 via the memory control circuit 22. 28 is an image display unit composed of an LCD or the like, and the image data for display written into the memory 30 is displayed by the image display unit 28 based on the control by the memory control circuit 22. By sequentially displaying the captured image data using the image display unit 28, an electronic viewfinder function can be realized.

[0028] Also, the image display unit 28 can turn the display ON / OFF according to an instruction from the system control circuit 50, and when the display is turned OFF, the power consumption of the imaging device 100 can be significantly reduced. 30 is a memory for temporarily storing the captured still images and moving images, and has a storage capacity sufficient to store a predetermined number of still images and a moving image for a predetermined time.

[0029] Thereby, even in the case of continuous shooting of a plurality of still images, it is possible to perform high-speed and large-volume image writing to the memory 30. Also, the memory 30 can also be used as an area for temporarily storing authentication feature data and as a work area for the system control circuit 50.

[0030] 31 is a non-volatile memory composed of a FlashROM or the like. The program code (computer program) executed by the system control circuit 50 is written into the non-volatile memory 31, and the system control circuit 50 executes various processes while sequentially reading out the program code. Also, in the non-volatile memory 31, areas for storing face feature data for authentication as dictionary data, areas for storing system information, and areas for storing user setting information are provided, so that various information and settings can be read out and restored at startup.

[0031] 32 is a compression and decompression unit that compresses and decompresses image data by means of an adaptive discrete cosine transform (ADCT) or the like. It reads the image stored in the memory 30, performs compression processing or decompression processing, and rewrites the processed data back into the memory 30. 40 is an exposure control unit that controls the mechanical shutter 12 having an aperture function, and can also perform flash light control by the flash 48.

[0032] 42 is a focus control unit that controls the focusing of the photographing lens 10, and 44 is a zoom control unit that controls the zooming of the photographing lens 10. 48 is a flash, which has a flash light control function. Note that based on the operation result obtained by the image processing circuit 20 calculating the captured image data, the system control circuit 50 controls the exposure control unit 40 and the focus detection control means 42.

[0033] The system control circuit 50 also performs autofocus (hereinafter referred to as AF) processing based on the pixel data for phase difference detection obtained from the imaging device 14. Here, AF is automatic focus detection that sets the main subject area selected by the user or the main subject area automatically set by the camera as the focus detection area and automatically detects the focus position.

[0034] The imaging device 14 is composed of a C-MOS sensor and its peripheral circuits, has a pixel array of m in the horizontal direction and n in the vertical direction, and one photoelectric conversion element is arranged in each pixel. The imaging device 14 is configured to be able to output signals independent of all pixels. Also, a part of all pixels are pixels for focus detection, and it is possible to perform AF (imaging plane phase difference AF) using the imaging plane phase difference detection method.

[0035] Specifically, as shown in FIG. 2, the imaging device 14 has a plurality of focus detection pixels 251 dispersed in the imaging pixels 250. FIG. 2 is a diagram showing an arrangement example of the imaging pixels and the focus detection pixels in the first embodiment. Further, each of the plurality of pixels 251 for focus detection is configured to receive a light beam passing through one of the regions of a pair of different exit pupils of the imaging optical system. Each of the imaging pixels 250 receives a light beam passing through the entire region of the exit pupil of the imaging optical system that forms an image of the subject, and generates an image of the subject.

[0036] In addition, a color filter of, for example, a Bayer array is disposed on the front surface of the imaging pixel 250. Also, in the example of FIG. 2, in the imaging device 14, among the 2 rows × 2 columns of pixels belonging to the first and second rows, a pair of G pixels (Gr pixel and Gb pixel) arranged diagonally are configured as imaging pixels, and the B pixels are replaced with pixels for focus detection. The 2 rows × 2 columns of pixels belonging to the fifth and sixth rows are configured in the same manner.

[0037] The system control circuit 50 performs focus detection processing in the phase difference AF method using the imaging signals of the pixels for focus detection discretely arranged in the imaging device 14. That is, a pair of image signals are formed from a plurality of pixels for focus detection by light beams passing through a pair of pupil regions of the imaging optical system. The pixel group for focus detection in the second row of FIG. 2 forms, for example, a right-eye image signal, and the pixel group for focus detection in the sixth row of FIG. 2 forms, for example, a left-eye image signal. Then, by performing a correlation operation on this pair of image signals, the amount of deviation between this pair of images is obtained. Since this deviation amount changes according to the distance to the subject, focus detection is performed based on this deviation amount.

[0038] In this embodiment, imaging surface phase difference AF is realized by replacing a part of the imaging pixel array disposed on the imaging surface with the pixels 251 for focus detection. However, the present invention is not limited to this method, and any configuration capable of focus detection may be used. For example, phase difference focus detection using a known dedicated focus detection sensor or contrast focus detection using a known method may be used for focus detection.

[0039] 60, 62, 66, 70, and 72 constitute operation means for inputting various operation instructions of the system control circuit 50, and are composed of one or a combination of a plurality of switches, dials, touch panels, pointing by gaze detection, voice recognition devices, etc. 60 is a mode dial, and can switch and set each function mode such as power off, automatic shooting mode, shooting mode, panorama shooting mode, video shooting mode, playback mode, PC connection mode, etc.

[0040] 62 is a two - stroke (SW1, SW2) shutter switch. When SW1 turns ON during the process of pushing in the shutter switch 62, operations such as AF (auto - focus) processing, AE (auto - exposure) processing, and AWB (auto - white balance) processing are started. When the shutter switch 62 is pushed in all the way, the shutter switch SW2 turns ON and still - image shooting starts. In the case of flash shooting, after performing the flash pre - emission processing for EF, the mechanical shutter 12 is operated to expose the imaging element 14 for the exposure time determined by the AE processing.

[0041] During this exposure period, the flash is made to emit light, and when the exposure period ends, the mechanical shutter 12 is blocked by the exposure control unit 40 to end the exposure of the imaging element 14. The signal read from the imaging element 14 is written as image data into the memory 30 via the A / D converter 16 and the memory control circuit 22, and development processing using the operations in the image processing circuit 20 and the memory control circuit 22 is performed. Also, the image data is read from the memory 30 and compressed by the compression - expansion unit 32.

[0042] After that, recording processing for writing the image data to the recording medium 200 is performed, and these series of still - image shooting and recording processing operations are performed by the ON of the shutter switch SW2. 66 is a display switching switch that can switch the display of the image display unit 28 (ON / OFF switching). With this function, when shooting using the optical viewfinder 104, power consumption can be reduced by turning off the image display unit 28 composed of an LCD or the like. 70 is an operation unit composed of various buttons, a touch panel, a rotary dial, etc., and includes a menu button, a set button, a macro button, a multi-screen playback page turning button, a flash setting button, a single shot / continuous shot / self-timer switching button, etc.

[0043] Also, the operation unit 70 includes a menu move + (plus) button, a menu move - (minus) button, a playback image move + (plus) button, a playback image - (minus) button, a shooting image quality selection button, an exposure compensation button, a date / time setting button, etc. 72 is a zoom switch as a zoom operation means for the user to give an instruction to change the magnification of the captured image. This zoom switch 72 includes a tele switch for changing the imaging angle of view to the telephoto side and a wide switch for changing it to the wide angle side.

[0044] By using the zoom switch 72, it becomes a trigger to instruct the zoom control unit 44 to change the imaging angle of view of the shooting lens 10 and perform an optical zoom operation. Also, when the optical zoom end is reached, it also becomes a trigger for electronic zooming change of the imaging angle of view by image cropping by the image processing circuit 20, pixel interpolation processing, etc.

[0045] 86 is a power supply means composed of a primary battery, a secondary battery, an AC adapter, etc. 90 is an interface with a recording medium 200 such as a memory card or a hard disk, and 92 is a connector for making an electrical connection with the recording medium 200. 104 is an optical viewfinder, which is provided separately from the image display unit 28, and it is possible to perform shooting using only the optical viewfinder.

[0046] 110 is a communication unit with a built-in GPS, which receives GPS radio waves via the connector 112 and acquires position information. 200 is a recording medium, which includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, etc., an interface 204 with the imaging device 100, and a connector 206 for making an electrical connection with the imaging device 100.

[0047] In the above configuration, depending on the type of the subject to be tracked, it may be difficult to track due to few feature amounts in the image or the like, and there may be a case of erroneously tracking another subject. In this embodiment, in order to reduce such erroneous tracking, the tracking continuation period is changed according to the type and part of the subject. Subsequently, FIG. 3 is a flowchart showing the operation of setting the tracking continuation time in the first embodiment, and a method for determining the tracking continuation period by the system control circuit 50 will be described with reference to FIG. 3.

[0048] In FIG. 3, when the system control circuit 50 determines that the subject detected during the live view has disappeared and switches to the tracking process, the system control circuit 50 switches the tracking continuation time while performing the tracking operation. Here, the system control circuit 50 functions as a switching means for switching the tracking continuation time (the time for continuing the tracking) according to the type of the subject detected from the image or the part of the subject.

[0049] In step S201, it is determined whether the subject to be tracked is a person. That is, the type of the subject is determined. If it is not a person, the process proceeds to step S207, and if it is a person, the process proceeds to step S202. In step S202, it is determined whether the part to be tracked is tracking either of the left and right pupils of the person. That is, it is determined whether the part of the subject is a pupil. If it is determined that the tracking part is a pupil, the process proceeds to step S203, and if it is not determined to be a pupil, the process proceeds to step S204.

[0050] In step S203, the tracking continuation time is set to A and the process ends. In step S204, it is determined whether the tracked part includes the back of the head of a person, i.e., whether the part of the subject is a face. If it is determined to be a face, the process proceeds to step S205, where the tracking duration is set to B and the process ends. If it is not determined to be a face, the process proceeds to step S206. In step S206, the tracked part is regarded as the whole body of the person, the tracking duration is set to C, and the process ends.

[0051] Here, the tracking duration and the set times A, B, and C are determined according to the general feature amounts of each part of the person. That is, since the relationship of the feature amount of the whole body > the feature amount of the face > the feature amount of the pupil generally holds, the tracking duration is set as C > B > A. In step S207, it is determined whether the subject to be tracked is a dog, a cat, or the like. That is, it is determined whether the type of the subject is a dog, a cat, or the like. If the subject to be tracked is a dog, a cat, or the like, the process proceeds to step S208; otherwise, the process proceeds to step S213.

[0052] In step S208, it is determined whether the tracked part is tracking one of the left and right pupils. If the tracked part is determined to be a pupil, the process proceeds to step S209, where the tracking duration is set to D and the process ends. If it is not determined to be a pupil in step S208, the process proceeds to step S210. In step S210, it is determined whether the tracked part includes the back of the head of a dog, a cat, or the like, i.e., whether it is a face. If it is determined to be a face, the process proceeds to step S211, where the tracking duration is set to E and the process ends. If it is not determined to be a face, the process proceeds to step S212.

[0053] In step S212, the tracked part is regarded as the whole body of a dog, a cat, or the like, the tracking duration is set to F, and the process ends. Here, the tracking duration and the set times D, E, and F are in the order of the general feature amounts corresponding to the parts of a dog, a cat, or the like, i.e., F > E > D. That is, also in the case of a dog, a cat, or the like, since the relationship of the feature amount of the whole body > the feature amount of the face > the feature amount of the pupil generally holds, the tracking duration is set as F > E > D.

[0054] In step S213, it is determined whether the subject to be tracked is a bird. That is, it is discriminated whether the type of the subject is a bird. If the subject to be tracked is a bird, the process proceeds to step S214; if it is not a bird, the process proceeds to step S219. In step S214, it is determined whether the part being tracked is one of the pupils on the left or right. If the part being tracked is determined to be a pupil, the process proceeds to step S215, where the tracking duration is set to G and the process ends. If it is not determined to be a pupil in step S214, the process proceeds to step S216.

[0055] In step S216, it is determined whether the part being tracked is a face including the back of the bird's head. If it is determined to be a face, the process proceeds to step S217, where the tracking duration is set to H and the process ends. If it is not determined to be a face, the process proceeds to step S218.

[0056] In step S218, the part being tracked is regarded as the entire body of the bird, the tracking duration is set to I, and the process ends. Here, the tracking duration and the set times G, H, and I are set such that I > H > G in the order of the amount of general feature quantities corresponding to the parts of the bird. That is, also in the case of a bird, the relationship of the feature quantity of the whole body > the feature quantity of the face > the feature quantity of the pupil generally holds, so the tracking duration is set to I > H > G.

[0057] In step S219, the subject to be tracked is regarded as a vehicle, the tracking duration is set to J, and the process ends. Here, the tracking duration and the set times C, F, I, and J are set such that J > C > F > I in the order in which it is assumed that there are more feature quantities corresponding to the type of the subject.

[0058] That is, in the order of the size of the whole body being large, the relationship of the feature quantity of the whole vehicle > the feature quantity of the whole body of a person > the feature quantity of the whole body of a dog, cat, etc. > the feature quantity of the whole body of a bird generally holds, so the tracking duration is set to J > C > F > I. Note that the above tracking duration may be set according to the general moving speed of the subject, etc. Alternatively, the above tracking duration may be set according to an appropriate combination of the size, feature quantity, and moving speed of the subject.

[0059] In addition, in steps S202, S208, and S214, instead of determining whether one of the left and right pupils is being tracked, for example, it may be replaced with a determination of whether an area (part) where the feature amount is assumed to be smaller than the face area, such as the mouth, is being tracked. Also, the magnitude relationship of A, B, C, D, E, F, G, H, I, and J may be changed according to the feature amount of the assumed area (part). Also, the values of A to J may be set manually by the user, or may be set automatically based on past learning data using AI.

[0060] As described above, in this embodiment, the tracking continuation time is switched according to the part (at least one of the whole body, face, and pupil) or the type of the subject (at least one of a person, dog, cat, bird, vehicle, etc.). Furthermore, in this embodiment, in the part of the subject, the tracking continuation time is switched so that the time for continuing the tracking of the part including the predetermined part is longer than the time for continuing the tracking of the included part.

[0061] That is, in the same subject, the larger the part, the longer the tracking continuation time is switched so that the tracking continuation time is longer, so that it is possible to further reduce the mis-tracking of the subject. Also, in this embodiment, the larger the size of each type of the subject, the longer the tracking continuation time is switched so that the tracking continuation time is longer, so that it is possible to reduce the mis-tracking of the subject.

[0062] Next, in the tracking process, when the system control circuit 50 determines that the tracking cannot be continued or when the tracking continuation period has elapsed, the system control circuit 50 re-determines the main subject according to FIG. 4. FIG. 4 is a flowchart showing the operation after the tracking continuation time of the first embodiment has elapsed. In step S301 of FIG. 4, it is determined whether the tracked subject is a person or an animal. If the type of the subject is a person or an animal, the process proceeds to step S302; otherwise, it proceeds to step S310.

[0063] In step S302, if the tracking part is the pupil, the process proceeds to step S303; otherwise, it proceeds to step S305. In step S303, it is determined whether the face of the same subject as the tracked subject can be detected. If the face part of the same subject can be detected, the process proceeds to step S304, where the main subject is set to the face of the same subject and the flow ends. If the face of the same subject cannot be detected, the process proceeds to step S306.

[0064] In step S305, it is determined whether the part of the tracked subject is a face. If the tracked subject is a face, the process proceeds to step S306; otherwise, it proceeds to step S308. In step S306, if the entire body of the same subject as the tracked subject can be detected, the process proceeds to step S307. Then, the main subject is set to the entire body of the tracked subject and the flow ends. If the entire body of the same subject as the tracked subject cannot be detected, the process proceeds to step S310.

[0065] In step S308, if the face part of the same subject as the tracked subject can be detected, the process proceeds to step S309. Then, the main subject is set to the face of the tracked subject and the flow ends. If the face of the same subject as the tracked subject cannot be detected, the process proceeds to step S310.

[0066] In step S310, when a specific subject (part) is no longer detected in a plurality of images sequentially acquired by the image acquisition means, the subject area (the area of the part) in the image in which the specific subject (part) was detected is referred to. Then, if any detection of a different type (or part) from the specific subject (part) being tracked can be made within the subject area (the area of the part), the process proceeds to step S311. And in step S311, the subject that could be detected by referring to the subject area (the area of the part) is determined as the main subject.

[0067] Note that as a method for determining the main subject in step S311, the system control circuit 50 may determine it from the size of the subject that could be detected by referring to the subject area (the area of the part) and the position within the screen. For example, a method where weights are assigned to the distance from the center of the angle of view of the main subject and the size of the subject, and the subject that is close to the center of the angle of view and large, and for a person or an animal, the detected parts in the order of the pupil, face, and body are used as the main subject. The method of determining the main subject may be another method, and the main subject may be determined according to the camera settings or the scene judged by the camera. Also, at that time, the user may touch an arbitrary subject position on the screen to determine the main subject.

[0068] If no other subject can be detected in step S310, the process proceeds to step S312, and the subject position on the side closer to the imaging device within the angle of view is set as the main subject area to be the AF target. For determining the position on the near side within the angle of view, there are methods such as using the defocus map to set the area estimated to be the closest from the imaging sensor.

[0069] Also, as a method for setting the subject area when no other subject can be detected, another method may be adopted, for example, searching for and determining an area having relatively prominent features from the area within the angle of view using AI or the like. As described above, by changing the tracking continuation period according to the type and part of the subject and appropriately changing the main subject, it is possible to reduce the mis - tracking of the subject.

[0070] That is, in this embodiment, when the time for continuing the tracking of a predetermined part of the subject has elapsed, if a part including the predetermined part is detected, the part including the part is tracked, so that it is possible to further reduce the mis-tracking of the subject. As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0071] For example, although this embodiment has taken as an example changing the tracking continuation period according to the part or the subject type, the tracking continuation period may be changed according to the size of the subject area to be tracked or the ratio of the subject area to the screen. In that case, when the subject area is large, it is changed so as to lengthen the tracking continuation period.

[0072] Also, it is determined whether the same or similar part is detected in the vicinity of the area being tracked, and when the same or similar part is detected in the vicinity of the area being tracked, the tracking continuation period may be changed so as to be lengthened. In addition, the amount of edges or the saliency (for example, contrast) of the subject area to be tracked is calculated, and when the amount of edges is large or the saliency is high, it is determined that there are many feature amounts of the subject and the tracking continuation period may be changed so as to be lengthened.

Embodiment

[0073] In addition, in an animal (for example, Dalmatian, etc.) having a pattern with features similar to the pupil all over the body, mis-tracking of the pupil is likely to occur. In the second embodiment, in order to reduce such mis-tracking of the pupil, when the position of the pupil part of the subject is outside the face area of the subject whose position was detected immediately before, the time for continuing the tracking is made shorter than when it is inside the face area of the subject. Subsequently, with reference to FIGS. 5 and 6, a processing method of the second embodiment in which the system control circuit 50 changes the pupil tracking continuation time based on the positional relationship between the face detection result and the pupil tracking position will be described.

[0074] FIG. 5 is a flowchart showing the operation of a method for reducing pupil mis-tracking using the face detection result of Example 2. Note that the process of FIG. 5 is performed after the pupil tracking duration setting steps (S203, S209, S215) for each subject type in FIG. 3. In FIG. 5, the system control circuit 50 changes the tracking duration based on the pupil tracking position and the face position of the subject detected from the immediately preceding image.

[0075] In step S401, it is determined whether the subject to be tracked is a pupil. If it is not a pupil, the process proceeds to step S406, and if it is a pupil, the process proceeds to step S402. In step S402, it is determined whether a face with a certain level of reliability or more has been detected in the frame immediately before the current tracking frame. The frame immediately before is a frame in which it is expected that the position of the subject does not deviate significantly from that of the current frame, and specifically refers to a frame within, for example, one frame from the current tracking frame.

[0076] Also, a certain level of reliability or more refers to, for example, a reliability level at which a face is detected with a correct answer rate of about 80%. If there is no face detection result with a certain level of reliability or more in the immediately preceding frame, the process proceeds to step S406, and if there is a face detection result, the process proceeds to step S403. In step S403, region expansion is performed on the face detection result (face range, face region) of S402.

[0077] Since there is a possibility that the face orientation of the subject has changed or the face position itself has changed due to movement between the previous frame and the current frame, the region of the face in the previous frame is expanded at an arbitrary magnification and used for determining the validity of the pupil tracking position. For example, the size of the face detection is multiplied by a magnification factor of 1.2 to 1.8, and the face region is set at a magnification factor that can cover the entire head of the subject to cope with changes in face orientation and face position.

[0078] Also, when the subject movement speed calculated by any method based on the difference in position from the previous frame is slow, it is desirable to set a low magnification for expanding the face region, and when the movement speed is fast, it is desirable to set a high magnification for expanding the face region. Similarly, when the distance from the subject to the image acquisition means (subject distance) is far, it is expected that the movement amount of the subject on the imaging device 14 is small, so the magnification for expanding the face region may be set low, and when the subject distance is close, the magnification for expanding the face region may be set high.

[0079] Further, according to the reliability of the face detection result, the magnification may be set higher as the reliability is lower. Also, the magnification may be set according to the type of the subject. For example, the magnification is set higher for a subject of a type that is difficult to distinguish. In the above method, the system control circuit 50 calculates the predicted face region of the current frame and transitions to step S404.

[0080] In step S404, it is determined whether the center coordinates of the tracking position of the pupil in the current frame are within the face region calculated in step S403. If the center coordinates of the tracking position are outside the range of the face region, the process transitions to step S405. If it is within the range (inside) of the face region, the process transitions to step S406.

[0081] In step S405, the system control circuit 50 sets a shorter pupil tracking duration. Basically, the tracking duration is set to zero so that the next subject can be tracked immediately, but depending on the distance from the face region to the pupil, settings such as making it shorter as the distance is farther may be made. After the setting of the tracking time is completed, the process ends. Note that during the tracking process, the system control circuit 50 determines whether shooting is continuing. If it is, for example, the process transitions to step S201, and if shooting ends, the flow of FIGS. 3 to 5 ends.

[0082] FIG. 6 is a diagram showing a method for reducing pupil mis-tracking using the face detection result of the second embodiment, and shows an operation in which the system control circuit 50 changes the tracking duration based on the pupil tracking position and the face position of the subject detected from the immediately preceding image. For frame 500 in the live view updated at 60fps (frames per second), frame 501 of the tracking result is output at 60fps intervals shifted by the tracking processing time. On the other hand, the detection result 502 such as a face is output at 30fps intervals because it requires more complex processing than tracking. Hereinafter, the determination of shortening the tracking time of the pupil tracking frame in frame 2 will be described as an example.

[0083] In the image 503 of frame 1, the face detection range 505 of the subject 504 is shown. In frame 2, an image of the case where the tracked pupil is within the range of the face area (the case where it is No in step S404) is shown in 509. Also, an image of the case where the pupil is outside the range of the face area (the case where it is Yes in step S404) due to mis-tracking the pattern of the subject's body as the pupil is shown in 513. In the image 509, what is obtained by applying the face detection range 505 of frame 1 to frame 2 is the face detection range 506, and the expanded face area calculated in step S403 of FIG. 5 is 507.

[0084] In the image 509, since the tracked pupil position 508 is inside the face area 507, the pupil tracking position is considered appropriate and the pupil tracking time is not changed. On the other hand, in the image 513, what is obtained by applying the face detection range 505 of frame 1 to frame 2 is 510, and the expanded face area calculated in step S403 of FIG. 5 is 511. Since the tracked pupil position 512 is outside the face area 507, the pupil tracking position is considered inappropriate, and the system control circuit 50 shortens the pupil tracking time (step S405).

[0085] As in 514, when there is no face detection result (No in step S402), the determination using the above-expanded face area is not performed, and the pupil tracking position is used as it is. That is, in this embodiment, when the pupil tracking position goes outside (outside) the range of the (expanded) face area, by setting the tracking continuation time short, it is possible to reduce the mis-tracking of the subject. As described above, the preferred embodiments of the present invention have been described, but the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

[0086] Note that a part or all of the control in this embodiment may be supplied to an imaging device or the like via a network or various storage media by a computer program that realizes the functions of the above-described embodiments. Then, a computer (or CPU, MPU, etc.) in the imaging device or the like may read and execute the program. In that case, the program and the storage medium storing the program will constitute the present invention.

Explanation of Reference Numerals

[0087] 10: Photographing lens 12: Mechanical shutter 14: Image sensor 16: A / D converter 18: Timing generation unit 20: Image processing circuit 22: Memory control circuit 28: Image display unit 30: Memory 32: Compression / expansion unit 40: Exposure control unit 42: Focus control unit 44: Zoom control unit 48: Flash 50: System control circuit 60: Mode dial switch 62: Shutter switch 66: Display switching switch 70: Operation unit 72: Zoom switch 90: Interface 92: Connector 100: Imaging device 104: Optical viewfinder 110: Communication unit 112: Connector 200: Recording medium 202: Recording unit 204: Interface 206: Connector

Claims

1. Image acquisition means for sequentially acquiring images, Tracking means for tracking a subject detected from the images acquired by the image acquisition means over a plurality of images sequentially acquired by the image acquisition means, Switching means for switching the time for continuing the tracking in the tracking means according to the type of the subject detected from the images, and having, The switching means further switches the time for continuing the tracking in the tracking means according to the part of the subject, The time for continuing the tracking when the subject is a human face is made larger than the time for continuing the tracking when the subject is one of a face or pupil of a dog, a cat, or a bird. A subject tracking device characterized by that.

2. Image acquisition means for sequentially acquiring images, Tracking means for tracking a subject detected from the images acquired by the image acquisition means over a plurality of images sequentially acquired by the image acquisition means, Switching means for switching the time for continuing the tracking in the tracking means according to the type of the subject detected from the images, and having, The switching means further switches the time for continuing the tracking in the tracking means according to the part of the subject, The time for continuing the tracking when the subject is a human face is made larger than the time for continuing the tracking when the subject is a human pupil. A subject tracking device characterized by that.

3. Having detection means for detecting a specific subject from the images, The tracking means performs tracking based on a subject area corresponding to the specific subject detected by the detection means. The subject tracking device according to claim 1 or 2, characterized by that.

4. The switching means switches the time for continuing the tracking according to the size or feature amount of the part. The subject tracking device according to claim 3, characterized by that.

5. When the specific subject is not detected by the detection means in a plurality of images sequentially acquired by the image acquisition means, the tracking means continues to track the specific subject based on the subject area in the image in which the specific subject was detected. The subject tracking device according to claim 3, characterized in that.

6. The switching means makes the time for continuing to track a part including a predetermined part among the parts of the subject longer than the time for continuing to track the part including the part. The subject tracking device according to any one of claims 1 to 5, characterized in that.

7. When the time for continuing to track a predetermined part of the subject has elapsed, if a part including the predetermined part is detected, the subject tracking device according to any one of claims 1 to 5, characterized in that the part including the part is tracked.

8. Imaging means for performing imaging; The subject tracking device according to any one of claims 1 to 7; An imaging device comprising:

9. A computer program for causing a computer to function as each means of the subject tracking device according to any one of claims 1 to 7.

10. A computer-readable storage medium storing the computer program according to claim 9.

11. An image acquisition step of sequentially acquiring images; A tracking step of tracking a subject detected from the images acquired in the image acquisition step over a plurality of images sequentially acquired in the image acquisition step; A switching step of switching the time for continuing the tracking in the tracking step according to the type of the subject detected from the images, and having: The switching step switches the time for continuing the tracking in the tracking step according to the part of the subject. A subject tracking method, characterized in that when the subject is a human face, the time for continuing the tracking is made longer than the time for continuing the tracking when the subject is one of the faces or pupils of a dog, a cat, or a bird. **Claim 12** An image acquisition step of sequentially acquiring images; A tracking step of tracking a subject detected from the images acquired in the image acquisition step over a plurality of images sequentially acquired in the image acquisition step; A switching step of switching the time for continuing the tracking in the tracking step according to the type of the subject detected from the images, and having: The switching step switches the time for continuing the tracking in the tracking step according to the part of the subject; A subject tracking method, characterized in that when the subject is a human face, the time for continuing the tracking is made longer than the time for continuing the tracking when the subject is a human pupil. **Claim 13** An image acquisition means for sequentially acquiring images; A detection means for detecting a specific subject from the images acquired by the image acquisition means; A tracking means for tracking the subject over a plurality of images sequentially acquired by the image acquisition means when the subject is no longer detected after being detected by the detection means; A switching means for switching the time for continuing the tracking in the tracking means according to the type of the subject detected by the detection means, and having: The switching means further switches the time for continuing the tracking in the tracking means according to the part of the subject; A subject tracking device, characterized in that when the subject is a human face, the time for continuing the tracking is made longer than the time for continuing the tracking when the subject is one of the faces or pupils of a dog, a cat, or a bird. **Claim 14** An image acquisition means for sequentially acquiring images; detection means for detecting a specific subject from the image acquired by the image acquisition means; tracking means for tracking the subject over a plurality of images sequentially acquired by the image acquisition means when the subject is no longer detected after being detected by the detection means; switching means for switching the time for continuing the tracking in the tracking means according to the type of the subject detected by the detection means, and the switching means further switches the time for continuing the tracking in the tracking means according to the part of the subject, wherein the time for continuing the tracking when the subject is a human face is made longer than the time for continuing the tracking when the subject is a human pupil. A subject tracking device characterized by this.

15. The subject tracking device according to claim 13 or 14, wherein the tracking means performs tracking based on a subject area corresponding to the specific subject detected by the detection means.

16. The subject tracking device according to claim 15, wherein the switching means switches the time for continuing the tracking according to the size or feature amount of the part.

17. The subject tracking device according to claim 15, wherein, in a plurality of images sequentially acquired by the image acquisition means, when the specific subject is no longer detected by the detection means, based on the subject area in the image where the specific subject was detected, the tracking of the specific subject is continued.

18. The subject tracking device according to any one of claims 13 to 17, wherein the switching means makes the time for continuing the tracking of a part including a predetermined part of the subject longer than the time for continuing the tracking of the included part.

19. The subject tracking device according to any one of claims 13 to 18, wherein when the time for continuing the tracking of a predetermined part of the subject has elapsed and a part including the predetermined part is detected, the part including the predetermined part is tracked.

20. Imaging means for performing imaging, the subject tracking device according to any one of claims 13 to 19, and an imaging device provided with the same.

21. A computer program for causing a computer to function as each means of the subject tracking device according to any one of claims 13 to 19.

22. A computer-readable storage medium storing the computer program according to claim 21.

23. An image acquisition step of sequentially acquiring images, a detection step of detecting a specific subject from the acquired images, a tracking step of tracking the subject over a plurality of images sequentially acquired by the image acquisition step when the subject is no longer detected after being detected in the detection step, and a switching step of switching the time for continuing the tracking in the tracking step according to the type of the subject detected in the detection step, and in the switching step, the time for continuing the tracking in the tracking step is switched according to the part of the subject, and a subject tracking method characterized in that the time for continuing the tracking when the subject is a human face is made longer than the time for continuing the tracking when the subject is one of a face or a pupil of a dog, a cat, or a bird.

24. An image acquisition step of sequentially acquiring images, a detection step of detecting a specific subject from the acquired images, a tracking step of tracking the subject over a plurality of images sequentially acquired by the image acquisition step when the subject is no longer detected after being detected in the detection step, and a switching step of switching the time for continuing the tracking in the tracking step according to the type of the subject detected in the detection step, and In the switching step, according to the part of the subject, the time for continuing the tracking in the tracking step is switched, A subject tracking method characterized in that the time for continuing the tracking when the subject is a person's face is made longer than the time for continuing the tracking when the subject is a person's pupil.

Images