Tracking device, control method thereof, and imaging device

Abstract

To enable a photographer, when tracking a subject, to compose a shot that shows a subject's movement.SOLUTION: A tracking device includes detection means for detecting a subject in an image obtained by photoelectric conversion of light incident through an imaging optical system by an imaging element, tracking means for tracking the subject in a plurality of predetermined directions, determination means for determining a tracking direction among the plurality of directions in which the subject is to be tracked, and control means for controlling the tracking means so as to track the subject in the tracking direction and to suppress tracking in a direction different from the tracking direction.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a tracking device, a control method thereof, and an imaging device. [Background technology]

[0002] Conventionally, imaging devices have been available that have a function (subject tracking function) in which a user specifies a main subject among subjects detected from an image and the angle of view tracks the specified main subject in order to support subject framing operations. In imaging devices equipped with such a subject tracking function, a user specifies a subject image displayed on a screen using a touch panel or an operation switch, etc., and a tracking frame is displayed in the area of ​​the specified subject image. Furthermore, the tracking function is realized by changing the image stabilization lens and the image cut-out position according to the position of the tracking frame.

[0003] Patent Document 1 discloses the configuration of a camera equipped with an anti-shake function that maintains the position of a subject image at the center of the angle of view when the position of the detected subject image is displaced from the center of the angle of view. In Patent Document 1, the amount of deviation between the center position of the subject image and the center position of the screen is calculated. Then, when the subject image is about to go out of the frame due to camera shake or subject movement, the anti-shake lens is driven so that the amount of deviation from the center of the screen to the subject image approaches zero, thereby preventing the subject image from going out of the frame. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-111430 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when taking continuous shots or videos in a scene where you want to record the continuous movement of a subject, if the subject position tracked by the subject tracking function is always at the center of the screen, an image may be taken with a composition in which the background, not the subject, is moving. For example, when shooting sports such as gymnastics or figure skating, it is desirable to be able to capture the athlete at the center of the screen in the direction in which the athlete is running, and capture the athlete's movement in the direction in which the athlete is jumping. However, in the configuration of Patent Document 1, since the subject is captured at the center of the angle of view, when the subject jumps, as shown in the center of the diagram in FIG. 2(a), the background ground (floor) appears to be moving downward.

[0006] The present invention has been made in consideration of the above problems, and has an object to enable shooting with a composition that allows the movement of a subject to be seen when the subject is being tracked. [Means for solving the problem]

[0007] In order to achieve the above object, the tracking device of the present invention includes a detection unit that detects a subject in an image obtained by photoelectrically converting light incident through an imaging optical system by an imaging element, and a tracking unit that tracks the subject in a plurality of predetermined directions. a vibration detection unit that detects a vibration amount in each of the plurality of directions; The tracking device includes a determination unit that determines a tracking direction in which the subject is tracked among the plurality of directions, and a control unit that controls the tracking unit to perform tracking in the tracking direction and suppress tracking in a direction different from the tracking direction. The determination means determines a shake angle based on the shake amount for each of the plurality of directions, and determines the tracking direction based on the shake angles in the plurality of directions. do. Effect of the Invention

[0008] According to the present invention, when a subject is being tracked, it is possible to capture an image with a composition that allows the movement of the subject to be seen. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the arrangement of an imaging apparatus according to an embodiment of the present invention. [Diagram 2]5A and 5B are diagrams showing examples of positions within the angle of view of a subject tracked by the subject tracking function. [Diagram 3] FIG. 2 is a block diagram showing the functional configuration of an image stabilization control in the imaging apparatus for carrying out processing in the first embodiment. [Figure 4] A graph showing the change in angular velocity ω and angle θ over time. [Diagram 5] 5 is a flowchart showing a processing example of a subject tracking function in the first embodiment. [Figure 6] 5 is a flowchart for explaining a subject designation process in the first embodiment. [Figure 7] 5 is a flowchart for explaining a target position setting process for subject tracking in the first embodiment. [Figure 8] FIG. 11 is a block diagram showing the functional configuration of an image stabilization control in an imaging apparatus for carrying out processing according to a second embodiment. [Figure 9] 10 is a flowchart illustrating a tracking control determination process in the second embodiment. [Figure 10] 13 is a graph showing angular velocity and angle, and swing direction within an image plane in the third embodiment. [Figure 11] FIG. 13 is a diagram showing a subject position and a target position in the third embodiment. [Figure 12] 13 is a flowchart illustrating a target position setting process for subject tracking according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (First embodiment) FIG. 1 is a block diagram showing an example of the configuration of a digital camera 100 having a subject tracking function, as an example of an imaging device according to this embodiment. Lens barrel 101 holds therein a group of lenses including zoom lens 102, focus lens 103, and anti-vibration lens 104, as well as an aperture / shutter 105 (photographing optical system). Zoom lens 102 adjusts the focal length by moving in the direction of the optical axis of lens barrel 101, and optically changes the angle of view. Focus lens 103 adjusts the focus by moving in the direction of the optical axis of lens barrel 101. Anti-vibration lens 104 corrects image blurring caused by camera shake and the like (optical vibration reduction). Aperture / shutter 105, which adjusts the amount of light, is used for exposure control.

[0012] In this embodiment, the digital camera 100 is described as an imaging device in which the lens barrel 101 and the camera body are integrally configured, but the present invention is not limited to this. The present invention is also applicable to an imaging system that is configured with a camera body and an interchangeable lens that is detachable from the camera body.

[0013] The imaging element 106 is an image sensor such as a CCD (charge-coupled device) type or a CMOS (complementary metal-oxide semiconductor) type, which receives light that has passed through the lens barrel 101 and converts the subject image into an electrical signal by photoelectric conversion to generate an image signal. Note that image blur can be corrected by driving the mechanical mechanism that holds the imaging element 106 in accordance with the amount of shake (imaging vibration reduction).

[0014] The image signal generated by the imaging element 106 is input to an image processing unit 107 where various processes such as pixel interpolation and color conversion are performed, and the image data after the various processes is stored in an image memory 108. The image memory 108 is a storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory).

[0015] The image data stored in image memory 108 as described above is sent to non-volatile memory 117 or to recording unit 116 via interface (I / F) unit 115 and recorded in a removable external recording medium (not shown), or is recorded in both memory 117 and the external recording medium. The external recording medium is a memory card or the like that is attached to digital camera 100 when used. Non-volatile memory 117 is a storage medium built into digital camera 100, and stores information such as settings for digital camera 100 in addition to program data and image data.

[0016] The display unit 109 is configured to include, for example, a TFT type LCD (thin film transistor driven liquid crystal display) and displays captured images (image data) and specific information (for example, shooting information, etc.). In addition, an electronic viewfinder (EVF) function that allows the user to adjust the angle of view can be provided by displaying information such as a live view that sequentially displays images captured periodically.

[0017] The operation unit 118 includes zoom operation members such as a zoom lever or a zoom button, a release switch that instructs the start of shooting, a touch panel or operation switches that specify a subject and configure settings on the digital camera 100, etc. Operation signals from the operation unit 118 are sent to the system control unit 150.

[0018] The system control unit 150 includes a calculation device such as a CPU (Central Processing Unit), and controls the entire digital camera 100 by sending control commands to each unit in response to a user's operation. In this embodiment, the system control unit 150 includes a camera control unit 120, a zoom control unit 121, an AF control unit 122, an optical image stabilization control unit 123, an AE control unit 124, an imaging image stabilization control unit 125, an electronic image stabilization control unit 126, a subject detection unit 127, and a frame display unit 128. By executing various control programs stored in the memory 117, the system control unit 150 performs, for example, control of the image sensor 106, AE / AF control, optical / electronic / imaging image stabilization control, zoom control, and the like.

[0019] The camera control unit 120 issues instructions for controlling the overall camera functions. For example, when the release switch is half-pressed, it issues instructions to perform shooting preparation operations such as AE / AF, and when it is fully pressed, it issues instructions to perform shooting operations for still images and videos. In addition, in the subject tracking function described below, the camera control unit 120 also issues instructions to start / stop subject tracking using optical image stabilization, electronic image stabilization, and imaging image stabilization.

[0020] The zoom control unit 121 detects the amount and direction of operation of the zoom operation member provided in the operation unit 118, and calculates the zoom drive speed and zoom drive direction. Then, the zoom lens drive unit 110 is driven according to the calculation result, thereby controlling the movement of the zoom lens 102 along the optical axis. The zoom lens drive unit 110 is composed of an actuator such as a direct current (DC) motor that drives the zoom lens 102, a position detection sensor such as a rotary encoder that detects the motor position, and a drive circuit that drives them. In addition, other configurations such as a configuration that does not require a position detection sensor such as a stepping motor may be used. In a collapsible type lens barrel in which the lens barrel 101 is extended from the camera when the power is turned on, it is necessary to detect an error if the lens barrel is pressed during the extension operation of the lens barrel, so the former configuration is often adopted. In an inner type lens barrel in which the zoom lens 102 is driven inside the lens barrel 101, the latter configuration, which is advantageous for noise reduction, is often adopted. Either configuration may be used as the configuration of the zoom lens drive unit 110 in the present invention.

[0021] An autofocus (AF) control unit 122 performs automatic focus adjustment (AF) control to control the focus lens driving unit 111 so as to focus on the subject, based on focus adjustment information obtained by a known method such as a contrast method or a phase difference method. The focus lens driving unit 111 is configured with an actuator such as a voice coil motor (VCM), a position detection sensor for feeding back the position, and a driving circuit for driving them, and drives the focus lens 103. Alternatively, other configurations such as a stepping motor that does not require a position detection sensor may be used.

[0022] The automatic exposure (AE) control unit 124 calculates exposure control values ​​(aperture value and shutter speed) based on the photometric value obtained by image processing in the image processing unit 107. The AE control unit 124 calculates the brightness value (photometric value) in an arbitrary area (photometric frame) in the screen according to a set photometric method. Photometric methods include spot photometric, which measures the light at a set specific position, evaluative photometric, which automatically determines the position of the photometric frame according to the scene, and center-weighted average photometric, which measures the light averagely over the entire screen with emphasis on the center of the screen. The AE control unit 124 controls the aperture / shutter driving unit 113 based on the photometric result. The aperture / shutter driving unit 113 is composed of actuators such as a stepping motor and an electromagnetic plunger and a driving circuit for driving them, and drives the aperture / shutter 105 to achieve automatic exposure adjustment.

[0023] The optical image stabilization control unit 123 calculates the amount of shake applied to the digital camera 100 based on shake detection information from an angular velocity sensor such as a gyro sensor of the shake detection unit 119. Then, according to the calculation result, the optical image stabilization is realized by driving the image stabilization lens 104 so as to cancel (or reduce) the amount of shake applied to the digital camera 100. More specifically, the optical image stabilization control unit 123 calculates a target position for controlling the image stabilization lens 104 at a predetermined calculation period based on the amount of shake, and issues a drive instruction to the image stabilization lens driving unit 112. Based on the instruction, the image stabilization lens driving unit 112 drives the image stabilization lens 104. The image stabilization lens driving unit 112 is composed of an actuator such as a voice coil motor (VCM), a driving circuit, and a position detection sensor such as a Hall element. The optical image stabilization control unit 123 feeds back the position of the image stabilization lens 104 detected by the position detection sensor of the image stabilization lens driving unit 112, and feedback control is performed so as to hold the image stabilization lens 104 at the target position.

[0024] Like the optical image stabilization control unit 123, the electronic image stabilization control unit 126 calculates the amount of shake applied to the digital camera 100 at a predetermined calculation cycle based on shake detection information from an angular velocity sensor such as a gyro sensor in the shake detection unit 119. Then, in accordance with the calculation result, a partial range of the image stored in the image memory 108 is read out so as to cancel (or reduce) the amount of shake applied to the digital camera 100, and the range of the image to be displayed on the display unit 109 or recorded in the recording unit 116 is changed for each frame image, thereby realizing electronic image stabilization.

[0025] The imaging stabilization control unit 125, like the optical stabilization control unit 123 and the electronic stabilization control unit 126, calculates the amount of shake applied to the digital camera 100 based on shake detection information from an angular velocity sensor such as a gyro sensor of the shake detection unit 119. Then, according to the calculation result, the imaging stabilization is realized by driving a mechanical mechanism that holds the imaging element 106 so as to cancel (or reduce) the amount of shake applied to the digital camera 100. More specifically, the imaging stabilization control unit 125 calculates a target position for controlling the imaging element 106 at a predetermined calculation period based on the amount of shake, and issues a drive instruction to the imaging element drive unit 114. Based on the instruction, the imaging element drive unit 114 drives the imaging element 106. The imaging element drive unit 114 is composed of an actuator such as a voice coil motor (VCM), a drive circuit, and a position detection sensor such as a Hall element. An imaging vibration isolation control unit 125 feeds back the position of the imaging element 106 detected by the position detection sensor of the imaging element driving unit 114, and executes feedback control so as to hold the imaging element 106 at a target position.

[0026] The subject detection unit 127 detects a subject area of ​​a predetermined subject from the image data stored in the image memory . Here, the subject detection process in this embodiment will be described. In this embodiment, a subject detection method (face detection process, color detection process) for detecting a subject (a face of a person or the like, or an object) based on face information or color information included in image data will be described.

[0027] The face detection process is a process that detects a face area present in image data using a known algorithm. For example, the subject detection unit 127 extracts features from a square partial area on the image data and compares the features with face features prepared in advance. If the correlation value between the two exceeds a predetermined threshold, the subject detection unit 127 determines that the partial area is a face area. This determination process is repeated while changing the combination of the size, arrangement position, and arrangement angle of the partial area, thereby making it possible to detect various face areas present in the image data.

[0028] In addition, when the device has a face recognition function, a pattern matching is performed using a known algorithm between the feature amount of a preregistered face image and the feature amount of the detected face area, and the registered face image with the highest correlation value is recognized as the detected face. In addition, when the correlation values ​​for all registered face images are less than a predetermined value, the detected face area is determined to be an unregistered face.

[0029] In the color detection process, a process is executed in which color information of a subject area designated according to a subject designation method described later is stored as a characteristic color. The color detection process is executed when the subject to be detected is an object. As color information, RGB signals, luminance signals (Y signals), and color difference (RY, BY) signals, which are output signals from the image processing unit 107, are used. When detecting a subject, the subject detection unit 127 divides the image data into a plurality of partial areas and calculates the average luminance and color difference for each partial area. In addition, the subject detection unit 127 compares the color information of each area at the time of detecting the subject with prestored characteristic color information, and sets a partial area in which the difference in luminance and color difference is equal to or less than a predetermined amount as a candidate for the subject area. Then, a group of adjacent partial areas in the candidate subject areas is set as a same-color area, and a region in which the same-color area is within a predetermined size range is set as the final subject area.

[0030] When the subject is an object, the shape of the object region may vary depending on the object. Therefore, the smallest quadrangular region that contains all the same color regions is taken as the object region. Therefore, the object region may be a rectangular region with different horizontal and vertical sizes. The position of the object region is taken as the center position of the quadrangular region. Frame display unit 128 displays the face region and object region detected in the subject detection process on display unit 109 using frames (subject detection frames).

[0031] Next, a description will be given of the tracking process of the subject detection frame detected by subject detection unit 127. The tracking process of the subject detection frame is a process of continuing to track a subject area that moves between successive frame images. When the subject tracking function is started, the area of ​​a person or object that is to be the main subject is specified according to a subject specification method described later.

[0032] If the specified subject is a person, that is, a face, all faces in the frame image are detected. If multiple faces are detected in the same image, the subject detection unit 127 determines that the face closest to the position of the face in the immediately preceding frame image is the person to be tracked. In addition, if the detected face is a face registered by the face recognition function, the recognized face is given priority and determined as the person to be tracked, regardless of the positional relationship between the frame images.

[0033] If the specified subject is an object, the comparison process of the characteristic color information starts from the position of the object area in the immediately preceding frame image. If the frame rate is sufficiently higher than the speed of the moving object, there is a high possibility that the same object will be detected in a nearby area between the previous and following frame images. Therefore, the subject detection unit 127 performs matching process of the same color area while shifting the partial area to be compared left and right, up and down, starting from the position of the immediately preceding object area, thereby making it possible to track the moving object area. Frame display unit 128 displays, on display unit 109, the face region and object region tracked by the tracking process of the subject detection frame as a subject tracking frame.

[0034] Next, control related to the subject tracking function in the camera control unit 120 of this embodiment will be described with reference to FIGS. The user specifies a main subject by operating a touch panel or a subject specification operation switch of operation unit 118. Then, a subject tracking function is executed so that the specified main subject is tracked to a target position set within the screen.

[0035] FIG. 2 is a diagram for explaining the position of a subject tracked by the subject tracking function. FIG. 2 shows an example of an image in which the subject jumps from right to left, captured in a predetermined range of the screen by the subject tracking function, is continuously photographed. As described above, FIG. 2(a) is an example of an image in which the target position is always set near the center of the screen and the angle of view is made to track the subject. In the central image among these images, the subject is always photographed near the center of the screen, even though there is vertical movement of the subject, so that the composition is such that it appears that the background has moved, not the subject. On the other hand, FIG. 2(b) is an example of an image obtained when the target position is set near the center of the screen in the horizontal direction and the detected subject position in the vertical direction (i.e., tracking control is disabled). Comparing FIG. 2(a) and FIG. 2(b), FIG. 2(b) has a composition in which the movement of the subject jumping in the vertical direction can be understood when viewed as a continuous image. The purpose of this embodiment is to perform subject tracking to obtain the composition of FIG. 2(b).

[0036] FIG. 3 is a block diagram showing the functional configuration of the vibration reduction control of the imaging device in the first embodiment. The mode selection unit 300 selects a setting mode for setting a target position for subject tracking in response to the user's operation of the operation unit 118. The setting mode of the target position includes a manual mode in which the user sets the target position to be tracked, and an auto mode in which the camera detects a framing operation and automatically sets the target position. Examples of the manual mode include a "center" mode in which tracking is performed near the center of the screen in both the vertical and horizontal directions, a "vertical center" mode in which tracking is performed near the center in the vertical direction and tracking is not performed in the horizontal direction, and a "horizontal center" mode in which tracking is performed near the center in the horizontal direction and tracking is not performed in the vertical direction. In the scene shown in FIG. 2, the user can obtain a tracking effect as shown in FIG. 2(b) by selecting the "horizontal center" mode in advance.

[0037] On the other hand, in auto mode, the target position setting unit 301 automatically sets the target position based on the angle of view obtained from the zoom control unit 121, the shake angle obtained from the shake detection unit 119, and the subject position information obtained from the subject detection unit 127.

[0038] The angle of view of the camera can be calculated from the focal length of the zoom lens 102 controlled by the zoom control unit 121 and the size of the image sensor 106. If the focal length is f and the horizontal size of the image sensor 106 is h, then the angle of view φ is φ=2×atan(h / 2f). The angle of view in the vertical direction can also be calculated in a similar manner.

[0039] The camera's swing angle can be calculated by integrating the angular velocity detected by an angular velocity sensor such as a gyro sensor of the shake detection unit 119. FIG. 4 is a graph showing the time change of the angular velocity ω (dashed line) and the angle θ (solid line). This graph shows the case where the camera is panned or tilted by about 45 deg in the horizontal or vertical direction at a maximum of about 3.5 deg / sec for about 4 seconds. If the angular velocity at time n detected by the shake detection unit 119 is ωn and the sampling time is T, the angle θn by which the camera moves during the sampling time T is θn=ωn×T. Furthermore, when it is detected that the angular velocity ω exceeds a predetermined angular velocity and the camera is panned or tilted, the angle θn during that time is added, that is, the swing angle θ can be calculated by θ=Σθn.

[0040] The target position setting unit 301 compares the angle of view φ and the swing angle θ, and determines that the direction in which the swing angle θ exceeds the angle of view φ (which may be a value obtained by multiplying the angle of view φ by a predetermined coefficient) is the direction in which the camera is swinging to track a moving subject. In the direction in which the subject is being tracked, the target position is set near the center of the screen and tracking control is enabled. In the direction in which the swing angle θ is smaller than the angle of view φ, it is determined that the direction is not the direction in which the subject is being tracked, and the target position is set to the detected subject position and tracking control is disabled. In other words, in the direction in which the camera is swinging to track the subject, the tracking control is enabled to prevent the subject from going out of frame, while in the direction in which the subject is not being tracked, the movement of the subject can be retained in the composition. In FIG. 2(b), the target position is automatically set so that tracking of the subject is enabled only in the horizontal direction based on the detection results of the angle of view φ and the swing angle θ.

[0041] Next, the image stabilization control unit 302 will be described. The image stabilization control unit 302 is composed of an optical image stabilization control unit 123, an imaging image stabilization control unit 125, and an electronic image stabilization control unit 126. In the subject tracking control, feedback control is performed so that the deviation between the target position set by the target position setting unit 301 and the subject position detected by the subject detection unit 127 becomes zero. The control method using feedback control will be described later together with the description of S502 in FIG. 5. Each image stabilization control unit performs control based on either or both of the deviation for subject tracking control and the amount of shake detected by the shake detection unit 119 for image stabilization control. The image stabilization control unit 302 assigns the subject tracking control for correcting the subject position and the image stabilization control for correcting the shake to each of the image stabilization control units 123, 125, and 126 according to the features of each of the image stabilization control units 123, 125, and 126.

[0042] The advantages of optical image stabilization and imaging image stabilization are that image stabilization is performed optically by driving the image stabilization lens 104 and the image sensor 106, so image stabilization can be performed with little degradation in image resolution. With optical image stabilization, the movable range of the image stabilization lens 104 is wide on the telephoto side, so a high effect can be obtained with telephoto. On the other hand, with imaging image stabilization, although the driving range of the driving part of the image sensor 106 is limited on the telephoto side, a high effect can be obtained from wide angle to medium telephoto because the resolution is high on the wide-angle side.

[0043] On the other hand, a feature of electronic image stabilization is that the controllable range of electronic image stabilization is increased and the image stabilization effect can be improved by reducing the size of the output image to the display unit 109 and the recording unit 116. However, reducing the output image size reduces the image resolution.

[0044] In view of the above characteristics, as an example of control allocation in the optical image stabilization control unit 123, the imaging image stabilization control unit 125, and the electronic image stabilization control unit 126, a form in which image blur correction is performed by optical image stabilization and imaging image stabilization, and subject tracking is performed by electronic image stabilization is considered. Furthermore, a control allocation method in which the ratio of control by optical image stabilization is set high on the telephoto side and the ratio of control by imaging image stabilization is set high on the wide-angle side is also considered. In addition, a form in which the ratio of allocation is changed depending on still images and moving images is also acceptable. That is, the present invention is applicable to a configuration in which subject tracking control and image stabilization control are performed by any of the optical image stabilization control unit 123, the imaging image stabilization control unit 125, and the electronic image stabilization control unit 126. A control amount obtained by combining the optical and electronic correction amounts by the optical image stabilization control unit 123, the imaging image stabilization control unit 125, and the electronic image stabilization control unit 126 is output as a control amount of subject tracking and image stabilization for the final captured image or live view image.

[0045] Next, the processing of the subject tracking function will be described with reference to Fig. 5 to Fig. 7. Fig. 5 is a flowchart showing an example of the overall processing of the subject tracking function. The subject tracking function described below is assumed to be performed based on a command from the camera control unit 120 unless otherwise specified explicitly.

[0046] When an instruction to start the subject tracking function is given by operating the subject tracking function start button, menu, etc., the camera control unit 120 performs subject designation processing in S500. Here, the subject designation processing in S500 will be described with reference to FIG.

[0047] 6(a) is a flowchart showing an example of a process in which a user specifies a desired subject using a touch panel constituting the operation unit 118. In this example, it is assumed that a user performs an operation of touching an image of a subject displayed on the display unit 109 to specify the subject. In S600, the camera control unit 120 determines whether or not the touch panel has been pressed. If the touch panel has been pressed, the process proceeds to S601, and if the touch panel has not been pressed, the determination process of S600 is repeated.

[0048] In S601, camera control unit 120 acquires information on the position touched by the user (touch position). In the next S602, camera control unit 120 notifies subject detection unit 127 of the touch position, and subject detection unit 127 performs face detection near the touch position. If a face is detected near the touch position in S602, the main subject is determined to be a person, and the process proceeds to S603. If no face is detected near the touch position in S602, the main subject is determined to be an object other than a person, and the process proceeds to S604.

[0049] In S603, the camera control unit 120 performs control to store face information of a person who is the target of automatic tracking in the memory 117. Specific face information includes information such as the face size and the detected position of the face when the subject is specified, and the face direction. In addition, in a camera having a face recognition function, identification information such as an authentication ID is also stored in the memory 117.

[0050] On the other hand, in S604, the camera control unit 120 performs control to store the characteristic color near the touch position as color information of the automatic tracking target in the memory 117. Specific color information includes the characteristic color at the time of designating the subject, its brightness, color difference value, size of the same color area, center of gravity position of the same color area, etc. In addition, in a camera having an object authentication function, identification information such as an authentication ID is also stored in the memory 117. In the following description, face information and color information are collectively referred to as subject information (including subject size, subject detection position, etc.).

[0051] After the subject information is stored in memory 117 in S603 or S604, the process proceeds to S605, where the subject tracking frame is displayed. In S605, frame display unit 128 performs control to display a subject tracking frame of a size corresponding to the subject size, with the subject detection position as the center, on display unit 109. This ends the subject designation process.

[0052] In this manner, in system control unit 150, subject detection unit 127 detects a subject at a position specified by the user on display unit 109 or in the vicinity of the specified position. Then, frame display unit 128 displays a subject tracking frame on display unit 109. According to the processing example of Fig. 6(a), the user can easily specify the subject that he or she wishes to track in an intuitive manner.

[0053] However, the method of specifying the subject is not limited to this method, and another method will be described with reference to FIG. 6B is a flowchart showing a processing example in which the user specifies a subject using a subject specification switch as an operation member constituting the operation unit 118. First, in S606, the frame display unit 128 displays a frame serving as a guide for subject specification near the center of the screen of the display unit 109. The user adjusts the camera orientation so that the image of the subject to be tracked is placed near the center of the screen using this frame as a guide. Next, in S607, the camera control unit 120 judges whether or not the subject specification switch, which is one of the operation members of the operation unit 118, has been pressed. If the subject specification switch has been pressed, the process proceeds to S608, and if the switch has not been pressed, the judgment process of S607 is repeated in a waiting state.

[0054] In S608, subject detection unit 127 performs face detection near the center of the screen. If a face is detected near the center of the screen, the main subject is determined to be a person, and the process proceeds to S603. On the other hand, if a face is not detected near the center of the screen, the main subject is determined to be an object other than a person, and the process proceeds to S604. Note that the processes from S603 onwards are the same as those in Fig. 6(a), and therefore the same step numbers are used and explanations are omitted.

[0055] 6(b), subject detection unit 127 detects a subject at or near the center of the screen of display unit 109. Then frame display unit 128 displays a subject tracking frame indicating the position of the subject on display unit 109. With this method, a user can easily specify a subject even in a camera that is not equipped with an operating member such as a touch panel.

[0056] Regarding the subject designation process, in the case of a camera having both a touch panel and a subject designation switch constituting the operation unit 118, both the method of Fig. 6(a) and the method of Fig. 6(b) may be applied simultaneously. Also, the subject designation process to be applied may be changed depending on the flow of the process.

[0057] 5 ends, the process proceeds to S501, where the target position for tracking the object is set. Note that while the object tracking function is continuing, the processes from S501 to S504 described below are repeated. The cycle for executing the processes from S501 to S504 is set to a cycle synchronized with the frame rate of live view or video (e.g., 30 fps), for example, since the object tracking control is based on the object position detected from the image.

[0058] The target position setting process for subject tracking in S501 will now be described with reference to Fig. 7. In the target position setting process for subject tracking, the swing angle and subject position are determined for each of the vertical and horizontal directions, and the target position setting process for subject tracking is executed.

[0059] First, in S700, it is determined whether the setting mode of the target position is the auto mode. If it is the auto mode, the process proceeds to S702. On the other hand, if it is not the auto mode, i.e., if it is the manual mode, the process proceeds to S701 to determine whether it is the center mode or the horizontal center mode (i.e., whether at least the horizontal direction is the tracking direction), and if it is either, the process proceeds to S705. If it is neither, i.e., if it is the vertical center mode, the process proceeds to S706.

[0060] In S702, the angle of view is obtained from the zoom control unit 121, the swing angle is obtained from the shake detection unit 119, and the swing angle for the angle of view in the horizontal direction is determined. If the swing angle is equal to or smaller than the angle of view (equal to or smaller than the threshold), the process proceeds to S703, and if the swing angle is larger than the angle of view (larger than the threshold) (i.e., the horizontal direction is the tracking direction), the process proceeds to S705. In S705, the target position in the horizontal direction is set to the center position of the screen.

[0061] In S703, it is determined whether the subject position acquired from the subject detection unit 127 is within a predetermined range in the screen in the horizontal direction. If it is determined that the subject position is within the predetermined range, the process proceeds to S704, and the horizontal target position is set to the current horizontal subject position. This makes it possible to disable subject tracking in the horizontal direction in conjunction with the process of S502 described later.

[0062] On the other hand, if it is determined that the subject does not fall within the frame, the process proceeds to S705, where the horizontal target position is set to the center of the screen, thereby preventing the subject from going out of the frame. Note that a relatively wide range is set as the range used for the determination in S703, since the purpose here is to prevent the subject from going out of the frame. This range becomes the allowable range of horizontal movement, and for example, when tracking the subject in the vertical direction, it becomes possible to express the horizontal movement of the subject within the allowable range.

[0063] Next, in S706, it is determined again whether the setting mode of the target position is the auto mode, and if it is the auto mode, the process proceeds to S708. On the other hand, if it is not the auto mode, that is, if it is the manual mode, the process proceeds to S707 to determine whether it is the center mode or the vertical center mode (that is, whether at least the vertical direction is the tracking direction), and if it is either of these, the process proceeds to S711. If it is neither of these, that is, if it is the horizontal center mode, the process of setting the tracking target position is terminated.

[0064] In S708, a swing angle relative to the angle of view is determined for the vertical direction based on the angle of view acquired from the zoom control unit 121 and the swing angle acquired from the shake detection unit 119. If the swing angle is equal to or smaller than the angle of view (equal to or smaller than a threshold), the process proceeds to S709, and if the swing angle is larger than the angle of view (larger than a threshold) (i.e., the vertical direction is the tracking direction), the process proceeds to S711. In S711, the target position in the vertical direction is set to the center position of the screen.

[0065] In S709, it is determined whether the subject position acquired from the subject detection unit 127 is within a predetermined range in the screen in the vertical direction. If it is determined that the subject position is within the predetermined range, the process proceeds to S710, where the target position in the vertical direction is set to the current subject position in the horizontal direction. This makes it possible to disable subject tracking in the vertical direction in conjunction with the process of S502 described later.

[0066] On the other hand, if it is determined that the subject does not fall within the frame, the process proceeds to S711, where the vertical target position is set to the center of the screen to prevent the subject from going out of the frame. Note that the range used for the determination in S709 is set to a relatively wide range, as in S703, in order to prevent the subject from going out of the frame. This range becomes the allowable range for vertical movement, and for example, when tracking the subject in the horizontal direction, it becomes possible to express the vertical movement of the subject within the allowable range.

[0067] In S702 and S708, the swing angle is compared with the angle of view, but the present invention is not limited to this. For example, the swing angle may be compared with a threshold value determined based on the angle of view, such as 50% of the angle of view, or may be compared with a predetermined fixed value instead of the angle of view.

[0068] When the tracking target position setting process of S501 ends, the process proceeds to S502, where subject tracking control is executed. In the subject tracking control of S502, the positions of the vibration-proof lens 104 and the image sensor 106, which are to be controlled, and the recording range of the image memory 108 are controlled by a control unit of the vibration-proof control unit 302 that is assigned to the subject tracking control. Then, feedback control is performed so that the deviation between the target position set in S501 and the subject position detected by the subject detection unit 127 becomes zero. As the feedback control method, a control method such as PID control can be applied. The control amount u of the PID control is calculated by the following formula (1).

[0069] TIFF0007625401000001.tif19121u:Control amount e:Deviation Kp: Proportional gain Ki: Integral gain Kd: Differential gain

[0070] The control amount of PID control is calculated by adding the proportional element, integral element, and differential element for the deviation between the target position and the subject position. The proportional element is a control amount proportional to the deviation, and as the subject position approaches the target position, the control amount gradually decreases, allowing for smooth subject tracking. The integral element can correct the steady-state deviation by integrating the deviation, as an error (steady-state deviation) between the target position and the subject position remains with only the proportional element. The differential element is the differential value (difference value) of the subject position between successive image frames, and has the effect of predicting changes in the subject position on the screen. Therefore, the differential element can improve the responsiveness of subject tracking.

[0071] The control gains Kp, Ki, and Kd are tuned to be appropriate values ​​taking into consideration the speed at which the subject is tracked to the center of the screen. Generally, increasing the gain can improve the responsiveness of tracking, but if the subject position on the screen moves too fast, framing may become difficult. In addition, if the gain is too high, the control may oscillate, that is, a hunting phenomenon in which the subject moves back and forth near the center of the screen may occur. Therefore, it is necessary to set parameters that take into account the responsiveness of subject tracking and the oscillation margin of control by measuring the step response and frequency response of the control. PID control is a widely applied control, and can realize control with good responsiveness by relatively simple calculation processing. In this embodiment, the control method of PID control has been described, but the present invention is not limited to this control method. If there is no need to increase the tracking accuracy and responsiveness of the subject position, only P control or PI control may be used, or a form that uses various other filters may be used.

[0072] Once subject tracking control is executed in S502, the process proceeds to S503. In S503, it is determined whether or not the subject detected by subject detection unit 127 can continue to be detected. If the subject has been lost, the process returns to S500 and the subject designation process is executed again. If the subject has not been lost and can be detected, the process proceeds to S504. In S504, it is determined whether or not the subject tracking function has been terminated. If it is determined in S504 that the subject tracking function has been terminated by operating the subject tracking function end button, menu, or the like, the subject tracking function is terminated.

[0073] As described above, according to the first embodiment, the target position setting unit 301 sets a target position for subject tracking in the vibration isolation control unit 302 based on the shake angle, angle of view, and subject position detected for each of the vertical and horizontal directions by the shake detection unit 119, the zoom control unit 121, and the subject detection unit 127. As a result, according to this embodiment, even when a subject is being tracked, it is possible to shoot with a composition that allows the movement of the subject to be understood, and it is possible to prevent the moving subject from going out of the frame within the screen.

[0074] Second embodiment Next, a second embodiment of the present invention will be described. Fig. 8 is a block diagram showing the functional configuration of an imaging device for performing processing in the second embodiment. Note that Fig. 8 differs from Fig. 3 described in the first embodiment in that a control gain setting unit 800 is provided instead of the target position setting unit 301. Since the other configurations are similar to those shown in Fig. 3, the same reference numbers are used and the description will be omitted.

[0075] In the first embodiment, a form was described in which the target position is set to the center of the screen when the effect of tracking control is enabled for each of the horizontal and vertical directions, and the target position is set to the subject position when the effect of tracking control is disabled. In contrast, in the second embodiment, the target position is always set to the center of the screen, the process of S501 in Fig. 5 is not performed, and the effect of tracking control is changed by adjusting the control gain for calculating the control amount u shown in formula (1) in S502.

[0076] 8 sets a control gain based on the angle of view acquired from the zoom control unit 121, the swing angle acquired from the shake detection unit 119, and information on the subject position acquired from the subject detection unit 127. The angle of view and swing angle of the camera are calculated by the same process as in the first embodiment.

[0077] Also, whether or not to enable tracking control is determined by a tracking control determination process shown in Fig. 9. Note that since the process is the same as that shown in Fig. 7 except for S904, S905, S910, and S911, the same step numbers are used and the description is omitted. In the process shown in Fig. 9, the horizontal target position is set in S704 and S705 in Fig. 7, whereas horizontal tracking is disabled in S904 and horizontal tracking is enabled in S905. Similarly, vertical tracking is disabled in S910 and vertical tracking is enabled in S911.

[0078] In this way, when it is determined that tracking control should be enabled based on the camera swing angle and subject position determination, the control gain setting unit 800 sets a control gain that enables subject tracking that has been tuned in advance for the vibration reduction control unit 302. When it is determined that tracking control should be disabled, the control gain setting unit 800 sets the control gain of the vibration reduction control unit 302 to a low gain.

[0079] In formula (1), decreasing the control gains Kp, Ki, and Kd decreases the control amount u, and the effect of the subject tracking control can be suppressed. In other words, lowering the control gain makes it possible to finely adjust the tracking performance, such as slowing down the tracking speed to the center of the screen set as the target position, or intentionally leaving a steady-state deviation. Furthermore, setting the control gains Kp, Ki, and Kd to 0 makes the control amount u 0, and it is also possible to completely disable the tracking control.

[0080] As described above, in the second embodiment, the target position for subject tracking is set to the center of the screen, and the control gain setting unit 800 can change the control gain for subject tracking based on the swing angle, the angle of view, and the subject position, thereby fine-tuning the tracking speed, etc. This makes it possible to shoot with a composition that allows the movement of the subject to be seen even when the subject is being tracked, and also makes it possible to prevent the moving subject from going out of the frame within the screen.

[0081] (Third embodiment) Next, a third embodiment of the present invention will be described. In the first embodiment, an example was described in which the target position of subject tracking is set to the center position of the screen or the subject position in the horizontal and vertical directions to enable / disable tracking. In contrast, in the third embodiment, a form is described in which the swing direction in an image plane perpendicular to the optical axis is calculated from the ratio of the swing angle in the horizontal direction to the swing angle in the vertical direction, and the target position is set according to the swing direction. Note that the imaging device in the third embodiment can use one having the functional configuration described in the first embodiment with reference to FIG. 3, so the description is omitted here.

[0082] Fig. 10 is a graph showing the angular velocity ω detected by the angular velocity sensor when tracking a subject while swinging the camera, the angle θ obtained by integrating the angular velocity ω, and the swing direction α within the image plane. Fig. 10(a) is a graph showing the angular velocity ωpan and angle θpan, which are the detection results in the horizontal direction of the angular velocity sensor. Similarly, Fig. 10(b) is a graph showing the angular velocity ωtilt and angle θtilt, which are the detection results in the vertical direction. In both graphs, the horizontal axis represents time t, and the vertical axis represents the angular velocity or angle, with the angular velocity ω indicated by a dashed line and the angle θ indicated by a solid line.

[0083] FIG. 10(c) is a graph in which the horizontal axis is the horizontal angle θpan shown in FIG. 10(a) and the vertical axis is the vertical angle θtilt shown in FIG. 10(b). The swing direction α in the image plane can be calculated by α=atan(θtilt / θpan) based on the ratio of the horizontal angle θpan to the vertical angle θtilt. This graph shows a graph when the camera is swung horizontally to the left at a maximum of about 3.5 deg / sec by about 45 deg and vertically upward at a maximum of about 2.0 deg / sec by about 25 deg in about 4 seconds. In other words, the swing direction α is calculated from α=atan(25 deg / -45 deg) to mean that the camera is swung in a direction of about 150 deg with the horizontal right direction as the reference.

[0084] Next, a method for setting a target position for subject tracking in this embodiment will be described with reference to FIG. In this embodiment, the target position is set so that subject tracking is enabled in a direction parallel to the camera swing direction α detected by the shake detection unit 119, and subject tracking is disabled in a direction perpendicular to the swing direction α. ​​By setting the target position based on the camera swing direction, the method of the present invention can also be applied to cases where a subject moving in a diagonal direction is tracked and photographed.

[0085] 11, the subject position S and the target position T are shown with xy representing an image plane perpendicular to the optical axis direction. The subject position S is a position detected by the subject detection unit 127, and the coordinates of the subject position S are S=(xs, ys). The target position T is a position calculated by the target position setting unit 301 and set in the vibration prevention control unit 302, and the coordinates of the target position T are T=(xt, yt). The target position T can be calculated from the swing direction α and the subject position S by the following formula (2).

[0086] TIFF0007625401000002.tif42135T:Target position S:Subject position α: Swing direction R: Rotation matrix R -1 : Rotation inverse matrix

[0087] R is a rotation matrix that rotates the coordinates by the swing direction α, and R -1 is the inverse rotation matrix. The target position T can be calculated by rotating the subject position S in the sweep direction α and then translating it in a direction parallel to the sweep direction α.

[0088] First, the target position setting unit 301 calculates the subject position S'=(xs', ys') on the image plane x'-y' rotated by the swing direction α. ​​Since the coordinates of the subject position S are coordinates on the image plane xy, the rotated subject position S' is calculated by applying the rotation inverse matrix R -1 Then, the target position setting unit 301 calculates a target position T' from the subject position S' so that tracking is enabled in the x' direction, which is parallel to the swing direction α, and tracking is disabled in the y' direction, in the image plane x'-y'. That is, the coordinates of the target position T' in the image plane x'-y' can be expressed as T'=(0, ys'). Finally, in order to return the target position T' in the image plane x'-y' to the coordinates in the image plane xy, the target position T' is multiplied by a rotation matrix R to calculate a target position T that has been rotated and transformed by α.

[0089] As described above, according to the third embodiment, the target position setting unit 301 calculates the swing direction on the image plane perpendicular to the optical axis from the swing angle detected by the shake detection unit 119, and sets the target position for subject tracking in the vibration compensation control unit 302 based on the swing direction. This makes it possible to shoot a subject that moves diagonally within the screen with a composition that allows the subject's movement to be understood, and also makes it possible to prevent the subject from going out of the frame within the screen.

[0090] (Fourth embodiment) Next, a fourth embodiment of the present invention will be described. Fig. 12 is a flowchart for explaining a target position setting process for subject tracking in the fourth embodiment. In Fig. 12, the same processes as those shown in Fig. 7 are given the same step numbers and will not be described.

[0091] In the first embodiment, a manual mode in which the user sets the target position to be tracked and an auto mode in which the camera detects a framing operation and automatically sets the target position are selected by the user. In contrast, in the fourth embodiment, the photographing scene is determined based on subject information acquired from the subject detection unit 127 and the like, and the tracking direction is selected.

[0092] 7, the horizontal setting mode and horizontal swing angle are determined, whereas in S700 to S702, a determination is made as to whether the scene being shot is a scene in which a subject moving in the horizontal direction is to be shot or not. For example, if it is determined that the scene being shot is one in which a subject moves horizontally while performing, such as in figure skating or gymnastics, tracking is enabled in the horizontal direction and disabled in the vertical direction.

[0093] 7, the vertical setting mode and vertical swing angle are determined, whereas in S706 to S708, it is determined whether the scene being photographed is a scene in which a subject moving vertically is to be photographed or not. For example, if it is determined that the scene is a competition scene in which the subject moves left and right while aiming vertically toward the goal, such as bouldering, tracking is enabled in the vertical direction and tracking is disabled in the horizontal direction.

[0094] The shooting scene can be determined by obtaining information such as subject information detected by the subject detection unit 127, the subject motion vector detection results, the shooting location, images previously taken at the same location, and the shooting time from the camera and the cloud, and estimating the scene using known scene recognition technology.

[0095] As described above, in the fourth embodiment, the photographing scene is determined and the direction in which tracking is enabled can be automatically selected. This allows the user to track and photograph a subject with a composition suited to the photographing scene without having to select a setting mode for the target position.

[0096] Fifth embodiment Next, a fifth embodiment of the present invention will be described. In the first embodiment, a form in which a target position for subject tracking is set during shooting has been described. In contrast, in the fifth embodiment, a form in which a tracking image is generated by a trimming process after shooting will be described. By trimming a specific range from continuous shot images or videos shot with a wide angle of view, it is possible to generate an image in which the subject is enlarged while retaining the movement of the subject.

[0097] In this embodiment, the target position setting process for subject tracking described in the first embodiment with reference to FIG. 7 is applied to the process for setting the center position of trimming.

[0098] The camera control unit 120 stores the shake angle obtained from the shake detection unit 119 during shooting and the time when the angle was obtained in the memory 117. The image data recorded in the recording unit 116 is transferred to the image memory 108, and the subject position on the transferred image is detected by the subject detection unit 127. Using this information, the target position of tracking is calculated by processing similar to the setting processing in FIG.

[0099] The camera control unit 120 generates a trimmed image centered on the calculated target position, and records the image again in the recording unit 116. The swing angle may be stored in advance in the memory 117 during shooting, or a method may be applied in which a motion vector is calculated from a video or continuous shot images, and the swing angle is estimated based on the motion vector information. The target position setting process for subject tracking may be the process shown in FIG. 9 or FIG. 12.

[0100] As described above, in the fifth embodiment, by applying the target setting process for subject tracking to the trimming process during playback, it is possible to generate an image with the same effect as when shooting. This makes it possible to generate an image in which the subject is enlarged from a captured video or continuous shot images, and also to automatically generate an image with a composition that shows the movement of the subject.

[0101] (Other embodiments) The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0102] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0103] 100: digital camera, 104: anti-vibration lens, 106: image sensor, 108: image memory, 110: zoom control unit, 112: anti-vibration lens drive unit, 118: operation unit, 119: shake detection unit, 120: camera control unit, 123: optical anti-vibration control unit, 125: imaging anti-vibration control unit, 126: electronic anti-vibration control unit, 127: subject detection unit, 128: frame display unit, 300: mode selection unit, 301: target position setting unit, 302: anti-vibration control unit, 800: control gain setting unit

Claims

1. a detection means for detecting a subject in an image obtained by photoelectrically converting light incident via a photographing optical system using an image sensor; A tracking means for tracking a subject in a plurality of predetermined directions; a vibration detection unit that detects a vibration amount in each of the plurality of directions; a determination means for determining a tracking direction in which the subject is tracked among the plurality of directions; A control means for controlling the tracking means so as to perform tracking in the tracking direction and suppress tracking in a direction different from the tracking direction, The determination means is determining a shake angle based on the shake amount for each of the plurality of directions; The tracking direction is determined based on the shaking angles in the plurality of directions. A tracking device characterized by:

2. The tracking device further includes an operation unit for designating at least one of the plurality of directions as the tracking direction, The tracking device according to claim 1 , wherein when a tracking direction is specified by the operation means, the determination means determines the specified tracking direction as the tracking direction for tracking the subject, regardless of the shake angle.

3. 3. The tracking device according to claim 2, wherein the plurality of directions are a first direction and a second direction perpendicular to the first direction.

4. The determination means: When the shaking angle is greater than a predetermined threshold value for each of the plurality of directions, the corresponding direction is detected as the tracking direction.

4. The tracking device according to claim 1, wherein the first and second sensors are arranged in a first direction.

5. The determination means is determining an angle of view of the image based on a focal length of the photographing optical system; Setting the threshold value based on the angle of view 5. The tracking device according to claim 4.

6. The tracking device according to claim 4 or 5, characterized in that the control means further enables tracking in a corresponding direction when the shake angle is equal to or less than the threshold value for each of the plurality of directions and the position of the subject in the image exceeds a predetermined range for each of the plurality of directions.

7. The determination means: The tracking direction is detected based on a ratio of the shaking angles in the plurality of directions.

4. The tracking device according to claim 1, wherein the first and second sensors are arranged in a first direction.

8. The control means is configured to: A target position of the subject in the tracking direction is set to the center of the image in the tracking direction; A target position of the subject in a direction other than the tracking direction is set as a position of the subject in the image.

8. The tracking device according to claim 1, wherein the tracking means is controlled by:

9. The tracking device according to any one of claims 1 to 7, characterized in that the control means controls the tracking means by setting a target position of the subject to the center of the image and increasing a control value for moving the subject to the target position by the tracking means in the tracking direction and decreasing a control value in a direction other than the tracking direction.

10. The tracking means includes: a means for moving an anti-vibration lens included in the photographing optical system on a plane perpendicular to an optical axis of the photographing optical system; a means for moving the image sensor on a plane perpendicular to the optical axis of the photographing optical system; 10. The tracking device according to claim 1, further comprising at least one of the following:

11. Further comprising a storage means for storing the image, 11. The tracking device according to claim 1, wherein the tracking means reads out a part of the image from the storage means, and moves the position of the subject in the read image.

12. The camera further includes a scene detection unit for detecting a photographed scene, The tracking device according to claim 1 , wherein, when a photographic scene is detected by the scene detection means, the determination means determines the tracking direction in accordance with the detected photographic scene, regardless of the shake angle.

13. the tracking means tracks the subject by image processing that outputs an enlarged image of a part of the recorded image; 2. The tracking device according to claim 1, wherein the control means controls the tracking means by controlling a range of an image to be enlarged.

14. an image sensor that photoelectrically converts light incident thereon via a photographing optical system to output an image; A tracking device according to any one of claims 1 to 13, An imaging device comprising:

15. 15. The imaging apparatus according to claim 14, further comprising the imaging optical system.

16. A method for controlling a tracking device having a tracking means for tracking a subject in a plurality of predetermined directions, comprising: a detection step in which a detection means detects a subject in an image obtained by photoelectrically converting light incident via a photographing optical system using an image sensor; a vibration detection unit that detects a vibration amount in each of the plurality of directions; a determination step of determining a tracking direction in which the subject is tracked among the plurality of directions; A control step in which a control means performs tracking in the tracking direction and controls the tracking means so as to suppress tracking in a direction different from the tracking direction, In the determination step, determining a shake angle based on the shake amount for each of the plurality of directions; The tracking direction is determined based on the shaking angles in the plurality of directions. A control method comprising:

17. A program for causing a computer to execute each step of the control method according to claim 16.

18. A computer-readable storage medium storing the program according to claim 17.

Images