Subject tracking device, method for controlling the same, program, and recording medium

Abstract

To provide a subject tracking device capable of reducing small image shake caused by the accuracy of detecting the position of a subject in performing subject tracking.SOLUTION: The subject tracking device includes: a subject detection unit for detecting a subject from a captured image by a plurality of detection techniques; a tracking unit for changing a capturing range on the basis of a target position in the captured image and tracking the subject; and a control unit for controlling the degree of tracking by the tracking unit according to a detection technique which the subject detection unit uses to detect the subject, the detection result being used by the tracking unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for tracking a subject in an imaging device. [Background technology]

[0002] In recent years, many imaging devices equipped with an image stabilization function have been developed, and these imaging devices are capable of stabilizing image shake in moving images. Image stabilization functions include optical image stabilization and electronic image stabilization. The former is a method of correcting image shake by driving part of the optical system or the image sensor to cancel out camera shake in response to a camera shake signal detected by a camera shake detection sensor. The latter is a method of correcting image shake by shifting the area from which the image is cut out using image processing in response to the camera shake signal.

[0003] However, when recording moving images using an imaging device equipped with the above-mentioned image stabilization function, the subject may still go out of frame even if image blur caused by camera shake is corrected. This is because, if the subject is moving rapidly, the correction of image blur caused by camera shake is unable to track the subject's movement. Therefore, to prevent a moving subject from going out of frame, the photographer must constantly focus on panning and tilting to change the shooting direction to match the subject's movement.

[0004] To address the above-mentioned issues, a subject tracking function is known that continuously tracks a subject using a mechanism typically used for image stabilization or electronic image stabilization. The subject tracking function is realized by moving the shooting range using the image stabilization mechanism or the like so that the subject remains positioned near a predetermined position within the image frame. This technology allows the photographer to capture a stable subject image without constantly concentrating on panning or tilting the camera to track the subject.

[0005] Generally, to stably control the shooting range so that the subject remains positioned near a predetermined position within the screen, a mechanism is required to stably detect the subject position and stably shift the shooting range. However, depending on the shooting conditions, the moving part that changes the shooting range may oscillate.

[0006] To address this issue, Patent Document 1 proposes that a control device that tracks a subject using a movable means that shifts the subject is provided with a detection means that detects the shooting conditions of the captured image, and changes the degree of tracking of the subject depending on the detected shooting conditions.

[0007] Patent Document 1 exemplifies the following shooting conditions: focal length, shooting magnification, subject distance, electronic zoom magnification, frame rate, amount of shake, and reliability indicating the likelihood of the subject being a subject. Specifically, the following control method is described: The degree of tracking is controlled to decrease as the focal length, shooting magnification, and electronic zoom magnification increase; The degree of tracking is controlled to decrease as the subject distance decreases; The degree of tracking is controlled to decrease as the frame rate decreases; and The degree of tracking is controlled to decrease as the subject reliability decreases. With this configuration, it is claimed that a good subject tracking device can be provided that suits the shooting conditions while suppressing oscillation of the movable means that shifts the subject. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-208252 Summary of the Invention [Problem to be solved by the invention]

[0009] However, in Patent Document 1, the accuracy of detecting the subject position in subject detection is not taken into consideration.

[0010] The subject detection functions installed in recent tracking devices are capable of detecting subjects using multiple methods. Examples of subject detection methods include methods that use image recognition to detect people, animals, and facial areas, methods that detect organs (parts) in the face such as the eyes, nose, and mouth, and methods that detect the subject's entire body (entire object). There are also known detection methods that do not use image recognition, but identify the subject based on the degree of match between a temporarily stored template image and histograms, color data, etc. It is known that the accuracy of detecting the subject's position varies depending on these detection methods.

[0011] When tracking a subject, if the detection accuracy of the subject position is low, i.e., the subject position contains a lot of noise components, and tracking control is performed at the same level as when the detection accuracy of the subject position is high, the noise components may cause the moving image to shake slightly, which is not desirable as an image.

[0012] Furthermore, if the subject detection method is changed during subject tracking, the detected subject position may fluctuate depending on the subject detection method, which may result in a momentary shake in the video when the subject detection method changes, resulting in a decrease in quality of the video.

[0013] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide a subject tracking device that can reduce small image shakes caused by the detection accuracy of the subject position when tracking a subject. [Means for solving the problem]

[0014] The subject tracking device according to the present invention is characterized by comprising: subject detection means capable of detecting a subject from a captured image using a plurality of detection methods; tracking means that changes the shooting range based on a target position within the captured image to track the subject; and control means that uses the detection results of the tracking means to control the degree of tracking at which the tracking means tracks the subject depending on the detection method used by the subject detection means to detect the subject. [Effects of the Invention]

[0015] According to the present invention, when tracking a subject, it is possible to reduce small fluctuations in an image caused by the accuracy of detecting the position of the subject. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram showing an example of the arrangement of an imaging apparatus according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of parts related to image stabilization control and subject tracking control. [Figure 3A] 10 is a flowchart showing an operation for calculating a subject tracking amount. [Figure 3B] 10 is a flowchart showing an operation for manipulating a cutoff frequency. [Figure 4] 10A to 10C are diagrams showing the time-series positions of a subject based on different subject detection methods. [Figure 5] 10A and 10B are diagrams showing the relationship between the subject detection method, the cutoff frequency, and the degree of tracking. [Figure 6] 10A and 10B are diagrams showing changes in cutoff frequency when the subject detection method is changed. [Figure 7A] 10 is a flowchart showing an operation for calculating a subject tracking amount in the second embodiment. [Figure 7B] 10 is a flowchart showing an adjustment operation of a subject position. [Figure 8] 10A to 10C are diagrams showing the time-series positions of a subject based on different subject detection methods. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0018] (First embodiment) In this embodiment, a method for reducing minute fluctuations in moving images caused by noise components at the subject position by changing the degree of tracking depending on the subject detection method will be described.

[0019] FIG. 1 is a block diagram showing the configuration of an image capturing apparatus 100 which is a first embodiment of a subject tracking device according to the present invention.

[0020] In FIG. 1, an imaging device 100 is configured with a camera body 1 and a photographing lens 2. A zoom lens 101 moves in the direction of the optical axis to optically change the focal length of a photographing optical system 200 that forms a subject image, thereby changing the photographing angle of view. An image stabilization lens 102 moves in a direction perpendicular to the optical axis, optically correcting image blur caused by shaking of the imaging device 100. A focus lens 103 moves in the direction of the optical axis to optically adjust the focus position. An aperture 104 and a shutter 105 can adjust the amount of light by opening and closing them, and are used for exposure control.

[0021] Light that has passed through the imaging optical system 200 is received by an image sensor 106 that uses a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor) sensor, and is converted from an optical signal into an electrical signal.

[0022] The AD converter 107 performs noise removal processing, gain adjustment processing, and AD conversion processing on the image signal read out from the image sensor 106 .

[0023] The timing generator 108 controls the drive timing of the image sensor 106 and the output timing of the AD converter 107 in accordance with instructions from the camera control unit 115 .

[0024] The image processing circuit 109 performs pixel interpolation processing, color conversion processing, etc. on the output from the AD converter 107, and then sends the processed image data to an internal memory 110. The image processing circuit 109 includes a circuit for aligning multiple images captured in succession, a geometric transformation circuit for performing cylindrical coordinate transformation and lens distortion correction, a synthesis circuit for performing trimming and synthesis processing, etc.

[0025] The display unit 111 displays the image data stored in the internal memory 110 as well as shooting information and the like.

[0026] The compression / decompression processing unit 112 performs compression or decompression processing according to the image format on the data stored in the internal memory 110. The storage memory 113 stores various data such as parameters.

[0027] The operation unit 114 is a user interface that allows the user to perform various menu operations and mode switching operations.

[0028] The camera control unit 115 is composed of a computing device such as a CPU (Central Processing Unit), and executes various control programs stored in the internal memory 110 in response to user operations via the operation unit 114. The control programs are programs for performing, for example, zoom control, image stabilization control, automatic exposure control, automatic focus adjustment control, and subject tracking processing. Note that each block shown in the camera control unit 115 is realized by the camera control unit 115 executing the control program stored in the internal memory 110. However, these blocks may be configured as circuits independent of the camera control unit 115.

[0029] In the case of a lens-interchangeable camera, information is transmitted between the camera body 1 and the photographic lens 2 via the camera side communication unit 140 and the lens side communication unit 128 .

[0030] The aperture driver 120 drives the aperture 104 , and the shutter driver 135 drives the shutter 105 .

[0031] A luminance signal detection unit 137 detects the luminance of the subject and scene from the signal read out from the image sensor 106 and passed through the AD converter 107. An exposure control unit 136 calculates the exposure value (aperture value and shutter speed) based on the luminance information obtained by the luminance signal detection unit 137, and notifies the diaphragm driving unit 120 and the shutter driving unit 135 of the calculation result. The exposure control unit 136 also simultaneously controls the amplification of the image signal read out from the image sensor 106, thereby performing automatic exposure control (AE control).

[0032] A zoom lens driving unit 124 drives the zoom lens 101 to change the angle of view. A zoom lens control unit 127 controls the position of the zoom lens 101 in accordance with a zoom operation instruction from the operation unit 114.

[0033] Focus lens driver 121 drives focus lens 103. Evaluation value calculator 138 extracts specific frequency components from the luminance information obtained by luminance signal detector 137, and then calculates a contrast evaluation value based on the extracted frequency components. Focus lens controller 139 issues a command to drive focus lens 103 by a predetermined drive amount over a predetermined range. At the same time, it acquires evaluation values, which are the calculation results of evaluation value calculator 138 at each focus lens position. As a result, it calculates a defocus amount in the contrast AF method from the focus lens position where the change curve of the contrast evaluation value reaches its peak, and notifies the focus lens driver 121. Focus lens driver 121 drives focus lens 103 according to the defocus amount, thereby performing automatic focusing control (AF control) to focus a light beam on the surface of image sensor 106.

[0034] Although the contrast AF method has been described here, a phase difference AF method may also be used, and the details of the phase difference AF method are well known, so a description thereof will be omitted.

[0035] The shake detection units (134, 125) detect shake and vibration applied to the imaging device. In this embodiment, in addition to the camera-side shake detection unit 134 arranged on the camera side, a lens-side shake detection unit 125 is also arranged on the lens side to detect shake and vibration applied to the lens.

[0036] An image stabilization lens position detection unit 123 detects the position of the image stabilization lens 102. An image stabilization lens anti-shake control unit 126 calculates an image stabilization amount for suppressing shake in response to a shake detection signal detected by the lens-side shake detection unit 125, the camera-side shake detection unit 134, or both, and notifies the image stabilization lens driving unit 122 of the amount. The image stabilization lens driving unit 122 controls the driving of the image stabilization lens 102 in a direction perpendicular to the optical axis.

[0037] The camera-side image stabilization control unit 133 can communicate with the image stabilization lens stabilization control unit 126 via the camera communication unit 140 and the lens communication unit 128 in the photographing optical system 200. The camera-side image stabilization control unit 133 calculates an image sensor shake correction amount for suppressing shake using the image sensor 106 based on shake detection signals detected by the camera-side shake detection unit 134, the lens-side shake detection unit 125, or both. Then, based on the calculated correction amount and the position of the image sensor 106 detected by the image sensor position detection unit 132, it transmits a drive signal for the image sensor 106 to the image sensor drive unit 130, thereby controlling stabilization by the image sensor 106. Based on the image sensor drive signal received from the image stabilization control unit 133, the image sensor drive unit 130 drives the image sensor 106 in a direction perpendicular to the optical axis.

[0038] The motion vector detection unit 131 uses a block matching method to calculate the correlation value between the current frame and the previous frame for each block obtained by dividing the frame, then searches for the block in the previous frame that minimizes the calculation result, and detects the deviation of other blocks relative to that block as a motion vector.

[0039] Subject information acquisition unit 141 is a block that acquires information necessary for tracking amount calculation from image processing unit 109. Subject tracking amount calculation unit 142 is a block that calculates the subject tracking amount. Note that subject tracking amount calculation unit 142 may acquire information directly from image processing unit 109 without going through subject information acquisition unit 141. Subject information acquisition unit 141 and subject tracking amount calculation unit 142 will be described later with reference to FIG. 2.

[0040] FIG. 2 is a block diagram showing the configuration of parts related to image blur correction control and subject tracking control in the first embodiment.

[0041] First, the image stabilization control unit 133 on the camera side will be described.

[0042] The shake angular velocity detected by the camera shake detection unit 134 is converted into a shake angle by being integrated by the camera integration unit 1331. Here, it is assumed that an integral low-pass filter (hereinafter referred to as an integral LPF) is used for the camera integration unit 1331.

[0043] The shake correction amount calculation unit 1332 calculates the correction amount for canceling the shake angle, taking into consideration the frequency band of the shake angle and the drivable range of the image sensor 106. Specifically, the shake correction amount is calculated by integrating the gain related to the zoom magnification and the subject distance with respect to the shake angle.

[0044] The ratio calculation unit 1333 calculates the correction ratio to be performed by the camera, assuming that the sum of the shake correction amounts on the camera side and the lens side is 100%. In this embodiment, the correction ratio is determined based on the movable ranges of the image sensor 106 and the image stabilization lens 102. In addition to the movable range of the shake correction member, the correction ratio may also be determined taking into account the movable range for correction by clipping in image processing (electronic image stabilization). The correction ratio calculation unit 1334 multiplies the shake correction amount by the result of calculation by the ratio calculation unit 1333, and calculates the correction amount based on the correction ratio.

[0045] The position control unit 1335 performs PID control (ratio control, integral control, differential control) on the deviation between the target position and the current position of the image sensor 106 in terms of the correction amount, converts it into an image sensor drive signal, and inputs it to the image sensor drive unit 130. The current position is the output result of the image sensor position detection unit 132. Since PID control is a common technique, a detailed description will be omitted. The image sensor drive unit 130 drives the image sensor 106 in accordance with the image stabilization drive signal.

[0046] Next, the image stabilization control unit 126 on the lens side will be described.

[0047] The shake angular velocity detected by the lens-side shake detection unit 125 is converted into a shake angle by integration processing by the lens integration unit 1261. Here, an integral LPF is used for the lens integration unit 1261. The shake correction amount calculation unit 1262 calculates the correction amount for canceling the shake angle, taking into account the frequency band of the shake angle and the drivable range of the image stabilization lens 102. Specifically, the shake correction amount is calculated by integrating the shake angle with gains related to the zoom magnification and subject distance.

[0048] The correction ratio accumulator 1263 calculates the amount of correction based on the correction ratio by multiplying the total of the shake correction amounts on the camera side and the lens side by the correction ratio provided by the lens side, assuming that the total is 100%. In this embodiment, the correction ratio provided by the lens side is calculated from the calculation result by the camera side ratio calculator 1333. The correction ratio provided by the camera side or the lens side is notified via the camera side communication unit 140 and the lens side communication unit 128.

[0049] The position control unit 1264 performs PID control (ratio control, integral control, differential control) on the deviation between the target position and the current position of the correction amount of the image stabilization lens 102, converts it into a drive signal for the image stabilization lens 102, and inputs it to the image stabilization lens drive unit 122. The current position is the output result of the image stabilization lens position detection unit 123. Since PID control is a commonly used technique, a detailed description will be omitted. The image stabilization lens drive unit 122 drives the image stabilization lens 102 in accordance with the image stabilization drive signal.

[0050] By driving the image stabilization lens 102 and the image sensor 106 in the above manner, image blur caused by camera shake can be reduced.

[0051] Next, the image processing circuit 109 will be further described with reference to FIG.

[0052] In this embodiment, the image processing circuit 109 includes a subject detection unit 1091. The subject detection unit 1091 is a block that detects an image area of ​​a subject included in a captured image based on an image signal output from the image sensor 106, and generates subject detection information. The subject detection unit 1091 can detect a subject from a captured image using a plurality of detection methods.

[0053] More specifically, the subject detection unit 1091 has a face detection function that detects people, animals, and faces by image recognition, a function that detects organs (parts) in a face such as the pupils, nose, and mouth, a whole body detection function that detects the entire body of the subject (whole object), etc. Then, the positions of the face, organs, and whole body are calculated from the results of the face detection, organ detection, and whole body detection.

[0054] The subject detection unit 1091 detects faces and whole bodies by storing the contour shapes of the faces and whole bodies internally as feature data, and then performs pattern matching processing to identify image areas that match the feature data for each image to be detected.

[0055] In face detection, a pattern matching process is performed to identify image areas that match feature data representing the shape of a face stored in advance from the area obtained by whole-body detection. The degree of match with the stored feature data is calculated, and areas with a degree of match equal to or greater than a predetermined value are identified as face or whole-body areas.

[0056] Furthermore, in order to increase the number of opportunities to detect the face or the entire body and improve detection accuracy, pattern matching processing is performed using multiple pieces of feature data stored inside the image processing circuit 109. Note that pattern matching processing may also be performed using feature data for only a portion of the shape of the face or the entire body. Furthermore, pattern matching processing may also be performed by changing the size of the feature data to detect the face or the entire body, regardless of its size.

[0057] In organ detection, a pattern matching process is performed to identify image areas that match feature data representing the shapes of organs that have been stored internally in advance, from areas determined by face detection.

[0058] As another detection method, detection using deep learning can also be performed. The object detection unit 1091 has multiple product-sum calculators and is also used as a block that performs processing for deep learning. The object detection unit 1091 applies object detection processing to image data using one learning model selected by the camera control unit 115 from multiple learning models stored therein. Furthermore, the object detection unit 1091 may switch learning models to perform multiple types of detection processing on one image data.

[0059] The subject detection unit 1091 also performs subject identification processing between captured images, such as images during live view. When the subject detection unit 1091 detects a subject, it temporarily stores an image of the area of ​​the detected subject as a template in the internal memory 110. Furthermore, based on the template information temporarily stored in the internal memory 110, the subject detection unit 1091 searches for an area that matches the template temporarily stored in the internal memory 110 from among the images generated during live view (template matching), and sets the matching area as the subject area.

[0060] One method for subject detection unit 1091 to search for an area that matches a template temporarily stored in internal memory 110 is to cut out the image into areas, calculate the absolute value of the difference from the template temporarily stored in internal memory 110, and determine the area with the smallest difference as the subject area. Another method is to find the area from the degree of similarity of a histogram, color data, etc. with the template temporarily stored in internal memory 110, but any other method may be used as long as it can identify an area in the image that matches the template temporarily stored in internal memory 110.

[0061] Thus, in this embodiment, the subject detection unit 1091 also functions as a means for identifying an area by comparing multiple images sequentially acquired by the image sensor 106 based on a specific subject or at least one part of the subject detected from the image.

[0062] As described above, the subject detection unit 1091 can detect a subject using multiple subject detection methods. However, when a subject cannot be detected using a specific subject detection method, it is also possible to switch to another subject detection method. For example, when subject detection using image recognition becomes impossible, it is also possible to identify the subject area by subject identification processing.

[0063] Next, the image processing circuit 109 also has a clipping processing unit 1092 that performs electronic image stabilization.

[0064] Here, we will explain the projective transformation method for electronic image stabilization performed by the cropping processor 1092. If the coordinates of each pixel in the image before correction are (X0, Y0) and the coordinates of each pixel in the image after correction are (X1, Y1), then the following (Equation 1) is obtained.

[0065]

number

[0066] Here, the 3x3 matrix in (Equation 1) is generally called a projective transformation matrix. The values ​​of each element of this matrix indicate the following quantities: h1, h2, h4, and h5 indicate the amount of shake correction for the rotational component, h3 and h6 indicate the amount of shake correction for the translational component, and h7 and h8 indicate the amount of shake correction for the tilt component. Transforming (Equation 1) yields (Equation 2) and (Equation 3), which enable the coordinates (X1, Y1) of each pixel in the corrected image to be calculated. X1=(h1·X0+h2·Y0+h3) / (h7·X0+h8·Y0+1)…(Formula 2) Y1=(h4·X0+h5·Y0+h6) / (h7·X0+h8·Y0+1)…(Formula 3) If the amount of angular shake in the pitch direction is θp, the amount of angular shake in the yaw direction is θy, the amount of angular shake in the roll direction is θr, the amount of translational correction in the horizontal direction is tx, the amount of translational correction in the vertical direction is ty, the amount of tilt correction in the horizontal direction is vx, the amount of tilt correction in the vertical direction is vy, and the focal length is f, then the elements h1 to h8 of the 3 × 3 matrix in (Equation 1) can be expressed as (Equation 4) to (Equation 11).

[0067] h1=cosθr…(Formula 4) h2=-sinθr…(Equation 5) h3=tx=f·tanθy …(Equation 6) h4=sinθr …(Equation 7) h5=cosθr…(Equation 8) h6=ty=f·tanθp …(Equation 9) h7=νx=-(tanθy) / f …(Equation 10) h8=νy=-(tanθp) / f …(Equation 11) To correct shake caused by angular shake θp in the pitch direction, the shake correction amount ty for the vertical translational component and the shake correction amount vy for the vertical tilt component are calculated from (Equation 9) and (Equation 11).To correct shake caused by angular shake θy in the yaw direction, the shake correction amount tx for the horizontal translational component and the shake correction amount vx for the horizontal tilt component are calculated from (Equation 6) and (Equation 10).To correct shake caused by angular shake θr in the roll direction, the shake correction amount in the rotational direction is calculated from (Equation 4), (Equation 5), (Equation 7), and (Equation 8).

[0068] The cutout processing unit 1092 sets each of the elements h1 to h8 and performs projective transformation, thereby correcting each shake component and enabling electronic image stabilization.

[0069] Next, the control block for tracking the subject will be described.

[0070] 2, the subject information acquisition unit 141 is a block that extracts and acquires information about the subject detected by the subject detection unit 1091, and acquires necessary information such as the subject type (person, animal, vehicle), body part (eyes, face, body), position, size, etc. In this embodiment, it is also possible to acquire information about the method used for subject detection.

[0071] The photographer can set any subject as a tracking target subject via the operation unit 114 by performing a touch operation or a button operation. The tracking target subject may also be determined by an automatic subject setting program of the camera without the photographer operating any components. The subject information acquisition unit 141 sets a subject selected by the photographer or the camera as a main subject, and is capable of acquiring detection information related to the main subject. In this embodiment, the subject position can be acquired by the subject position acquisition unit 1411, and the subject detection method can be acquired by the subject detection method acquisition unit 1412.

[0072] The subject tracking amount calculation unit 142 performs a process of changing the imaging range based on the subject detection information acquired from the subject information acquisition unit 141 so that the subject is stabilized at a predetermined position. The process of changing the imaging range can be performed by changing the image crop position as in electronic image stabilization, or by moving the housing itself. More specifically, the process of changing the imaging range is performed by a means for shifting the imaging range, and can use a mechanism for moving optical components such as a lens or image sensor, a mechanism for panning and tilting the imaging device, a gimbal mechanism, or a method for changing the image crop position as in electronic image stabilization. In this embodiment, the cropping processing unit 1092 is described as changing the image crop position.

[0073] The subject target position setting unit 1421 performs processing to set an arbitrary position within the photographing angle of view as the subject target position. The photographer can set the subject target position via the operation unit 114. For example, the coordinates of the center of the angle of view or a subject target position arbitrarily set by the photographer is set. In this embodiment, for ease of explanation, the subject target position is set to the center of the angle of view.

[0074] The subject position adjustment unit 1422 performs processing to adjust the subject position as necessary based on the subject position and subject detection method from the subject information acquisition unit 141. In this embodiment, the subject detection information is acquired via the subject information acquisition unit 141, but it is also possible to acquire information directly from the subject detection unit 1091 in the image processing unit 109.

[0075] Tracking amount calculation unit 1423 calculates the amount of object tracking according to the object target position set by object target position setting unit 1421 and the object position adjusted by object position adjustment unit 1422. As described above, the object target position is the center of the angle of view, so the difference between the output of object position adjustment unit 1422 and the center position of the angle of view is calculated.

[0076] The tracking amount calculation unit 1423 further performs filtering to reduce noise related to the subject position. In this embodiment, the degree of subject tracking can be changed by changing the cutoff frequency of this filter depending on the subject detection method. This processing will be described in detail later.

[0077] The subject tracking amount after filtering is input to the clipping processing unit 1092 of the image processing circuit 109, which performs projective transformation processing to enable subject tracking. If it is desired to perform camera shake component correction and subject tracking in the clipping processing unit 1092, both can be achieved by adding the subject tracking amount to the translational component shake correction amounts tx and ty described above. The subject tracking amount is also taken into account for the tilt correction amounts vx and vy as necessary.

[0078] 3A and 3B are flowcharts showing the operation of the subject tracking process. The process of this flowchart is basically executed in a cycle in which subject detection is performed. The process of this flowchart is realized by the camera control unit 115 executing a control program stored in the internal memory 110. Note that in FIG. 1, each block shown in the camera control unit 115 is basically realized by the camera control unit 115 executing a control program stored in the internal memory 110. Therefore, in the following description, each block in the camera control unit 115 may be expressed as the subject of the operation.

[0079] First, in step S301, camera control unit 115 performs processing to acquire subject detection information. As described above, subject detection unit 1091 detects the image area of ​​the subject included in the captured image based on the image signal output from image sensor 106, and generates subject detection information. The subject detection information includes information such as the type of subject (person, animal, vehicle, etc.), the detected body part (eyes, face, body, etc.), and the position and size of each organ or body part.

[0080] In step S302, the camera control unit 115 performs a process of selecting information to be used for the subject tracking process from the information acquired in step S301. The subject position acquisition unit 1411 acquires the subject position from the subject detection information, and the subject detection method acquisition unit 1412 acquires the method used to calculate the selected subject position from the subject detection information. Note that if subject detection is performed using multiple detection methods, the subject position and the method for calculating the subject position are selected in step S302, but the process of step S302 differs depending on how the subject detection is performed. For example, it is possible to perform subject detection using one detection method, and if subject detection is not possible using that detection method (the reliability of the detection result is low), start subject detection using another detection method, and if subject detection is possible, do not perform subject detection using the other detection method. In such a case, in step S301, the detection method that enabled subject detection and the subject position detected using that detection method are acquired as subject detection information, and there is no need to select information to be used for the subject tracking process in step S302.

[0081] In step S303, camera control unit 115 sets the subject target position. The subject target position is set in advance by the photographer via operation unit 114. Unless the photographer resets the subject target position, it is not necessary to update the subject target position every time.

[0082] In step S304, the camera control unit 115 performs processing to manipulate the cutoff frequency of a low-pass filter (LPF) related to object tracking amount calculation according to the ON / OFF of the object tracking function and the object detection method. The low-pass filter (LPF) performs processing to extract low-frequency components of information related to the position of the object. Here, the manipulation of the cutoff frequency will be described with reference to FIG. 3B.

[0083] FIG. 3B is a flowchart showing the operation of the cutoff frequency manipulation process in this embodiment.

[0084] First, in step S3041, camera control unit 115 determines whether the subject tracking function is ON or OFF. If the subject tracking function is ON (step S3041: Yes), camera control unit 115 proceeds to step S3042, and if the subject tracking is OFF (step S3041: No), camera control unit 115 proceeds to step S3047.

[0085] In step S3042, camera control unit 115 determines whether there has been a change in the subject detection method. This can be determined by comparing the subject detection method acquired in step S302 of Fig. 3A with the previous subject detection method. If the subject detection method has changed (step S3042: Yes), camera control unit 115 proceeds to step S3043, and if the subject detection method has not changed (step S3042: No), camera control unit 115 proceeds to step S3044.

[0086] In step S3043, the camera control unit 115 sets a target cutoff frequency according to the subject detection method. The target cutoff frequency is the maximum cutoff frequency in a low-pass filter (LPF calculation) related to the subject tracking amount calculation determined for each subject detection method.

[0087] As mentioned above, the purpose of the low-pass filter (LPF) processing related to the subject tracking amount calculation is to remove noise from the subject detection position. In particular, if the subject position contains noise greater than a predetermined frequency, the subject tracking processing will cause the shooting range of the moving image to fluctuate, leading to shaking of the moving image. This will be explained using Figure 4.

[0088] Figure 4 shows an example of how position data changes depending on the subject detection method. Figure 4(a) shows subject position data calculated using face detection processing. Figure 4(b) shows subject position data calculated using subject identification processing without face detection processing.

[0089] Figures 4(a) and 4(b) show that the degree of variation in subject position, i.e., the amount of noise, differs depending on the subject detection method. Furthermore, if subject tracking is performed using a signal that has been low-pass filtered (LPF) with a high cutoff frequency on the subject position data in Figure 4(b), the resulting video will have a steady shaking.

[0090] Therefore, it is preferable to cut out noise components to prevent steady shaking. However, when using an LPF to reduce noise at the subject's position, lowering the cutoff frequency increases the likelihood of the camera being unable to track the subject's movement. This is because the LPF mentioned above acts not only on noise components but also on the subject's movement components, reducing the degree of tracking. For this reason, it is best to tune the cutoff frequency of the low-pass filter (LPF) to balance subject tracking ability and the effects of noise.

[0091] For these reasons, it is advisable to determine the target cutoff frequency in advance depending on the subject detection method. When the subject position variation is relatively small, as in Figure 4(a), the target cutoff frequency can be set high, but when the subject position variation is relatively large, as in Figure 4(b), it is best to set the target cutoff frequency low.

[0092] Regarding subject detection methods, even if the same face detection method is used, the amount of noise contained at the subject position may differ between humans and animals. This is thought to be due to the difference in feature data used in the pattern matching process performed during subject detection. Similarly, in detection using deep learning, differences in the learning model may result in differences in the amount of noise contained at the subject position. Therefore, it is advisable to set the target cutoff frequency taking into account the feature data and reference model used in the subject detection process.

[0093] Based on the above, the target cutoff frequency is determined for each subject detection method, taking into consideration the noise contained in the subject position data and the subject tracking ability. In this way, it is possible to achieve both the degree of tracking and the blocking of noise components according to the subject detection method.

[0094] When the process of step S3043 ends, the camera control unit 115 advances the process to step S3044.

[0095] In step S3044, the camera control unit 115 determines whether the current cutoff frequency is equal to the target cutoff frequency. If the current cutoff frequency is equal to the target cutoff frequency (step S3044: Yes), the camera control unit 115 ends the cutoff frequency operation process. If the current cutoff frequency is not equal to the target cutoff frequency (step S3044: No), the process proceeds to step S3045.

[0096] In step S3045, the camera control unit 115 determines whether the current cutoff frequency is smaller than the target cutoff frequency. If the current cutoff frequency is smaller than the target cutoff frequency (step S3045: Yes), the camera control unit 115 proceeds to step S3046, where a predetermined amount is added to the current cutoff frequency. The predetermined amount to be added is not increased to the target cutoff frequency all at once, but is added in small amounts determined in advance so that the cutoff frequency changes gradually. When the processing of step S3046 is completed, the cutoff frequency manipulation processing ends.

[0097] On the other hand, if the current cutoff frequency is higher than the target cutoff frequency (step S3045: No), the camera control unit 115 proceeds to step S3047 and subtracts a predetermined amount from the current cutoff frequency. The predetermined amount to be subtracted is not to reduce the cutoff frequency to the target cutoff frequency all at once, but rather to subtract a small predetermined amount so that the cutoff frequency changes gradually.

[0098] Even if object tracking is OFF in step S3041 (step S3041: No), camera control unit 115 performs processing to lower the cutoff frequency in step S3047. As the cutoff frequency decreases, the degree of object tracking gradually decreases, and object tracking becomes disabled. After completing the processing of step S3047, camera control unit 115 ends the processing to operate the cutoff frequency.

[0099] This completes the description of the flow relating to the process of manipulating the cutoff frequency in FIG. 3B.

[0100] Returning to the explanation of FIG. 3A, after the camera control unit 115 operates the cutoff frequency in step S304, the process advances to step S305.

[0101] In step S305, camera control unit 115 calculates the difference between the current subject position and the subject target position. The current subject position has already been acquired in step S302, and the subject target position has already been acquired in step S303.

[0102] In step S306, the camera control unit 115 performs a filter (LPF) calculation process on the calculation result calculated in step S305 using the cutoff frequency determined in step S304.

[0103] In step S307, the camera control unit 115 performs resolution adjustment, limit processing, etc. for subsequent input to the projective transformation circuit. By applying the output of step S307 to the projective transformation circuit, it is possible to keep the subject positioned near the center of the screen (subject target position).

[0104] This concludes the description of the subject tracking process in FIG.

[0105] Next, the relationship between the subject detection method, the cutoff frequency, and the degree of tracking will be described with reference to FIGS.

[0106] 5 is a diagram showing the object detection methods, cutoff frequencies, and tracking levels. Now, assume that the relationship of object detection method 1<object detection method 2<object detection method 3 is satisfied with respect to the amount of noise contained in the object position.

[0107] If the target cutoff frequency for object detection method 1 is Fc1, the target cutoff frequency for object detection method 2 is Fc2, and the target cutoff frequency for object detection method 3 is Fc3, the relationship between Fc1, Fc2, and Fc3 can be determined as Fc1>Fc2>Fc3 based on the above-mentioned noise amount relationship.

[0108] FIG. 6 is a time series graph showing the change in cutoff frequency when the subject detection method changes from subject detection method 1 to subject detection method 2, and from subject detection method 2 to subject detection method 3 during subject tracking, and then when tracking ends.

[0109] Time T1 is the timing when a tracking start operation is performed. At that time, assuming that the subject is detected using subject detection method 1, the cutoff frequency is gradually increased toward Fc1 during the period from time T1 to time T2. As the cutoff frequency increases, the tracking level also increases.

[0110] At time T2, the target cutoff frequency Fc1 is reached, and the cutoff frequency is maintained at the target cutoff frequency Fc1. During the period from time T2 to time T3, the tracking level is maintained at "high" (see FIG. 5).

[0111] At time T3, object detection method 1 changes to object detection method 2. In response to this change in object detection method, the target cutoff frequency changes from Fc1 to Fc2, so in the period from time T3 to time T4, the cutoff frequency gradually decreases from the current frequency toward Fc2. As the cutoff frequency decreases, the tracking level also decreases.

[0112] At time T4, the target cutoff frequency Fc2 is reached, and the cutoff frequency is maintained at the target cutoff frequency Fc2. From time T4 to time T5, the tracking level is maintained at "medium" (see FIG. 5).

[0113] At time T5, object detection method changes from method 2 to method 3. In response to this change in object detection method, the target cutoff frequency changes from Fc2 to Fc3, so in the period from time T5 to time T6, the cutoff frequency is gradually lowered from the current frequency toward Fc3. As the cutoff frequency decreases, the tracking level also decreases.

[0114] At time T6, the target cutoff frequency Fc3 is reached, and the cutoff frequency is maintained at the target cutoff frequency Fc3. From time T6 to time T7, the tracking level is maintained at "low" (see FIG. 5).

[0115] At time T7, an operation to end object tracking is performed. When object tracking ends, the cutoff frequency is gradually reduced to bring the tracking component closer to zero.

[0116] This concludes the description of the relationship between the subject detection method, the cutoff frequency, and the degree of tracking using Figures 5 and 6. The change rate (amount of change per unit time) of the cutoff frequency may be variable depending on the situation. For example, the change rate may be different when the cutoff frequency is increased and when it is decreased. Furthermore, the change rate may be different depending on the magnitude of the required change in the cutoff frequency.

[0117] As described above, according to this embodiment, by changing the degree (level) of tracking depending on the subject detection method, it is possible to reduce fluctuations in the amount of subject tracking due to the subject detection method, and as a result, it is possible to reduce fluctuations in the shooting range.

[0118] (Second embodiment) In this embodiment, a method will be described in which the subject position is adjusted depending on the subject detection method, thereby reducing fluctuations in the shooting range when the subject tracking method is changed during subject tracking.

[0119] If the subject tracking method changes during subject tracking, and the subject detection position of the subject detection unit 1091 is used as is, the subject position may change in a step-like manner. Furthermore, the step-like change in the subject position also affects the subject tracking amount, resulting in a sudden change in the shooting range and an unattractive video. This is thought to be because a change in the subject detection method may also change the size of the area recognized as the subject.

[0120] Therefore, when the subject detection method changes, it is desirable to reduce fluctuations in the subject tracking amount by adjusting the subject position accordingly.

[0121] The imaging device of the second embodiment will be described below. The configuration of the imaging device is the same as that of the first embodiment shown in Figures 1 and 2. In this embodiment, too, for ease of explanation, the center of the screen is set as the subject target position.

[0122] 7A and 7B are flowcharts showing the operation of subject tracking processing in the second embodiment. The processing of this flowchart is realized by camera control unit 115 executing a control program stored in internal memory 110. Note that in FIG. 1, each block shown in camera control unit 115 is basically realized by camera control unit 115 executing a control program stored in internal memory 110. Therefore, in the following description, each block in camera control unit 115 may be expressed as the subject of the operation.

[0123] In FIG. 7A, steps S701 to S704 are the same as steps S301 to S304 in FIG. 3A in the first embodiment, and therefore a description thereof will be omitted.

[0124] In step S705, camera control unit 115 adjusts the subject position. Subject position adjustment unit 1422 shown in Fig. 2 performs processing to adjust the subject position as needed based on subject position information and subject detection method information from subject information acquisition unit 141. The adjustment of the subject position by subject position adjustment unit 1422 will be described below using the flowchart in Fig. 7B.

[0125] In step S70501, camera control unit 115 determines whether the subject detection method has changed. This can be determined by comparing the subject detection method acquired in step S302 of Fig. 3A with the previous subject detection method. If the subject detection method has changed (step S70501: Yes), camera control unit 115 proceeds to step S70502, and if the subject detection method has not changed (step S70501: No), camera control unit 115 proceeds to step S70508.

[0126] In step S70502, the camera control unit 115 calculates the offset of the subject position. The offset of the subject position is the difference between the previous subject position, i.e., the subject position using the previous subject detection method, and the current subject position, i.e., the subject position using the subject detection method after the change. After completing the processing of step S70502, the camera control unit 115 proceeds to step S70503.

[0127] In step S70503, the camera control unit 115 determines whether the offset of the subject position calculated in step S70502 is smaller than a predetermined threshold value Th. If the offset amount of the subject position is smaller than the predetermined threshold value (step S70503: Yes), the camera control unit 115 proceeds to step S70504.

[0128] In step S70504, camera control unit 115 assumes that the change in subject position due to the change in subject detection method is negligibly small, and substitutes the subject position after the change in subject detection method as is for subject position'. After completing the processing of step S70504, camera control unit 115 proceeds to step S70505.

[0129] In step S70505, the camera control unit 115 sets the count number for smoothly connecting the subject positions (hereinafter referred to as the connecting count), which is set when the offset amount of the subject position is larger than a predetermined threshold, to 0. In the case of step S70504, there is no need to smoothly connect the subject positions, so the connecting count may be set to 0. When the processing of step S70505 ends, the camera control unit 115 ends the adjustment of the subject position.

[0130] On the other hand, in step S70503, if the offset amount of the subject position is equal to or greater than the predetermined threshold value Th (step S70503: No), the camera control unit 115 advances the process to step S70506.

[0131] In step S70506, the camera control unit 115 calculates the number of counts (joining counts) required for the joining process and the amount of offset change for each joining count based on the offset amount of the subject position in order to adjust for changes in subject position due to changes in the subject detection method. After completing the process of step S70506, the camera control unit 115 proceeds to step S70507.

[0132] In step S70507, camera control unit 115 assigns a value obtained by adding the amount of offset change to the subject position after the subject detection method change as subject position'. After the processing of step S70507 ends, camera control unit 115 ends adjustment of the subject position.

[0133] Step S70508 is a determination process that is carried out when the subject detection method has not changed. In step S70508, the camera control unit 115 determines whether the transition count is 0 or less. If the transition count is 0 or less (step S70508: Yes), the camera control unit 115 proceeds to step S70509.

[0134] In step S70509, the camera control unit 115 substitutes the current subject position into subject position' as is, and proceeds to step S70510. In step S70510, the camera control unit 115 sets the transition count to 0, and ends the adjustment of the subject position.

[0135] On the other hand, if the connection count is greater than 0 in step S70508 (step S70508: No), the camera control unit 115 advances the process to step S70511.

[0136] In step S70511, camera control unit 115 assigns a value obtained by adding the offset operation amount to the current object position to object position'. Here, since this is a transition period for the object position after the object detection method was changed in the previous loop of the flowchart, calculations are performed to gradually transition to the object position after the change in object detection method. When the processing of step S70511 is completed, camera control unit 115 ends adjustment of the object position.

[0137] This concludes the explanation of FIG. 7B.

[0138] Returning to the explanation of FIG. 7A, after adjusting the subject position in step S705, camera control unit 115 advances the process to step S706. In step S706, camera control unit 115 calculates the difference between the adjusted subject position' and the subject target position. After completing the process of step S706, camera control unit 115 advances the process to step S707.

[0139] Steps S707 to S708 are similar to steps S306 to S307 in FIG. 3A in the first embodiment, and therefore a description thereof will be omitted.

[0140] This concludes the explanation regarding FIGS. 7A and 7B.

[0141] FIG. 8 is a graph showing the subject position over time.

[0142] 8(a), dashed line L801 indicates the subject position in subject detection method 1, and is represented as p1(t) using time t. Similarly, dashed line L802 indicates the subject position in subject detection method 2, and is represented as p2(t) using time t.

[0143] The subject position in subject detection method 1 has been calculated since time t0 and is no longer calculable at time t2. On the other hand, the subject position in subject detection method 2 begins to be calculated from time t1 and has been continuously calculated since then. There are various reasons why the subject position cannot be calculated, such as when there is accumulated blur in the captured image and the subject cannot be recognized, or when the subject cannot be recognized due to its movement.

[0144] Time t2 is the timing when the subject detection method changes. As shown in Figure 8(a), there are cases where multiple subject detection methods are in operation at the same time, and each method calculates a subject position.

[0145] In this embodiment, subject position data from subject detection method 1 contains less noise than subject detection method 2, and so long as the subject position is calculated using subject detection method 1, that position data is used. This is because, as described in the first embodiment, a smaller amount of noise allows for a higher degree of tracking.

[0146] However, when the subject detection method is switched at time t2, switching from the subject position obtained using subject detection method 1 to the subject position obtained using subject detection method 2 results in an instantaneous difference between position p1(t2) and position p2(t2). This difference causes a sudden change in the subject tracking amount, resulting in shaking of the shooting range (captured image). This difference is shown as the subject position offset in Figure 8(a).

[0147] Therefore, in this embodiment, the change in subject detection method is detected and the subject position is adjusted at that timing. The subject position after adjustment is represented as subject position' by the solid line L803 in FIG. 8(a). Furthermore, if the subject position is represented as p'(t) using time t, p'(t) is calculated as follows:

[0148] First, from time t0 to time t2, p'(t)=p1(t).

[0149] Next, from time t2 to time t3, p'(t)=p2(t)+offset(t), where offset(t) is the amount of change in offset. The amount of change in offset will be explained using FIG. 8(b).

[0150] The offset change amount is a value determined based on the subject position offset, and its absolute value monotonically decreases, serving to smoothly bridge the subject position differences between different subject detection methods. Here, the offset change amount is set to change linearly, but this is not limiting and any other value may be used as long as the absolute value of offset(t) monotonically decreases. Furthermore, the polarity of the offset change amount may change depending on the subject detection method used before and after the transition, so it may also be a negative value.

[0151] The period from time t2 to time t3 is the offset adjustment period To, and after this period the subject position in subject detection method 1 and the subject position in subject detection method 2 are connected. The longer the offset adjustment period To, the smoother the subject position can be connected. However, if the subject detection method is switched frequently or depending on the interval between tracking processes, the degree of tracking may decrease. Therefore, the offset adjustment period To may be changed depending on the frame rate and scene.

[0152] The aforementioned transition count may be determined in accordance with the offset adjustment period To, or a predetermined transition count may be set in advance, and the offset adjustment period To may be determined based on the offset of the subject position and the transition count.

[0153] Returning to the explanation of FIG. 8(a), finally, after time t3, p'(t)=p2(t).

[0154] By adjusting the subject position' according to the subject detection method as described above, it is possible to reduce fluctuations in the shooting range (captured image) when the subject tracking method is changed during subject tracking.

[0155] Although the use of filter (LPF) processing can be considered to smoothly connect the subject positions and suppress fluctuations in the subject tracking amount, this can be disadvantageous in terms of tracking delay, and therefore it is considered that it is often better to adjust the subject position as in this embodiment. However, depending on the shooting situation, it is possible to switch between the subject position adjustment method described in this embodiment and the use of filter (LPF) processing.

[0156] In this embodiment, the subject position adjustment unit 1422 in the subject tracking amount calculation unit 142 adjusts the subject position according to the subject detection method, but similar processing may also be performed in the image processing unit 109 or the subject detection unit 1091.

[0157] The disclosure of this specification includes the following subject tracking device, method, program, and storage medium.

[0158] (Item 1) a subject detection means capable of detecting a subject from a captured image using a plurality of detection methods; a tracking means for tracking the subject by changing a photographing range based on a target position within a photographed image; a control means for controlling a degree of tracking of the subject by the tracking means in accordance with a detection method by which the subject detection means detects the subject, the detection result of which is used by the tracking means; An object tracking device comprising:

[0159] (Item 2) 2. The subject tracking device according to item 1, further comprising a setting unit for setting the target position.

[0160] (Item 3) 3. The subject tracking device according to item 1 or 2, characterized in that the control means has a low-pass filter that extracts low-frequency components of information related to the position of the subject in order to adjust the degree of tracking, and the subject detection means changes the cutoff frequency of the low-pass filter depending on the accuracy of the detection method used to detect the subject.

[0161] (Item 4) 4. The subject tracking device according to item 3, wherein the control means sets the cutoff frequency of the low-pass filter higher as the accuracy of the detection method for detecting the subject increases.

[0162] (Item 5) 5. The subject tracking device according to item 3 or 4, characterized in that the control means sets a target cutoff frequency of the low-pass filter in accordance with a detection method for detecting the subject, and gradually brings the cutoff frequency of the low-pass filter closer to the target cutoff frequency at a predetermined change amount.

[0163] (Item 6) 6. The subject tracking device according to any one of items 1 to 5, wherein the tracking means adjusts the position of the subject according to a detection method for detecting the subject, and calculates the amount of adjustment of the shooting range based on the adjusted position of the subject.

[0164] (Item 7) 7. The subject tracking device according to item 6, wherein the tracking means adjusts the position of the subject according to the amount of change in the position of the subject at the timing when the detection method for detecting the subject is switched.

[0165] (Item 8) 8. The object tracking device according to any one of items 1 to 7, wherein the plurality of detection methods include a method of detecting a specific object from the captured image and a method of identifying the object by comparing images between a plurality of frames.

[0166] (Item 9) 9. The subject tracking device according to any one of items 1 to 8, wherein the plurality of detection methods include template matching, deep learning, and pattern matching.

[0167] (Item 10) 10. The subject tracking device according to any one of items 1 to 9, wherein the tracking means changes the photographing range by cutting out an image from the photographed image.

[0168] (Item 11) 10. The subject tracking device according to any one of items 1 to 9, wherein the tracking means changes the photographing range by moving a lens.

[0169] (Item 12) 10. The subject tracking device according to any one of items 1 to 9, wherein the tracking means changes the imaging range by moving an imaging element.

[0170] (Item 13) 10. The subject tracking device according to any one of items 1 to 9, wherein the tracking means changes the imaging range by panning and tilting an imaging device.

[0171] (Item 14) a subject detection step capable of detecting a subject from a captured image using a plurality of detection methods; a tracking step of tracking the subject by changing a photographing range based on a target position in the photographed image; a control step of controlling a degree of tracking of the subject in the tracking step in accordance with a detection method for detecting the subject in the subject detection step, the detection result of which is used in the tracking step; A subject tracking method comprising:

[0172] (Item 15) Item 15. A program for causing a computer to execute the subject tracking method according to Item 14.

[0173] (Item 16) Item 15. A computer-readable storage medium storing a program for causing a computer to execute the subject tracking method according to item 14.

[0174] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions.

[0175] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0176] 1: camera body, 2: photographing lens, 100: imaging device, 106: imaging element, 109: image processing circuit, 110: internal memory, 115: camera control unit, 133: image stabilization control unit, 142: subject tracking amount calculation unit

Claims

1. a subject detection means capable of detecting a subject from a captured image using a plurality of detection methods; a tracking means for tracking the subject by changing a photographing range based on a target position within a photographed image; a control means for controlling a degree of tracking of the subject by the tracking means in accordance with a detection method by which the subject detection means detects the subject, the detection result of which is used by the tracking means; An object tracking device comprising:

2. 2. The subject tracking device according to claim 1, further comprising a setting unit for setting the target position.

3. 2. The subject tracking device according to claim 1, wherein the control means has a low-pass filter that extracts low-frequency components of information relating to the position of the subject in order to adjust the degree of tracking, and the control means changes a cutoff frequency of the low-pass filter depending on the accuracy of a detection method used by the subject detection means to detect the subject.

4. 4. The subject tracking device according to claim 3, wherein the control means sets a cutoff frequency of the low-pass filter higher as the accuracy of the detection method for detecting the subject increases.

5. 4. The subject tracking device according to claim 3, wherein the control means sets a target cutoff frequency of the low-pass filter in accordance with a detection method for detecting the subject, and gradually brings the cutoff frequency of the low-pass filter closer to the target cutoff frequency at a predetermined change amount.

6. 2. The subject tracking device according to claim 1, wherein the tracking means adjusts the position of the subject in accordance with a detection method for detecting the subject, and calculates the amount of adjustment of the shooting range based on the adjusted position of the subject.

7. 7. The subject tracking device according to claim 6, wherein the tracking means adjusts the position of the subject in accordance with the amount of change in the position of the subject at the timing when the detection method for detecting the subject is switched.

8. 2. The subject tracking device according to claim 1, wherein the plurality of detection methods include a method of detecting a specific subject from the captured image and a method of identifying the subject by comparing images between a plurality of frames.

9. The subject tracking device according to claim 1 , wherein the plurality of detection methods include any one of template matching, deep learning, and pattern matching.

10. 2. The subject tracking device according to claim 1, wherein the tracking means changes the photographing range by cutting out an image from the photographed image.

11. 2. The subject tracking device according to claim 1, wherein said tracking means changes said photographing range by moving a lens.

12. 2. The subject tracking device according to claim 1, wherein the tracking means changes the photographing range by moving an image sensor.

13. 2. The subject tracking device according to claim 1, wherein said tracking means changes said photographing range by panning and tilting an imaging device.

14. a subject detection step capable of detecting a subject from a captured image using a plurality of detection methods; a tracking step of tracking the subject by changing a photographing range based on a target position in the photographed image; a control step of controlling a degree of tracking of the subject in the tracking step in accordance with a detection method for detecting the subject in the subject detection step, the detection result of which is used in the tracking step; A subject tracking method comprising:

15. A program for causing a computer to execute the subject tracking method according to claim 14.

16. A computer-readable storage medium storing a program for causing a computer to execute the subject tracking method according to claim 14.

Images