Control apparatus, control method thereof, image pickup apparatus, computer readable storage medium, and computer program product

Abstract

The invention discloses a control apparatus, a control method thereof, an image pickup apparatus, a computer readable storage medium, and a computer program product. The control apparatus detects a region of a subject in an image that has been photographed, determines a target position of the subject in the image in a case where the detected subject is tracked, and changes a viewing angle of photographing such that the position of the subject in the image becomes the target position. The apparatus determines the target position by a first process in a case where the entire subject is included in the captured image, and determines the target position by a second process different from the first process in a case where the entire subject is not included in the captured image.

Description

Technical Field

[0001] The present invention relates to a control device and a control method thereof, a camera device, a computer-readable storage medium and a computer program product. Background Art

[0002] Recently, in an imaging device such as a digital still camera, a technique is known in which a subject in an image is identified using a subject recognition technique, and the identified subject is tracked so as to keep the subject at a predetermined position within the angle of view and stabilize the framing. Japanese Patent Laid-Open No. 2017-121042 discloses a technique in which a tracking correction amount in a correction lens for tracking a subject is calculated so that the center position of the identified subject area is the center of the image, and then the tracking correction amount is changed based on the likelihood of the identified subject. The technique disclosed in Japanese Patent Laid-Open No. 2017-121042 makes it possible to prevent erroneous tracking operations and perform highly responsive tracking.

[0003] By the way, for example, when shooting a moving subject during panning, the subject can be shot in various compositions according to the photographer's intention at the time of shooting. However, the above-mentioned past technology does not consider appropriately tracking the subject throughout the scene shot with different intentions. Summary of the invention

[0004] The present invention has been made in view of the above-mentioned problems, and provides a technique capable of appropriately photographing a subject as a tracking target according to a scene.

[0005] In order to solve the above-mentioned problem, one aspect of the present disclosure provides a control device, which includes: a detection unit, which is configured to detect an area of ​​a subject in an image that has been captured; a determination unit, which is configured to determine a target position of the subject in the image when the detected subject is tracked; and a control unit, which is configured to change the viewing angle for shooting so that the position of the subject in the image becomes the target position, wherein, in a case where the subject is included in the captured image as a whole, the determination unit determines the target position through a first processing, and in a case where the subject is not included in the captured image as a whole, the determination unit determines the target position through a second processing different from the first processing.

[0006] Another aspect of the present disclosure provides an imaging device, comprising: an imaging unit configured to capture an image; a detection unit configured to detect an area of ​​a subject in an image that has been captured; a determination unit configured to determine a target position of the subject in the image when the detected subject is tracked; and a control unit configured to change the viewing angle for capturing so that the position of the subject in the image becomes the target position, wherein, in a case where the subject is included in its entirety in the captured image, the determination unit determines the target position through a first process, and in a case where the subject is not included in its entirety in the captured image, the determination unit determines the target position through a second process that is different from the first process.

[0007] Another aspect of the present disclosure provides a control method for a control device, the control method comprising: detecting an area of ​​a subject in an image that has been captured; determining a target position of the subject in the image when the detected subject is tracked; and changing the viewing angle for shooting so that the position of the subject in the image becomes the target position, wherein, in a case where the subject is included in its entirety in the captured image, the target position is determined by a first processing in the determination, and in a case where the subject is not included in its entirety in the captured image, the target position is determined by a second processing in the determination that is different from the first processing.

[0008] Another aspect of the present disclosure provides a computer-readable storage medium, which includes instructions for performing a control method for a control device, the control method including: detecting an area of ​​a subject in an image that has been captured; determining a target position of the subject in the image when the detected subject is tracked; and changing the viewing angle for shooting so that the position of the subject in the image becomes the target position, wherein, in a case where the subject is included in the captured image as a whole, the target position is determined by a first processing in the determination, and in a case where the subject is not included in the captured image as a whole, the target position is determined by a second processing in the determination that is different from the first processing.

[0009] Another aspect of the present disclosure provides a computer program product, which includes instructions for performing a control method for a control device, the control method including: detecting an area of ​​a subject in an image that has been captured; determining a target position of the subject in the image when the detected subject is tracked; and changing the viewing angle for shooting so that the position of the subject in the image becomes the target position, wherein, in a case where the subject is included in its entirety in the captured image, the target position is determined by a first processing in the determination, and in a case where the subject is not included in its entirety in the captured image, the target position is determined by a second processing in the determination that is different from the first processing.

[0010] According to the present invention, a subject to be tracked can be appropriately photographed according to the scene.

[0011] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram showing an example of a functional configuration of a digital camera serving as an example of an image pickup apparatus according to the embodiment.

[0013] Figure 2 is shown in more detail with Figure 1 Block diagram of the configuration related to image blur correction in .

[0014] FIG. 3A to FIG. 3D is a diagram showing a method for detecting a subject vector according to the present embodiment.

[0015] Figure 4 is a diagram showing an example of a method for detecting the motion of a subject according to the present embodiment.

[0016] FIG. 5A to FIG. 5C : is a diagram showing an example of composition after tracking control of a subject according to the present embodiment.

[0017] FIG. 6A to FIG. 6C is a diagram illustrating a method for determining the whole body size of a subject within a shooting angle according to the present embodiment.

[0018] 7A to 7C : is a diagram showing a subject tracking control method according to the present embodiment when the whole body of the subject falls within the shooting angle.

[0019] FIG. 8A to FIG. 8D : is a diagram showing a subject tracking control method according to the present embodiment when the whole body of the subject does not fall within the shooting angle.

[0020] Fig. 9: is a flowchart showing the image blur correction control according to the present embodiment. DETAILED DESCRIPTION

[0021] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but are not limited to inventions requiring all of these features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar configurations, and redundant descriptions thereof are omitted.

[0022] Note that the following embodiments will describe the case where an interchangeable lens digital camera is used as a camera device as an example. However, the present invention can be applied to any electronic device with a camera function. Examples of such electronic devices include cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smart phones, game consoles, robots, drones, and dashboard cameras. However, these are merely examples, and the present invention can also be applied to other electronic devices. The control device according to the present embodiment is also configured to include a part of the constituent elements of the camera device. The control device can be composed of a part of the configuration of the main body of the camera device, or can be composed of a part of the configuration of the interchangeable lens.

[0023] Digital camera configuration

[0024] Will refer to Figure 1 An example of a configuration of a digital camera 10 according to the present embodiment is described. The digital camera 10 according to the present embodiment includes a main body 100 and a lens unit 200 that can be attached to and detached from the main body 100. The main body 100 and the lens unit 200 are connected by a mount that mechanically attaches the main body and the lens unit to each other in a removable manner. The mount provided in the lens unit 200 and the mount provided in the main body 100 are provided with terminals configured to contact each other when the lens unit 200 is mounted to the main body 100. The terminals include a terminal for supplying power from the main body 100 to the lens unit 200, a terminal for communication between the camera side communication unit 135 and the lens side communication unit 128, and the like.

[0025] The lens unit 200 includes an imaging optical system (or imaging lens) 150 for generating an optical image of a subject on an imaging plane of an image sensor 106 provided in the main body 100. The imaging optical system 150 includes a plurality of optical lenses (including a movable lens) and an aperture stop 104. For simplicity, Figure 1 Only the zoom lens 101, the image blur correction lens 102, and the focus lens 103 are shown as movable lenses among the optical lenses. Note that each of these movable lenses may be actually constituted by a plurality of lenses.

[0026] The zoom lens 101 is driven in the optical axis direction by the zoom lens driving unit 124, and changes the focal length (angle of view) of the imaging optical system 150. The image blur correction lens 102 can be moved in the direction perpendicular to the optical axis by the correction lens driving unit 122, and optically corrects (suppresses) image blur and the like generated by the movement of the digital camera 10. The focus lens 103 is driven in the optical axis direction by the focus lens driving unit 121, and changes the focal length of the imaging optical system 150. The size of the opening in the aperture stop 104 is controlled by the aperture driving unit 120 that adjusts the amount of light entering the main body 100 from the imaging optical system 150.

[0027] The lens control unit 160 includes, for example, one or more processors capable of executing programs. The one or more processors may be at least one of a CPU, an MPU, a microprocessor, etc. The lens control unit 160 controls the operation of the lens unit 200 by loading the program stored in the ROM 141 into the RAM 142 and executing the program. The lens control unit 160 also controls the operation of the lens unit 200 in response to instructions, requests, etc. from the camera control unit 115, outputs information related to the lens unit 200 to the camera control unit 115, etc.

[0028] exist Figure 1 , the function blocks indicated inside the lens control unit 160 schematically indicate various functions implemented by the lens control unit 160 executing the program. In this way, the operations performed by the function blocks inside the lens control unit 160 according to the present embodiment are actually performed by the lens control unit 160. The lens-side motion detection unit 125, the lens-side anti-vibration control unit 126, and the lens information management unit 129 implemented in the lens control unit 160 will be described later.

[0029] The ROM 141 is a rewritable nonvolatile memory, and stores programs executed by the lens control unit 160, information and setting values ​​of the lens unit 200, etc. The RAM 142 is used to load programs to be executed by the lens control unit 160, temporarily store information related to the lens unit 200, etc. Note that at least one of the ROM 141 and the RAM 142 may be integrated with the lens control unit 160.

[0030] For example, the shutter 105 is a mechanical focal plane shutter. Note that an electronic shutter may be used instead of or in addition to the shutter 105. The “electronic shutter” refers to a shutter operation achieved by controlling the operation of the image sensor 106.

[0031] The image sensor 106 may be, for example, a well-known CCD or CMOS color image sensor having a primary color Bayer array filter. The image sensor 106 includes a pixel array in which a plurality of pixels are arranged two-dimensionally and a peripheral circuit for reading out signals from the pixels. Each pixel accumulates a charge corresponding to the amount of incident light by photoelectric conversion. By reading out a signal having a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing an image of a subject formed on an imaging plane is obtained.

[0032] The image sensor 106 according to the present embodiment is also configured to be movable in a direction perpendicular to the optical axis of the imaging optical system 150. The position of the image sensor 106 is controlled by the sensor driving unit 130. In this way, the digital camera 10 according to the present embodiment has an optical image blur correction function in both the lens unit 200 and the main body 100.

[0033] The AD converter 107 applies noise removal processing, gain adjustment processing, and AD conversion processing to the analog image signal read out from the image sensor 106, and generates a digital image signal (image data). The AD converter 107 outputs the digital image data to the image processing circuit 109. Note that the AD converter 107 may be provided in the image sensor 106.

[0034] The timing generator 108 supplies a signal for controlling the timing of operation to the image sensor 106 and the AD converter 107 in response to an instruction from the camera control unit 115 .

[0035] The image processing circuit 109 generates signals and image data for different purposes, obtains and / or generates various types of information, etc. by applying predetermined image processing to the image data output by the AD converter 107. The image processing circuit 109 may be, for example, a dedicated hardware circuit such as an application-specific integrated circuit (ASIC) designed to implement a specific function. Alternatively, the image processing circuit 109 may be composed of a processor such as a digital signal processor (DSP) or a graphics processing unit (GPU) that executes software to implement a specific function. The image processing circuit 109 outputs the obtained or generated information, data, etc. to the camera control unit 115, the RAM 110, etc. according to the purpose of use.

[0036] The image processing applied by the image processing circuit 109 may include, for example, preprocessing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, etc. Preprocessing includes signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when the image sensor is provided with a color filter, and the values ​​of the color components not included in the individual pixel data constituting the image data are interpolated. Color interpolation processing is also called "demosaicing". Correction processing may include white balance adjustment, hue adjustment, correction of image degradation caused by optical aberrations of the imaging optical system 150 (image restoration), correction of vignetting effects in the imaging optical system 150, and color correction, etc. Detection processing includes detecting feature areas (e.g., facial areas or human areas) or motion in such areas, processing for identifying people, etc. Data processing may include cropping areas (trimming), synthesis, scaling, encoding and decoding, and header information generation (data file generation). The generation of display image data (including image data for live view display), recording image data, etc. is also included in data processing. The evaluation value calculation process may include processes such as generating a signal used in automatic focus detection (AF), an evaluation value, etc., generating an evaluation value used in automatic exposure control (AE), etc. Special effect processing includes adding a bokeh effect, changing color tones, relighting processing, etc.

[0037] Note that these are merely examples of processing applicable by the image processing circuit 109 and the processing applied by the image processing circuit 109 is not limited thereto. In addition, some of the image processing mentioned here may be performed by a signal processing circuit built into the image sensor 106 or by the camera control unit 115.

[0038] The camera control unit 115 includes, for example, one or more processors capable of executing programs. The one or more processors may be at least one of a CPU, an MPU, a microprocessor, etc. The camera control unit 115 implements the functions of the digital camera 10 by loading a program stored in the ROM 113 into the RAM 110 and executing the program to control the operation of the main body 100 and the lens unit 200. The camera control unit 115 controls the operation of the lens unit 200 by communicating with the lens control unit 160. Figure 1 In FIG. 1 , the function blocks indicated inside the camera control unit 115 schematically indicate various functions implemented by the camera control unit 115 executing the program. Thus, the operations of the function blocks in the camera control unit 115 described here are actually performed by the camera control unit 115.

[0039] The ROM 113 is a rewritable nonvolatile memory, and stores programs executed by the camera control unit 115, various types of setting values ​​of the digital camera 10, GUI data, etc. The RAM 110 is used to load programs executed by the camera control unit 115, temporarily store data to be processed by the image processing circuit 109, intermediate data, and data resulting from the processing, etc. The RAM 110 is also used as a buffer memory for image data, a video memory for the display unit 111, and the like.

[0040] The display unit 111 is provided on, for example, a surface of a housing of the digital camera 10 , and displays a live view image, information of the digital camera, information related to an imaging scene, a menu screen, etc. The display unit 111 may be a touch screen.

[0041] The "operation unit 114" is a general term for input devices (buttons, switches, dials, etc.) provided for the photographer to input various types of instructions to the digital camera 10. The input devices constituting the operation unit 114 are named according to the functions assigned to them. For example, the operation unit 114 includes a release switch, a moving image recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, a direction key, an OK key, etc. The release switch is a switch for recording a still image, and the camera control unit 115 recognizes that the release switch is in a half-pressed state as a shooting preparation instruction, and recognizes that the release switch is in a fully pressed state as a shooting start instruction. In addition, the camera control unit 115 recognizes that the moving image recording switch is pressed during a shooting standby state as a moving image recording start instruction, and recognizes that the moving image recording switch is pressed during recording a moving image as a recording stop instruction. Note that the functions assigned to the same input device may be variable.

[0042] Upon detecting an operation performed on the operation unit 114, the camera control unit 115 performs an operation corresponding to the detected operation. For example, when an operation for changing the angle of view of the lens unit 200 is detected, the camera control unit 115 periodically transmits an angle of view change instruction to the lens control unit 160 while the operation is being detected. Each time the angle of view change instruction is received, the zoom lens driving unit 124 of the lens control unit 160 drives the zoom lens 101 by a set amount.

[0043] The exposure control unit 138 determines the exposure conditions (aperture value, shutter speed (or exposure time), and imaging sensitivity) based on, for example, the AE evaluation value generated by the image processing circuit 109 and the settings of the digital camera (for example, imaging mode). When the exposure conditions for capturing a still image have been determined, the exposure control unit 138 outputs the shutter speed to the shutter drive unit 136 and the aperture value to the lens control unit 160, respectively. In addition, when the exposure conditions for capturing a moving image have been determined, the exposure control unit 138 outputs the exposure time to the timing generator 108. When the aperture value changes, the exposure control unit 138 outputs the aperture value to the lens control unit 160. The exposure control unit 138 outputs the imaging sensitivity to the image sensor 106 or the image processing circuit 109. The aperture drive unit 120 of the lens control unit 160 drives the aperture stop 104 according to the aperture value received from the camera control unit 115.

[0044] The focus lens control unit 139 determines the drive amount and drive direction for moving the focus lens 103 from the current focus lens position to the target focus lens position based on, for example, the AF evaluation value generated by the image processing circuit 109. If the AF evaluation value is a contrast evaluation value, the focus lens control unit 139 uses the focus lens position with the highest evaluation value as the target position to determine the drive amount and drive direction. If the AF evaluation value is a defocus amount, the focus lens control unit 139 converts the defocus amount into a drive amount and drive direction. The focus lens control unit 139 sends the determined drive amount and drive direction to the lens control unit 160. The focus lens drive unit 121 of the lens control unit 160 drives the focus lens 103 according to the drive amount and drive direction received from the camera control unit 115.

[0045] The digital camera 10 performs both correction of image blur (hand shake) caused by movement of the main body 100 and the lens unit 200 and correction of image blur (subject blur) caused by movement of the subject during the exposure period.

[0046] The camera-side motion detection unit 134 detects motion of the main body 100, and outputs a signal based on the detected motion to the camera-side anti-vibration control unit 133. The lens-side motion detection unit 125 detects motion of the lens unit 200, and outputs a signal based on the detected motion to the lens-side anti-vibration control unit 126. Each of the camera-side motion detection unit 134 and the lens-side motion detection unit 125 may be, for example, a gyro sensor for outputting a signal based on angular velocity around each axis of a Cartesian coordinate system, a combination of a gyro sensor and an accelerometer for outputting a signal based on acceleration along each axial direction of the Cartesian coordinate system, or the like.

[0047] Although in the present embodiment, the main body 100 and the lens unit 200 have separate motion detection units, the motion detection unit may be provided in only one of the main body 100 and the lens unit 200. For example, if the lens unit 200 does not have a motion detection unit, an output signal of the motion detection unit provided in the main body 100 may be sent to both the camera-side anti-vibration control unit 133 and the lens-side anti-vibration control unit 126.

[0048] The sensor position detection unit 132 detects the current position of the image sensor 106. For example, the current position may be a relative displacement relative to the reference position. The position when the center of the pixel array of the image sensor 106 is perpendicular to the optical axis may be taken as the reference position. Note that the reference position may be set based on other well-known methods. The displacement may be a displacement in two directions parallel to the short side and the long side of the image sensor 106, respectively.

[0049] The camera-side anti-vibration control unit 133 determines the correction amount for moving the image sensor 106 so as to cancel the motion detected by the camera-side motion detection unit 134. Note that the camera-side anti-vibration control unit 133 determines the correction amount within a range not exceeding the maximum amount of movement in each direction relative to the reference position. Then, the camera-side anti-vibration control unit 133 corrects image blur by moving the image sensor 106 (changing the shooting angle) by the sensor driving unit 130 according to the determined correction amount.

[0050] The correction lens position detection unit 123 detects the current position of the image blur correction lens 102. For example, the current position may be a relative displacement relative to the reference position. The position when the center of the image blur correction lens 102 is perpendicular to the optical axis may be taken as the reference position. Note that the reference position may be set based on other well-known methods. The displacement may be displacement in two directions parallel to the short side and the long side of the image sensor, respectively.

[0051] The lens-side anti-vibration control unit 126 determines the correction amount of moving the image blur correction lens 102 to offset the motion detected by the lens-side motion detection unit 125. Note that the lens-side anti-vibration control unit 126 determines the correction amount within a range that does not exceed the maximum amount of movement in each direction relative to the reference position. Then, the lens-side anti-vibration control unit 126 corrects the image blur by moving the image blur correction lens 102 (changing the shooting angle) by the correction lens driving unit 122 according to the determined correction amount.

[0052] The image blur correction described herein may be performed in the main body 100, the lens unit 200, or both. For example, when image blur correction is performed in both the main body 100 and the lens unit 200, the amount of movement exceeding the maximum amount of movement of one of the elements is compensated by the other element. Alternatively, the necessary amount of movement may be allocated to the main body 100 and the lens unit 200 at a predetermined ratio. Alternatively, the blur to be offset by the main body 100 and the lens unit 200 may be distributed between the main body 100 and the lens unit 200 according to the components of the blur.

[0053] Instead of or in addition to the optical image blur correction for moving the anti-vibration member (here, the image blur correction lens 102, the image sensor 106, or both), electronic image blur correction may be performed. The electronic image blur correction may be achieved by moving the position where the effective area is cropped from the captured image so as to offset the movement of the digital camera 10. For example, the movement amount of the image sensor 106 determined by the camera-side anti-vibration control unit 133 may be converted into the movement amount on the pixel array, and the cropping position may be changed (the shooting angle may be changed). In the present embodiment, the image synthesis processing unit 131 calculates the movement amount and performs cropping.

[0054] The motion vector detection unit 143 detects motion between frames as a motion vector using, for example, template matching. Although the motion vector detection unit 143 can also be used to detect motion in the digital camera 10, in the present embodiment, the motion vector detection unit 143 is mainly used to detect subject blur. A method for detecting subject blur will be described later. When the motion vector of the entire frame is detected, electronic image blur correction can be performed by changing the cropping position based on the amount of movement represented by the motion vector.

[0055] The camera information management unit 137 is used to store information related to the subject 100 and information obtained by the subject 100, such as setting information of the subject 100, the current position of the image sensor 106, drive limit (maximum movement in each direction), subject blur correction amount, etc.

[0056] The lens information management unit 129 is used to store information related to the lens unit 200 such as the optical characteristics, current position, driving limit (maximum movement amount) of the image blur correction lens 102, and information obtained by the lens unit 200.

[0057] Configuration related to image blur correction

[0058] Will refer to Figure 2 The configurations of the lens unit 200 and the body 100 related to image blur correction are described in more detail. Figure 2, the function blocks indicated inside the lens-side anti-vibration control unit 126 schematically indicate various functions implemented by the lens control unit 160 executing the program. Thus, the operations performed by the function blocks inside the lens-side anti-vibration control unit 126 described below are actually performed by the lens control unit 160. Likewise, the operations performed by the function blocks indicated inside the camera-side anti-vibration control unit 133 are actually performed by the camera control unit 115. The camera control unit 115 and the lens control unit 160 operate in cooperation to function as a control unit that performs image blur correction control.

[0059] First, a configuration related to image blur correction of the lens unit 200 will be described. The lens-side integration unit 151 integrates the angular velocity signal output from the lens-side motion detection unit 125 to convert the angular velocity signal into an angle signal.

[0060] The lens-side blur correction amount determination unit 152 determines a correction amount corresponding to the movement amount of the image blur correction lens 102 required to correct image blur based on the angle signal output by the lens-side integration unit 151 .

[0061] The adder 153 applies the subject blur correction amount determined by the main body 100 and obtained by the subject blur correction amount obtaining unit 157 and the subject tracking amount determined by the main body 100 and obtained by the subject tracking amount obtaining unit 158 ​​to (adds to) the correction amount determined by the lens side blur correction amount determining unit 152. A method for determining the subject blur correction amount and the subject tracking amount will be described later.

[0062] The lens-side ratio determination unit 154 determines the ratio of the amount of movement to be handled by the lens unit 200 assuming that the total amount of movement required to correct hand shake and subject blur and perform subject tracking is 100%. Note that when only one of the lens unit 200 and the main body 100 is used, the ratio determined by the lens-side ratio determination unit 154 is 100% or 0%.

[0063] Whether the image blur correction is to be performed by the lens unit 200 or the main body 100, or both, may be determined according to any desired conditions. For example, the image blur correction may be performed according to user settings, or if the total amount of movement of the image blur correction lens 102 or the image sensor 106 exceeds the maximum amount of movement, it may be determined that the image blur correction is to be performed by both the main body 100 and the lens unit 200. The lens side ratio determination unit 154 may make this determination based on information stored in the camera information management unit 137 and the lens information management unit 129.

[0064] The lens-side driving range limiting unit 155 limits the correction amount so that the image blur correction lens 102 is driven within a range that does not exceed the maximum movement amount. The correction lens controller unit 156 controls the position of the image blur correction lens 102 through the correction lens driving unit 122. The correction lens controller unit 156 performs feedback control based on the current position of the image blur correction lens 102 detected by the correction lens position detection unit 123 to move the image blur correction lens 102 to a target position based on the movement amount.

[0065] Next, a configuration related to image blur correction of the main body 100 will be described. The camera-side integrating unit 161 integrates the angular velocity signal output from the camera-side motion detecting unit 134 to convert the angular velocity signal into an angle signal.

[0066] The camera-side blur correction amount determination unit 162 determines a correction amount corresponding to the amount of movement of the image sensor 106 required to correct image blur based on the angle signal output by the camera-side integration unit 161. The subject vector detection unit 163 detects a subject vector indicating subject blur based on the motion vector between frames detected by the motion vector detection unit 143. The operation for detecting the subject vector will be described in detail later. The subject blur correction amount determination unit 164 determines a correction amount for correcting subject blur based on the subject vector.

[0067] The subject reference position determination unit 169 determines whether the whole body of the subject is within the shooting angle based on the information from the object recognition unit 140, and determines the subject tracking detection position. The operation of the subject reference position determination unit 169 will be described in detail later. The subject target position determination unit 170 determines a predetermined position within the shooting angle as a position for tracking the subject based on the information from the subject reference position determination unit 169. The operation of the subject target position determination unit 170 will be described in detail later.

[0068] The adder 171 adds the subject blur correction amount for subject blur correction determined by the subject blur correction amount determination unit 164 and the subject tracking amount for tracking the subject at a desired predetermined position within the shooting angle determined by the subject target position determination unit 170. The adder 165 adds the correction amount for correcting hand shake determined by the camera-side blur correction amount determination unit 162 to the result obtained by adding the subject blur correction amount and the subject tracking amount by the adder 171.

[0069] The camera-side ratio determination unit 166 determines the ratio of the amount of movement to be handled by the main body 100 assuming that the total amount of movement required to correct hand shake and subject blur and perform subject tracking is 100%. Note that when only one of the lens unit 200 and the main body 100 is used, the ratio determined by the camera-side ratio determination unit 166 is 100% or 0%. The camera-side ratio determination unit 166 can make this determination based on information stored in the camera information management unit 137 and the lens information management unit 129.

[0070] The camera-side driving range limiting unit 167 limits the correction amount so that the image sensor 106 is driven within a range not exceeding the maximum movement amount. The camera-side controller unit 168 controls the position of the image sensor 106 through the sensor driving unit 130. The camera-side controller unit 168 performs feedback control based on the current position of the image sensor 106 detected by the sensor position detection unit 132 to move the image sensor 106 to a target position based on the movement amount.

[0071] Object vector detection

[0072] Here we will refer to FIG. 3A to FIG. 3D The detection of the subject vector by the subject vector detection unit 163 is described. Note that the "subject" refers to an object that exists closer to the photographer than the background, and is generally a person, etc. Here, it is assumed that a region that moves between frames is detected as a subject region, and a vector representing the movement of the subject region is detected as a subject vector.

[0073] First, the motion vector detection unit 143 detects a motion vector with respect to a reference frame for each region set in a target frame. Figure 3A An example of setting an area for detecting a motion vector in a target frame is shown. Here, in order to reduce the processing load, a motion vector detection range 301 may be set in a part of the target frame. The motion vector detection unit 143 detects a motion vector for each area into which the detection range 301 is divided. The detection of the motion vector may be performed, for example, by using an image of each area as a template and detecting an area with the highest correlation in a reference frame as a template matching of the destination area.

[0074] Figure 3B Schematically shows the Figure 3AThe object vector detection unit 163 detects the object vector based on the histogram. The object vector detection unit 163 converts the angle signal output by the camera side integration unit 161 or the lens side integration unit 151 into the movement amount on the imaging plane. For example, the conversion can be performed using the focal length of the imaging optical system 150, the imaging time difference between the target frame and the reference frame or its frame rate, or the pixel pitch of the image sensor 106.

[0075] In the camera-side motion detection unit 134 or the lens-side motion detection unit 125, the origin of the output varies due to external disturbances such as temperature drift, etc. (offset phenomenon). Therefore, the subject vector detection unit 163 uses a predetermined range centered on the movement amount 302 as the background determination area 303. The subject vector detection unit 163 then determines that the motion vector whose magnitude is within the background determination area 303 is the background vector 304. On the other hand, the subject vector detection unit 163 determines that the motion vector whose magnitude is not within the background determination area 303 is the subject vector 305. The subject vector 305 can be used to correct the subject blur, and the background vector 304 can be used to correct the hand shake.

[0076] Although the foregoing example describes a method of separating a motion vector into the subject vector 305 and the background vector 304 based on an angular velocity signal (or an angle signal), other methods may be used instead. For example, if the digital camera 10 or the subject moves only a small amount, it is difficult to separate the subject vector and the background vector using a method based on an angular velocity or an angle.

[0077] For example, using the subject distance information makes it possible to separate the motion vector into a subject vector and a background vector even when the digital camera 10 or the subject moves only a small amount. The subject distance information may be, for example, a depth map indicating the subject distance for each region or each pixel into which the imaging range has been divided. As a more convenient method, an example of separating the subject region and the background region based on the AF frame with the highest focus will be described here. Figure 3C, 306 indicates a plurality of AF frames (focus detection areas) that can be set by the digital camera. 307 indicates an AF frame having the highest focusing degree among the plurality of AF frames 306. In this case, the subject vector detection unit 163 sets the subject area 308 based on the subject distance of the AF frame 307, takes the motion vector detected in the subject area 308 as the subject vector, and takes the motion vector detected in other areas as the background vector. For example, the subject vector detection unit 163 may set, as the subject area 308, an AF frame whose focusing degree difference with the AF frame 307 is within a predetermined threshold range, or an AF frame having a depth of field within a predetermined threshold range (e.g., ±2) with respect to the subject depth of field of the AF frame 307.

[0078] Note that the above method using the angular velocity signal (or angle signal) can be used in combination with the method using the subject distance information. In this case, a representative value (average value, mode value, etc.) of the same type of vector determined by the corresponding method can be used. The subject area can also be detected based on machine learning, a well-known feature area detection technology, etc., and the motion vector of the subject area can be detected as the subject blur. Figure 3D A state in which a predetermined subject region 309 (here, a face region of a human being) has been detected is schematically shown. Subject blur can be detected using an evaluation image captured for live view display.

[0079] Calculation of the amount of movement on the imaging plane

[0080] Figure 4 Schematically shows a case where a subject moving in a direction parallel to the imaging plane has been detected by comparing two consecutive distance information. For example, assuming that distance information is obtained at a cycle of 1 / 30 seconds, the time t required for the subject to move from position A to B is 33.3 ms.

[0081] exist Figure 4 In the figure, assuming that the principal point (single point) of the imaging optical system 150 is C, the distance between positions A and B (subject movement amount Δ) represents one side of the triangle ABC, similarly to the movement amount δ of the subject on the imaging plane represents one side of the triangle abC. When the subject distance is represented by L and the focal length of the imaging optical system 150 is represented by f, the ratio between the triangle ABC and the triangle abC is Lf:f. Therefore, since "the movement amount δ of the subject on the imaging plane = the movement amount Δ of the subject × f / Lf", the movement amount can be converted into the movement amount on the imaging plane. This relationship holds even when the speed of the subject is different between point A and point B.

[0082] For example, regarding the amount of movement on the imaging plane as a motion vector in the horizontal direction allows conversion into angular velocity by reversing the process for converting the movement of the digital camera 10 into the amount of movement on the imaging plane. Note that the method described here is merely an example, and the movement of the subject may be detected by other methods without using the image sensor 106.

[0083] Example of subject tracking

[0084] Next, we will refer to FIG. 5A to FIG. 5C An example of subject tracking according to the present embodiment is described. Figure 5A The schematic diagram shows a shooting scene in which a moving object 501 is being followed during panning. The photographer is not following the object in a stable manner, so a local component (e.g., the cockpit) in the object 501 will be out of the frame from the shooting angle. When the tracking detection position 502 is tracked to the tracking target position 503 within the object 501, correction is performed so that the tracking detection position 502 becomes the tracking target position 503 without considering the whole body of the object relative to the shooting angle, and the composition will be as follows. Figure 5B In this case, the Figure 5A The local components of the subject 501 (e.g., the cockpit) that are to exit the frame can be kept within the shooting angle. Figure 5B In the case of Figure 5A Other local components of the subject 501 within the shooting angle (for example, the tail) have now exited the frame from the shooting angle. Figure 5A The image at the shooting angle shown is taken under the shooting condition that the whole body of the subject 501 is within the shooting angle, so the composition after tracking control expected by the photographer is as follows: Figure 5C shown.

[0085] Therefore, in the present embodiment, the subject target position determination unit 170 determines the tracking target position in consideration of not only the subject 501 or a partial constituent element that is a part of the subject 501 but also the entire body of the subject.

[0086] Next, we will refer to FIG. 6A to FIG. 6C An example of determining whether the whole body of the subject 601 falls within (is included in) the shooting angle is described. Note that the subject reference position determination unit 169 determines whether the whole body of the subject 601 falls within the shooting angle. This determination can be made by various methods. Fig. 6AIn the example shown, the object recognition unit 140 detects the whole body of the subject 601 based on machine learning such as deep learning or a well-known feature region detection technology. For example, it is possible to determine whether the whole body of the subject 601 falls within the shooting angle by detecting the region of the subject 601 or a local component 602 (e.g., a cockpit or a tail) constituting the subject 601 using a trained model trained by supervised learning. Figure 6B An example of a case where a motion vector is used is shown. The subject reference position determination unit 169 determines whether the subject vector frame exists as a single individual based on the distribution of the vector frame 603 determined as the subject vector by the subject vector detection unit 163 and the vector frame 604 determined as the background vector. When the subject vector frame exists as a single individual, it can be determined that the entire body of the subject 601 falls within the shooting angle. In addition, as described above, by detecting the subject vector based on the subject depth or subject distance, etc. using a depth map or a LiDAR sensor, etc. instead of the camera-side motion detection unit 134, it can be determined that the entire body of the subject falls within the shooting angle. Additionally, the subject distance can be used in combination with the camera-side motion detection unit 134. As Figure 6C As shown, it is possible to determine whether the subject 601 falls within the shooting angle by extracting the contour (edge) 605 of the subject. The contour of the subject can be detected using an image filter (such as a Laplacian filter) for applying differentiation to the evaluation image. Additionally, FIG. 6A to FIG. 6C The methods can be used in combination to determine whether the subject 601 falls within the shooting angle.

[0087] Next, we will refer to 7A to 7C A method for determining a target position in an image to track a subject is described. Specifically, a method will be described in which the subject target position determination unit 170 determines a target position for tracking a subject when the subject reference position determination unit 169 has determined that the entire body of the subject falls within the shooting angle.

[0088] Fig. 7A The composition is schematically shown when the subject target position determination unit 170 controls tracking so that the whole body of the subject is in an equal position in the shooting angle. Fig. 7A In the horizontal direction of the evaluation image in FIG. 7 , the subject target position determination unit 170 determines the X coordinate of the tracking target position at a position where the right margin 703 and the left margin 704 are equal. Next, with respect to the vertical direction of the evaluation image, the subject target position determination unit 170 determines the Y coordinate of the tracking target position at a position where the upper margin 701 and the lower margin 702 are equal. In other words, Fig. 7A A composition in which tracking is controlled so that the entire body of the subject is located at a position equivalent to that in the evaluation image is shown.

[0089] Figure 7B The composition is schematically shown when the subject target position determination unit 170 determines the X coordinate of the tracking target position so that the traveling direction of the subject is arranged to be equal in the shooting angle. Fig. 7A The difference is, for example, that a larger margin is provided in the upper or lower direction of the shooting angle to create a composition such as shooting elements other than a subject such as the runway approach lights in a scene where an airplane is taking off from an airport at night. Fig. 7A In the same manner as in FIG. 1 , the subject target position determination unit 170 determines the X coordinate of the tracking target position so that the right margin 703 and the left margin 704 corresponding to the moving direction of the subject are equal. Fig. 7A Instead, the object target position determination unit 170 determines the Y coordinate position of the tracking target position so as to be closer to the lower margin 702 of the object. Alternatively, the object target position determination unit 170 may determine the Y coordinate position so as to be closer to the upper margin 701 of the object.

[0090] exist Figure 7C In the example shown, the subject type is different from Fig. 7A and Figure 7B , and is tracking a person. When photographing a person, the margin in the horizontal direction is generally wide relative to the shooting angle. In this case, the subject target position determination unit 170 determines the Y coordinate of the tracking target position so that the position in the vertical direction of the evaluation image is equal, and determines the X coordinate so that it is closer to the right margin 703 or closer to the left margin 704. Figure 7B The aircraft shown and Figure 7C For the person shown, in the case where the coordinates for keeping the tracking target position equal are made different, the position can be kept equal in the direction where the horizontal margin and the vertical margin are smaller depending on the type of the subject.

[0091] Next, we will refer to FIG. 8A to FIG. 8D Other methods for the subject target position determination unit 170 to determine the target position of the subject to be tracked are described below. FIG. 8A to FIG. 8D In the method shown, when the above reference FIG. 6A to FIG. 6C The method described herein determines a target position of the subject to be tracked when it is determined that the entire body of the subject is not within the shooting angle.

[0092] Fig. 8AThe composition in which the photographer is following the subject at a long focal length in a stable manner is schematically shown. The photographer is shooting at a shooting angle (focal length) at which the entire body of the subject is not included, and there are upper margins 901 and lower margins 902, but there is no right margin 903. On the other hand, there is a space of at least a predetermined value for the left margin 904. If this state remains stable for at least a predetermined number of frames, tracking control is not performed. In this composition, the position of a local component of the subject (for example, a cockpit) remains stable.

[0093] Figure 8B A composition in which the photographer is not following the subject at a long focal length in a stable manner is schematically shown. In this example, the photographer is following the subject too late with the camera, so the subject is gradually exiting the frame in the direction of travel of the subject (left side of the shooting angle). In this composition, the subject target position determination unit 170 performs tracking control in the direction opposite to the direction of travel of the subject so that a partial constituent element of the subject (e.g., the cockpit) falls within the shooting angle. Figure 8C Shows from Figure 8B Starting from the state shown, a predetermined amount of driven composition for tracking is performed in the direction opposite to the direction of travel of the subject (the left side of the shooting angle) (the right side of the shooting angle). Although a partial constituent element of the subject (for example, the cockpit) falls within the shooting angle, there is no space in the left margin 904 and the subject cannot be seen. In this way, subject tracking control is performed continuously. Note that the partial constituent elements of the subject to be tracked can be determined in advance. For example, the cockpit of an airplane can be predetermined as the partial constituent element to be tracked. Fig.8D In the tracking control, Figure 8C The illustrated state continues, and the subject is continuously tracked while the left margin 904 is maintained at a predetermined value (eg, 100 pixels).

[0094] A series of operations in image blur correction

[0095] Next, we will refer to Fig. 9A series of operations in image blur correction according to the present embodiment are described. Note that the series of operations described here are implemented by the camera control unit 115 loading a program stored in the ROM 113 or the like into the RAM 110 and executing the program so that various units within the camera control unit 115 (including various units of the camera-side anti-vibration control unit 133) function. However, the lens control unit 160 may be implemented by executing a program stored in the ROM 141 or the like, or the camera control unit 115 and the lens control unit 160 may be implemented in collaboration. For example, the processing may also be started when an image (evaluation image) captured by the image sensor 106 is loaded into the camera control unit 115. At this time, the image to be loaded is an image captured by the photographer while panning the camera to capture at least a part of a subject that is a moving body.

[0096] In step S901, the camera-side motion detection unit 134 (and / or the lens-side motion detection unit 125) detects the motion (hand shake) of the digital camera 10 as an angular velocity signal. In step S902, the camera-side blur correction amount determination unit 162 (and / or the lens-side blur correction amount determination unit 152) determines a correction amount (hand shake correction amount) corresponding to the amount of movement of the image sensor 106 and the image blur correction lens 102 required to cancel the motion detected in step S901.

[0097] In step S903, the subject vector detection unit 163 uses the above reference FIG. 3A to FIG. 3D The subject vector indicating the subject blur is detected for the evaluation image by any of the methods described above. For example, the subject vector detection unit 163 detects the subject vector based on the histogram of the magnitude of the motion vector detected for each region. In step S904, the subject blur correction amount determination unit 164 converts the subject vector detected in step S903 into the angular velocity of the subject. This can be done by referring to Figure 4 The inversion is used to convert the motion of the digital camera into the amount of movement on the imaging plane. Note that the angular velocity of the subject is used to predict the motion of the subject within the frame. Thus, the subject blur correction amount determination unit 164 stores the angular velocity of the subject obtained for the most recent predetermined number of frames in, for example, the RAM 110.

[0098] In step S905, the camera control unit 115 determines whether the tracking control function is valid. The setting of the tracking control function may be, for example, a manual setting instructed by the photographer through the operation unit 114, or the tracking state may be automatically detected and set by the camera control unit 115. If the camera control unit 115 determines that the tracking control function is valid, the sequence moves to step S906, otherwise it moves to step S912.

[0099] In step S906, the subject reference position determination unit 169 determines the subject reference position by referring to the above. FIG. 6A to FIG. 6C The subject reference position determination unit 169 detects a part of the subject or the whole body of the subject based on, for example, machine learning (such as deep learning of the object recognition unit 140) or a known feature region detection technique.

[0100] In step S907, the subject reference position determination unit 169 determines whether the whole body of the subject falls within (is included in) the shooting angle based on the result of step S906. If the subject reference position determination unit 169 determines that the whole body of the subject falls within the shooting angle, the sequence moves to step S908, otherwise it moves to step S909.

[0101] In step S908, when the whole body of the subject falls within the shooting angle, the subject target position determination unit 170 determines the position of the subject (the coordinates of the target position to be tracked of the subject) so that the margins in the frame (i.e., the distances from the ends of the subject to the ends of the angle of view) are equal (substantially equal). Note that, as described above with reference to Figure 7B and Figure 7C As described above, the subject target position determination unit 170 may conversely assume a composition including elements other than the subject, and may determine the coordinates so that the distance of the margins is equal only for the horizontal direction or the vertical direction in the image. For example, the subject target position determination unit 170 may determine the coordinates of the target position so that the distance of the margins (the distance from the end of the subject to the end of the viewing angle) is equal in the horizontal direction and the vertical direction in the direction where the image height of the subject is larger. In the present embodiment, the distances of the margins are approximately equal to achieve a composition in which the subject is located at the center of the shooting angle. However, the distances of the margins at both ends of the subject may be different as long as the subject is within the range in which the composition can be considered to place the subject at the center of the shooting angle. For example, even if the distances of the margins at both ends of the subject are different, as long as these distances are within 5% of the effective area in the horizontal direction, these distances may be considered to be substantially equal.

[0102] The subject target position determination unit 170 may also determine the coordinates of the target position so that the subject area is closer to the end of the angle of view in a direction in which the distance from the end of the subject to the end of the angle of view in the horizontal direction and the vertical direction is larger. Alternatively, the subject target position determination unit 170 may determine the coordinates of the target position so that the subject area is closer to the end of the angle of view in a direction in which the image height of the subject in the horizontal direction and the vertical direction is smaller.

[0103] For example, when Figure 7BWhen capturing an image with the composition shown, the subject is captured so that the long side direction of the subject area is substantially consistent with the horizontal direction of the viewing angle. In addition, the distance from the end of the subject to the end of the viewing angle in the short side direction of the subject area is greater than the distance from the end of the subject to the end of the viewing angle in the long side direction of the subject area. The subject target position determination unit 170 generally sets the distance of the margins in the long side direction relative to the distance of the subject area (for example, from the left end of the subject to the left end of the viewing angle, and from the right end of the subject to the right end of the viewing angle) to be equal. The subject target position determination unit 170 then sets one of the margins in the short side direction (for example, the vertical direction of the viewing angle) relative to the distance of the subject area (for example, from the upper end of the subject to the upper end of the viewing angle, and from the lower end of the subject to the lower end of the viewing angle) to be smaller than the other margin. Of course, the method is not limited thereto, and the subject target position determination unit 170 may determine the coordinates of the target position so that the subject area is not offset in the horizontal direction of the viewing angle, but is vertically offset in one direction in the image.

[0104] In addition, when Figure 7C When capturing an image with the composition shown, the subject is captured so that the long side direction of the subject area is substantially consistent with the vertical direction of the viewing angle. In addition, the distance from the end of the subject to the end of the viewing angle in the short side direction of the subject area is greater than the distance from the end of the subject to the end of the viewing angle in the long side direction of the subject area. The subject target position determination unit 170 generally sets the distance of the margins in the long side direction relative to the subject area (for example, from the upper end of the subject to the upper end of the viewing angle, and from the lower end of the subject to the lower end of the viewing angle) to be equal. The subject target position determination unit 170 also sets one of the margins in the short side direction (for example, the horizontal direction of the viewing angle) relative to the distance of the subject area (for example, from the left end of the subject to the left end of the viewing angle, and from the right end of the subject to the right end of the viewing angle) to be smaller than the other margin. Also in this case, the method is not limited thereto, and the subject target position determination unit 170 may determine the coordinates of the target position so that the subject area is not offset in the vertical direction of the image, but is horizontally offset in one direction in the image.

[0105] In step S909, the subject reference position determination unit 169 determines whether the partial constituent elements of the subject fall within the shooting angle. If the subject reference position determination unit 169 determines that the partial constituent elements of the subject fall within the shooting angle, the sequence moves to step S910, otherwise it moves to step S911. This processing is performed in consideration of a state in which, for example, the partial constituent elements of the subject are photographed at a full shooting angle at a long focal length. In this state, both the situation in which the photographer is following the constituent elements and the situation in which the photographer is not following the constituent elements (because the long focal length means that the subject easily exits the frame) are conceivable, and processing is performed accordingly.

[0106] For example, when the object recognition unit 140 detects a specific partial component of the subject (e.g., cockpit) within the shooting angle, the subject reference position determination unit 169 may determine that the specific partial component falls within the shooting angle. In addition, if the specific partial component is not detected within the shooting angle (even if a partial component of the subject not to be tracked is detected within the shooting angle), the subject reference position determination unit 169 may (also) determine that the partial component of the subject does not fall within the shooting angle.

[0107] In step S910, the camera-side anti-vibration control unit 133 skips tracking correction. This is done because it is assumed that the photographer tracks the partial constituent elements of the subject within the shooting angle in a stable manner even at a shooting angle of a long focal length, and therefore no assistance from tracking control is required. When a "yes" determination is made for a predetermined number of frames in step S909, the camera-side anti-vibration control unit 133 may skip tracking correction.

[0108] Note that if a local constituent element (e.g., a cockpit) determined to be a subject falls within the shooting angle, the subject target position determination unit 170 may further perform processing that takes the margin into consideration. For example, the subject target position determination unit 170 determines whether the distance of the margin in the direction of travel of the subject (e.g., from the left end of the subject to the left end of the viewing angle) is greater than a predetermined value (e.g., 0). This is done to determine whether the subject is being cut off even though the local constituent element is included in the shooting angle. When the distance of the margin (e.g., from the left end of the subject to the left end of the viewing angle) is not greater than the predetermined value, the subject target position determination unit 170 may perform the same processing as in S911. On the other hand, when the distance of the margin is greater than the predetermined value, tracking correction may be skipped.

[0109] In step S911, the subject target position determination unit 170 performs tracking control until the partial constituent elements of the subject come into view at the shooting angle. This is done because it is assumed that the photographer does not track the partial constituent elements of the subject within the shooting angle in a stable manner at a long focal length, and the partial constituent elements of the subject can be appropriately tracked by the processing performed in this step. For example, the subject target position determination unit 170 changes the position of the subject in the direction opposite to the direction of travel of the subject so that the distance of the margin in the direction of travel of the subject (for example, from the left end of the subject to the left end of the angle of view) is at least a predetermined threshold.

[0110] In step S912, the camera-side anti-vibration control unit 133 determines an image blur correction amount based on the hand shake correction amount, the subject blur correction amount, and the coordinates of the target position of the subject to be tracked determined from steps S901 to S911. In step S913, the camera-side anti-vibration control unit 133 and / or the lens-side anti-vibration control unit 126 drives the image sensor 106 and / or the image blur correction lens 102 as described above based on the correction amount and the correction amount for hand shake correction. When the processing of step S913 ends, the camera control unit 115 ends the sequence of operations in this processing.

[0111] As described above, according to the present embodiment, it is determined whether the entirety of the subject is included in the captured image, and when the entirety of the subject is included in the image, the target position for tracking within the image is determined so that the distance of the margin of the subject is equalized by the first processing. If the entire subject is not included in the image, the target position is determined so that the specific constituent elements of the subject are included in the image by the second processing. In other words, the target position for tracking the subject is determined so that the composition is appropriate, depending on whether the entire subject is included in the image. This enables tracking control that provides an appropriate composition intended by the photographer, which enables provision of a high-quality captured image. In other words, the subject that is the tracking target can be appropriately photographed according to the scene. Note that when it is determined in step S907 that the entirety of the subject does not fall within the shooting angle, the angle of view may be expanded (on the wide angle side) so that the entirety of the subject falls within the shooting angle, and thereafter the sequence may move to step S908. The method for expanding the angle of view may be a method of driving the zoom lens 101 to change the focal length of the imaging optical system 150 or a method of changing the crop size of the effective area. When this processing is performed, the whole body of the subject falls within the shooting angle, but the size of the subject image within the effective area automatically changes. In this way, a configuration may be adopted in which, when it is determined that the whole body of the subject does not fall within the shooting angle, the photographer can select whether to expand the angle of view using the operation unit 114. In addition, the configuration may be such that, when the photographer has made a selection to enable expansion of the angle of view, the photographer can also select the degree to which the angle of view is to be expanded. When it is necessary to expand the angle of view beyond the maximum value for expansion so that the whole body of the subject falls within the shooting angle, the sequence may move to step S909.

[0112] Other embodiments

[0113] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0114] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A control device, comprising: a detection unit configured to detect a region of a subject in an image that has been captured; a determining unit configured to determine a target position of the detected object in the image when the detected object is tracked; as well as a control unit configured to change a viewing angle for shooting so that a position of the object in the image becomes the target position, Wherein, when the entire subject is included in the captured image, the determination unit determines the target position through a first process, and when the entire subject is not included in the captured image, the determination unit determines the target position through a second process different from the first process.

2. The control device according to claim 1, in, In the first processing, the determination unit determines the target position so that in a first direction among the horizontal and vertical directions of the field of view, the distance from a first end of the field of view to a first end of the subject is substantially equal to the distance from a second end of the field of view to a second end of the subject.

3. The control device according to claim 2, in, The first direction is a direction in which the image height of the object is greater, among the horizontal direction and the vertical direction of the viewing angle.

4. The control device according to claim 2, in, In the first process, the determination unit determines the target position so that the subject becomes closer to one end of the angle of view in a second direction different from the first direction, of a horizontal direction and a vertical direction of the angle of view.

5. The control device according to claim 1, in, In the second process, the determination unit determines the target position so that at least a specific constituent element of the object is included in the captured image.

6. The control device according to claim 5, in, In the second process, when the specific constituent element of the subject is not included in the captured image, the determination unit determines the target position so that the subject moves in a direction opposite to a traveling direction of the subject.

7. The control device according to claim 5, in, In the second processing, when a specific component of the subject is included in the captured image and the distance from the end of the angle of view corresponding to the moving direction of the subject to the subject is not greater than a predetermined value, the determination unit determines the target position so that the distance becomes greater than the predetermined value.

8. The control device according to claim 5, in, The control unit does not change an angle of view for tracking the subject when the entire subject is not included in the captured image and when a specific component of the subject is included in the captured image.

9. A camera device, comprising: a camera unit configured to capture images; a detection unit configured to detect a region of a subject in an image that has been captured; a determining unit configured to determine a target position of the detected object in the image when the detected object is tracked; as well as a control unit configured to change a viewing angle for shooting so that a position of the object in the image becomes the target position, Wherein, when the entire subject is included in the captured image, the determination unit determines the target position through a first process, and when the entire subject is not included in the captured image, the determination unit determines the target position through a second process different from the first process.

10. The imaging device according to claim 9, in, In the first processing, the determination unit determines the target position so that in a first direction among the horizontal and vertical directions of the field of view, the distance from a first end of the field of view to a first end of the subject is substantially equal to the distance from a second end of the field of view to a second end of the subject.

11. The imaging device according to claim 10, in, The first direction is a direction in which the image height of the object is greater, among the horizontal direction and the vertical direction of the viewing angle.

12. The imaging device according to claim 10, in, In the first process, the determination unit determines the target position so that the subject becomes closer to one end of the angle of view in a second direction different from the first direction, of a horizontal direction and a vertical direction of the angle of view.

13. The imaging device according to claim 9, in, In the second process, the determination unit determines the target position so that at least a specific constituent element of the object is included in the captured image.

14. The imaging device according to claim 13, in, In the second process, when the specific constituent element of the subject is not included in the captured image, the determination unit determines the target position so that the subject moves in a direction opposite to a traveling direction of the subject.

15. The imaging device according to claim 13, in, In the second processing, when a specific component of the subject is included in the captured image and the distance from the end of the angle of view corresponding to the moving direction of the subject to the subject is not greater than a predetermined value, the determination unit determines the target position so that the distance becomes greater than the predetermined value.

16. The imaging device according to claim 13, in, The control unit does not change an angle of view for tracking the subject when the entire subject is not included in the captured image and when a specific component of the subject is included in the captured image.

17. A control method for a control device, the control method comprising: Detecting the area of ​​the subject in the image that has been captured; determining a target position of the detected subject in the image if the detected subject is tracked; as well as changing the viewing angle for shooting so that the position of the subject in the image becomes the target position, In which, when the entire subject is included in the captured image, the target position is determined by a first process in the determination, and when the entire subject is not included in the captured image, the target position is determined by a second process in the determination that is different from the first process.

18. A computer-readable storage medium comprising instructions for performing a control method for controlling a device, the control method comprising: Detecting the area of ​​the subject in the image that has been captured; determining a target position of the detected subject in the image if the detected subject is tracked; as well as changing the viewing angle for shooting so that the position of the subject in the image becomes the target position, In which, when the entire subject is included in the captured image, the target position is determined by a first process in the determination, and when the entire subject is not included in the captured image, the target position is determined by a second process in the determination that is different from the first process.

19. A computer program product comprising instructions for performing a control method for controlling a device, the control method comprising: Detecting the area of ​​the subject in the image that has been captured; determining a target position of the detected subject in the image if the detected subject is tracked; as well as changing the viewing angle for shooting so that the position of the subject in the image becomes the target position, In which, when the entire subject is included in the captured image, the target position is determined by a first process in the determination, and when the entire subject is not included in the captured image, the target position is determined by a second process in the determination that is different from the first process.

Images