Camera device, control method thereof, and storage medium

By calculating and stopping the rotation of the rotating parts in the camera device, the problem of excessive power consumption in panning and tilting functions when image blur correction is not required is solved, thereby reducing power consumption and extending automatic recording time.

CN114902131BActive Publication Date: 2024-11-19CANON KK
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202080090370.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-12-27
Filing Date
2020-12-03
Publication Date
2024-11-19
Estimated Expiration
2040-12-03

AI Technical Summary

Technical Problem

When the pan and tilt cameras are in a state where image blur correction is not required, continuous operation of the pan and tilt functions results in excessive power consumption, reducing the amount of automatic video recording time.

Method used

The camera device includes a camera component, a rotating component, first and second calculation components, and a stopping component. By calculating the amount of rotation and stopping the rotating component when it is below a threshold, power consumption is reduced.

Benefits of technology

In camera equipment using pan and tilt functions for image blur correction, power consumption is significantly reduced, and the automatic recording time of the equipment is extended.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114902131B_ABST
    Figure CN114902131B_ABST
Patent Text Reader

Abstract

The present invention enables to reduce power consumption as much as possible in an image pickup device that performs an image blur correction operation by using a pan / tilt function. The image pickup device is provided with: an image pickup unit for capturing an image of a subject; a rotation unit capable of rotating the image pickup unit; a first calculation unit for calculating a first rotation amount of the rotation unit for correcting image blur caused by shaking of the image pickup device; a second calculation unit for calculating a second rotation amount of the rotation unit for searching for an object from an image captured by the image pickup unit; and a stop unit for stopping the rotation unit when both the first rotation amount and the second rotation amount are less than threshold values ​​set for the first rotation amount and the second rotation amount, respectively.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a camera apparatus comprising a pan and tilt mechanism. Background Art

[0002] In recent years, cameras with pan and tilt functions have become smaller and smaller, making it easier to capture images with the camera attached to the body (wearable) or hand-held. Capturing images with the camera attached to the body or hand-held often results in blurred images. This has led to the development of image blur correction technology, which drives a motor to move the pan and tilt mechanism in a direction that cancels out the image blur. This is not a problem in the presence of a stable power supply, such as in the case of surveillance cameras. However, in the case of a limited total power capacity, such as in the case of a battery-driven camera, running the system for a long time while always driving the pan and tilt functions becomes a problem.

[0003] The technology in Patent Document 1 is a method for operating a wearable lifelogging automatic camera device equipped with a motion sensor, the method including transitioning to a low power consumption state in response to a signal from the motion sensor. With this technology, the device appropriately transitions to a low power consumption state when automatic camera recording is not required, thereby enabling automatic camera recording for a long time.

[0004] In Patent Document 2, a photographing method including searching for a subject using a pan and tilt mechanism is described. The technique described in the photographing method includes changing the center position of the angle of view of a photographed image using a changing unit for changing the center position of the angle of view of a photographed image based on the importance level of the subject calculated for each area to search for the subject. With this technique, since the subject is searched in an efficient manner, the search time and the power consumed by the search can be reduced. This enables automatic photographing for a long time.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application No. 2016-536868

[0008] Patent Document 2: Japanese Patent Application Laid-Open No. 2018-50146 Summary of the invention

[0009] Problem that the invention aims to solve

[0010] However, there is a problem that when the pan and tilt camera is in a state where image blur correction is not required (such as when in a low power consumption state or when no search is performed), continuously running the pan and tilt function results in large power consumption and a reduced amount of time for automatic shooting.

[0011] The present invention has been made in view of the above-mentioned problems, and aims to significantly reduce power consumption in an image pickup apparatus that performs an image blur correction operation using a pan and tilt function.

[0012] Solutions for solving problems

[0013] According to the present invention, a camera device includes: a camera component for capturing an image of a subject; a rotating component capable of rotating the camera component; a first calculating component for calculating a first rotation amount of the rotating component to correct image blur based on the shaking of the camera device; a second calculating component for calculating a second rotation amount of the rotating component to search for a subject from an image captured by the camera component; and a stopping component for stopping the rotating component when the first rotation amount or the second rotation amount is less than a threshold value set for the first rotation amount and the second rotation amount respectively.

[0014] Effects of the Invention

[0015] According to the present invention, it is possible to significantly reduce power consumption in an image pickup apparatus that performs image blur correction using a pan and tilt function.

[0016] Other features and advantages of the present invention will be apparent from the following description in conjunction with the accompanying drawings. Note that throughout the accompanying drawings, the same reference numerals represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0018] Figure 1A is a diagram schematically showing an image pickup apparatus according to an embodiment of the present invention.

[0019] Figure 1B is a diagram schematically showing an image pickup apparatus according to an embodiment of the present invention.

[0020] Figure 2 is a diagram showing an image pickup apparatus of the embodiment.

[0021] Figure 3A is a flowchart for explaining the operation of the image pickup apparatus according to the embodiment.

[0022] Figure 3Bis a flowchart for explaining the operation of the image pickup apparatus according to the embodiment.

[0023] Figure 4 : is a diagram showing a block configuration for calculating the absolute angle of the imaging device and calculating the image blur correction amount.

[0024] Figure 5A is a diagram showing the relationship between an imaging device and coordinate axes.

[0025] Figure 5B is a diagram showing the relationship between an imaging device and coordinate axes.

[0026] Fig. 6A It is a diagram for explaining area division of angle regions in space coordinates of an imaging device.

[0027] Figure 6B It is a diagram for explaining the area division into angle zones in the space coordinates of the imaging device.

[0028] Figure 6C It is a diagram for explaining the area division into angle zones in the space coordinates of the imaging device.

[0029] Fig.6D It is a diagram for explaining the area division into angle zones in the space coordinates of the imaging device.

[0030] Fig. 7A A diagram for explaining region segmentation in a captured image.

[0031] Figure 7B A diagram for explaining region segmentation in a captured image.

[0032] Figure 7C A diagram for explaining region segmentation in a captured image.

[0033] Fig. 8A A diagram for explaining region segmentation in a captured image.

[0034] Figure 8B A diagram for explaining region segmentation in a captured image.

[0035] Fig. 9 It is a diagram for explaining the detection of a person in each area in a captured image.

[0036] Fig.10 It is a diagram for explaining object detection for each area in a captured image.

[0037] Fig.11 It is a diagram for explaining scene detection for each area in a captured image.

[0038] Fig. 12A is a flowchart for explaining speech detection.

[0039] Fig. 12B is a flowchart for explaining speech detection.

[0040] Fig.13A A diagram for explaining motion detection in a captured image.

[0041] Fig. 13B A diagram for explaining motion detection in a captured image.

[0042] Fig. 13C A diagram for explaining motion detection in a captured image.

[0043] Fig.14 This is a diagram for explaining management of the number of captured images for each area.

[0044] Fig.15 This is a diagram for explaining management of the number of captured images of each subject registered through personal authentication.

[0045] Fig.16 This is a diagram for explaining management of the number of captured images of each subject registered by object recognition.

[0046] Fig.17 This is a diagram for explaining management of the number of captured images for each scene.

[0047] Fig.18 This is a flowchart for explaining calculation using the search elapsed time.

[0048] Fig.19A It is a diagram for explaining the calculation of the search target angle for each scene.

[0049] Fig.19B It is a diagram for explaining the calculation of the search target angle for each scene.

[0050] Fig. 20A is a diagram showing an example of the position in an image of a subject detected by a camera, and a method of setting a threshold value and a second rotation correction amount for tracking and searching.

[0051] Fig. 20B is a diagram showing an example of the position in an image of a subject detected by a camera, and a method of setting a threshold value and a second rotation correction amount for tracking and searching.

[0052] Fig.21 : is a diagram showing an example of a time-series change in the amount of vibration detected by a camera, and a method of setting the first and second thresholds for image blur correction and the first rotation correction amount. DETAILED DESCRIPTION

[0053] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but the invention is not limited to the invention requiring all of these features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar structures, and redundant descriptions thereof are omitted.

[0054] Camera structure

[0055] Figure 1A and Figure 1B is a diagram schematically showing an image pickup apparatus according to an embodiment of the present invention.

[0056] Figure 1A The imaging device 101 shown is provided with an operating member capable of operating a power switch (hereinafter referred to as a power button, but may be an operation such as tapping, flicking, or sliding on a touch panel). A lens barrel 102, which is a housing including an imaging lens group and an image sensor for capturing an image of a subject, is attached to the imaging device 101, and a rotation mechanism is provided that enables the lens barrel 102 to rotate relative to a fixing portion 103. A pitch rotation unit 104 is a unit capable of rotating the lens barrel 102 in a certain direction. Figure 1B The motor drive mechanism 105 is a motor drive mechanism that can rotate the lens barrel 102 in the pitch direction shown in the figure, and the pan rotation unit 105 is a motor drive mechanism that can rotate the lens barrel 102 in the yaw direction. In this way, the lens barrel 102 can rotate around one or more axes. Note that Figure 1B , the axes are defined with the origin being the position of the fixed portion 103. An angular velocity meter 106 and an accelerometer 107 are mounted on the fixed portion 103 of the image pickup device 101. The vibration of the image pickup device 101 is detected based on the angular velocity meter 106 and the accelerometer 107, and the pitch rotation unit and the pan rotation unit are rotationally driven based on the detected shake angle. With this structure, the shake in the lens barrel 102 as the movable portion is corrected and the tilt is corrected.

[0057] Figure 2 : is a block diagram showing the structure of the image pickup apparatus of this embodiment.

[0058] like Figure 2As shown, the control unit 215 includes a processor (e.g., a CPU, a GPU, a microprocessor, or an MPU, etc.) and a memory (e.g., a DRAM or an SRAM, etc.). With this structure, various processes can be performed to control each block of the imaging device 101, and data transfer between each block can be controlled. The non-volatile memory (EEPROM) 216 is an electrically erasable and recordable memory, and stores constants and programs used for the operation of the control unit 215, etc.

[0059] like Figure 2 As shown, the zoom unit 201 includes a zoom lens for performing magnification change. The zoom drive control unit 202 controls the drive of the zoom unit 201. The focus unit 203 includes a lens for performing focus adjustment. The focus drive control unit 204 controls the drive of the focus unit 203.

[0060] The imaging unit 205 includes an image sensor and an A / D converter. The image sensor receives light incident through a lens group, and outputs information related to the charge corresponding to the amount of received light to the image processing unit 206 as an analog image signal. The image processing unit 206 is an arithmetic circuit equipped with a plurality of arithmetic and logic units (ALUs), and applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to the digital image data output after A / D conversion, and outputs the processed digital image data. The digital image data output from the image processing unit 206 is converted into a recording format such as a JPEG format, and then recorded on a recording medium such as a nonvolatile memory 216 by the image recording unit 207. In addition, the digital image data output from the image processing unit 206 is converted into a format that can be displayed externally and sent to an external display device.

[0061] The lens barrel rotation driving unit 212 drives the tilt rotation unit 104 and the pan rotation unit 105 , thereby driving the lens barrel 102 to rotate by a required amount in the tilt direction and the pan direction.

[0062] The device shake detection unit 208 is equipped with, for example, an angular velocity meter (gyro sensor) 106 for detecting the angular velocity in the three-axis direction of the camera 101 and an accelerometer (acceleration sensor) 107 for detecting the acceleration in the three-axis direction of the device. The device shake detection unit 208 calculates the rotation angle of the device, the offset of the device, etc. based on the detected signals.

[0063] The device movement detection unit 209 detects movement of the imaging device 101 using position information from a global positioning system (GPS) or using output from the acceleration sensor 107. GPS may be provided on the imaging device 101, or position information detected by an external GPS detection device may be sent to the imaging device 101.

[0064] The displacement of the camera 101 is detected by the device shake detection unit 208. However, due to the characteristics of the acceleration sensor, it is difficult to detect large movements. Therefore, movement within a small range is detected by the device shake detection unit 208, and large movement is detected by the device movement detection unit 209.

[0065] The voice input unit 210 acquires an audio signal from the periphery of the imaging apparatus 101 from a microphone provided on the imaging apparatus 101. The microphone may be provided on the imaging apparatus 101, or an audio signal detected by an external microphone device may be sent to the imaging apparatus 101.

[0066] The operation unit 211 is used to operate the imaging apparatus 101, and is provided with a power button and a button for changing the settings of the camera. When the power button is operated, power is supplied to the entire system according to the intended use, and the camera is activated. The communication unit 213 communicates with an external device (smartphone, etc.) not shown via wired or wireless communication.

[0067] Operation of camera equipment

[0068] Figure 3A and Figure 3B is a flowchart for explaining the operation of the image pickup apparatus according to the present embodiment.

[0069] Figure 3A and Figure 3B The operation shown is started by turning on the main power of the imaging apparatus 101, and automatic subject search or automatic imaging is performed. In addition, in response to a user operation command input by voice from the voice input unit 210 and a user instruction input by confirming a composition on an external device (not shown) via the communication unit 213, manual pan and tilt driving, tracking and searching for an instructed subject, and manual imaging at an instructed timing can be performed.

[0070] When the power button of the operation unit 211 is operated, first, in step S301, the control unit 215 performs various initial settings (for example, imaging or automatic search). When the initial settings are completed and image information can be acquired from the image sensor, the process proceeds to step S302.

[0071] In step S302, the image processing unit 206 performs image processing for subject detection on the image signal captured by the camera unit 205. Using the processed image, person detection, object detection, and other similar subject detections are performed. In the case of person detection, the face or body of the subject is detected. In the face detection process, a pattern for judging the face of a person is predetermined, and a section included in the captured image that is consistent with the pattern can be detected as the face area of ​​the person. In addition, the reliability indicating the certainty that the section is the face of the subject can be calculated at the same time. The reliability is calculated based on the size of the face area in the image and / or the degree of consistency with the face pattern, etc. In the case of object recognition, an object consistent with a pre-registered pattern can be recognized in the same manner. In addition, there is a method of extracting a feature subject by using a method such as a histogram of hue or saturation in a captured image. With respect to an image of a subject captured in the angle of view of a captured image, a distribution derived from, for example, a histogram of hue or saturation in the captured image is divided into a plurality of sections, and a process for classifying the captured image for each section is performed. For example, a histogram of multiple color components is generated for a captured image, the image is segmented by a mountain-shaped distribution range, the captured image is classified by a combination of regions belonging to the same partition, and the image region of the subject is identified. An evaluation value is calculated for each image region of the identified subject, so that the image region of the subject with the highest evaluation value can be determined as the main subject region. The above method can be used to obtain each subject information from the captured image information. After the captured image information such as the subject detection information is acquired in step S302, the process proceeds to step S303.

[0072] In step S303, the angular velocity information in the device shake detection unit 208 is acquired, and the process proceeds to step S304. In step S304, the acceleration information in the device shake detection unit 208 is acquired, and the process proceeds to step S305. In step S305, the absolute angle of the camera 101 is calculated based on the detected acceleration information and angular velocity information, and the process proceeds to step S306.

[0073] In step S306 , an image blur correction amount is calculated for preventing or minimizing image blur caused in a state where the imaging apparatus 101 is hand-held or attached to a person's body.

[0074] Now refer to Figure 4 The block diagram shown and Figure 5A and 5B The diagram shown is used to explain a method of calculating the absolute angle of the imaging device in step S305 and a method of calculating the image blur correction amount in step S306.

[0075] like Figure 1AAs shown in FIG. 1 , the angular velocity meter 106 and the accelerometer 107 are arranged on the fixed portion 103 of the image pickup device 101. Therefore, since information about the angular velocity and acceleration at the position of the fixed portion 103 is output, the absolute angle of the image pickup device calculated based on such information is also calculated as the absolute angle at the position of the fixed portion 103. Therefore, in order to correct the rotational shake of the lens barrel 102 (shake of the image sensor), the correction angle is calculated based on the angular velocity information at the position of the fixed portion 103, thereby calculating the image blur correction amount. Then, the pitch rotation unit 104 and the pan rotation unit 105 are driven based on the image blur correction amount to correct the blur.

[0076] Figure 4 : is a diagram showing a block configuration for calculating the absolute angle of the imaging device and calculating the image blur correction amount.

[0077] A method of calculating the absolute angle of the imaging device will be described below.

[0078] The output of the angular velocity meter 106 (pitch direction, yaw direction, and roll direction) and the output of the accelerometer 107 (X axis, Y axis, and Z axis) are respectively input to the pitch absolute angle calculation unit 401, the yaw absolute angle calculation unit 402, and the roll absolute angle calculation unit 403. Then, each absolute angle in the pitch direction, the yaw direction, and the roll direction is calculated, thereby calculating the absolute angle at the position of the angular velocity meter 106 and the accelerometer 107 (in other words, the camera absolute angle at the fixing part 103).

[0079] First, using the gravitational acceleration, the absolute angles of the camera device in the roll direction, the pitch direction, and the yaw direction can be calculated based on the relationship between the outputs of the axes of the accelerometer 107, thereby calculating the acceleration-operated pitch absolute angle, the acceleration-operated yaw absolute angle, and the acceleration-operated roll absolute angle. However, the tilting angle can be accurately detected only when the device remains stationary and is not affected by the acceleration from the outside (in other words, only when the gravitational acceleration dominates the acceleration detected by the accelerometer). For example, consider a shooting operation performed by an operator holding a camera by hand during walking, a shooting operation in a state where the camera is fixedly attached to a part of the human body, or a shooting operation in a state where the camera is attached to a vehicle such as a car or a motorcycle. In these cases, the influence of accelerations other than the gravitational acceleration becomes large. Therefore, it is difficult to perform accurate absolute angle calculation. In addition, in the case of estimating the camera angle only by the angular velocity meter 106, the posture and angle can also be estimated by integral calculation of the angular velocity output, but since the error caused by the integral is also accumulated, it is also difficult to perform accurate absolute angle calculation.

[0080] Therefore, the noise area that is not properly processed by the angular velocity meter 106 and the accelerometer 107 is removed, and the signal is synthesized using sensor fusion to calculate the absolute angle. For example, a Kalman filter or a complementary filter is used to perform sensor fusion. The low-frequency noise that has the greatest influence on the integral error of the angular velocity meter 106 is removed, and the high-frequency noise that has the greatest influence on the operation error caused by the interference in the accelerometer 107 is removed. Then, the signal is synthesized. In this way, the absolute angle can be accurately calculated with the noise removed.

[0081] Therefore, the pitch absolute angle calculation 401 is performed based on the pitch angular velocity of the gyro from the angular velocity meter 106 and the sensor fusion of the acceleration to calculate the pitch absolute angle, thereby calculating the pitch absolute angle. In addition, the yaw absolute angle calculation 402 is performed based on the yaw angular velocity of the gyro from the angular velocity meter 106 and the sensor fusion of the acceleration to calculate the yaw absolute angle, thereby calculating the yaw absolute angle. In addition, the roll absolute angle calculation 403 is performed based on the roll angular velocity of the gyro from the angular velocity meter 106 and the sensor fusion of the acceleration to calculate the roll absolute angle, thereby calculating the roll absolute angle.

[0082] In addition, in the case where the absolute angle of the camera device cannot be detected from the accelerometer 107 (for example, in Figure 5B The absolute angle is calculated based on the angular velocity meter 106 only in the yaw rotation axis as the axis used for rotation about the Y axis when the Y direction in the image pickup device is completely consistent with the gravity direction. The last absolute angle obtained when it is determined that the absolute angle can be calculated based on the angle change of the image pickup device is set as the start, and the absolute angle is calculated using the angular velocity integral in the time period when the absolute angle cannot be calculated. The absolute angle of the image pickup device can be calculated according to the above method.

[0083] Next, a method of calculating the image blur correction amount will be described.

[0084] Since the absolute angle of the camera device can be calculated using the absolute angle calculation method of the camera device described above, image blur correction can be performed by rotationally driving the pitch rotation unit 104 and the pan rotation unit 105 based on the signal representing the absolute angle. However, in the case of image blur correction control based on the absolute angle of the camera device, control is performed to continuously maintain the same posture. This creates problems such as the composition not changing when the operator captures the image while moving or image blur correction control beyond the movable end of the correction unit cannot be performed. Therefore, it is best to perform image blur correction control only on the high-frequency component without performing blur correction on the low-frequency component. In the calculation of the image blur correction amount, since blur correction control is not performed for the low-frequency component, only the angular velocity meter 106 is used to calculate the correction amount.

[0085] The output of the angular velocity meter 106 is integrated after its DC component is cut off by a high-pass filter (HPF), and then converted into an angle signal, thereby calculating the blur correction angle. The angular velocity meter 106 is arranged on the fixed part 103, and the rotation axis of the angular velocity meter 106 in the yaw direction coincides with the rotation axis of the pan rotation unit 105. Therefore, the pan blur correction angle calculation unit 405 calculates the blur correction angle in the pan direction (yaw direction) by performing angle conversion on the yaw angular velocity output from the angular velocity meter 106 using the HPF and integration, and rotationally drives the pan rotation unit 105. In this way, blur correction in the pan direction is performed. Regarding the pitch direction, since the angular velocity meter 106 is arranged on the fixed part 103, the control for performing blur correction in the pitch direction varies according to the rotation angle of the pan rotation unit 105. It is assumed that at the position of the angular velocity meter 106 (in other words, at the position of the fixed part 103), a signal such as Figure 1B The angular velocity in the direction of the axis defined by the diagram. Figure 5A , when the X-axis is always located in a direction perpendicular to the optical axis), the pitch blur correction angle calculated by the pitch blur correction angle calculation unit 406 is directly calculated as the pitch blur correction angle. Figure 5B In the case of (when the Z axis is always located in a direction perpendicular to the optical axis), the roll blur correction angle calculated by the roll blur correction angle calculation unit 407 is directly calculated as the pitch blur correction angle. Between these two cases, based on the pan rotation angle, the pitch blur correction angle at the pan angle rotation position is calculated from the pitch blur correction angle and the roll blur correction angle. The pitch blur correction angle corresponding to the pan angle rotation position can be calculated by the following formula (1).

[0086] θtl=θpi×cosθap+θri×sinθap...(1)

[0087] (θtl: pitch blur correction angle, θpi: pitch blur correction angle, θri: roll blur correction angle, and θap: pan rotation angle)

[0088] In this manner, the pitch blur correction angle is calculated by the pitch blur correction angle calculation unit 404 according to the pan rotation position.

[0089] In equation (1), the pitch blur correction angle is calculated by converting the pitch angle and the roll angle. However, the pitch blur correction angle may be calculated by calculating the pitch angular velocity from the pan rotation angle and the pitch angular velocity and the roll angular velocity from the angular velocity meter 106, and performing angle conversion on the pitch angular velocity using HPF and integration processing.

[0090] In the above method, the pan blur correction angle and the pitch blur correction angle can be calculated, and image blur correction can be performed by rotationally driving the tilt rotation unit 104 and the pan rotation unit 105 according to the respective blur correction angles (image blur correction amounts).

[0091] Furthermore, the absolute angle of the lens barrel 102 can also be calculated based on the absolute angle of the imaging device obtained above and the rotational positions of the tilt rotation unit 104 and the pan rotation unit 105. More specifically, the optical axis-based camera angle in the yaw direction (the yaw absolute angle of the lens barrel 102) can be calculated by subtracting the rotational angle of the pan rotation unit 105 from the yaw absolute angle output from the yaw absolute angle calculation unit 402.

[0092] Furthermore, the lens barrel rotation angle in the pitch direction and the lens barrel rotation angle in the roll direction, which are converted into the position of the fixing portion 103, can be calculated from the rotation position of the pan rotation unit 105 and the rotation position of the tilt rotation unit 104. Therefore, the camera angle based on the optical axis in the pitch direction (the absolute tilt angle of the lens barrel 102) can be calculated by subtracting the lens barrel rotation angle in the pitch direction from the absolute tilt angle output from the absolute tilt angle calculation unit 401. Furthermore, the camera angle based on the optical axis in the roll direction (the absolute roll angle of the lens barrel 102) can be calculated by subtracting the lens barrel rotation angle in the roll direction from the absolute roll angle output from the absolute roll angle calculation unit 403.

[0093] In this way, if the angle of the imaging device based on the optical axis can be calculated, it is possible to determine, for example, which angular direction the lens barrel (camera optical axis) is currently pointing to based on the angle obtained when the imaging device is activated.

[0094] In the case where the absolute angle of the imaging apparatus and the image blur correction amount are calculated according to the above method, the process proceeds to step S307, in which camera movement detection is performed. The imaging apparatus 101 detects whether there is a large movement via the GPS position information or the acceleration information in the apparatus movement detection unit 209. Alternatively, in a structure having an external device capable of acquiring GPS position information, and the imaging apparatus 101 and the external device being able to communicate with each other, in the case where the GPS position information has changed, it is possible to detect whether the imaging apparatus 101 has moved by receiving movement information sent from the external device to the camera 101.

[0095] In step S308, the camera state is determined. Based on the camera angle or camera movement amount detected from the angular velocity information, acceleration information and / or GPS position information, it is determined what kind of vibration or movement state the camera is currently in. For example, in the case of taking pictures using the camera device 101 installed on a vehicle, the subject information such as the surrounding scenery changes greatly depending on the moving distance. Therefore, it is determined whether the device is in a "vehicle moving state" (the device is moving at a high speed in a state such as being installed on a vehicle), and this can be used in the automatic subject search described below. In addition, it is determined whether the change in the camera angle is large, and it is determined whether the camera device 101 is in a "placed shooting state" with a small shake angle. Since when the device is in the "placed shooting state", it can be considered that the angle of the camera device 101 itself has not changed, so a subject search for placed shooting can be performed. In addition, when the change in the camera angle is relatively large, it can be determined as a "handheld state", and a subject search for handheld shooting can be performed. The following describes the subject search method in each state.

[0096] In step S309, it is determined whether the camera angle detected in step S305 is accurately calculated, and in the case where it is difficult to detect the absolute angle of the imaging device, it is determined that detection is not possible, and the process proceeds to step S310.

[0097] In step S310, the automatic search process is initialized, and the processes described below such as initialization of the importance level of the subject are performed. If the absolute angle of the camera device can be detected, the process proceeds to step S311. The state in which the absolute angle of the camera device cannot be detected includes: a case in which the impact on the camera is very large and the calculation of the camera angle based on the output of the accelerometer fails, and a case in which the camera angular velocity is very large and exceeds the detectable range of the accelerometer. When such a state is detected, it is determined that the absolute angle of the camera device cannot be detected.

[0098] In step S311, region division based on the absolute angle of the imaging apparatus 101 obtained at the time of initial setting in step S301 or initialization of the automatic subject search in step S310 is performed. Then, block division of the current image acquired by the imaging apparatus 101 is performed based on the region division.

[0099] refer to FIG. 6A to FIG. 6D To illustrate the region segmentation. Fig. 6A As shown in FIG. 1 , the entire spherical surface is segmented with the camera position as the origin (the origin O is set as the camera position). Fig. 6A In the example shown, the pitch direction and the pan direction are divided into 22.5 degrees each. Fig. 6A When segmentation is performed as shown in FIG. 1 , as the angle in the pitch direction moves away from 0 degrees, the circumference in the horizontal direction becomes smaller and the area region becomes smaller. Figure 6B As shown, in the case where the pitch angle is 45 degrees or more, the area zone in the horizontal direction is set to be larger than 22.5 degrees.

[0100] Figure 6C and 6D 6 shows an example of performing region division within a shooting field angle. Axis 601 represents the direction of the image pickup device 101 at the time of initialization, and region division is performed using the direction angle as a reference position. Region 602 represents the field angle region of the captured image, and Fig.6D An example of the image captured at this time is shown in FIG. Based on the area segmentation, the image captured within the field angle is divided into Fig.6D Image partitions 603 to 618 are shown.

[0101] Fig. 7A , Figure 7B and Figure 7C 601 obtained at the time of initialization. These are area divisions based on the absolute angle of the imaging device 101 at the time of initialization of the automatic subject search. Figure 7B , a captured image obtained when the pitch angle is 0 degrees is shown, wherein the camera capture field angle is represented by 701 and the field angle center is represented by 702. Figure 7C , a captured image obtained when the pitch angle is 55 degrees is shown, wherein the camera capturing field angle is represented by 703 and the field angle center is represented by 704 .

[0102] Although the size difference between the regions is small when the pitch angle is 0 degrees because the angle range in the lateral (horizontal) direction is not much different, when the pitch angle is 55 degrees, the angle range in the lateral direction changes greatly depending on the angle. Therefore, when the pitch angle is 45 degrees or more, the region range in the horizontal direction is set to be greater than 22.5 degrees, thereby preventing the region range from becoming too small as the pitch angle increases.

[0103] Fig. 8A and Figure 8B The area obtained when the camera is rotated 11.25 degrees in the pan direction from the camera initial position is shown, and Figure 8B , a captured image obtained when the pitch angle is 0 degrees is shown, wherein the camera capture field angle is represented by 801 and the field angle center is represented by 802. Fig. 8A As shown, the area is set with 11.25 degrees as the center in the horizontal direction.

[0104] The above-mentioned area segmentation within the shooting angle is obtained by the following equations (2) and (3), and all the segmentation areas existing within the shooting angle are calculated, thus: Figure 7B or Figure 7C or Figure 8B As shown, region segmentation is performed in the image.

[0105] θay=θy+θy'...(2)

[0106] θax=θx'×cosθay...(3)

[0107] θx: pan angle (based on the initial camera position),

[0108] θy: pitch angle (based on the initial camera position),

[0109] θx': The angle from the pan angle position (the center of the image in the horizontal direction) to the center of the region segmentation, θy': The angle from the pitch angle position (the center of the image in the vertical direction) to the center of the region segmentation,

[0110] θax: The length of the horizontal angle from the horizontal center of the image to the horizontal region segmentation center,

[0111] θay: the length of the vertical angle from the vertical center of the image to the center of the vertical region segmentation, and

[0112] The initial position of the camera is based on 0 degrees.

[0113] Although each area division angle is set to 22.5 degrees, the horizontal area division angle within the range of 45 degrees to 67.5 degrees in the vertical direction is set to 45 degrees. Within the range of 67.5 degrees to 90 degrees in the vertical direction, no horizontal division is performed, thereby setting only one area.

[0114] In this manner, after the region division is performed in step S311 , the process proceeds to step S312 , where the importance level of the subject is calculated.

[0115] In the following, for example, based on Fig.6D The importance level of the subject is calculated for each area by acquiring the subject information in the image such as the image shown in the figure. Fig.6D In the areas 603, 604, 605, 606, 607, 610, 611, 614, 615, 616, 617, and 618, etc. shown in the figure, the importance level of the subject is not determined or updated (for example, when the size of the area is assumed to be 100%, when the captured image area of ​​the area occupies 50% or less). In each area set as described above, the importance level of the subject is set under various conditions. The importance level of the subject is calculated according to, for example, the following items.

[0116] Importance level setting corresponding to person information

[0117] For each area position within the scene angle, the importance level is set according to the character information.

[0118] Known face detection techniques include methods using knowledge related to the face (skin color information, parts such as eyes, nose and mouth, etc.), and methods using recognition devices for face detection via learning algorithms represented by neural networks, etc. In addition, in order to improve detection accuracy, a combination of face detection methods is generally used in face detection. According to face detection, the size of the face, the orientation of the face, and the reliability of the certainty of the face are calculated. In addition, there is a known technology for detecting facial expressions from information related to the detected facial organs, so that the degree of eye opening can be detected or the degree of smile can be detected. More specifically, feature values ​​required for detecting facial expressions are obtained from, for example, the contours of facial organs (e.g., eyes, nose and mouth) and the positions of the inner corners of the eyes, the outer corners of the eyes, the nostrils, the corners of the mouth and the lips. Methods for obtaining feature values ​​include, for example, a method for performing template matching using a template of a facial component and a method for performing a learning algorithm using a large number of sample images of a facial component. For example, the degree of smile, eye closure, blinking, facial expression estimation (happiness, surprise, anger, sadness and expressionlessness) and gestures can be detected based on the detected feature values.

[0119] In addition, personal facial data can be pre-registered, and person facial authentication can also be performed, which is used to further detect whether the person is a specific person based on the detected face. Determine whether the detected subject is in the same state as the object pre-registered in the database, or determine the degree of consistency. Extract feature information for identifying the subject area as the object to be authenticated from the image data of the detected subject. Then, compare the extracted feature information with the feature information of the image of the subject pre-registered in the database. Authentication is performed using an authentication evaluation value representing the similarity obtained by comparison to determine which registered object is the object to be authenticated, or whether there is no corresponding registered object. For example, a predetermined threshold is set for the authentication evaluation value, and when the authentication evaluation value is equal to or greater than the predetermined threshold, it can be determined that the object to be authenticated is the corresponding object registered in the database.

[0120] Kr is a value indicating the level of face reliability (for example, Kr increases as reliability increases from a low level to a high level), and reliability is judged using factors such as the size of the face, the orientation of the face, and the certainty of the face.

[0121] Ka is a value indicating the level of personal face authentication information, and is set for each face based on the importance level of each registered person's face (for example, a pre-registered importance level) or past imaging information described below.

[0122] Ks is a value indicating the level of facial expression, and is expressed by a level for each face based on a level corresponding to a preset facial expression (e.g., smile, happiness, and surprise are set to a high level, and anger, sadness, expressionless face, and closed eyes are set to a low level). Here, the level of each facial expression may be configured to vary according to the degree of facial expression (e.g., the degree of smile in the case of smile).

[0123] Based on the Kr, Ka, and Ks obtained above, the level Flvl corresponding to the facial expression of the person is obtained by the following formula (4).

[0124] Flvl=Kr×Ka×Ks...(4)

[0125] Now refer to Fig. 9 As described above, a small area captured in an image (such as Fig. 9Areas 901, 904, 905, and 908, etc. in the image are not considered to be areas to be searched, and therefore are not judged. Here, a case where images of three persons (909, 910, and 911) are captured within the field angle is described as an example. Subject 909 in this example is a subject that has not been registered for personal facial authentication, and is a subject whose facial expression is not smiling. Subject 910 in this example is a subject that has not been registered for personal facial authentication, and is a subject whose facial expression has a smile. Subject 911 in this example is a subject that has been registered for personal facial authentication, and is a subject whose facial expression has a smile.

[0126] Since the image of person 909 is captured in areas 902 and 906, both areas 902 and 906 are level-set according to the person information related to person 909. Since person 909 is not registered for personal face authentication, gain Ka is "1", and since person 909 does not have a smile, gain Ks is "1", the level of person 909 is Kr, and the importance level of subject 909 is determined according to the face reliability.

[0127] Since images of persons 910 and 911 are captured in areas 903 and 907, both areas 903 and 907 are level-set according to person information related to persons 910 and 911. Since person 910 is not registered for personal face authentication, gain Ka is "1", and since person 910 is a subject with a smile, gain Ks is set to a value equal to or greater than "1". Since person 911 is registered for personal face authentication, gain Ka is 1 or greater, and since person 911 is a subject with a smile, gain Ks is set to a value equal to or greater than "1".

[0128] Therefore, when the levels of face reliability of persons 909 , 910 , and 911 are the same, the magnitude of the importance levels of the objects corresponds to 911 > 910 > 909 .

[0129] In addition, the level setting is performed with the ratio (rate) of the photographed face. In the area where the ratio of the facial image is large, the level obtained above is directly set, and in the area where the ratio of the facial image is small, the level is changed according to the ratio of the facial image. For example, in the case where the ratio of the facial images in area 903 and area 907 is 8:2, their levels are set to 10:2.5 respectively.

[0130] In this way, the importance level is set for each area based on the person information.

[0131] Importance level setting corresponding to object recognition information

[0132] When a pre-registered subject is detected, the importance level is set based on the object recognition information. For example, the importance level using object recognition is set using general object category recognition such as "dog" or "cat" or the degree of coincidence with the pre-registered subject image. Object recognition includes, for example, a method of configuring a discriminator for "dog" or "cat" etc. based on a learning algorithm represented by a neural network.

[0133] Now refer to Fig.10 The level setting corresponding to the pre-registered object recognition information is described. Here, a case where an image of three subjects (1009, 1010, and 1011) is captured on the screen is described as an example. Subject 1009 is judged as a dog, and subject 1010 is judged as a cat by object recognition. Subject 1011 is a person, and since the person is judged in the "importance level setting corresponding to person information", the person is a subject excluded from the objects of the level setting corresponding to the object recognition information. The level obtained in the case where "dog" is recognized and the level obtained in the case where "cat" is recognized are registered in advance, and the dog is an important subject, for example. In the case where "cat" is not defined as an important subject, the level of the area 1007 where "dog" is detected is set to be higher than the level of the area 1006 where "cat" is detected. In addition, the reliability of object recognition can be obtained, and the level can be changed according to the detected reliability.

[0134] Here, the level setting is also performed according to the proportion of the photographed object. In the area where the proportion of the subject image is large, the importance level obtained above is directly set, and in the area where the proportion of the subject image is small, the level is changed according to the proportion.

[0135] Importance level settings corresponding to the scenario

[0136] Judgments such as "blue sky scene", "natural green scene" and "evening scene" are made by analyzing image data, and level setting is performed based on such scene judgment information. In addition, regarding scene judgment related to the sky, since information related to the tilt of the camera can be obtained from the absolute angle information of the camera, judgments related to the sky, such as "blue sky scene" and "evening scene", are made in images existing in an area facing upward relative to a direction perpendicular to the direction of gravity.

[0137] First, the 12-bit RAW data of one frame captured is divided into blocks consisting of n×m (n and m are integers), and the average value of pixels of each color R, G, and B in each divided area is calculated. The R, G, and B average values ​​of each block are subjected to temporary development processing including white balance correction processing, gamma correction processing, and color conversion matrix processing.

[0138] The judgment of "blue sky scene" is performed by calculating the proportion of blue sky blocks in the upper area of ​​the picture. The judgment of whether it is a blue sky block is performed by defining a blue sky judgment area in the UV color space and counting the blocks in the blue sky judgment area.

[0139] The "evening scene" is determined by calculating the proportion of evening blocks in the upper area of ​​the screen. The evening scene determination area is defined in the UV color space and the blocks in the evening scene determination area are counted to determine whether it is an evening scene block.

[0140] The "natural emerald green scene" is determined by detecting the ratio of natural emerald green blocks to all blocks in the picture. The natural emerald green determination area is defined in the UV color space, and the blocks in the natural emerald green determination area are counted to determine whether it is a natural emerald green block.

[0141] Now refer to Fig.11 The setting of the importance level corresponding to the scene detection information is explained.

[0142] The scene in this example shows a blue sky in the upper right area of ​​the screen, a building in the left area of ​​the screen, and natural verdant green in the lower right area of ​​the screen. "Blue sky scene" and "natural verdant green scene" are detected by performing scene judgment in the screen, and a level corresponding to blue sky area recognition is set in area 1103 and a level corresponding to natural verdant green area recognition is set in area 1107. With respect to area 1102, since about 40% of the area is a blue sky area, the level corresponding to blue sky area recognition is set to 40%. With respect to area 1106, since about 30% of the area is a natural verdant green area, the level corresponding to natural verdant green area recognition is set to 30%.

[0143] Although the scene determination method using color space information is described above, a method of determining a scene using brightness values ​​may also be used. Using the "night scene" determination as an example, the following situation is determined to correspond to a night scene: only subjects with extremely high brightness levels and extremely low brightness levels are distributed in the histogram of the entire screen. In addition, point light sources may be determined using contrast evaluation based on high-frequency components of the brightness signal of the image, and the "night scene" may be determined using the brightness distribution and the detection results of the point light sources.

[0144] The respective importance levels of "blue sky scene", "evening scene", "natural green scene" and "night scene" are registered in advance, and the importance level corresponding to the scene is set for each area according to the registered importance levels.

[0145] Importance level setting corresponding to voice information

[0146] Judgments such as "voice direction", "sound level (voice level)" and "voice recognition" are performed by analyzing voice information data, and setting of importance levels based on voice information is performed.

[0147] Now refer to Fig. 12A and Fig. 12B The setting of the importance level corresponding to the voice information is explained with reference to the flowchart of FIG.

[0148] In step S1201, the voice acquisition unit of the control unit 215 determines whether the audio of the voice from the outside is acquired. If the voice is acquired, the process proceeds to step S1202. If the voice is not acquired, the process returns to step S1201.

[0149] In step S1202, the voice direction detection unit of the control unit 215 detects the source direction of the acquired voice. In step S1203, the sound level is detected. In step S1204, the voice recognition unit of the control unit 215 recognizes the acquired voice.

[0150] In step S1205, it is determined whether the recognized voice is a specific voice command. If the recognized voice is a voice command, the process proceeds to step S1206 and the voice recognition level Sc is set to Ac1. If the recognized voice is not a voice command, the process proceeds to step S1207 and the voice recognition level Sc is set to 0.

[0151] In step S1208, it is determined whether the detected sound level is equal to or greater than a predetermined value. If the sound level is equal to or greater than the predetermined value, the process proceeds to step S1209, and the voice level Ss is set to Ac2. If the sound level is less than the predetermined value, the process proceeds to step S1210, and the voice level Ss is set to 0.

[0152] In step S1211, the direction of the voice source detected in step S1202 is calculated. FIG. 6A to FIG. 6D Corresponding sound direction area. For example, considering the voice direction recognition error, all areas within ±45 degrees from the determined direction angle are determined as the sound direction area.

[0153] Next, in step S1212, the speech recognition level Sc and the speech level Ss are added together to calculate the total sound level S1, and the process proceeds to step S1213.

[0154] In step S1213, it is determined whether the total sound level S1 has changed to a value greater than the total sound level S1 obtained at the time of the previous sampling. In the case where the value has changed to a larger value, the process proceeds to step S1214, the time count Xt is set to a predetermined time parameter Ta, and the process proceeds to step S1215. In step S1215, the level S1 is set to the area set in step S1211, and the process returns to step S1201.

[0155] In a case where it is determined in step S1213 that the total sound level S1 has changed to a value equal to or less than the total sound level S1 obtained at the previous sampling time, the process advances to step S1216, the predetermined time parameter Ta is decremented, and the process advances to step S1217.

[0156] In step S1217, it is determined whether the time Ta is equal to or less than 0. In the case where the time Ta is equal to or less than 0 (a case where Ta has passed since the total sound level S1 was changed to a smaller value), in step S1218, the total sound level S1 is set to 0, and the total sound level S1 of the area set in step S1211 is set to 0. In step S1217, in the case where the time Ta is greater than 0, the process proceeds to step S1215, and the total sound level S1 calculated so far is set as the importance level corresponding to the voice information in the area of ​​the voice direction.

[0157] Importance level setting corresponding to image motion information

[0158] By judging in FIG. 6A to FIG. 6D The importance level corresponding to the image motion information is set based on whether or not there is a moving subject in each area obtained by the area division performed as shown.

[0159] The difference between frames or the motion vector between frames is detected for each region. The method of detecting the motion vector includes: dividing the screen into a plurality of regions; comparing the image of the previous frame and the current image (two consecutive images) stored in advance with each other; and calculating the image motion amount based on the information related to the relative offset in the image.

[0160] Here, since the angle of the lens barrel (in the optical axis direction of the imaging surface) can be obtained based on the difference between the absolute angle of the imaging device and the rotation positions of the pitch rotation unit 104 and the pan rotation unit 105, the motion vector value affected by the change in the camera angle can be detected from the difference in the lens barrel angle between frames.

[0161] In passing Fig.13AIn the area 1301 obtained by further dividing each area as shown, moving pixels between frames are detected, and frequency distribution processing is performed based on the vector signal obtained by subtracting the motion vector value affected by the change in the camera angle from the detected moving pixels. Note that in the case where it is difficult to detect a vector (such as in a low contrast case), vector information in a block where a vector cannot be detected is not reflected in the frequency distribution processing.

[0162] exist Fig. 13B and Fig. 13C An example of frequency distribution is shown in . Fig. 13B An example of frequency distribution in a certain frame in the case where there is no moving subject in a certain area is shown. Since the range of threshold 1302 is a vector range where the vector value is small and there is little motion, vector information obtained within the range of threshold 1302 is not used. When the amount of movement other than threshold 1302 exceeds frequency threshold 1303, it is determined that there is a moving subject in the area. Fig. 13B In the case of , the movement amount other than the threshold 1302 does not exceed the frequency threshold 1303, so it is determined that there is no moving subject. Fig. 13C An example of frequency distribution in a certain frame in the case where there is no moving subject in the area is shown. Since the vector movement amount outside the range of threshold 1302 exceeds the frequency threshold 1303, it is determined that there is a moving subject in the area. When it is determined that the moving subject has been continuously present during the past multiple frames, the importance level of the current moving subject is set.

[0163] Level settings corresponding to past camera information

[0164] The importance level is set based on information related to past imaging.

[0165] The camera is configured to perform the driving for panning and tilting according to the method described below to perform automatic subject search. When the camera detects an automatic imaging trigger from image information during the search, automatic imaging is performed. For example, the automatic imaging trigger may be the detection of a facial expression (such as a smile) of a person, or may be the size of the final importance level of the subject described below. In addition, the operator can operate the release switch SW provided on the camera to manually capture an image. When the camera captures an image, past imaging information is stored and managed.

[0166] First, setting of a level corresponding to past image pickup information for each area will be described.

[0167] In passing FIG. 6A to FIG. 6D In each region obtained by the region division performed as shown, the number of captured images in each region is stored and managed. Fig.14 The previous management methods in each area are shown in Fig.14 The number of captured images is managed by region along the vertical axis and by time along the horizontal axis. Fig.14 The importance level for each area is set based on the past information in each area such as the past information shown in the figure. "Current time to time T1" indicates the number of captured images in the time period from the current time to the past time T1, and its weighting coefficient is, for example, 1. "Time T1 to time T2" ​​indicates the number of captured images in the time period from the past time T1 to the past time T2, and its weighting coefficient is, for example, 0.8. "Time T2 to time T3" indicates the number of captured images in the time period from the past time T2 to the past time T3, and its weighting coefficient is, for example, 0.5. "Time T3 to time T4" indicates the number of captured images in the time period from the past time T3 to the past time T4, and its weighting coefficient is, for example, 0.2.

[0168] The number of captured images is multiplied by the corresponding weighting coefficient, and the multiplication results in each time period are added together, thereby calculating the weighted total number of captured images in each area.

[0169] The total number of captured images in region 1 corresponds to 0×1+0×0.8+0×0.5+2×0.2=0.4 images, and the total number of captured images in region 3 corresponds to 3×1+4×0.8+2×0.5+0×0.2=7.2 images.

[0170] Level setting corresponding to past imaging information of each area is performed by multiplying the total number of captured images in each area by a level coefficient (the level coefficient is a negative value and is set in such a way that the importance level decreases as the number of captures increases).

[0171] Furthermore, past camera information is fed back to, for example, "importance level setting corresponding to person information", "importance level setting corresponding to object recognition information", and "importance level setting corresponding to scene", and also affects each level setting.

[0172] Fig.15 This is a table for managing past image-taking information used for feedback to "importance level setting corresponding to person information". The number of past image-taking images of each subject (Asan, Bsan, Csan, Dsan, ...) registered as a person is stored and managed. Fig.14As in the method described above, a weighting coefficient is set in each time period, and the total number of captured images is calculated for each registered subject. The result obtained by multiplying the total number of captured images by the level coefficient for level setting (the coefficient is a negative value and is set in such a way that the level decreases as the number of captured images increases) is added to the gain Ka. In this way, past imaging information is fed back to the "importance level setting corresponding to person information". Note that the gain Ka is set in such a way that it does not become equal to or less than 0.

[0173] Fig.16 This is a table for managing past image-taking information used for feedback to "importance level setting corresponding to object recognition information". The number of past image-taking images of each registered subject (dog, cat, ...) is stored and managed. Fig.14 As in the method described above, a weighting coefficient is set in each time period, and the total number of captured images is calculated for each registered subject. The result obtained by multiplying the total number of captured images by the level coefficient for level setting (the coefficient is a negative value and is set in such a way that the level decreases as the number of captured images increases) is added to the level setting corresponding to each subject. In this way, past imaging information is fed back to the "importance level setting corresponding to object recognition information". Note that the importance level corresponding to each subject is set in such a way that it does not become equal to or less than 0.

[0174] Fig.17 This is a table for managing past image capture information used for feedback to "importance level setting corresponding to the scene". The number of past captured images for each scene (blue sky, evening scene, natural green, night scene, etc.) is stored and managed. Fig.14 As in the method described above, a weighting coefficient is set in each time period, and the total number of captured images is calculated for each registered subject. The result obtained by multiplying the total number of captured images by the level coefficient for level setting (the coefficient is a negative value and is set in such a way that the level decreases as the number of captured images increases) is added to the level setting corresponding to each scene. In this way, past imaging information is fed back to the "importance level setting corresponding to the scene". Note that the importance level corresponding to each scene is set in such a way that it does not become equal to or less than 0.

[0175] Importance level setting corresponding to unsearched time

[0176] Conduct and pass FIG. 6A to FIG. 6D The importance level corresponding to the elapsed time from the last search at each area position obtained by the area segmentation shown in FIG. Fig.18 A processing flow chart is shown in FIG.

[0177] When the process starts in step S1801, the current pan and tilt positions are acquired, and the process proceeds to step S1802.

[0178] In step S1802, obtain the above reference Figure 5A and 5B The absolute angle of the camera device calculated by the method described above, and then the process proceeds to step S1803. In step S1803, the absolute angle of the lens barrel 102 is calculated based on the pan and tilt positions acquired in step S1801 and the absolute angle of the camera device acquired in step S1802, and then the process proceeds to step S1804.

[0179] In step S1804, "1" is substituted into "Area" which is a variable used for loop operation, and the process proceeds to step S1805.

[0180] In step S1805, it is determined whether the absolute angular velocity calculated by differentiating the absolute angle of the lens barrel 102 is within a predetermined velocity, and whether the absolute angular velocity calculated by differentiating the absolute angle of the lens barrel 102 is within a predetermined velocity. Figures 6A to 6D The variable Area corresponds to the area number of an area in the field angle by using the area number defined in each area obtained by the segmentation performed as shown. If it is determined that Area is within the field angle and the absolute angular velocity of the lens barrel is a velocity that enables detection of each subject (an angular velocity at which the delay time until the subject is detected from the image is within a predetermined value in consideration of the imaging delay and the detection time delay), the process proceeds to step S1806.

[0181] In step S1806, the importance level corresponding to the non-searched time of the current Area is set to 0, and the process proceeds to step S1808.

[0182] If it is determined in step S1805 that the current Area does not correspond to an area included in the field angle, or the barrel absolute angular velocity is greater than a predetermined angular velocity, the process advances to step S1807 to increment the time level of the current Area, and then to step S1808.

[0183] In step S1808, Area is incremented, and the process proceeds to step S1809. Then, it is determined whether Area is greater than the total number of regions, and if it is determined that Area is greater than the total number of regions, the process ends. If it is determined that Area is less than the total number of regions, the process returns to step S1805, so that the processes of steps S1805 to S1809 can be repeated.

[0184] In the above method, an importance level corresponding to the non-search time is set for each area, and the time level of each area becomes larger according to the elapsed time (non-search time of the area) since the last search for the subject in the area. In this way, if there is an area that has not been searched for a long time, the importance level of the area is increased so that the search in the area is started with panning and tilting.

[0185] Importance level setting corresponding to camera vibration status

[0186] Make settings for the importance level according to the vibration status of the camera.

[0187] The vibration state of the camera (vehicle moving state, placed shooting state, or hand-held state) is determined based on the determination result of the camera state calculated in step S308.

[0188] When the camera vibration state is "placed shooting state", since the calculation of the absolute angle of the camera device is not performed incorrectly and the reliability of the calculation of the importance level of each area is high, the importance level of the subject in each area calculated so far is directly used when performing subsequent search control.

[0189] In the case where the camera vibration state is the "vehicle moving state", since the moving speed is high, although there is little change in the area of ​​the person sitting in the vehicle, the subject such as the scenery changes all the time. Therefore, in the case of the vehicle moving state, the importance level of the "importance level setting corresponding to the person information" is directly used, but the importance level of the "importance level setting corresponding to the image motion information" is not used. In addition, since the "importance level setting corresponding to the scene" and the "importance level setting corresponding to the object recognition information" are very likely to change quickly, the importance level is reduced. However, in automatic photography, for example, a method of immediately photographing at the time of detection can be designed. In the "importance level setting corresponding to the voice information" and the "level setting corresponding to the past photography information", the importance level is directly used. In addition, calculations can be performed, for example, to detect the forward direction of the camera moving at high speed using, for example, an acceleration sensor, and to increase the importance level of the area located in the forward direction.

[0190] In the case where the vibration state of the camera is a "handheld state" (a state in which blur vibration is large), the operator may manipulate the direction of the camera. Therefore, in an area within a range of ±45 degrees relative to the camera's normal position (where the pitch angle and the pan angle are 0 degrees), the importance level is increased, and in an area within a range from ±45 degrees to ±90 degrees, the importance level is reduced. In addition, in an area farther from ±90 degrees, the importance level is even further reduced. In this way, the importance level of each area is set. A method of adding the importance levels of each area together or a method of weighting the importance levels calculated for each area according to the range may be used. For example, the weighting coefficient may be set to 1 in an area within a range of ±45 degrees, the weighting coefficient may be set to 1 or less in an area within a range from ±45 degrees to ±90 degrees, and the weighting coefficient may be set to an even lower value in an area farther from ±90 degrees.

[0191] According to the above method, the calculation of the importance level is changed according to the camera vibration state, thereby enabling a subject search suitable for the vibration state.

[0192] The "importance level corresponding to the person information", "importance level corresponding to the object recognition information", "importance level corresponding to the scene", "importance level corresponding to the voice information", "importance level setting corresponding to the image motion information", "importance level corresponding to the past camera information", and "importance level corresponding to the unsearched time" of each area obtained by the above method are added together to calculate the subject importance level, and the processing proceeds to step S313. Note that in the case of detecting a subject whose subject importance level is equal to or greater than a predetermined threshold, a tracking judgment is made, and the thresholds used for tracking and searching are changed to make tracking easier.

[0193] In step S313, the target angle for pan and tilt search is calculated based on the subject importance level of each area, and the process proceeds to step S314. The target angle for pan and tilt search is calculated by the following method.

[0194] First, the search final level is calculated based on the importance level of the subject in the area adjacent to each area. Fig.19AThe search final level in area 1901 is calculated based on information about area 1901 and its surrounding areas (1902, 1903, 1904, 1905, 1906, 1907, 1908 and 1909) shown in the figure. Then, the weighting coefficient of area 1901 is set to 1, and the weighting coefficients of other areas are set to 1 or less (for example, 0.5), and the subject importance level of each area is multiplied by the associated weighting coefficient, and the calculated values ​​obtained in all areas are added together to calculate the search final level. This operation is performed on all areas, and the search final level in each area is calculated thereby. Next, the area with the largest search final level is set as the search target area. When the search target area is Fig.19A and 19B In the case of the region 1901 shown, the pan and pitch search target angles are calculated from the search final level of the regions 1901 to 1909 and the center angle of the regions 1901 to 1909 according to equation (5).

[0195] [Mathematical formula 1]

[0196]

[0197] In equation (5), n is the number 9 of regions 1901 to 1909, i is a variable for defining regions 1901 to 1909 as 1 to 9, b is the center angle of each region, and a is the search final level of each region. The pan and pitch search target angles can be calculated by calculating the center of gravity position y (y is calculated in each of the pitch direction and the pan direction) based on the angle of the search final level calculated between regions 1901 to 1909.

[0198] Note that, in the case where all values ​​of the search final level in the respective areas are equal to or smaller than a predetermined threshold value, since it is determined that no important subject exists at this time even when searching these areas, pan and tilt driving is not performed.

[0199] At this time, the target angle is calculated based on the subject importance level calculated while excluding the "level setting corresponding to the past imaging information". Then, after the pan and tilt drive is performed to reach the target angle, the camera is kept at the angle position until any of the search final levels of the respective areas becomes greater than a predetermined threshold value. In this way, the pan and tilt search target angle is calculated.

[0200] Next, the second rotation correction amount for tracking is calculated, and the process proceeds to step S314.

[0201] Fig. 20A and Fig. 20B: is a diagram showing the position in an image of a subject detected by a camera, and a method of setting a threshold value and a second rotation correction amount for tracking and searching.

[0202] C0 is the tracking target position in the captured image (field angle), C1 is the center position of the subject area in the captured image, a circle P centered on C0 is the tracking target area, and the radius of the circle P is a threshold value for determining whether to perform tracking and searching. When the subject tracking remaining amount (the distance in the captured image between C1 and C0) is greater than the threshold value for tracking and searching, tracking is performed by rotating the mechanism so that C1 is moved to a position closer to C0.

[0203] When calculating the pan and pitch rotation correction amounts (Dp, Dt) for tracking, the difference between the coordinates of C0 (x0, y0) and the coordinates of C1 (x1, y1) is obtained, and the difference is multiplied by the conversion coefficients for pan and pitch to obtain the drive amount. The pan conversion coefficient is represented by kp, the pitch conversion coefficient is represented by kt, and the pan rotation correction amount Dp and the pitch rotation correction amount Dt are represented by the following formulas.

[0204] Dp=(x1-x0)×kp...(6)

[0205] Dt=(y1-y0)×kt...(7)

[0206] In equations (6) and (7), conversion coefficients kp and kt are used to convert position errors in images into pan and tilt target position differences, and therefore vary according to the size of the image sensor, the focal length, and the detection resolution of the pan and tilt position detection unit.

[0207] In step S314, the pan and tilt drive amount is calculated based on the image blur correction amount obtained in step S306, the pan and tilt search target angle obtained up to step S313, and the rotation correction amount for tracking. The pan and tilt drive amount (both the drive angle for pan and the drive angle for tilt) is calculated by adding the image blur correction amount, the rotation correction amount for tracking, and the drive angle used for control sampling based on the pan and tilt search target angle, and then the processing proceeds to step S315.

[0208] Fig.21 : is a diagram showing an example of a time-series change in the amount of vibration detected by a camera, and a method of setting the first and second thresholds for image blur correction and the first rotation correction amount. Fig.21The threshold 1 in is the first threshold for image blur correction related to the vibration amount for determining whether the camera is in the placed shooting state, and the threshold 1 is measured in advance and stored in the camera. In the case where the camera is in the placed shooting state, the need to operate the rotation function for image blur correction is low. If the detected vibration amount is less than the threshold 1 (e.g. Fig.21 t0 to t1 in FIG. 1 ), it is determined that the camera is in the placed shooting state, and the first rotation correction amount for image blur correction is set to 0 (zero).

[0209] Fig.21 Threshold 2 in is a second threshold value for image blur correction related to the amount of vibration for determining whether the camera is within a range in which image blur correction can be performed, and is measured in advance and stored in the camera. In the case where the amount of vibration is greater than the upper limit of the range in which camera image blur correction can be performed, since image blur equal to or greater than the upper limit of the range cannot be corrected even when the image blur correction operation is performed, it is preferable not to perform the image blur correction operation from the perspective of the effect of the image blur correction relative to power consumption. For the camera, if the detected amount of vibration is equal to or greater than Fig.21 Threshold 2 in Fig.21 t1 to t2 in the figure), the first rotation correction amount for image blur correction is set to 0.

[0210] For cameras, if the amount of vibration detected is equal to or greater than Fig.21 The threshold value is 1 but less than Fig.21 Threshold 2 in Fig.21 The first rotation correction amount for image blur correction is maintained at the same value calculated until step S314 (as in the case from time t2 to time t3 in step S314).

[0211] Fig. 20A 3 is a diagram showing an example in which a subject with high imaging importance exists within the field angle. The range of the tracking target area is set narrower corresponding to the importance, the second rotation correction amount used for tracking and searching is kept the same value as the value calculated up to step S314, so that C1 that is not in the tracking target area is continuously photographed at the position of C0, and tracking using rotational drive is continued.

[0212] Fig. 20B 1 is a diagram showing an example in which a subject with low imaging importance exists within the field angle. The range of the tracking target area is set wider corresponding to the importance, and since C1 is in the tracking target area, the second rotation correction amount for tracking and searching is set to 0, and tracking using rotational drive is not performed.

[0213] In a case where the object does not exist within the field angle, the second rotation correction amount for tracking and searching is set to 0, and tracking using rotational driving is not performed.

[0214] In addition, the rotation correction amount when there is a rotation instruction via user operation is defined as a third rotation correction amount. When panning and tilting rotations are performed in response to user operation, it is preferable to disconnect the power supply to the rotation mechanism that does not need to be rotated (stop the rotation mechanism) from the perspective of power saving. Here, the user operation includes "instructions via external device operation", "voice instructions" and "gesture instructions". "Instructions via external device operation" is an instruction sent from an external device not shown via the communication unit 213. When an instruction from a user operating an external device is received, the rotation mechanism can be rotated in the indicated direction. In addition, "voice instructions" are instructions via voice input from the voice input unit 210. The source direction of the voice can be detected, and the rotation mechanism can be rotated in that direction. In addition, "gesture instructions" are instructions via gesture operations, which have a specific meaning from a person in the subject in the image captured by the camera unit in step S312. When the instruction is detected, the rotation mechanism is rotated based on the gesture. When a rotation instruction is performed via user operation, the power supply to the rotation unit of the axis that does not need to be rotated can be disconnected. In the case where the operation amount in the pan or tilt direction is zero as a rotation instruction via the user operation, the third rotation correction amount by the user operation is set to 0, and the rotation driving is not performed.

[0215] Here, in the present embodiment, when the threshold value of the operation amount of the user operation related to whether the rotation is performed according to the user operation (the threshold value of the user operation) is zero (in other words, the user operation has occurred), the rotation correction amount is calculated each time. However, when a small operation amount of the user operation is detected, a threshold value for providing a dead zone may be set so that the third rotation correction amount is set to 0.

[0216] In this manner, by setting the first to third rotation correction amounts, when any of the first to third rotation amounts is smaller than the respective threshold values ​​set for the first to third rotation amounts, power supply to the rotation mechanism can be cut off.

[0217] In step S315, a threshold for determining whether to disconnect power to any of the pan and tilt axes is calculated using the imaging conditions and shooting mode set by the user and the camera state detected inside the imaging apparatus, and the process proceeds to step S316.

[0218] Here, the shooting conditions set by the user include "image blur correction intensity setting", "tracking intensity setting", "search intensity setting", "automatic shooting frequency setting", "shutter speed setting" and "power saving setting", and the thresholds used for image blur correction and tracking and searching change according to the shooting conditions.

[0219] For example, in a case where the "image blur correction strength setting" is set to "low", the first threshold value for image blur correction is changed upward to make it easier to disconnect the power supply. In a case where the "tracking strength setting" is set to "low", the threshold values ​​for tracking and searching are changed upward to make it easier to disconnect the power supply. In a case where the "search strength setting" is set to "low", the threshold values ​​for tracking and searching are changed upward to make it easier to disconnect the power supply. In a case where the "automatic camera frequency setting" is set to "low", the first threshold value for image blur correction and the threshold value for tracking and searching are changed upward. In a case where the "shutter speed setting" is set to a speed higher than the hand shake threshold, since the image blur correction effect is minimal, the first threshold value for image blur correction is changed upward. In a case where the "power saving setting" is set to "on", the first threshold value for image blur correction, the threshold value for tracking and searching, and the threshold value for user operation are changed upward.

[0220] Furthermore, the shooting modes set by the user include “panorama shooting mode”, “time-lapse shooting mode”, “tracking shooting mode”, and “first-person shooting mode”, and the threshold value changes according to the operation of the set shooting mode.

[0221] For example, in the "panoramic shooting mode", a predetermined number of images are captured while panning and tilting are rotated by predetermined rotation angles in the horizontal and vertical directions, and the captured images are connected and synthesized into a still image. When a user instruction via an external device operation is received and the field angle is changed by a predetermined rotation angle in the horizontal or vertical direction, power supply to the rotation unit of the axis that does not need to be rotated can be disconnected. The threshold for user operation is changed in a direction that makes it easier to disconnect power supply to the rotation mechanism that does not need to be rotated.

[0222] In the "time-lapse shooting mode", a moving image is created from a plurality of images captured while rotating at a predetermined angle from the start position to the end position of the pan and tilt and changing the field angle. When a user instruction via an external device operation is received and the shooting direction is changed between the indicated start angle and end angle for the horizontal direction and the vertical direction, the power supply to the rotation unit of the axis that does not need to be rotated can be disconnected. The threshold value for the user operation is changed in a direction that makes it easier to disconnect the power supply to the rotation mechanism that does not need to be rotated.

[0223] In the "tracking shooting mode", a moving subject is tracked while panning and tilting to capture an image. When a user instruction is received via an external device operation and the indicated subject is tracked, the power supply to the rotation unit of the axis that does not need to be rotated can be disconnected. The threshold value for user operation is changed in a direction that makes it easier to disconnect the power supply to the rotation mechanism that does not need to be rotated. When the subject to be tracked is specified and tracked, the threshold value for tracking and searching is changed downward, thereby performing the desired tracking. However, in the case where the subject to be tracked is not indicated or the indicated subject no longer exists, the threshold value for tracking and searching is changed upward to make it easier to disconnect the power supply.

[0224] Here, in the "panoramic shooting mode", "time-lapse shooting mode" and "tracking shooting mode", when it is determined in step S308 that the camera vibration state is the placement shooting state, the image blur correction operation is not required, so the first threshold value for image blur correction is changed upward. In this way, it is easier to cut off the power supply to the rotation unit. On the other hand, when it is determined that the camera vibration state is not the placement shooting state, the vibration is caused by the camera being handheld, etc. Therefore, the first threshold value for image blur correction remains unchanged.

[0225] In the "first-person shooting mode", video recording is performed while panning and tilting are being rotated so that a subject moving on the screen can be followed. At this time, the power supply to the rotation unit of the axis that does not need to be rotated can be disconnected. When a user instruction is received via an external device operation, and the pan and tilt are rotated so that the indicated subject or the identified main subject can be followed, the threshold value for user operation is changed in a direction that makes it easier to disconnect the power supply to the rotation mechanism that does not need to be rotated. When the subject to be followed is specified and tracked, the threshold value for tracking and searching is changed downward, thereby performing a tracking rotation operation at a desired speed. However, in the case where there is no indicated subject or the subject being followed no longer exists, the threshold value for tracking and searching is changed upward to make it easier to disconnect the power supply.

[0226] In addition, the camera status detected inside the camera device includes "camera scene judgment", "remaining battery power", "total number of captured images or total shooting time within a predetermined amount of time", "focal length", "subject brightness" and "captured image transmission status", and the power supply to the rotating mechanism can be disconnected according to the camera status.

[0227] For example, in "Camera scene determination", in the case where it is determined to be an important camera scene, in order to make it possible to reliably shoot a subject without blur, the first threshold for image blur correction is changed downward so that the power supply to the pan and tilt drive motors can be easily turned on. In another case, the first threshold for image blur correction is changed upward so that the power supply can be easily disconnected. In a similar manner, in the case where it is determined to be an important camera scene, the threshold for tracking and searching is changed downward so that the tracking and searching of the subject can be reliably performed. In another case, the threshold for tracking and searching is changed upward so that the power supply can be easily disconnected.

[0228] Here, the important imaging scene determination is performed by determining whether the scene currently being photographed is an important imaging scene based on person information, object recognition information, audio information, image movement information, and past imaging information recognized by the camera.

[0229] In the case where the "remaining battery power" is less than a predetermined value, increasing the camera operation time will make it more likely that automatic imaging of an important subject can be achieved compared to the presence or absence of the image blur correction function. Therefore, in the case where the remaining battery power is less than a predetermined value, the first threshold value for image blur correction is changed so that the power supply to the pan and tilt drive motors can be easily disconnected. In the same way, the threshold values ​​for tracking and searching are changed so that the power supply can be easily disconnected.

[0230] In the case where the "total number of captured images or total capturing time within a predetermined amount of time" is equal to or greater than a predetermined value, it can be assumed that an image of the desired subject can be captured by automatic imaging up to now. Therefore, in the case where the remaining battery power is less than a predetermined value, the first threshold value for image blur correction is changed so that the power supply to the pan and tilt drive motors can be easily disconnected. In the same manner, the threshold values ​​for tracking and searching are changed so that the power supply can be easily disconnected.

[0231] In the case where the "focal length" is on the wide angle side, it is highly possible that an image with minimal blur can be captured using a shutter speed without driving pan and tilt for image blur correction. When the focal length of the lens barrel 102 is on the wide angle side of a predetermined threshold or on the wide angle side corresponding to the focal length, the threshold for determining whether to disconnect the power supply is greatly changed by a predetermined coefficient. In this way, the first threshold for image blur correction is changed so that the power supply to the pan and tilt drive motors can be easily disconnected. In the same way, the thresholds for tracking and searching are changed so that the power supply can be easily disconnected.

[0232] In the case where the "subject brightness" is equal to or greater than a predetermined threshold, an image with minimal subject blur can be captured using a shutter speed without driving pan and tilt for image blur correction. Therefore, the threshold for determining whether to disconnect the power supply can be changed according to the brightness information of the image captured by the imaging unit 205. For example, in the case where the brightness of the subject is greater than a predetermined brightness, the first threshold for image blur correction is changed so that the power supply to the pan and tilt drive motors can be easily disconnected. In the same way, the thresholds for tracking and searching are changed so that the power supply can be easily disconnected.

[0233] In step S316 , the rotation correction amount of the axis calculated in step S314 is calculated again based on the vibration detection amount detected in the camera and the subject importance level, and the process proceeds to step S317 .

[0234] In the case where the vibration detection amount detected in the camera is small enough to make it determined that the camera is in the placed shooting state, the need to operate the rotation function is low. Therefore, when the vibration amount corresponding to the placed shooting state is set in advance as a threshold value ( Fig.21 In the case where the amount of vibration detected in the imaging device 101 is less than the threshold value 1, Fig.21 ), the first rotation correction amount for image blur correction is set to 0.

[0235] On the other hand, in the case where the vibration detection amount detected in the camera is a very large amount and is a vibration amount exceeding the upper limit of the range in which the image pickup device 101 can perform image blur correction, even if the image blur correction operation is performed, the subject blur exceeding the upper limit of the range cannot be suppressed. Therefore, from the perspective of the effect of image blur correction relative to power consumption, it is preferable not to perform the image blur correction operation. Here, the vibration amount corresponding to the upper limit for enabling image blur correction is set in advance as a threshold value ( Fig.21 Therefore, when the vibration amount detected in the camera 101 is greater than the set threshold value (in Fig.21 ), the first rotation correction amount for image blur correction is set to 0.

[0236] Furthermore, in the case where the subject is not detected in the image captured by the imaging unit 205, or in the case where the detected subject has a low importance, it is preferable not to perform automatic tracking and imaging and not to operate the rotation mechanism from the viewpoint of operating the system for a long time. Here, in the case where the subject is not detected, the second rotation correction amount for tracking and searching is set to 0. On the other hand, in the case where the subject is detected, the area (tracking target area) for determining the range for continuously capturing the subject from the center of the field angle is set to a narrow area ( Fig. 20A ), and is set to a wider area when the subject imaging importance is low ( Fig. 20B ). Then, in the case where the centroid of the object area detected by the imaging apparatus 101 exists in the tracking target area, the second rotation correction amount for tracking and searching is set to 0.

[0237] In step S317, the rotation correction amount calculated in step S315 is compared with the threshold for determining whether to disconnect the power supply calculated in step S316, and when any rotation correction amount is less than the predetermined time threshold, the power supply to the rotation mechanism of the axis is disconnected, and then the processing enters step S318.

[0238] In step S318, the tilt rotation unit 104 and the pan rotation unit 105 are driven and controlled by the barrel rotation driving unit 212 according to the driving angles for pan and tilt, and the process proceeds to step S319.

[0239] In step S319, it is determined whether there has been an image capture instruction manually input via the operation of the operation unit 211. If there has been an image capture instruction, the process proceeds to step S322, image capture is started, a still image is acquired, and the captured image is recorded in the image recording unit 207. In step S319, if there has been no manual image capture instruction, the process proceeds to step S320, and automatic image capture determination processing is performed.

[0240] In the automatic camera judgment of step S320, when the subject importance level obtained until step S316 is greater than a predetermined value, it is judged that the automatic camera operation is turned on. In addition, even in the case where the subject importance level is not greater than a predetermined value, the automatic camera judgment can be made using the degree of smile on the detected face, the proximity in terms of distance between the detected multiple faces can be used to make the automatic camera judgment, or if an object (e.g., a dog or a cat) or a scene (e.g., a blue sky, an evening scene, a night scene, or natural green) that has not been previously photographed is detected, the automatic camera judgment can be made. In step S320, when the automatic camera judgment is made, the processing then enters step S321.

[0241] In step S321, if the automatic imaging determination result is automatic imaging on, the process proceeds to step S322 and still image shooting is performed. In step S321, if it is determined that automatic imaging is off, the process returns to step S302 and the automatic subject search and automatic imaging process are repeated.

[0242] In step S322, after still image shooting is performed, the process then proceeds to step S323. Then, using the subject information from the time point of the shooting in step S310, management of past shooting information (such as the number of shot images of important subject information in the past, and the planned consumption rate calculated from the number of shot images per unit time and the target number of shot images, etc.) is performed. In the case of shooting moving images, management of past shooting information is performed in the same manner. In the case of moving images, management of past shooting information is performed using shooting information such as subject detection in the captured moving images.

[0243] In step S323 , when the past imaging information is updated, the process returns to step S302 , and the automatic subject search and automatic imaging processes are repeated.

[0244] As described above, according to the above-described embodiment, it is possible to provide an image pickup apparatus configured to appropriately turn off the image blur correction operation when the image blur correction drive is not required, thereby enabling automatic image pickup for a long period of time.

[0245] Other embodiments

[0246] In addition, the present invention can be implemented by the following process: a program for implementing one or more functions of the above-mentioned embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0247] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, in order to inform the public of the scope of the present invention, the following claims are added.

[0248] This application claims priority from Japanese Patent Application No. 2019-239279, filed on December 27, 2019, which is hereby incorporated by reference herein.

Claims

1. A camera device, comprising: A camera component, used for capturing an image of a subject; A rotating component capable of rotating the camera component; a first calculation unit for calculating a first rotation amount of the rotation unit for correcting image blur based on shaking of the image pickup device; a second calculating component, used for calculating a second rotation amount of the rotating component for searching for a subject from the image captured by the imaging component; as well as a stopping member for stopping the rotating member when both the first rotation amount and the second rotation amount are smaller than threshold values ​​set for the first rotation amount and the second rotation amount, respectively; Wherein, when tracking a subject so as to keep the subject at a predetermined position within the screen, the second calculation component calculates, for a subject existing outside a predetermined radius from the predetermined position, the amount of rotation used by the rotation component to move the subject to the predetermined position as the second rotation amount, and for a subject existing within the predetermined radius from the predetermined position, calculates the second rotation amount to be zero, and changes the predetermined radius based on the importance level of the subject.

2. The imaging device according to claim 1, wherein: The first calculation section sets the first rotation amount to zero when the shake amount of the image pickup apparatus is smaller than a first threshold value.

3. The imaging device according to claim 2, wherein: The first calculation section sets the first rotation amount to zero when the shake amount of the image pickup apparatus is larger than a second threshold value that is larger than the first threshold value.

4. The imaging device according to claim 2, wherein: The first calculation section changes the first threshold value according to an imaging condition set by a user.

5. The imaging device according to claim 4, wherein: The imaging condition set by the user includes at least one of an image blur correction intensity setting, a tracking intensity setting, an automatic imaging frequency setting, a shutter speed setting, and a power saving setting.

6. The imaging device according to claim 2, wherein: The first calculation section changes the first threshold value according to a state of the imaging device.

7. The imaging device according to claim 6, wherein: The state of the imaging device includes at least one of an imaging scene, a remaining battery charge, a total number of captured images or a total capturing time within a predetermined amount of time, a focal length, a subject brightness, and a captured image transmission state.

8. The imaging device according to claim 1, wherein: The second calculation section sets the second rotation amount based on the importance of the object.

9. The imaging device according to claim 8, wherein: When the importance of the object is smaller than a second threshold value, the second calculation unit sets the second rotation amount to zero.

10. The imaging device according to claim 9, wherein: The second calculation unit changes the second threshold value according to the importance of the object.

11. The imaging device according to claim 9, wherein: The second calculation section changes the second threshold value according to an imaging condition set by a user.

12. The imaging device according to claim 11, wherein: The imaging condition set by the user includes at least one of an image blur correction intensity setting, a tracking intensity setting, an automatic imaging frequency setting, a shutter speed setting, and a power saving setting.

13. The imaging device according to claim 9, wherein: The second calculation section changes the second threshold value according to a state of the imaging device.

14. The imaging device according to claim 13, wherein: The state of the imaging device includes at least one of an imaging scene, a remaining battery charge, a total number of captured images or a total capturing time within a predetermined amount of time, a focal length, a subject brightness, and a captured image transmission state.

15. The imaging device according to claim 8, wherein: The second rotation amount includes a rotation amount for tracking so that the subject stops at a predetermined position on the screen.

16. The imaging device according to claim 1, further comprising: a third calculating component, configured to calculate a third rotation amount of the rotating component in response to a user instruction; Wherein, when the first rotation amount, the second rotation amount, and the third rotation amount are all smaller than thresholds set for the first rotation amount, the second rotation amount, and the third rotation amount, respectively, the stopping component stops the rotating component.

17. The imaging device according to claim 16, wherein: The user instruction includes at least one of an instruction operated via an external device, a voice instruction, and a gesture instruction.

18. The imaging device according to claim 16, wherein: The third calculation section changes a third threshold value for the third rotation amount according to a shooting mode set by a user.

19. The imaging device according to claim 18, wherein: The shooting mode includes at least one of a panoramic shooting mode, a time-lapse shooting mode, a tracking shooting mode, and a first-person shooting mode.

20. The imaging device according to claim 1, wherein The stopping member stops the rotating member by cutting off power supply to the rotating member.

21. The imaging device according to claim 1, wherein: The rotating component includes at least one rotating mechanism for rotating the imaging component around an axis.

22. A control method for an imaging device, wherein the imaging device is provided with an imaging component for capturing an image of a subject and a rotating component capable of rotating the imaging component, characterized in that: The control method comprises: calculating a first rotation amount of the rotating member for correcting image blur based on shaking of the imaging device; calculating a second rotation amount of the rotating member for searching for a subject from an image captured by the imaging member; and When both the first rotation amount and the second rotation amount are smaller than threshold values ​​set for the first rotation amount and the second rotation amount, respectively, the rotating member is stopped, Wherein, when calculating the second rotation amount, when tracking the subject so that the subject remains at a predetermined position within the screen, for a subject existing outside a predetermined radius from the predetermined position, the rotation amount of the rotating component used to move the subject to the predetermined position is calculated as the second rotation amount, and for a subject existing within the predetermined radius from the predetermined position, the second rotation amount is calculated to be zero, and the predetermined radius is changed based on the importance level of the subject.

23. A computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 22.

Citation Information

Patent Citations

  • Search device, imaging device thereof, and search method

    JP2018050146A

  • Imaging apparatus and method for controlling the same, program, and storage medium

    JP2021107874A

  • Vibration-proofing controller for camera

    JP1995191360A

  • Image capturing apparatus and subject tracking method

    JP2010154374A

  • Imaging apparatus and method of controlling the same

    JP2011217333A