Imaging device, control method and program thereof

Abstract

To achieve both photographing of a number of subjects and photographing by tracking a specific subject.SOLUTION: An imaging apparatus 100 can change the pan angle, the tilt angle, and the zoom magnification of a lens barrel 101, and controls respective units of the imaging apparatus 100 to detect subject candidates from an image picked up by the imaging apparatus set to have a wide angle of view, track the subject candidates with the wide angle of view based on the amount of change in the positions and sizes of the detected subject candidates, determine the presence or absence of a subject candidate satisfying a predetermined photographing condition while tracking the subject candidates, when the subject candidate satisfying the photographing condition is present, photograph the subject candidate with the wide angle of view, determine the presence or absence of a subject candidate satisfying a predetermined gaze condition, when the subject candidate satisfying the gaze condition is present, tack the subject candidate with a telephoto angle of view, when the photographing condition is satisfied, photograph the subject candidate with the telephoto angle of view, and subsequently return the angle of view to the wide angle of view.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus, and a control method and program thereof. [Background technology]

[0002] Automatic imaging devices that take pictures periodically and continuously without the user giving instructions to take pictures are becoming more and more practical. For example, there is a life log camera that is used while being attached to the user's body with a strap or the like, and records scenes that the user sees in daily life as images at regular intervals. There are also automatic imaging devices that recognize a certain facial expression on a person's face, a dog or a cat, a sudden movement, etc. and release the shutter, and cameras that have a PTZ drive mechanism that controls the pan angle, tilt angle, optical zoom magnification, etc. and automatically take pictures while freely switching the shooting range. Some automatic imaging devices with a PTZ drive mechanism have a function to automatically change the subject.

[0003] In imaging by an automatic imaging device, an imaging method is often adopted in which the pan angle and tilt angle are controlled according to the movement of the subject, and the imaging direction is adjusted so that the subject is in the center of the angle of view, thereby keeping the subject large within the screen. In this case, it is common to control the zoom magnification and adjust the subject to an appropriate size. As a technology for realizing such a function, for example, Patent Document 1 describes a method for determining a subject to be tracked based on a priority set for feature information included in pre-registered tracking target information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4957721 Summary of the Invention [Problem to be solved by the invention]

[0005] It is not easy to detect or photograph other subjects while controlling the optical zoom to a telephoto angle of view to track and capture a specific subject, as in the technology described in Patent Document 1. On the other hand, if the photographing angle of view is fixed to a wide angle of view in order to capture many subjects, it is not easy to capture the minute movements and expressions of a specific subject.

[0006] An object of the present invention is to provide an imaging apparatus that is capable of both capturing images of many subjects at a wide angle of view and capturing images of a specific subject by tracking the subject at a telephoto angle of view. [Means for solving the problem]

[0007] The imaging device according to the present invention is characterized in that it comprises an imaging means, a driving means for changing the pan angle, tilt angle and zoom magnification of the imaging means, a detection means for detecting a subject candidate from an image captured by setting the imaging means to a wide-angle angle of view, a tracking means for tracking the subject candidate at a wide-angle angle of view based on a change in the position and size of the subject candidate detected by the detection means, a first determination means for determining whether or not there is a subject candidate that satisfies a predetermined shooting condition while tracking the subject candidate by the tracking means, a second determination means for determining whether or not there is a subject candidate that satisfies a predetermined gaze condition while tracking the subject candidate by the tracking means, a tracking means for tracking the subject candidate detected by the detection means at a telephoto angle of view, and a control means for controlling the tracking shooting process to perform an immediate shooting process in which the subject candidate is shot at a wide-angle angle of view if there is a subject candidate that satisfies the shooting condition, and to perform a tracking shooting process in which the angle of view is returned to the wide-angle angle of view after shooting according to the determination result by the first determination means while tracking the subject candidate by the tracking means if there is a subject candidate that satisfies the gaze condition. Effect of the Invention

[0008] According to the imaging device of the present invention, it is possible to capture images of many subjects at a wide angle of view and to capture images of a specific subject by tracking the subject at a telephoto angle of view. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing a schematic configuration of an imaging apparatus according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating functional blocks of the imaging device. [Diagram 3] FIG. 2 is a diagram showing an example of sub-image data handled by the imaging device. [Figure 4] FIG. 2 is a diagram showing an example of a subject candidate list handled by the imaging apparatus. [Diagram 5] 10 is a flowchart of a subject candidate detection process performed by the imaging apparatus. [Figure 6] 4 is a flowchart of automatic shooting processing by the imaging device according to the first embodiment. [Figure 7] 13 is a flowchart of the immediate shooting process in S605. [Figure 8] 13 is a flowchart of the tracking shooting process in S607. [Figure 9] 10 is a flowchart of automatic shooting processing by an imaging device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Fig. 1(a) is a diagram showing a schematic configuration of an image capturing device 100 according to an embodiment of the present invention. The image capturing device 100 has a lens barrel 101, a tilt unit 102, a panning unit 103, and a control unit 104.

[0011] The lens barrel 101 has an imaging optical system composed of an imaging lens group and an imaging element, etc., captures images according to the control of the control unit 104, and outputs the obtained image data to the control unit 104. The imaging lens group includes movable lenses such as a focus lens and a zoom lens. The lens barrel 101 includes a drive mechanism that drives the movable lenses according to instructions from the control unit 104. The lens barrel 101 is attached to the tilt unit 102 so as to be rotatable about a horizontal axis (hereinafter referred to as "tilt rotation axis T") orthogonal to the imaging optical axis (hereinafter referred to as "optical axis") of the imaging optical system.

[0012] The tilt unit 102 drives the lens barrel 101 to rotate about a tilt rotation axis T in accordance with instructions from the control unit 104. By the tilt unit 102 rotating the lens barrel 101 about the tilt rotation axis T, it is possible to change the elevation and depression angles of the lens barrel 101 (the optical axis of the imaging optical system).

[0013] Tilt unit 102 is attached to pan unit 103 so as to be rotatable about an axis (hereinafter referred to as the "pan rotation axis") P perpendicular to the ground surface of imaging device 100. Pan unit 103 drives tilt unit 102 to rotate about pan rotation axis P in accordance with instructions from control unit 104. By pan unit 103 rotating tilt unit 102 about pan rotation axis P, the azimuth angle of lens barrel 101 (optical axis of the imaging optical system) attached to tilt unit 102 can be changed.

[0014] 2, the control unit 104 is composed of a control unit 207 (CPU), a ROM 208, a system memory 209, etc., and controls the start / stop of automatic shooting in response to an input (instruction) through an operation unit (not shown) of the imaging device 100. The control unit 104 also controls the drive of the movable lens contained in the lens barrel 101, the shooting operation by the imaging element, and the drive of the tilt unit 102 and the pan unit 103. The control unit 104 has a storage device such as a non-volatile memory (EEPROM, etc.) that stores various information such as image data output from the lens barrel 101 and specifications and functions of the imaging device 100.

[0015] 1(b) is a diagram for explaining the definition of the rotation axes of the imaging device 100. The optical axis of the imaging optical system of the lens barrel 101 is defined as the Z axis. The tilt unit 102 has a tilt rotation axis T that rotates around the X axis (pitch direction). The pan unit 103 has a pan rotation axis P that rotates around the Y axis (yaw direction).

[0016] 2 is a diagram illustrating the functional blocks of the imaging device 100. The lens barrel 101 has a lens unit 201 and an imaging section 202. The lens unit 201 forms an optical image on an imaging surface of an imaging element in the imaging section 202. The control unit 104 has an image processing section 203, an image recording section 204, a lens driving section 205, a lens barrel driving section 206, a control section 207, a ROM 208, and a system memory 209.

[0017] Movable lenses such as a zoom lens and a focus lens (not shown) of the lens unit 201 are controlled by a lens driving unit 205 of the control unit 104. The imaging unit 202 has an imaging element such as a CMOS image sensor, and converts an optical image formed by the lens unit 201 into an analog image signal. The imaging unit 202 performs A / D conversion and noise reduction processing on the analog image signal to generate image data consisting of a digital image signal, and outputs the image data to an image processing unit 203 of the control unit 104. In this embodiment, the imaging element is a CMOS image sensor capable of capturing 8K (7680 pixels horizontally × 4320 pixels vertically), but is not limited to this. The imaging elements of the lens unit 201 and the imaging unit 202 configure the imaging optical system of the lens barrel 101.

[0018] Tilt unit 102 has a drive mechanism that rotates barrel 101 in the pitch direction (see FIG. 1(b)). Pan unit 103 has a drive mechanism that rotates tilt unit 102 in the yaw direction (see FIG. 1(b)). Tilt unit 102 and pan unit 103 change the elevation / depression angle and azimuth angle of barrel 101 according to drive instructions (drive signals) input via barrel drive section 206 of control unit 104, making it possible to set the shooting direction to any direction.

[0019] The ROM 208 included in the control unit 104 is a rewritable non-volatile memory such as an EEPROM, and stores programs executed by the control unit 207 and various setting values ​​of the imaging device 100. The system memory 209 is a volatile memory (RAM) used when the control unit 207 executes programs.

[0020] The image processing unit 203 performs image processing such as distortion correction, white balance adjustment, color interpolation processing, and encoding processing on the image data output from the imaging unit 202 of the lens barrel 101 to generate main image data for recording and sub-image data for performing various processing. The image processing unit 203 can also generate a cropped image by cropping a portion of the main image data. The sub-image data may be main image data with a lower resolution. The image processing unit 203 may generate only the main image data or only the sub-image data. The main image data and cropped image data are output to the image recording unit 204, and the sub-image data is output to the control unit 207.

[0021] The image recording unit 204 encodes the original image data and trimmed image data output from the image processing unit 203 in a predetermined format such as JPEG or MPEG, and records (saves) the data as a data file in a storage medium such as a non-volatile memory provided within the image recording unit 204.

[0022] The control unit 207 is, for example, a microprocessor such as a CPU, and controls the operation of each unit of the imaging device 100 by loading a program stored in a ROM 208 into a system memory 209. This realizes various functions of the imaging device 100. For example, the control unit 207 performs a process of detecting subject candidates and subject features.

[0023] The subject candidate detection process is a process of detecting the type of subject candidate present within the shooting angle of view and subject candidate information, which is area information of the subject candidate, by applying a known object detection process to the sub-image data input from the image processing unit 203. In this embodiment, among the objects detected by the object detection process, a predetermined type is set as the subject candidate. The type of subject candidate is, for example, an unspecified person or an animal such as a dog or cat. The area information of the subject candidate is represented by, for example, the x coordinate and y coordinate of the upper left corner, the width (x direction) and the height (y direction) of the area occupied by the subject candidate in the entire image. Note that the x direction is the horizontal direction of the image, and the y direction is the vertical direction of the image.

[0024] The subject feature detection process is a process for detecting feature information of a subject candidate based on subject candidate information (type of subject candidate and area information). When the type is a person, the feature information includes relative coordinates, width and height of a face area within the area of ​​the subject candidate, face direction, and facial expression parameters that digitize facial expressions. Based on the subject candidate information and the feature information, the control unit 207 generates control instructions for the lens driving unit 205 and the lens barrel driving unit 206 for PTZ (pan, tilt, zoom) control, and outputs them to the lens driving unit 205 and the lens barrel driving unit 206, respectively.

[0025] The lens driving unit 205 drives the zoom lens and focus lens included in the lens unit 201 according to commands such as a target position and a driving speed input from the control unit 207. By driving the zoom lens, the angle of view can be adjusted from the wide-angle side where the subject appears small (a wide range is captured) to the telephoto side where the subject appears large (a narrow range is enlarged and captured), or vice versa. In addition, by driving the focus lens, the focus on a subject candidate can be adjusted. The lens barrel driving unit 206 drives the tilt unit 102 and the pan unit 103 according to commands such as a rotation direction, a rotation amount, and a rotation speed input from the control unit 207.

[0026] Next, a control method for the imaging device 100 will be described. First, the subject candidate list will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a diagram showing an example of sub-image data output from the image processing unit 203 to the control unit 207. Fig. 4(a) is a diagram showing an example of a subject candidate list as a result of detecting subject candidates from the sub-image data of Fig. 3. In this embodiment, the resolution of the sub-image data is 7680 pixels x 4320 pixels, but is not limited to this.

[0027] The subject candidate list is a list used in subject candidate detection processing and automatic shooting processing (details will be described later) and is a list for managing subject candidates detected within the angle of view. The sub-image data includes people 301, 302, 303, and 304 and a dog 305, which are registered in the subject candidate list as subject candidates. Note that areas 306 and 307 shown in FIG. 3 will be described later. The subject candidate list is composed of ID, type, and area information, as shown in FIG. 4(a). The area information includes the top left coordinates (x, y) and their size (width, height), the top left coordinates (x, y) and their size (width, height) of the previous area information, and parameters (Param1 to 4) defined for each type. The parameters (Param1 to 4) store the facial expression parameters described above.

[0028] The subject candidate list in Fig. 4(a) shows the result of performing an initial subject candidate detection process (S502 described below) on the sub-image data in Fig. 3 and updating the subject candidate list (S503 described below). Therefore, in the subject candidate list in Fig. 4(a), IDs 1 to 4 correspond to people 301 to 304, and ID 5 corresponds to dog 305.

[0029] The previous region information is used to hold the immediately preceding value when updating the region information in S602 (see FIG. 6) and S802 (see FIG. 8) described later. Fig. 4(b) shows an example of the result after the update process of the subject candidate list, that is, the result after the feature detection process of the subject candidates for the second or subsequent times, the details of which will be described later.

[0030] Fig. 5 is a flowchart of the subject candidate detection process. Each process (step) indicated by an S number in the flowchart of Fig. 5 is realized by the control unit 207 (CPU) loading a predetermined program stored in the ROM 208 into the system memory 209 and comprehensively controlling each unit of the imaging device 100.

[0031] When this process is started by a user operating an operation unit (not shown) provided in the imaging device 100 to start automatic shooting, periodic output of sub-image data from the image processing unit 203 to the control unit 207 is started.

[0032] In S501, the control unit 207 determines whether or not sub-image data has been acquired from the image processing unit 203. If the control unit 207 determines that the sub-image data has not been acquired (NO in S501), it repeats the process of S501, and if it determines that the sub-image data has been acquired (YES in S501), it advances the process to S502.

[0033] In S502, the control unit 207 performs a subject candidate detection process. Details of the subject candidate detection process have been described above with reference to Figs. 3 and 4, and therefore will not be described here. In S502, the control unit 207 further holds the area of ​​the detected subject candidate as a template image for use in S602 (see Fig. 6) described later.

[0034] In S503, the control unit 207 performs an update process of the subject candidate list using the subject candidate information. Specifically, when the subject candidate list is blank, all detected subject candidate information is added to the subject candidate list. When there is a registration in the subject candidate list, if the overlap between the area information of the subject candidate of the same type among the registrations and the area information of the subject candidate detected this time is less than a certain amount (for example, if it is less than 50% in terms of area), the subject candidate detected this time is added to the subject candidate list.

[0035] For example, assume that a person was detected in region 306 (top left coordinates (1100, 1900), width (800), and height (1400)) in Fig. 3 in S502. In this case, the overlap (overlap between the region of person 301 and region 306) in area conversion to the region information of person 301, who is a subject candidate of ID.1 (top left coordinates (1000, 1800), width (800), height (1400)) is 50% or more. As a result, control unit 207 determines that the person appearing in region 306 is the same person 301 as ID.1, and does not add the person appearing in region 306 to the subject candidate list.

[0036] On the other hand, suppose that a person was detected in area 307 in FIG. 3 (top left coordinates (4000, 3000), width (800), and height (1400)) in S502. In this case, the overlap of area 307 with all of the area information of the subject candidates with IDs. 1 to 4 whose type is "person" and registered in the subject candidate list is less than 50% in terms of area, so a new subject candidate with ID. 6 is added. At that time, the management number of the template image of the subject candidate area (area 307) held in S502 is set to the same number as the ID in the subject candidate list, and they are associated with each other.

[0037] In S504, the control unit 207 determines whether or not an operation to end automatic shooting has been performed on the operation unit of the imaging device 100. If the control unit 207 determines that the end operation has not been performed (NO in S504), the process returns to S501, and if the control unit 207 determines that the end operation has been performed (YES in S504), the process ends. With the end of the process, the periodic output of sub-image data from the image processing unit 203 to the control unit 207 stops.

[0038] First Embodiment Fig. 6 is a flowchart of the automatic shooting process according to the first embodiment. Each process (step) indicated by an S number in the flowchart in Fig. 6 is realized by the control unit 207 loading a predetermined program stored in the ROM 208 into the system memory 209 and comprehensively controlling each unit of the imaging device 100. This process is started when a user operates an operation unit (not shown) provided in the imaging device 100 to start automatic shooting.

[0039] In S601, the control unit 207 determines whether or not one or more subject candidates are registered in the subject candidate list. Note that the subject candidate detection process (FIG. 5) is started at the same time as the start of this process, and the subject candidate detection process and the automatic shooting process are started under the same conditions and executed in parallel. If the control unit 207 determines that no subject candidates are registered (NO in S601), it repeats the determination of S601, and if it determines that one or more subject candidates are registered (YES in S601), it proceeds to S602.

[0040] In S602, the control unit 207 executes a tracking process for all subject candidates registered in the subject candidate list. Specifically, in the tracking process, a certain range (hereinafter referred to as a "peripheral area image") including at least a part of the area is first cut out for each area information of an ID registered in the subject candidate list. Next, template matching is performed between a template image having the same management number as the ID in the subject candidate list and a peripheral area image having the same ID. Then, for each subject candidate registered in the subject candidate list, the top left coordinate, width, and height with the highest similarity are written as new area information in the subject candidate list, thereby updating the subject candidate list. At that time, the template image is also updated with the peripheral area image used in the template matching. In addition, the area information before the update is registered as the previous area information (top left coordinate (x, y) and previous size (width, height).

[0041] In S603, the control unit 207 performs feature detection for all subject candidates registered in the subject candidate list according to the type of the subject candidate, and updates the subject candidate list based on the result. For example, if the type is "person", the top left coordinate of the person's face area, the size of the face area (width, height), the angle of the face, and the facial expression evaluation value are detected as features. Figure 4(b) shows an example of a state in which the subject candidate list in Figure 4(a) has been updated based on the results of the processes in S602 and S603.

[0042] The top left coordinate of the face region is calculated with the top left coordinate of the entire person's region including the face region as the origin (0,0). The detection results of these features correspond to Param1 to 4 shown in Fig. 4. That is, Param1 represents the top left coordinate (x,y) of the face region, Param2 represents the size (width, height) of the face region, Param3 represents the angle of the face, and Param4 represents the facial expression evaluation.

[0043] The angle of the face is 0 (zero) when facing directly ahead in relation to the shooting plane, - (negative) when facing to the left, and + (plus) when facing to the right. A face facing directly to the left (profile) takes a value of -90, with a maximum value of -180. A face facing directly to the right (profile) takes a value of +90, with a maximum value of +180. A face angle of ±180 indicates that the face is facing directly backwards.

[0044] The facial expression evaluation value is a numerical value obtained by classifying facial expressions. Facial expressions can be classified into, for example, laughter, surprise, sadness, fear, disgust, anger, etc., and an evaluation value is obtained by evaluating each facial expression. The evaluation value is expressed as a value between 0 and 1, with a higher value indicating a higher evaluation. As a result of such evaluation, the facial expression with the highest evaluation value and that evaluation value are adopted as the facial expression evaluation value.

[0045] In S604, the control unit 207 judges the shooting conditions. Specifically, the control unit 207 calculates shooting condition evaluation values ​​(details will be described later) for all the subject candidates registered in the subject candidate list, and judges whether or not there is a subject candidate having a shooting condition evaluation value exceeding a predetermined threshold value (hereinafter referred to as a "first threshold value"). Note that the first threshold value used in the judgment in S604 can be a different value for each type of subject candidate.

[0046] Here, a method for determining the shooting condition evaluation value will be specifically described. When the type is "person", the facial expression evaluation value is used as the shooting condition evaluation value. For example, the value of Param4 in FIG. 4(b) is checked, and the presence or absence of a subject candidate exceeding 0.6 for "smile" and 0.7 for "surprise" is determined. Similarly, the presence or absence of a subject candidate exceeding 0.85 for "sadness", 0.85 for "fear", 0.95 for "disgust", and 0.95 for "anger" is determined.

[0047] For the shooting condition evaluation value when the type is "dog", the inverse of the value obtained by dividing the distance from the subject candidate whose type is "person" by a fixed number can be used. For example, when determining the shooting condition evaluation value of ID.5 in FIG. 4(b), the distance between the center of gravity coordinate of ID.5 whose type is "dog" and the center of gravity coordinate of ID.1 to 4 whose type is "person" is calculated. The center of gravity coordinate can be calculated by x = x of the upper left coordinate + width / 2, y = y of the upper left coordinate + height / 2. In the case of the coordinates shown in FIG. 4(b), the distance between the center of gravity coordinate of ID.5 and ID.1 to 4 is 2059, 2316, 2178, and 3067, respectively, with the decimal points rounded down. It is determined whether the inverse of the value obtained by dividing these distances by a fixed number of 100 is 1 or more. However, the calculation method of the shooting condition evaluation value is not limited to the above example.

[0048] If there is at least one subject candidate whose shooting condition evaluation value exceeds the first threshold value (YES in S604), the control unit 207 proceeds to S605; if there is no subject candidate whose shooting condition evaluation value exceeds the first threshold value (NO in S604), the control unit 207 proceeds to S606.

[0049] In S605, the control unit 207 executes an immediate shooting process to obtain the actual image data. The immediate shooting process is a shooting process that combines cut-out shooting and shooting involving PTZ control, and details thereof will be described later as a subroutine with reference to FIG.

[0050] In S606, the control unit 207 judges the gaze condition. In judging the gaze condition, for all subject candidates registered in the subject candidate list, a gaze condition evaluation value based on changes in the position and shape patterns is calculated, and it is judged whether or not the gaze condition evaluation value exceeds a predetermined threshold value (hereinafter referred to as the "second threshold value"). This judgment is performed after S604 using the sub-image used in judging the shooting condition in S604. Thereby, when there is a subject candidate whose shooting condition evaluation value exceeds the first threshold value, the time lag from when the sub-image is acquired to when the main shooting for acquiring the main image data is performed can be shortened compared to the case where S606 is performed before S604.

[0051] Here, the first to sixth determination methods will be described as specific examples of the method for determining the gaze condition evaluation value. In the first determination method for the gaze condition evaluation value, first, the inverse of the value obtained by dividing the distance between the center coordinates of two or more subject candidates by a fixed value is obtained as the "gaze value". Then, the number of times the gaze value exceeds a certain value indicating that two or more subject candidates remain within a certain range for a certain period or more is set as the gaze condition evaluation value. Then, the IDs of the subject candidates whose gaze value is equal to or greater than the certain value, the gaze condition evaluation value, and the type of the gaze condition evaluation value are managed in a list (not shown). For each ID, if the gaze value is equal to or greater than the certain value, the gaze condition evaluation value of the ID in the list is incremented, and if the gaze value is less than the certain value, the gaze condition evaluation value of the ID in the list is set to 0 (zero).

[0052] In the second method of determining the gaze condition evaluation value, a value representing the magnitude of change in the aspect ratio of the area information of the subject candidate is calculated by (previous height / previous width)-(current height / current width), and the absolute value is set as the gaze value. The number of times the gaze value exceeds a predetermined fixed value is set as the gaze condition evaluation value. The IDs of the subject candidates whose gaze values ​​are equal to or greater than the fixed value, the gaze condition evaluation values, and the types of the gaze condition evaluation values ​​are managed in a list (not shown). For each ID, if the gaze value is equal to or greater than the fixed value, the gaze condition evaluation value of the ID in the list is incremented, and if the gaze value is less than the fixed value, the gaze condition evaluation value of the ID in the list is set to 0.

[0053] In the third method of determining the gaze condition evaluation value, a gaze value is used that indicates whether or not the subject candidate moves at a predetermined speed or more for a certain period of time. Specifically, the movement speed of the subject candidate is calculated as the gaze value, which is the change amount of the upper left coordinate of the subject candidate (the distance between the previous upper left coordinate and the current upper left coordinate divided by the time difference between the acquisition of the previous and current images). The number of times the gaze value exceeds a predetermined value is set as the gaze condition evaluation value. The IDs of the subject candidates whose gaze value is equal to or greater than the certain value, the gaze condition evaluation value, and the type of the gaze condition evaluation value are managed in a list (not shown). For each ID, if the gaze value continues to be equal to or greater than the certain value for a certain period of time, the gaze condition evaluation value of the ID in the list is incremented, and if it is less than the certain value or for less than the certain period of time, the gaze condition evaluation value of the ID in the list is set to 0.

[0054] In a fourth method for determining the gaze condition evaluation value, when the facial angles (Param3 in Figure 4(b)) of multiple subject candidates are facing the same area, the number of those faces is used as the gaze condition evaluation value, and the gaze condition evaluation value and the IDs of the corresponding subject candidates are managed in a list (not shown).

[0055] In the fifth method of determining the gaze condition evaluation value, the ratio of the area information of the subject candidate included in a designated area designated in advance within the angle of view (area of ​​the subject candidate included in the designated area / area of ​​the area information of the subject candidate) is set as the gaze condition evaluation value. The gaze condition evaluation value, the ID of the corresponding subject candidate, and the type of the gaze condition evaluation value are managed in a list (not shown).

[0056] In the sixth determination method of the gaze condition evaluation value, a value representing the result that none of the conditions used in the first to fifth determination methods are satisfied is set as the gaze value, and when none of the conditions are satisfied, it is set as 1, and when at least one condition is satisfied, it is set as 0. The number of times the gaze value exceeds a predetermined fixed value is set as the gaze condition evaluation value. The gaze condition evaluation value, the ID of the subject candidate whose gaze value is equal to or greater than the fixed value, the gaze condition evaluation value, and the type of the gaze condition evaluation value are managed in a list (not shown). For each ID, if the change in the gaze value is equal to or greater than the fixed value, the gaze condition evaluation value of the ID in the list is incremented, and if it is less than the fixed value, the gaze condition evaluation value of the ID in the list is set to 0.

[0057] The control unit 207 updates the list for determining the gaze condition using any one of the first to sixth determination methods illustrated, and determines whether or not the gaze condition evaluation value exceeds a threshold value (hereinafter referred to as the "second threshold value") that is predetermined for each type of gaze condition evaluation value. If the control unit 207 determines that there is a subject candidate whose gaze condition evaluation value exceeds the second threshold value (YES in S606), the process proceeds to S607, and if it determines that there is no subject candidate whose gaze condition evaluation value exceeds the second threshold value (NO in S606), the process proceeds to S608. Note that, although it is sufficient to execute at least one of the first to sixth determination methods in S606, when executing a plurality of determination methods, the process proceeds to S607 when it is determined that there is a subject candidate whose gaze condition evaluation value exceeds the second threshold value in at least one of the first to sixth determination methods.

[0058] In S607, the control unit 207 executes tracking shooting processing accompanied by PTZ control. Note that details of the tracking shooting processing will be described later as a subroutine with reference to FIG.

[0059] In S608, the control unit 207 determines whether or not an operation to end automatic shooting has been performed on the operation unit of the imaging device 100, similarly to S504 in the subject candidate detection process. If the control unit 207 determines that the end operation has not been performed (NO in S608), the control unit 207 returns the process to S601, and if the control unit 207 determines that the end operation has been performed (YES in S608), the control unit 207 ends this process.

[0060] Next, the contents of the immediate shooting process of S605 will be described. Fig. 7 is a flowchart of the immediate shooting process of S605. When the control unit 207 determines in S604 that there is a subject candidate whose shooting condition evaluation value exceeds the first threshold, the control unit 207 starts shooting at the maximum resolution in S701. As a result, image data at the maximum resolution that can be shot by the imaging device 100 is output from the imaging unit 202 to the image processing unit 203.

[0061] In S702, the control unit 207 instructs the image processing unit 203 to execute the cut-out and save process, and the image processing unit 203 executes the cut-out and save process. The cut-out and save process is a process of cutting out an image so as to include a region indicated by region information of a subject candidate determined to match the determination condition in the shooting condition determination in S604 (the shooting condition evaluation value exceeds the first threshold), generating image data, and saving the image data in the image recording unit 204.

[0062] In S703, the control unit 207 calculates a PTZ control value. Here, the PTZ control value is calculated so that the area including the area information of the subject candidate having the maximum shooting condition evaluation value calculated in S604 (determination of shooting conditions) is set as the target angle of view, and shooting can be performed at that angle of view.

[0063] Here, when the subject candidate with the maximum shooting condition evaluation value is determined by comparing the shooting condition evaluation values ​​of different conditions, the subject candidate with the maximum shooting condition evaluation value is identified by comparing the shooting condition evaluation values ​​with a value weighted in advance. The PTZ control value is a specific control amount sent to the lens barrel driving unit 206 and the lens driving unit 205 to set the main image data output from the imaging unit 202 of the lens barrel 101 and generated by the image processing unit 203 to a target angle of view. In addition, in S703, the Z control value (zoom control value) is calculated to be a value that substantially makes the shooting angle of view more telephoto than the current shooting angle of view.

[0064] In S704, the control unit 207 holds the current angle of view setting information. Specifically, the control unit 207 acquires and holds position information of the movable lenses constituting the lens unit 201 and angle information of the tilt unit 102 and the pan unit 103 via the lens driving unit 205 and the lens barrel driving unit 206.

[0065] In S705, the control unit 207 controls the driving device. Specifically, the control unit 207 controls the position of the movable lens constituting the lens unit 201 and the angles of the tilt unit 102 and the pan unit 103 using the PTZ control value calculated in S703 via the lens driving unit 205 and the lens barrel driving unit 206. This completes preparations for shooting the subject candidate with the maximum shooting condition evaluation value.

[0066] In S706 , the control unit 207 performs shooting processing, and stores the main image data generated by the image processing unit 203 in the image recording unit 204 .

[0067] In S707, the control unit 207 restores the angle of view setting information. That is, the control unit 207 first calculates a PTZ control value for returning to the position information and angle information held in S704. Then, the control unit 207 immediately restores the position of the movable lens constituting the lens unit 201 and the angles of the tilt unit 102 and pan unit 103 to the state before the shooting process via the lens driving unit 205 and the lens barrel driving unit 206. The control unit 207 ends this process after the process of S707, and the process is returned to S601.

[0068] Next, the contents of the tracking photography process in S607 will be described. FIG. 8 is a flowchart of the tracking photography process in S607. When the control unit 207 determines in S606 that there is a subject candidate whose gaze condition evaluation value exceeds the second threshold, the control unit 207 starts the tracking photography process, and holds the view angle setting information in S801. Specifically, the control unit 207 acquires and holds position information of the movable lens constituting the lens unit 201 and angle information of the tilt unit 102 and the pan unit 103 via the lens driving unit 205 and the lens barrel driving unit 206. Note that the control unit 207 starts the tracking photography process in S801 and holds the system time at the start of the tracking photography process in order to calculate the elapsed time used for timeout determination in S807 described later.

[0069] In S802, the control unit 207 executes a tracking process for the subject candidate with the highest gaze condition evaluation value in the gaze condition determination in S606. However, when determining the subject candidate with the highest gaze condition evaluation value by comparing the gaze condition evaluation values ​​of different conditions, the comparison is performed using a value obtained by weighting each gaze condition evaluation value in advance. The tracking process in S802 is performed in the same manner as the tracking process in S602, and a description thereof will be omitted here.

[0070] In S803, the control unit 207 detects features according to the type of subject for the subject candidates tracked in S802, and updates the subject candidate list using the results. For example, as shown in Fig. 4, when the type is "person", the upper left coordinates (x, y), size (width, height), angle (Param3), and facial expression evaluation value (Param4) of the face are detected and updated.

[0071] In S804, the control unit 207 calculates a PTZ control value. In S804, the area including area information of the subject candidate tracked in S802 is set as the target angle of view, and the PTZ control value is calculated so that the subject candidate can be photographed at that angle of view. The definition of the PTZ control value is in accordance with the explanation in S703, and the explanation here is omitted. In S805, the control unit 207 controls the drive device. The control method of the drive device in S805 is the same as the control method of the drive device in S705. In S804, as in S703, a Z control value is calculated that is substantially a telephoto angle of view more than the current photographing angle of view.

[0072] In S806, the control unit 207 determines the photographing conditions. Specifically, the control unit 207 calculates a photographing condition evaluation value for the subject candidate tracked from S802, and determines whether the calculated photographing condition evaluation value exceeds a predetermined threshold value (hereinafter referred to as a "third threshold value"). The photographing condition evaluation value is as described in conjunction with the processing content in S604, and for example, the facial expression evaluation value is used as the photographing condition evaluation value when the type is "person". In addition, a different value can be used for the third threshold value for each type of subject candidate. The third threshold value may be the same value as the first threshold value used in S604, or may be a different value. If the control unit 207 determines that the photographing condition evaluation value does not exceed the third threshold value (NO in S806), the process proceeds to S807, and if the control unit 207 determines that the photographing condition evaluation value exceeds the third threshold value (YES in S806), the process proceeds to S808.

[0073] In S807, the control unit 207 calculates the elapsed time from the system time held in S801, and determines whether the calculated elapsed time exceeds a predetermined time (whether a timeout has occurred). If the control unit 207 determines that a timeout has occurred (YES in S807), the process proceeds to S809, and if the control unit 207 determines that a timeout has not occurred (NO in S807), the process returns to S802.

[0074] In S808, the control unit 207 executes shooting processing, and stores the main image data generated by the image processing unit 203 in the image recording unit 204. In the following S809, the control unit 207 restores the angle of view setting information. That is, the control unit 207 calculates a PTZ control value for returning to the angle of view setting information (position information and angle information) held in S801. Then, the control unit 207 restores the position of the movable lens constituting the lens unit 201 and the angles of the tilt unit 102 and pan unit 103 to the state before the tracking shooting processing via the lens driving unit 205 and the lens barrel driving unit 206.

[0075] In S810, the control unit 207 initializes the subject candidate list. Specifically, the control unit 207 erases all contents registered in the subject candidate list by the subject candidate detection process in S502. However, S810 does not stop the subject candidate detection process, and the subject candidate detection process continues to be executed. After the subject candidate list is initialized in S810, the process returns to S601.

[0076] As described above, the imaging device 100 according to this embodiment can adjust the shooting direction so that a predetermined subject candidate is at the center by controlling the pan angle and tilt angle according to the movement of the subject candidate. Also, the imaging device 100 can arbitrarily change the zoom magnification between a telephoto angle of view at which the subject candidate is large and a wide angle of view at which the subject candidate is small. In this way, the imaging device 100 can adjust the subject candidate to an appropriate size in the image and track it. Then, by performing the automatic shooting process as described with reference to the flowcharts of Figs. 5 to 8, it is possible to shoot a plurality of subject candidates with a wide angle of view without bias, and to shoot a subject candidate that satisfies a predetermined shooting condition with a telephoto angle of view in a large (detailed) size.

[0077] <Second embodiment> In the second embodiment, compared to the first embodiment described above, only a part of the flow of the automatic photography process is different from the flow of the automatic photography process (FIG. 6) in the first embodiment, and the rest of the configuration and processing contents are the same as those in the first embodiment. Therefore, the following description will focus on the differences from the first embodiment, and a description of the common contents will be omitted.

[0078] Fig. 9 is a flowchart of the automatic shooting process in the second embodiment. Each process (step) indicated by an S number in the flowchart in Fig. 9 is realized by the control unit 207 (CPU) loading a predetermined program stored in the ROM 208 into the system memory 209 and comprehensively controlling each unit of the imaging device 100.

[0079] The flowchart in Fig. 9 differs from the flowchart in Fig. 6 only in that a determination process in S901 is performed between S604 and S606 in the flowchart in Fig. 6. Therefore, in the flowchart in Fig. 9, the same S numbers are omitted for the same processes as those in the flowchart in Fig. 6, and their explanations are omitted. In addition, the immediate shooting process in S605 is performed according to the flowchart in Fig. 7, and the tracking shooting process in S607 is performed according to the flowchart in Fig. 7.

[0080] In S901, the control unit 207 performs a frequency check. The frequency check is a process of checking whether the shooting condition determination in S604 has been performed a predetermined number of times (hereinafter referred to as the "preset number of times"). For example, a count is incremented each time S901 is passed. In other words, the number of times the determination result in S604 is "NO" is counted. Then, it is determined whether the count value exceeds, for example, five times, which is an example of the preset number of times.

[0081] When the control unit 207 determines that the frequency of the determination process of S604 has exceeded the preset number of times (YES in S901), the process proceeds to S606, and when the control unit 207 determines that the frequency of the determination process of S604 has not exceeded the preset number of times (NO in S901), the process returns to S604. This allows the shooting condition determination, which is more important than the gaze condition determination, to be performed more frequently.

[0082] As described above, in this embodiment, when it is determined that a situation requires tracking of a subject candidate at a telephoto angle of view using zoom control, control is performed so that tracking shooting processing with optical zoom is performed for only the required period. In addition, shooting conditions are determined even while tracking a subject candidate at a wide angle of view, and when it becomes necessary to suddenly change the shooting conditions, shooting involving cut-out shooting (S702) and PTZ control (S703) is performed. This makes it possible to obtain a more significant effect of being able to shoot a plurality of subject candidates at a wide angle of view without bias than in the first embodiment.

[0083] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0084] For example, in the above embodiment, in the subject candidate detection process (S502), the object detection process may be performed on the entire surface of the sub-image data each time. Also, when the subject candidate list is blank, the object detection process may be performed on the entire surface of the sub-image data, and when the subject candidate list is not blank (when subject candidates have been detected), only objects that have entered the screen may be detected. Furthermore, in the shooting condition determination in S604 or S806, when at least two subject candidates are at a distance equal to or less than a predetermined value, the determination may be performed using a value obtained by adding up the facial expression evaluation values ​​of the subject candidates.

[0085] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0086] 100 Imaging device 101 Telescope tube 102 Tilt unit 103 Bread Unit 104 Control Unit 201 Lens unit 202 Imaging unit 206 Tube Drive Unit 207 Control Unit

Claims

1. An imaging means; A driving means for changing a pan angle, a tilt angle and a zoom magnification of the imaging means; a detection means for detecting a subject candidate from an image captured by the imaging means with a wide angle of view; a tracking means for tracking the subject candidate at a wide angle of view based on the amount of change in position and size of the subject candidate detected by the detection means; a first determination means for determining whether or not a subject candidate that satisfies a predetermined photographing condition exists while tracking the subject candidate by the tracking means; a second determination means for determining whether or not a subject candidate that satisfies a predetermined gaze condition is present while tracking the subject candidate by the tracking means; a tracking means for tracking the subject candidate detected by the detection means at a telephoto angle of view; and a control means for controlling the subject candidate to perform an immediate shooting process in which the subject candidate is shot at a wide-angle angle of view when a subject candidate that satisfies the shooting conditions is present, and to perform a tracking shooting process in which the angle of view is returned to the wide-angle angle of view after shooting the subject candidate using the tracking means in accordance with a determination result by the first determination means when a subject candidate that satisfies the gaze condition is present.

2. 2. The image pickup apparatus according to claim 1, wherein the first determination means performs the determination more frequently than the second determination means.

3. The determination by the second determination means of the presence or absence of a subject candidate satisfying the gaze condition based on the first image acquired by the imaging means is 3. The imaging apparatus according to claim 1, wherein the determination is performed after the first determination means determines whether or not there is a subject candidate that satisfies the photographing condition based on the first image.

4. The imaging device according to any one of claims 1 to 3, characterized in that the detection means detects an object present within the angle of view as a subject candidate when a subject candidate has not been detected, and detects an object that has entered the angle of view as a subject candidate when a subject candidate has already been detected.

5. The imaging device according to any one of claims 1 to 4, characterized in that, in the immediate shooting process, the control means performs extraction of a subject candidate in an image captured at a wide-angle angle of view, shooting one of the subject candidate in the image at a telephoto angle of view, and returning to the wide-angle angle of view after shooting at the telephoto angle of view.

6. 6. The imaging device according to claim 1, wherein in the tracking shooting process, if a subject candidate that satisfies the gaze condition satisfies the shooting condition, the subject candidate is photographed with a telephoto angle of view, and if a predetermined time has elapsed without the shooting condition being satisfied, the subject candidate is not photographed.

7. The imaging device according to any one of claims 1 to 6, characterized in that the second determination means determines that there is a subject candidate that satisfies the gaze condition when a distance between two or more subject candidates detected by the detection means remains within a certain range for a certain period of time or more.

8. The imaging device according to any one of claims 1 to 6, characterized in that the second determination means determines that there is a subject candidate that satisfies the gaze condition when the aspect ratio of the subject candidate detected by the detection means continues to change for a certain period of time or more.

9. The imaging device according to any one of claims 1 to 6, characterized in that the second determination means determines that there is a subject candidate that satisfies the gaze condition when the subject candidate detected by the detection means is moving at or above a predetermined speed for or above a certain period of time.

10. The imaging device according to any one of claims 1 to 6, characterized in that the second determination means determines that there is a subject candidate that satisfies the gaze condition when a number of subject candidates that are facing the same direction among the multiple subject candidates detected by the detection means is equal to or greater than a certain value.

11. The imaging device according to any one of claims 1 to 6, characterized in that the second determination means determines that there is a subject candidate that satisfies the gaze condition when the subject candidate detected by the detection means falls within a designated area that is specified in advance within the angle of view.

12. A method for controlling an imaging device capable of changing a pan angle, a tilt angle, and a zoom magnification of an imaging means, comprising: a detection step of detecting a subject candidate from an image captured by the imaging means with a wide angle of view; a tracking step of tracking the subject candidate at a wide angle of view based on a change in position and size of the detected subject candidate; a first determination step of determining whether or not there is a subject candidate that satisfies a predetermined photographing condition while tracking the subject candidate; a first photographing step of photographing a subject candidate that satisfies the photographing condition with a wide-angle view; a second determination step of determining whether or not there is a subject candidate that satisfies a predetermined gaze condition; a second photographing step of photographing the subject candidate when the photographing condition is satisfied while tracking the subject candidate with a telephoto angle of view when the photographing condition is satisfied; a control step of returning the imaging means to a wide angle of view after the second photographing step.

13. A program for causing a computer to function as each of the means of the imaging apparatus according to any one of claims 1 to 11.

Images