Imaging device, control method thereof, and program

Abstract

To control a timing of performing automatic authentication registration of a subject in an imaging device capable of automatic photographing.SOLUTION: An imaging device can perform automatic photographing and automatic authentication registration of a subject. The imaging device includes a driving unit that causes a lens tube 102 to rotate in a panning direction and a tilting direction, and can change an imaging direction by control of the driving unit. The imaging device can search for a subject that is detected from captured image data and perform automatic authentication registration in which the subject is authenticated and stored. A first control unit 223 of the imaging device makes a determination of whether a condition for performing automatic authentication registration is satisfied and a determination of whether a condition for performing automatic photographing is satisfied. The first control unit 223 executes automatic authentication registration determination processing and automatic photographing determination processing while performing a search for performing automatic photographing; and determines a timing of performing automatic authentication registration based on a determination result.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an automatic photography technique in an imaging device. [Background technology]

[0002] When taking still or moving images using an imaging device, it is common for the photographer to determine the subject of the image through a viewfinder, etc., check the shooting conditions by himself, and adjust the framing of the captured image. Conventional technology includes a mechanism that detects user operation errors or the external environment, notifies the user that the conditions are not suitable for shooting, or controls the camera to make the conditions suitable for shooting.

[0003] In contrast to an imaging device that performs shooting by user operation, Patent Document 1 discloses a life log camera that periodically and continuously shoots without the user giving a shooting instruction. The life log camera is used while being attached to the user's body with a strap or the like, and records scenes that the user sees in daily life as images at regular intervals. When shooting with a life log camera, the image is not taken at the timing intended by the user, such as pressing the shutter button, but is taken at regular intervals. Therefore, it is possible to record images of unexpected moments that the user would not normally shoot. In addition, an imaging device that automatically shoots an object is known. Patent Document 2 discloses a device that automatically shoots when a certain condition is met.

[0004] On the other hand, Patent Document 3 discloses an imaging device equipped with a personal authentication function that stores information about a subject and uses it to determine who is being photographed. This imaging device prioritizes focusing on a subject based on stored information. Personal authentication is a process of identifying an individual by quantifying features such as a face, but the features of a face change as the person grows, and change depending on the angle of the face and how the light hits it. Since it is difficult to identify an individual using only a single piece of feature data, a method is used to improve authentication accuracy by using multiple feature data for the same person. Patent Document 4 also discloses a method of separating a camera for storing information about a subject from a camera for photographing the subject. It is possible to control the timing of photographing the subject for personal authentication and the timing of photographing the subject independently. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2016-536868 [Patent Document 2] JP 2001-51338 A [Patent Document 3] Patent No. 4634527 [Patent Document 4] JP 2007-325285 A Summary of the Invention [Problem to be solved by the invention]

[0006] With conventional technology, when the requirements for automatic photography and the requirements for automatic authentication and registration are different, it is difficult to satisfy both requirements in a single photography session. An object of the present invention is to control the timing of automatic authentication and registration of a subject in an image capturing device capable of automatic photography. [Means for solving the problem]

[0007] The imaging device according to an embodiment of the present invention is an imaging device capable of automatic photography and automatic authentication and registration, and includes an imaging unit for imaging a subject, a search unit for searching for a subject detected from image data acquired by the imaging unit, and a search unit for searching the detected subject. Automatic authentication and registration and a control means for performing an authentication registration judgment by the authentication registration means as to whether a first condition for performing the automatic authentication registration is satisfied or not, and a photographing judgment as to whether a second condition for performing the automatic photographing is satisfied or not, and controlling the timing of the automatic photographing and the automatic authentication registration. The control means determines the timing of the automatic authentication registration by performing the authentication registration judgment and the photographing judgment related to the detected subject while controlling the search by the search means. If the first condition is satisfied as a result of the authentication / registration judgment and the photographing judgment, the authentication / registration means performs automatic authentication / registration of the detected subject, and if the first condition is not satisfied and the second condition is satisfied, the control means controls the automatic photographing. It is characterized by: Effect of the Invention

[0008] According to the imaging device of the present invention, it is possible to control the timing of automatic authentication and registration of a subject in an imaging device capable of automatic photography. [Brief description of the drawings]

[0009] [Figure 1] 1A and 1B are diagrams illustrating the appearance and driving direction of a camera according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing the overall configuration of a camera according to an embodiment of the present invention; [Diagram 3] FIG. 1 is a diagram illustrating an example of the configuration of a wireless communication system between a camera and an external device. [Figure 4] FIG. 4 is a block diagram showing a configuration of an external device in FIG. [Diagram 5] FIG. 2 is a diagram showing the configuration of a camera and an external device. [Figure 6] FIG. 6 is a block diagram showing a configuration of an external device in FIG. 5. [Figure 7] 6 is a flowchart illustrating the operation of a first control unit. [Figure 8] 10 is a flowchart illustrating the operation of a second control unit. [Figure 9]11 is a flowchart illustrating a shooting mode process. [Figure 10] FIG. 2 is an explanatory diagram of area division within a captured image. [Figure 11] 11 is a table showing execution decisions based on automatic authentication registration decisions and automatic photography decisions. [Figure 12] 11A and 11B are explanatory diagrams of subject placement in composition adjustment. [Figure 13] FIG. 1 is an explanatory diagram of a neural network. [Figure 14] FIG. 13 is a diagram showing a state in which an image is viewed on an external device. [Figure 15] 10 is a flowchart illustrating a learning mode determination. [Figure 16] 13 is a flowchart illustrating a learning mode process. [Figure 17] FIG. 1 is a block diagram showing a configuration of an imaging apparatus. [Figure 18] 11 is a table showing an example of person information. [Figure 19] FIG. 13 is a diagram showing an example of a screen of person information displayed on an external device. [Figure 20] 4A and 4B are diagrams illustrating examples of image data and subject information. [Figure 21] 11 is a flowchart for explaining an outline of a periodic operation performed by the imaging device. [Figure 22] 13 is a flowchart and a table illustrating a provisional registration determination process. [Figure 23] 13A and 13B are diagrams and a table showing image data after the angle of view has been adjusted based on provisional registration determination. [Figure 24] 13 is a flowchart and a table illustrating a main registration determination process. [Diagram 25] 13 is a flowchart illustrating a first main registration count determination process. [Figure 26] 13 is a flowchart illustrating a second main registration count determination process. [Figure 27] 13 is a flowchart and a table illustrating a photographing subject determination process. [Figure 28] 4A and 4B are diagrams illustrating examples of image data and subject information. [Figure 29]11A and 11B are diagrams illustrating an example of an image after the angle of view is adjusted based on the subject to be photographed. [Diagram 30] FIG. 11 is a diagram showing an example of registered person information. [Diagram 31] 4A and 4B are diagrams illustrating examples of image data and subject information. [Diagram 32] 11 is a flowchart for explaining an outline of a periodic operation performed by the imaging device. [Diagram 33] 13 is a flowchart and a table illustrating an importance determination process. [Diagram 34] 13 is a flowchart and a table illustrating a photographing subject determination process. [Diagram 35] 13A and 13B are diagrams illustrating examples of image data and subject information according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. First, the technical background of the present invention will be described. For example, in the case of shooting for the purpose of life logging, shooting is performed periodically and continuously, so there is a possibility that image information that is uninteresting to the user will be recorded. Therefore, there is a method of automatically performing panning and tilting operations of the imaging device to search for surrounding subjects and shoot with an angle of view that includes the detected subject. This can increase the possibility of recording image information that is preferable to the user.

[0011] An imaging device that can automatically control the shooting direction is required to search for subjects to be photographed and not miss the right timing to shoot at the same time. It is necessary to adjust the shooting composition using the panning, tilting and zooming mechanisms while taking into account the number of subjects, their movement direction and the background, and to perform the shooting operation as soon as the right timing is grasped.

[0012] Furthermore, by using the personal authentication information, it is possible to detect subjects that should be given priority in the search and to use it to determine subjects that should be included in the angle of view when taking a picture, thereby increasing the possibility of recording an image that is more preferable to the user.

[0013] However, in an imaging device capable of automatic shooting, if the registration of personal authentication is not performed automatically, the convenience may be significantly reduced. In personal authentication, the identification process of an individual is performed by quantifying the feature amount obtained from the image of the face. However, if the numerical value changes due to changes accompanying the growth of the person, a slight change in the angle of the face, or a slight change in the amount of light irradiated on the face, the person may not be considered as the same person when they should be the same person. In this case, if the subject tracking control erroneously recognizes a different person as a result of erroneous authentication, the imaging device tracks the different person, and as a result, a problem occurs in that the opportunity to photograph the person that should be photographed is missed. Therefore, in an imaging device capable of automatic shooting, the reliability of personal authentication is directly linked to the reliability of automatic shooting. Regarding the registration information of personal authentication for the same person, it is important to maintain and improve the authentication accuracy by using multiple registration information by adding the registration information as needed, and the registration information should be updated automatically. In order to realize automatic shooting with higher performance and higher convenience, automatic registration of personal authentication becomes very important.

[0014] More accurate personal authentication registration requires highly accurate face image data. In other words, it is assumed that the composition is arranged at the optical center, which is least affected by the aberration of the optical lens. In addition, an image that captures a large area of ​​the face is required, and it is necessary to use the still image shooting function of the imaging device to obtain a high-resolution image focused on the subject. However, in automatic shooting, the composition is adjusted taking into account multiple subjects and the background so as not to miss a shutter opportunity. Therefore, there may be cases where the conditions required for automatic shooting and the composition adjustment conditions required for personal authentication registration cannot be satisfied at the same time. Therefore, in this embodiment, an example of an imaging device that can control the timing to perform automatic registration for personal authentication without hindering the shooting opportunity of automatic shooting will be described.

[0015] FIG. 1A is a diagram showing a schematic view of the appearance of the imaging device of this embodiment. In addition to a power switch, the camera 101 is provided with an operating member for operating the camera. The lens barrel 102 integrally includes a group of photographing lenses and an imaging element as an imaging optical system for capturing an image of a subject, and is movably attached to a fixed part 103 of the camera 101. Specifically, the lens barrel 102 is attached to the fixed part 103 via a first rotation unit 104 and a second rotation unit 105, which are mechanisms that can be rotated relative to the fixed part 103, and the shooting direction can be changed. The first rotation unit 104 is a unit that drives the lens barrel 102 in the tilting direction (hereinafter referred to as a tilt rotation unit). The second rotation unit 105 is a unit that drives the lens barrel 102 in the panning direction (hereinafter referred to as a pan rotation unit). The angular velocity meter 106 and the accelerometer 107 are disposed on the fixed part 103 of the camera 101. For example, the angular velocity meter 106 includes a gyro sensor, and the accelerometer 107 includes an acceleration sensor.

[0016] 1(B) is a schematic diagram showing the relationship between a three-dimensional orthogonal coordinate system (X-axis, Y-axis, Z-axis) and three directions (pitch, yaw, roll). The X-axis (horizontal axis), Y-axis (vertical axis), and Z-axis (depth direction axis) are each defined relative to the position of fixed part 103. The direction around the X-axis is the pitch direction, the direction around the Y-axis is the yaw direction, and the direction around the Z-axis is the roll direction.

[0017] Tilt rotation unit 104 includes a motor drive mechanism capable of rotating barrel 102 in the pitch direction shown in Fig. 1(B). Pan rotation unit 105 includes a motor drive mechanism capable of rotating barrel 102 in the yaw direction shown in Fig. 1(B). In other words, camera 101 has a mechanism for rotating barrel 102 in two axial directions.

[0018] The angular velocity meter 106 and the accelerometer 107 output an angular velocity detection signal and an acceleration detection signal, respectively. Vibrations of the camera 101 are detected based on the output signals of the angular velocity meter 106 and the accelerometer 107, and the tilt rotation unit 104 and the pan rotation unit 105 are rotated. This corrects the shake and inclination of the lens barrel 102. In addition, based on the output signals of the angular velocity meter 106 and the accelerometer 107, movement of the camera 101 is detected based on the measurement results for a certain period of time.

[0019] 2 is a block diagram showing the overall configuration of the camera 101. The first control unit 223 includes an arithmetic processing unit. The arithmetic processing unit is a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The memory 215 includes a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory), etc. The first control unit 223 executes various processes according to a program stored in a non-volatile memory (EEPROM) 216 to control each block of the camera 101 and control data transfer between each block. The non-volatile memory 216 is an electrically erasable and storeable memory, and stores constants, programs, etc. for the operation of the first control unit 223.

[0020] The zoom unit 201 includes a zoom lens that changes magnification (enlarges and reduces the image of a subject that is formed). The zoom drive control unit 202 drives and controls the zoom unit 201, and detects the focal length during drive control. The focus unit 203 includes a focus lens that adjusts the focus. The focus drive control unit 204 drives and controls the focus unit 203. The imaging unit 206 includes an image sensor, receives light incident through each lens group, and outputs charge information according to the amount of light to the image processing unit 207 as an analog image signal. The zoom unit 201, focus unit 203, and imaging unit 206 are disposed inside the lens barrel 102.

[0021] An image processing unit 207 performs image processing on digital image data obtained by A / D conversion of an analog image signal. The image processing includes distortion correction, white balance adjustment, color interpolation processing, etc., and the image processing unit 207 outputs the digital image data after the image processing. An image recording unit 208 acquires the digital image data output from the image processing unit 207. The digital image data is converted into a recording format such as JPEG (Joint Photographic Experts Group) format. The converted data is stored in a memory 215, and is also transmitted to a video output unit 217, which will be described later.

[0022] The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to rotate the lens barrel 102 in the tilting direction and the panning direction. The device vibration detection unit 209 includes an angular velocity meter 106 that detects the angular velocity of the camera 101 in the three axial directions, and an accelerometer 107 that detects the acceleration of the camera 101 in the three axial directions. The first control unit 223 calculates the rotation angle of the device, the shift amount of the device, and the like based on the detection signal from the device vibration detection unit 209.

[0023] The audio input unit 213 acquires audio signals from the surroundings of the camera 101 using a microphone provided in the camera 101, converts the signals into digital audio signals, and transmits the signals to the audio processing unit 214. The audio processing unit 214 performs audio-related processing such as optimization of the input digital audio signals. The audio signals processed by the audio processing unit 214 are transmitted to the memory 215 by the first control unit 223. The memory 215 temporarily stores the image signals and audio signals obtained by the image processing unit 207 and the audio processing unit 214.

[0024] The image processing unit 207 and the audio processing unit 214 read out the image signal and the audio signal temporarily stored in the memory 215, and encode the image signal and the audio signal, etc., to generate a compressed image signal and a compressed audio signal. The first control unit 223 transmits the generated compressed image signal and compressed audio signal to the recording and playback unit 220.

[0025] The recording and reproducing unit 220 records the compressed image signal and compressed audio signal generated by the image processing unit 207 and the audio processing unit 214, control data related to shooting, etc., on the recording medium 221. When the audio signal is not compression-encoded, the first control unit 223 transmits the audio signal generated by the audio processing unit 214 and the compressed image signal generated by the image processing unit 207 to the recording and reproducing unit 220 to record them on the recording medium 221.

[0026] The recording medium 221 is a recording medium built into the camera 101 or a removable recording medium. The recording medium 221 can record various data such as compressed image signals, compressed audio signals, and audio signals generated by the camera 101. Generally, a medium with a larger capacity than the nonvolatile memory 216 is used for the recording medium 221. For example, any type of recording medium can be used for the recording medium 221, such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a nonvolatile semiconductor memory, or a flash memory.

[0027] The recording and reproducing unit 220 reads out and reproduces the compressed image signal, compressed audio signal, audio signal, various data, and programs recorded on the recording medium 221. The first control unit 223 transmits the read out compressed image signal and compressed audio signal to the image processing unit 207 and audio processing unit 214, respectively. The image processing unit 207 and audio processing unit 214 temporarily store the compressed image signal and compressed audio signal in the memory 215, decode them in a predetermined procedure, and transmit the decoded signal to the video output unit 217.

[0028] A plurality of microphones are arranged in the audio input unit 213 of the camera 101. The audio processing unit 214 can detect the direction of a sound relative to a plane on which the plurality of microphones are arranged, and the detection information is used for searching for a subject and automatic shooting, which will be described later. The audio processing unit 214 detects a specific audio command. The audio command is, for example, several commands registered in advance, or, in an embodiment in which the user can register a specific voice in the camera, a command based on a registered voice. The audio processing unit 214 also performs sound scene recognition. In sound scene recognition, a determination process of a sound scene is performed by a network that has been machine-learned based on a large amount of voice data in advance. For example, a network for detecting specific scenes such as "cheers are being raised," "claps are being made," and "voice is being uttered" is set in the audio processing unit 214, and a specific sound scene or a specific voice command is detected. When the audio processing unit 214 detects a specific sound scene or a specific voice command, it outputs a detection trigger signal to the first control unit 223 or the second control unit 211.

[0029] The second control unit 211 is provided separately from the first control unit 223 that controls the entire camera system, and controls the power supply to the first control unit 223. The first power supply unit 210 and the second power supply unit 212 supply power for operating the first control unit 223 and the second control unit 211, respectively. When the power button provided on the camera 101 is pressed, power is first supplied to both the first control unit 223 and the second control unit 211. As described later, the first control unit 223 also controls to turn off its own power supply to the first power supply unit 210. Even when the first control unit 223 is not operating, the second control unit 211 operates, and information from the device vibration detection unit 209 and the audio processing unit 214 is input to the second control unit 211. The second control unit 211 determines whether or not to start the first control unit 223 based on various input information. When it is determined that the first control unit 223 is to be started, the second control unit 211 instructs the first power supply unit 210 to supply power to the first control unit 223 .

[0030] Audio output unit 218 has a speaker built into camera 101, and outputs audio of a preset pattern from the speaker, for example, during shooting. LED control unit 224 controls LEDs (light emitting diodes) provided in camera 101. Furthermore, during shooting, the LEDs are controlled based on preset lighting patterns and blinking patterns.

[0031] The video output unit 217 has, for example, a video output terminal, and outputs an image signal to display a video on a connected external display, etc. The audio output unit 218 and the video output unit 217 may be combined into one terminal, for example, an HDMI (registered trademark: High-Definition Multimedia Interface) terminal.

[0032] The communication unit 222 is a processing unit that performs communication between the camera 101 and an external device. For example, the communication unit 222 transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. The communication unit 222 receives control signals related to shooting, such as commands to start and end shooting, pan / tilt, and zoom drive, and outputs them to the first control unit 223. This allows the camera 101 to be driven based on instructions from the external device. The communication unit 222 also transmits and receives information such as various parameters related to learning processed by the learning processing unit 219 described later between the camera 101 and the external device. The communication unit 222 includes a wireless communication module, such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a Wireless USB (registered trademark), and a GPS receiver.

[0033] The environmental sensor 226 periodically detects the state of the environment surrounding the camera 101. The environmental sensor 226 is configured using, for example, the sensors listed below. A temperature sensor that detects the temperature around the camera 101. An air pressure sensor that detects the air pressure around the camera 101. An illuminance sensor that detects the brightness around the camera 101. A humidity sensor that detects the humidity around the camera 101. A UV sensor that detects the amount of ultraviolet light around the camera 101. In addition to the various detected information (temperature, air pressure, illuminance, humidity, UV), the rate of change at a specified time interval can be calculated from the various information. In other words, the amount of change in temperature, air pressure, illuminance, humidity, and UV can be used to determine automatic photography, etc.

[0034] Communication between the camera 101 and the external device 301 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of a wireless communication system between the camera 101 and the external device 301. The camera 101 is a digital camera with a photographing function, and the external device 301 is a smart device including a Bluetooth (registered trademark) communication module and a wireless LAN communication module.

[0035] In FIG. 3, the communication between the camera 101 and the external device 301 is shown as a first communication 302 (see the solid arrow) and a second communication 303 (see the dotted arrow). For example, the first communication 302 is a communication by wireless LAN (Local Area Network) conforming to the IEEE802.11 standard series. The second communication 303 is a communication having a master-slave relationship such as a control station and a dependent station, such as Bluetooth (registered trademark) Low Energy (hereinafter referred to as "BLE"). Note that wireless LAN and BLE are examples of communication methods. Each communication device has two or more communication functions, and other communication methods may be used as long as one communication function that communicates in the relationship between a control station and a dependent station can control the other communication function. However, the first communication 302 by wireless LAN or the like is capable of faster communication than the second communication 303 by BLE or the like. In addition, the second communication 303 is at least one of lower power consumption and shorter communication distance than the first communication 302.

[0036] Next, the configuration of the external device 301 will be described with reference to Fig. 4. The external device 301 includes, for example, a wireless LAN control unit 401 for wireless LAN, a BLE control unit 402 for BLE, and a public wireless control unit 406 for public wireless communication.

[0037] The wireless LAN control unit 401 performs RF control of the wireless LAN, communication processing, driver processing for performing various controls of communication by the wireless LAN compliant with the IEEE802.11 standard series, and protocol processing related to communication by the wireless LAN. The BLE control unit 402 performs RF control of the BLE, communication processing, driver processing for performing various controls of communication by the BLE, and protocol processing related to communication by the BLE. The public wireless control unit 406 performs RF control of public wireless communication, communication processing, driver processing for performing various controls of public wireless communication, and protocol processing related to public wireless communication. The public wireless communication is, for example, communication compliant with the IMT (International Multimedia Telecommunications) standard or the LTE (Long Term Evolution) standard.

[0038] The external device 301 further includes a packet transmission / reception unit 403. The packet transmission / reception unit 403 performs processing for performing at least one of transmission and reception of packets related to communication by wireless LAN and BLE, and public wireless communication. Note that the external device 301 of this embodiment will be described as performing at least one of transmission and reception of packets in communication, but other communication formats such as circuit switching, other than packet switching, may be used.

[0039] The control unit 411 included in the external device 301 includes a CPU and the like, and controls the entire external device 301 by executing a control program stored in the storage unit 404. The storage unit 404 stores, for example, the control program executed by the control unit 411 and various information such as parameters necessary for communication. Various operations described below are realized by the control unit 411 executing the control program stored in the storage unit 404.

[0040] The GPS (Global Positioning System) receiver 405 receives a GPS signal notified from an artificial satellite, analyzes the GPS signal, and estimates the current position (longitude and latitude information) of the external device 301. Alternatively, there is an embodiment in which the current position of the external device 301 is estimated based on information of a wireless network present in the vicinity using a WPS (Wi-Fi Positioning System) or the like. For example, it is assumed that the current GPS position information acquired by the GPS receiver 405 is located within a preset position range (within a range of a predetermined radius centered on the detected position) or that the GPS position information has undergone a position change of a predetermined amount or more. In these cases, the movement information is notified to the camera 101 via the BLE control unit 402, and is used as a parameter for automatic shooting and automatic editing, which will be described later.

[0041] The display unit 407 has a function capable of outputting visually recognizable information, such as an LCD (liquid crystal display device) or LED, or outputting sound, such as a speaker, and presents various information. The operation unit 408 includes, for example, buttons for accepting operations of the external device 301 by a user. The display unit 407 and the operation unit 408 may be configured, for example, by a touch panel.

[0042] The voice input voice processing unit 409 acquires information on the voice uttered by the user, for example, by a general-purpose microphone built into the external device 301. A configuration may be adopted in which a user's operation command is identified by voice recognition processing. There is also a method of acquiring a voice command by the user's pronunciation using a dedicated application in the external device 301. In this case, a specific voice command to be recognized by the voice processing unit 214 of the camera 101 can be registered via the first communication 302 by wireless LAN. The power supply unit 410 supplies the necessary power to each unit of the external device 301.

[0043] The camera 101 and the external device 301 transmit and receive data through communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, data such as audio signals, image signals, compressed audio signals, and compressed image signals are transmitted and received. In addition, the external device 301 transmits shooting instructions to the camera 101, voice command registration data, a predetermined position detection notification based on GPS position information, a location movement notification, and the like. In addition, learning data is transmitted and received using a dedicated application in the external device 301.

[0044] 5 is a diagram showing a schematic configuration example of an external device 501 capable of communicating with the camera 101. For example, the camera 101 is a digital camera having a photographing function. The external device 501 is a wearable device including various sensing units capable of communicating with the camera 101 via a Bluetooth (registered trademark) communication module or the like.

[0045] The external device 501 is configured to be wearable on the user's arm, etc. The external device 501 is equipped with sensors that detect biological information such as the user's pulse, heart rate, and blood flow at a predetermined cycle, an acceleration sensor that can detect the user's motion state, and the like.

[0046] The biological information detection unit 602 included in the external device 501 includes, for example, a pulse sensor, a heartbeat sensor, and a blood flow sensor for detecting the user's pulse, heartbeat, and blood flow, respectively, and a sensor using a conductive polymer for detecting a change in potential due to contact with the skin. In this embodiment, the heartbeat sensor included in the biological information detection unit 602 will be used for explanation. The heartbeat sensor detects the user's heartbeat by irradiating infrared light onto the skin using, for example, an LED, detecting the infrared light that has passed through the body tissue with a light receiving sensor, and processing the signal. The biological information detection unit 602 outputs a signal of the detected biological information to the control unit 607 (see FIG. 6).

[0047] The shake detection unit 603 included in the external device 501 detects the motion state of the user. The shake detection unit 603 includes, for example, an acceleration sensor and a gyro sensor, and acquires movement information and motion detection information. The movement information is information indicating whether or not the user is moving, the moving speed, etc., based on the acceleration information. The motion detection information is detection information of motion, such as whether the user is performing an action by swinging their arms around.

[0048] The external device 501 includes a display unit 604 and an operation unit 605. The display unit 604 outputs visually perceptible information such as an LCD or LED. The operation unit 605 accepts an operation instruction for the external device 501 from a user.

[0049] 6 is a block diagram showing the configuration of the external device 501. The external device 501 includes a control unit 607, a communication unit 601, a biological information detection unit 602, a shaking detection unit 603, a display unit 604, an operation unit 605, a power supply unit 606, and a storage unit 608.

[0050] The control unit 607 includes a CPU and the like, and controls the entire external device 501 by executing a control program stored in a storage unit 608. The storage unit 608 stores, for example, the control program executed by the control unit 607 and various information such as parameters necessary for communication. Various operations described below are realized by the control unit 607 executing the control program stored in the storage unit 608. The power supply unit 606 supplies power to each unit of the external device 501.

[0051] The operation unit 605 accepts an operation instruction of the external device 501 by the user and notifies the control unit 607. The operation unit 605 also acquires a voice uttered by the user, for example, by a general-purpose microphone built into the external device 501, identifies the user's operation command by voice recognition processing, and notifies the control unit 607. The display unit 604 presents various information to the user by outputting visually recognizable information or by sound output from a speaker or the like.

[0052] The control unit 607 acquires and processes detection information from the biometric information detection unit 602 and the shaking detection unit 603. The various detection information processed by the control unit 607 is transmitted to the camera 101 by the communication unit 601. For example, the external device 501 can transmit detection information to the camera 101 when a change in the user's heart rate is detected, and can also transmit detection information when a change in the moving state occurs, such as walking, running, or stopping. The external device 501 can also transmit detection information to the camera 101 when a preset arm swinging motion is detected, and can also transmit detection information when a movement of a preset distance is detected.

[0053] An operation sequence of camera 101 will be described with reference to Fig. 7. Fig. 7 is a flowchart for explaining an example of processing performed by first control unit 223 (Main CPU) of camera 101. When a user operates a power button provided on camera 101, power is supplied from first power supply unit 210 to first control unit 223 and each component of camera 101. Power is also supplied from second power supply unit 212 to second control unit 211. Details of the operation of second control unit 211 will be described later with reference to the flowchart of Fig. 8.

[0054] 7 starts after power is supplied to the device, and in S701, the start-up conditions are read in. In this embodiment, there are the following three cases regarding the conditions for starting the power supply. (1) When the power button is manually pressed to turn on the power. (2) When a startup instruction is sent from an external device (eg, external device 301) via external communication (eg, BLE communication) and the power supply is started. (3) When the power supply is turned on by an instruction from the second control unit 211. Here, in the case of (3), that is, when the power is turned on by an instruction from the second control unit 211, the start-up conditions calculated within the second control unit 211 are read. The details will be described later with reference to FIG. 8. The start-up conditions read here are also used as one parameter element during subject search and automatic shooting, which will also be described later. When the reading of the start-up conditions in S701 is completed, the process proceeds to S702.

[0055] In S702, detection signals from various sensors are read in. The sensor signals read in here are as follows: A signal from a sensor that detects vibrations, such as a gyro sensor or an acceleration sensor in the device vibration detection unit 209. Signals of each rotation position of the tilt rotation unit 104 and the pan rotation unit 105. A voice signal detected by the voice processing unit 214, a detection trigger signal for specific voice recognition, and a sound direction detection signal. A detection signal of environmental information by the environmental sensor 226. After the detection signals from the various sensors are read in S702, the process proceeds to S703.

[0056] In S703, the first control unit 223 detects whether a communication instruction is transmitted from the external device, and when a communication instruction is received, controls communication with the external device. For example, a process of reading various information from the external device 301 is executed. The various information includes remote operation via wireless LAN or BLE, transmission and reception of audio signals, image signals, compressed audio signals, compressed image signals, and the like, operation instructions such as shooting from the external device 301, and transmission of voice command registration data. In addition, there is information such as a predetermined position detection notification based on GPS position information, a location movement notification, and transmission and reception of learning data. In addition, when it is necessary to update biometric information such as user's motion information, arm action information, and heart rate from the external device 501, a process of reading information via BLE is executed. Note that, although an example in which the environmental sensor 226 is mounted on the camera 101 has been described, it may be mounted on the external device 301 or the external device 501. In that case, a process of reading environmental information via BLE is executed in S703. After communication reading is performed in S703, the process proceeds to S704.

[0057] In S704, a mode setting determination is performed. Examples of "automatic shooting mode" (S710), "automatic editing mode" (S712), "automatic image transfer mode" (S714), "learning mode" (S716), and "automatic file deletion mode" (S718) will be described. In the next S705, a determination process is performed as to whether or not the operation mode is set to the low power consumption mode in S704. The low power consumption mode is a mode that is set when the mode is not any of the "automatic shooting mode", "automatic editing mode", "automatic image transfer mode", "learning mode", and "automatic file deletion mode". If it is determined in S705 that the mode is the low power consumption mode, the process proceeds to S706, and if it is determined in S705 that the mode is not the low power consumption mode, the process proceeds to S709.

[0058] In S706, a process is performed to notify the second control unit 211 (SubCPU) of various parameters related to the start-up cause determined within the second control unit 211. The various parameters are parameters for vibration detection determination, sound detection, and time passage detection, and the parameter values ​​change by learning in a learning process described below. When the process of S706 ends, the process proceeds to S707, where the power supply of the first control unit 223 (MainCPU) is turned off, and the series of processes ends.

[0059] In S709, a determination process is performed as to whether the mode setting in S704 is the automatic shooting mode. Then, in S711, S713, S715, and S717, a determination process is performed for each corresponding mode. Here, the mode setting determination process in S704 will be described. In the mode setting determination, a mode is selected from the modes shown in (1) to (5) below.

[0060] (1) Automatic shooting mode <Mode Judgment Condition> The condition is that it is determined that automatic photography should be performed based on information such as each detection information that has been learned, the time elapsed since switching to automatic photography mode, past photography information, and the number of photos taken, etc. Each detection information is information such as image, sound, time, vibration, location, physical changes, and environmental changes. <Processing within the mode> If it is determined in S709 that the automatic shooting mode is selected, the process proceeds to automatic shooting mode processing (S710). Based on the above-mentioned detection information that has been learned and set, pan / tilt and zoom are driven, and an automatic search for a subject is performed. When it is determined that it is the right time to take a picture according to the photographer's preference, the picture is taken automatically.

[0061] (2) Automatic editing mode <Mode Judgment Condition> The condition is that it is determined that automatic editing should be performed based on the time that has elapsed since the previous automatic editing was performed and on information about past photographed images. <Processing within the mode> If it is determined in S711 that the automatic editing mode is selected, the process proceeds to automatic editing mode processing (S712). A selection process for still images and moving images based on learning is performed, and an automatic editing process is performed to create a highlight video that is compiled into one video based on image effects, the duration of the edited video, and the like, based on the learning.

[0062] (3) Automatic image transfer mode <Mode Judgment Condition> When the automatic image transfer mode is set by an instruction using a dedicated application in the external device 301, it is determined that automatic transfer should be performed based on the elapsed time since the last image transfer and information on past captured images. <Processing within the mode> If it is determined in S713 that the automatic image transfer mode is selected, the process proceeds to automatic image transfer mode processing (S714). The camera 101 automatically extracts images that are likely to match the user's preferences, and automatically transfers the images that are likely to match the user's preferences to the external device 301. Extraction of images that match the user's preferences is performed based on a score that determines the user's preferences and is added to each image, which will be described later.

[0063] (4) Learning mode <Mode Judgment Condition> The condition is that it is determined that automatic learning should be performed based on the time elapsed since the previous learning process was performed, the information integrated with the image that can be used for learning, the number of learning data, etc. Alternatively, the learning mode is set when an instruction to set the learning mode is given via communication from the external device 301. <Processing within the mode> If the learning mode is determined to be selected in S715, the process proceeds to learning mode processing (S716). Learning tailored to the user's preferences is performed using a neural network based on each piece of operation information in the external device 301, notification of learning information from the external device 301, and the like. The operation information includes image acquisition information from the camera, information manually edited via a dedicated application, and judgment value information input by the user for an image in the camera. In addition, learning related to detection, such as personal authentication registration, voice registration, sound scene registration, and general object recognition registration, and learning of the above-mentioned low power consumption mode conditions are also performed at the same time.

[0064] (5) Automatic file deletion mode <Mode Judgment Condition> The condition is that it is determined that automatic file deletion should be performed based on the time that has elapsed since the previous automatic file deletion and the remaining capacity of the non-volatile memory 216 in which the image data is recorded. <Processing within the mode> If it is determined in S717 that the automatic file deletion mode is selected, the process proceeds to automatic file deletion mode processing (S718), in which files to be automatically deleted are designated from among the images in the non-volatile memory 216 based on the tag information of each image and the date and time of shooting, and are then deleted.

[0065] When the processes of S710, S712, S714, S716, and S718 in FIG. 7 are completed, the process returns to S702 and continues. Details of the processes in each mode (S710, S716) will be described later. If it is determined in S709 in FIG. 7 that the mode is not the automatic shooting mode, the process proceeds to S711. If it is determined in S711 that the mode is not the automatic editing mode, the process proceeds to S713. If it is determined in S713 that the mode is not the automatic image transfer mode, the process proceeds to S715. If it is determined in S715 that the mode is not the learning mode, the process proceeds to S717. If it is determined in S717 that the mode is not the automatic file deletion mode, the process returns to S702 and repeats. Note that the automatic editing mode, automatic image transfer mode, and automatic file deletion mode are not directly related to the gist of the present invention, so detailed explanations will be omitted.

[0066] 8 is a flowchart for explaining an example of processing performed by second control unit 211 of camera 101. When a user operates a power button provided on camera 101, power is supplied from first power supply unit 210 to first control unit 223 and each component of camera 101. In addition, power is supplied from second power supply unit 212 to second control unit 211.

[0067] After power is supplied, the second control unit (SubCPU) 211 starts up, and the process of Fig. 8 begins. In S801, a determination process is performed as to whether or not a predetermined sampling period has elapsed. The predetermined sampling period is set to, for example, 10 msec (milliseconds), and the process proceeds to S802 according to the determination result of the 10 msec period (when the predetermined sampling period has elapsed). If it is determined that the predetermined sampling period has not elapsed, the second control unit 211 waits until the determination process of S801 is executed again.

[0068] In S802, learning information is read in. The learning information is information transferred when communicating information to the second control unit 211 in S706 in Fig. 7, and includes, for example, information used in the following determinations. (1) Information for determining whether a specific shaking state is detected (S804 described below). (2) Information for determining whether a specific sound is detected (S805 described below). (3) Information for determining time passage detection (S807 described below).

[0069] After the process of S802, the process proceeds to S803, where a shaking detection value is acquired. The shaking detection value is an output value of a gyro sensor, an acceleration sensor, or the like in the device shaking detection unit 209. Next, the process proceeds to S804, where a detection process of a specific shaking state set in advance is performed. Here, several examples in which the determination process is changed according to the learning information read in S802 will be described.

[0070] <Tap detection> The tap state is, for example, a state in which a user taps the camera 101 with a fingertip or the like, and can be detected from the output value of an acceleration sensor attached to the camera 101. The output of the three-axis acceleration sensor is processed by passing through a band pass filter (BPF) set in a specific frequency domain at a predetermined sampling period, and a signal domain component of the acceleration change due to the tap is extracted. The number of times that the acceleration signal after passing through the BPF exceeds a predetermined threshold value (denoted as ThreshA) during a predetermined time (denoted as TimeA) is measured. A tap is determined based on whether the measured number of times is a predetermined number (denoted as CountA). For example, in the case of a double tap, the value of CountA is set to 2, and in the case of a triple tap, the value of CountA is set to 3. The values ​​of TimeA and ThreshA can also be changed by learning information.

[0071] <Detection of shaking state> The shaking state of the camera 101 can be detected from the output value of a gyro sensor or acceleration sensor attached to the camera 101. The output of the gyro sensor or acceleration sensor is subjected to absolute value conversion after its high-frequency components are cut by a high-pass filter (HPF) and its low-frequency components are cut by a low-pass filter (LPF). The number of times that the calculated absolute value exceeds a predetermined threshold value (denoted as ThreshB) during a predetermined time (denoted as TimeB) is measured. Vibration detection is performed based on whether the measured number of times is equal to or greater than a predetermined number (denoted as CountB). For example, it is possible to determine whether the camera 101 is placed on a desk or the like, that is, whether the shaking is small, or whether the camera 101 is worn by a user as a wearable camera and walking, that is, whether the shaking is large. In addition, by setting multiple conditions regarding the conditions of the judgment threshold value and the judgment count number, it is also possible to detect a detailed shaking state according to the shaking level. The values ​​of TimeB, ThreshB, and CountB can be changed by learning information.

[0072] In the above example, a method for detecting a specific shaking state by judging the detection value of a shaking detection sensor has been described. In addition, there is a method for inputting data sampled from the shaking detection sensor within a predetermined time into a shaking state judger using a neural network (NN), and detecting a specific shaking state that has been registered in advance by a trained NN. In this case, the weight parameters of the NN are read in S802 (reading learning information).

[0073] After the detection process in S804, the process proceeds to S805, where a detection process for a specific sound that has been set in advance is performed. Here, several examples will be described in which the detection determination process is changed depending on the learning information read in S802.

[0074] <Specific voice command detection> In the process of detecting a specific voice command, the specific voice command may include some commands that are registered in advance and commands based on a specific voice that the user has registered in the camera.

[0075] <Specific sound scene recognition> The sound scene is determined by a network that has undergone machine learning based on a large amount of audio data in advance. For example, it is possible to detect specific scenes such as "cheering," "clapping," and "speaking." The scenes to be detected change through learning.

[0076] <Sound level judgment> The sound level is detected by determining whether the sound level exceeds a predetermined level (threshold) for a predetermined time (threshold time). The threshold time and the threshold change through learning.

[0077] <Sound direction determination> A number of microphones arranged on a plane are used to detect the direction of a sound of a given magnitude. The above-mentioned determination process is carried out within the sound processing unit 214, and in S805, it is determined whether or not a specific sound has been detected based on each setting learned in advance.

[0078] After the detection process of S805 is performed, the process proceeds to S806, where the second control unit 211 judges whether the power supply of the first control unit 223 is in an OFF state. If it is judged that the first control unit 223 (Main CPU) is in an OFF state, the process proceeds to S807, and if it is judged that the first control unit 223 (Main CPU) is in an ON state, the process proceeds to S811. In S807, a process of detecting the passage of a preset time is performed. Here, the detection judgment process is changed according to the learning information read in S802. The learning information is information transferred when communicating information to the second control unit 211 in S706 described in FIG. 7. The elapsed time from when the first control unit 223 transitioned from an ON state to an OFF state is measured. If the measured elapsed time is equal to or longer than a predetermined time (denoted as TimeC), it is judged that the predetermined time has elapsed. If the measured elapsed time is shorter than TimeC, it is judged that the predetermined time has not elapsed. TimeC is a parameter that changes according to the learning information.

[0079] After the detection process of S807, the process proceeds to S808, where it is determined whether or not the condition for canceling the low power consumption mode is satisfied. Cancellation of the low power consumption mode is determined based on the following conditions. (1) A specific shaking was detected. (2) A specific sound was detected. (3) The specified time has elapsed. For (1), it is determined in S804 (specific shaking state detection processing) whether a specific shaking has been detected. For (2), it is determined in S805 (specific sound detection processing) whether a specific sound has been detected. For (3), it is determined in S807 (time lapse detection processing) whether a predetermined time has passed. If at least one of the conditions shown in (1) to (3) is satisfied, it is determined to cancel the low power consumption mode. If it is determined in S808 that the low power consumption mode should be cancelled, the process proceeds to S809, and if it is determined that the conditions for cancelling the low power consumption mode are not satisfied, the process returns to S801 and continues.

[0080] In S809, the second control unit 211 turns on the power supply of the first control unit 223, and in S810 notifies the first control unit 223 of the condition (one of vibration, sound, and time) for determining whether to cancel the low power consumption mode. Then, the process returns to S801 to continue the process.

[0081] On the other hand, when the process proceeds from S806 to S811 (when it is determined that the first control unit 223 is in the ON state), the process proceeds to S811. In S811, the information acquired in S803 to S805 is notified to the first control unit 223, and then the process returns to S801 and continues.

[0082] In this embodiment, even when the first control unit 223 is in the ON state, the second control unit 211 detects vibration and a specific sound, and notifies the detection result to the first control unit 223. This is not limiting, and a configuration may be adopted in which, when the first control unit 223 is in the ON state, the processes of S803 to S805 are not performed, and vibration detection and specific sound detection are performed in the process of the first control unit 223 (S702 in FIG. 7).

[0083] As described above, by performing S704 to S707 in Fig. 7 and the processing in Fig. 8, the conditions for switching to the low power consumption mode and the conditions for canceling the low power consumption mode are learned based on the user's operation. In other words, it becomes possible to perform camera operation suited to the convenience of the user who owns the camera 101. The learning method will be described later.

[0084] In the above example, a method for canceling the low power consumption mode based on the detection of shaking, sound, and the passage of time has been described in detail, but the low power consumption mode may also be cancelled based on environmental information. The decision to cancel the low power consumption mode may be made based on whether or not the absolute values ​​or changes in the environmental information such as temperature, air pressure, illuminance, humidity, and amount of ultraviolet light exceed a predetermined threshold, and the threshold value may be changed by learning, which will be described later. In addition, the detection information of shaking, sound, and the passage of time, and the absolute values ​​and changes in each environmental information may be determined based on a neural network, and a decision to cancel the low power consumption mode may be made. In this decision process, the decision conditions may be changed by learning, which will be described later.

[0085] 7 will be described with reference to Fig. 9. First, in S901 (image recognition processing), the image processing unit 207 performs image processing on a signal captured by the imaging unit 206 to generate an image for subject detection. Subject detection processing for detecting people, objects, etc. is performed on the generated image.

[0086] When detecting a person as a subject, the face or body of the subject is detected. In face detection processing, a pattern for determining the face of a person is predefined, and a portion in a captured image that matches the pattern can be detected as the face area of ​​the person. In addition, a reliability indicating the likelihood that the face is the face of the subject is calculated at the same time. The reliability is calculated, for example, from the size of the face area in the captured image and the degree of match with the face pattern. Object recognition is performed in a similar manner, and an object that matches a preregistered pattern can be recognized.

[0087] There is also a method of extracting a characteristic subject using a histogram of hue, saturation, etc. in a captured image. With respect to an image of a subject captured within a shooting angle of view, a distribution derived from a histogram of the hue, saturation, etc. is divided into a plurality of sections, and a process of classifying the captured image for each section is executed. For example, a histogram of a plurality of color components is created for the captured image, and the image is divided into sections based on the mountain-shaped distribution range. The captured image is classified in areas that belong to the same combination of sections, and the image area of ​​the subject is recognized. By calculating an evaluation value for each recognized image area of ​​the subject, the image area of ​​the subject with the highest evaluation value can be determined as the main subject area. With the above method, each subject information can be obtained from the imaging information.

[0088] In S902, a calculation process of the image blur correction amount is performed. Specifically, first, the absolute angle of the camera shake is calculated based on the angular velocity and acceleration information acquired by the device shake detection unit 209. The tilt rotation unit 104 and the pan rotation unit 105 are driven in an angular direction that cancels the absolute angle to obtain an angle at which the image blur is corrected, and the image blur correction amount is obtained. Note that the calculation method of the image blur correction amount calculation process here can be changed by a learning process described later.

[0089] In S903, the state of the camera is determined. The current vibration / movement state of the camera is determined based on the angular velocity information, acceleration information, GPS position information, and the like, and the camera angle and the amount of camera movement are used to determine what state the camera is in. For example, assume that the camera 101 is attached to a vehicle and is used for shooting. In this case, subject information such as the surrounding scenery changes significantly depending on the distance the vehicle moves. Therefore, it is determined whether the camera 101 is attached and moving at high speed in a "vehicle moving state," and the determination result is used for automatic subject search, which will be described later. It is also determined whether the angle of the camera 101 has changed significantly. It is determined whether the camera 101 is in a "stationary shooting state" in which there is almost no shaking, and if it is in a "stationary shooting state," it can be determined that there is no change in the position of the camera 101 itself. In this case, a subject search for stationary shooting can be performed. In addition, if the angle change of the camera 101 is relatively large, it is determined to be in a "handheld state." In this case, a subject search for handheld shooting can be performed.

[0090] In S904, a subject search process is performed. The subject search process includes the following processes. (1) Area division. (2) Calculation of the importance level for each area. (3) Determining the area to be searched. Each process will be explained below in order.

[0091] (1) Area division The area division will be described with reference to FIG. 10. The origin O of the three-dimensional orthogonal coordinate system is set as the camera position. FIG. 10(A) is a schematic diagram showing an example of area division all around the camera position (origin O). In the example of FIG. 10(A), the area is divided into areas every 22.5 degrees in the tilting direction and the panning direction. In this type of division, as the tilting angle moves away from 0 degrees, the circumference in the horizontal direction becomes smaller and the area becomes smaller. In contrast, FIG. 10(B) is a schematic diagram showing an example in which the area range in the horizontal direction is set to be larger than 22.5 degrees when the tilting angle is 45 degrees or more. FIGS. 10(C) and (D) are schematic diagrams showing examples of areas divided into areas within the shooting angle of view. An axis 1301 shown in FIG. 10(C) represents the orientation of the camera 101 at the time of initialization, and area division is performed with the direction of the axis 1301 as the reference direction. An example of an image corresponding to the area 1302 of the angle of view of a captured image is shown in Fig. 10(D). Within the image of the captured angle of view, the image is divided based on the area division as shown in Fig. 10(D). Examples of multiple divided regions 1303 to 1318 are shown.

[0092] (2) Calculation of importance level for each area For each divided area, an importance level indicating the priority of searching is calculated according to the situation of the subject existing in the area and the situation of the scene. The importance level based on the situation of the subject is calculated based on, for example, the number of people existing in the area, the size of the person's face, the direction of the face, the certainty of face detection, the expression of the person, the result of personal authentication of the person, etc. Also, the importance level according to the situation of the scene is calculated based on, for example, the general object recognition result, the scene discrimination result (blue sky, backlight, evening scene, etc.), the sound level detected from the direction of the area, the voice recognition result, the motion detection information within the area, etc.

[0093] Also, when camera vibration is detected in the camera state determination (S903) in FIG. 9, the importance level can be configured to change according to the vibration state. For example, assume that the state is determined to be "still photography state". In this case, it is determined that subject search is performed mainly on subjects with high priority (for example, the owner of the camera) among those registered in face authentication. Also, for automatic photography, which will be described later, photography is performed with priority given to the face of the owner of the camera, for example. As a result, even if the owner of the camera carries the camera around and takes pictures for a long time, many images in which the owner is captured can be recorded by removing the camera and placing it on a desk, etc. At this time, since face search is possible by panning and tilting, the user does not need to consider the angle at which the camera is placed, and can record images in which the owner is captured and group photos in which many faces are captured by simply setting the camera appropriately. In addition, with only the above conditions, unless there is a change in each area, there is a possibility that the area with the highest importance level will be the same. As a result, the area to be searched will never change. Therefore, a process is performed to change the importance level according to past shooting information. Specifically, a process is performed to lower the importance level of an area that has been continuously specified as a search area for a predetermined time, and a process is performed to lower the importance level of an area where shooting was performed in S910 described later for a predetermined time.

[0094] (3) Determining the search area Based on the importance level of each area calculated as described above, a process is executed to determine an area with a high importance level as a search target area, and a search target angle of panning and tilting required to capture the search target area in the field of view is calculated.

[0095] In S905 of Fig. 9, panning and tilting are driven. Specifically, the panning drive amount and tilting drive amount are calculated by adding the image blur correction amount at the control sampling frequency and the drive angle based on the search target angle of panning and tilting. The lens barrel rotation drive unit 205 drives and controls the tilt rotation unit 104 and the pan rotation unit 105.

[0096] In S906, the zoom unit 201 is controlled to perform zoom driving. Specifically, the zoom driving is performed according to the state of the search target subject determined in S904. For example, assume that the search target subject is a person's face. In this case, if the face size on the image is too small, it may be below the minimum detectable size and may not be detected, and the subject may be lost. In such a case, the zoom control to the telephoto side is performed to increase the size of the face on the image. On the other hand, if the face size on the image is too large, the subject may easily fall out of the angle of view due to the movement of the subject or the camera itself. In such a case, the zoom control to the wide-angle side is performed to reduce the size of the face on the screen. By performing the zoom control in this way, it is possible to maintain a state suitable for tracking the subject. The zoom control includes optical zoom control, which is performed by driving the lens, and electronic zoom control, which changes the angle of view by image processing. There are forms in which either one of the controls is performed, and forms in which both controls are combined.

[0097] S907 is a process for determining whether automatic registration of personal authentication is possible or not, depending on the detection status of the subject. If the face detection reliability is high and the face detection reliability is maintained at a high level, a more detailed determination is made. That is, if the face is not in profile but facing the camera, and the size of the face is equal to or larger than a predetermined value, it is determined that the state is suitable for automatic registration of personal authentication.

[0098] The next step S908 is automatic shooting determination processing. In the automatic shooting determination, it is determined whether to perform automatic shooting and the shooting method (determining which of still image shooting, video shooting, continuous shooting, panoramic shooting, etc. to perform). The determination of whether to perform automatic shooting will be described later.

[0099] In S909, it is determined whether or not a manual shooting instruction has been given. Manual shooting instructions include an instruction by pressing a shutter button, an instruction by tapping the camera housing with a finger or the like, an instruction by inputting a voice command, and an instruction from an external device. For example, a shooting instruction triggered by a tap operation is determined by detecting continuous high-frequency acceleration in a short period of time by the device shake detection unit 209 when the user taps the camera housing. The voice command input method is a shooting instruction method in which, when the user utters a password (for example, "take a picture") instructing a specific shooting, the voice processing unit 214 recognizes the voice and triggers shooting. The instruction method from an external device is, for example, a shooting instruction method in which a shutter instruction signal transmitted from a smartphone connected to the camera via Bluetooth (registered trademark) using a dedicated application is used as a trigger.

[0100] If it is determined in S909 that a manual shooting instruction has been given, the process proceeds to S910. Also, if it is determined in S909 that a manual shooting instruction has not been given, the process proceeds to S914. In S914, a determination is made as to whether or not to execute automatic authentication and registration. Using the result of the determination as to whether or not to execute automatic authentication and registration in S907 and the result of the determination as to whether or not to execute automatic shooting in S908, a determination is made as to whether or not to execute automatic authentication and registration. If it is determined in S914 that automatic authentication and registration is to be executed, the process proceeds to S915, and if it is determined that automatic authentication and registration is not to be executed, the process proceeds to S916. A specific example will be described with reference to FIG. 11.

[0101] Fig. 11 is a table for explaining the execution judgment of automatic authentication registration and automatic photography. The automatic authentication registration judgment result shall be either "registration possible" or "registration not possible", and the automatic photography judgment result shall be either "photography possible" or "photography not possible". If it is judged that the person is suitable for registration for personal authentication, the person shall register for personal authentication regardless of the judgment result for automatic photography. If it is judged that the person is not suitable for registration for personal authentication and the conditions for automatic photography are met ("photography possible"), the automatic photography shall be performed.

[0102] The reason why the possibility of automatic authentication and registration is given priority is that stable frontal face information is required for automatic authentication and registration. In automatic photography, it may be determined that photography should be performed when the subject is in profile, when there is a momentary smile, or when the time has elapsed since the previous photography, etc. However, conditions suitable for automatic authentication and registration are met only rarely. Therefore, in this embodiment, the algorithm gives priority to cases where conditions suitable for automatic authentication and registration are obtained.

[0103] It is possible to think that prioritizing automatic authentication and registration will hinder opportunities for automatic photography. However, this is incorrect because automatic authentication and registration improves the accuracy of personal authentication, and the accuracy of searching and tracking the priority subject is improved, which is very useful for finding photography opportunities in automatic photography. In addition, in this embodiment, if it is determined that the subject is suitable for personal authentication registration, it is always given priority over the results of automatic photography. Not limited to this, the priority may be changed according to the number of times photography is performed within a specified time by automatic photography or the photography interval. For example, when the frequency of photography by automatic photography is low, it is also possible to control so that automatic photography is temporarily given priority.

[0104] S915 in Fig. 9 is a registration process for personal authentication. A series of processes is executed to perform a photographing process by controlling the photographing state to be suitable for personal authentication, and to digitize and store the facial feature amount. A specific description will be given with reference to Fig. 12.

[0105] FIG. 12 is a schematic diagram for explaining subject placement in composition adjustment. FIG. 12(A) shows the composition when a still image is automatically captured, and FIG. 12(B) shows the composition when a still image is captured for personal authentication. By adjusting the composition, a state suitable for personal authentication as shown in FIG. 12(B) is achieved. In order to obtain facial feature quantities with higher accuracy, it is important to position the subject at the center of the image, which is less susceptible to the effects of optical aberration, and adjust the composition so that the face is captured large. On the other hand, when a still image is automatically captured in S910 described later, a more satisfying photo can be obtained by adjusting the composition so that the main subject and background fit together as shown in FIG. 12(A).

[0106] If a manual shooting instruction is issued by the user during the personal authentication registration process, it is possible to temporarily halt the process of S915, end the shooting mode process, and execute the shooting mode process again. The control of the composition adjustment is an operation of repeating panning, tilting, and zoom lens drive, and confirmation of the face position by face detection. By checking the manual shooting instruction as needed during this repetitive operation, and interrupting the personal authentication registration process when an interrupt is confirmed, the user's intention can be quickly reflected.

[0107] Automatic photography is photography in which image data output by an imaging unit is automatically recorded. In S916 of FIG. 9, the determination as to whether or not to perform automatic photography is performed as follows. Specifically, it is determined to perform automatic photography in the following two cases. In the first case, the importance level exceeds a predetermined value based on the importance level by area obtained in S904. In the second case, the determination result based on a neural network is used, which will be described later. Note that recording in automatic photography is recording of image data in memory 215 or recording of image data in non-volatile memory 216. It is also assumed that automatic transfer of image data to external device 301 and recording of image data in external device 301 are also included.

[0108] In this embodiment, control is performed so that photography is performed automatically by automatic photography judgment processing based on a neural network. Depending on the situation of the photography location and the situation of the camera, it may be better to change the judgment parameters of automatic photography. Unlike photography at fixed time intervals, automatic photography control based on situation judgment tends to be preferred in a form that meets the following requests. (1) I want to take a lot of pictures, including people and objects. (2) I don’t want to miss any memorable scenes. (3) I want to shoot in a power-saving manner, taking into consideration the remaining battery charge and the remaining capacity of the recording media. The evaluation value is calculated based on the state of the subject, and compared with a threshold. Automatic photography is performed if the evaluation value exceeds the threshold. The evaluation value for automatic photography is determined by judgment using a neural network.

[0109] Next, judgment based on a neural network (NN) will be explained. As an example of an NN, an example of a network using a multi-layer perceptron is shown in FIG. 13. An NN is used to predict an output value from an input value. By learning an input value and a model output value for that input in advance, it is possible to estimate an output value that follows the model that has been learned for a new input value. The learning method will be described later.

[0110] In FIG. 13, node 1201 and multiple nodes indicated by circles vertically arranged thereon represent neurons in the input layer. Node 1203 and multiple nodes indicated by circles vertically arranged thereon represent neurons in the hidden layer. Node 1204 represents a neuron in the output layer. Arrows 1202 represent connections connecting the neurons. In NN-based judgment, features based on the subject captured in the current angle of view and the state of the scene and camera are given as input to the neurons in the input layer. A value output from the output layer is obtained after calculation based on the forward propagation rule of the multi-layer perceptron. If the output value is equal to or greater than a threshold, a judgment is made to perform automatic shooting.

[0111] As the features of the subject, for example, the following information is used: Information on the general object recognition results at the current zoom ratio and current angle of view. Face detection results, number of faces in the current field of view, degree of smiling face, degree of closed eyes, face angle, face recognition ID number, and gaze angle of the subject. Scene detection results, time elapsed since the previous capture, current time, GPS location information, and amount of change from the previous capture location. Information about the current audio level, who is speaking, and whether there is applause or cheering. Vibration information (acceleration information, camera status), environmental information (temperature, air pressure, illuminance, humidity, amount of UV rays), etc.

[0112] Furthermore, when there is information notification from the external device 501, the notification information (user's motion information, arm action information, bio-information such as heart rate, etc.) is also used as feature information. The feature information is converted into a numerical value within a predetermined range and given to each neuron in the input layer as a feature. Therefore, each neuron in the input layer is required for the number of features to be used.

[0113] In judgments based on neural networks, the output value can be changed by changing the connection weights between each neuron in the learning process described below, and the judgment results can be adapted to the learning results.

[0114] In addition, the determination of automatic photography also changes depending on the activation conditions of the first control unit 223 read in S702 in Fig. 7. For example, in the case of activation by tap detection or activation by a specific voice command, it is highly likely that the user's intention is an operation to instruct photography. Therefore, the photography frequency is set to be high.

[0115] In the determination of the shooting method, the execution of shooting is determined based on the state of the camera and the state of the surrounding subject detected in S901 to S904. It is determined which of still image shooting, video shooting, continuous shooting, panoramic shooting, etc. to perform. For example, when a person who is a subject is stationary, still image shooting is selected and performed. When the subject is moving, video shooting or continuous shooting is performed. In addition, when multiple subjects exist so as to surround the camera, or when it is determined that the place is a scenic spot based on GPS information, panoramic shooting processing is performed. The panoramic shooting processing is a process of generating a panoramic image by synthesizing images taken in sequence while driving the panning and tilting of the camera. Note that, similar to the method of determining whether to perform automatic shooting, various information detected before shooting can be determined based on a neural network, and the shooting method can also be determined. In addition, in this determination processing, the determination conditions can be changed by a learning processing described later.

[0116] 9, if it is determined in the automatic photography determination process of S908 that automatic photography is to be performed, the process proceeds to S910. If it is determined in S916 that automatic photography is not to be performed, the photography mode process ends. After S915 (automatic authentication registration process), the photography mode process ends.

[0117] In S910, automatic shooting is started. That is, shooting is started by the shooting method determined in S908. At this time, the focus drive control unit 204 performs autofocus control. In addition, exposure control is performed using an aperture control unit, a sensor gain control unit, and a shutter control unit (not shown), so that the subject is adjusted to an appropriate brightness. After shooting, the image processing unit 207 performs various known image processing such as auto white balance processing, noise reduction processing, and gamma correction processing, and generates image data.

[0118] During the automatic shooting in S910, if a predetermined condition is satisfied, the camera may notify the person to be shot that shooting will be performed before shooting. The predetermined condition is set based on the following information, for example. The number of faces within the field of view, the degree to which the face is smiling, the degree to which the eyes are closed, the subject's gaze angle and facial angle, and face recognition ID number. -Personal authentication: Number of people registered, recognition results of general objects when photographing. Scene recognition results, time elapsed since the previous photo was taken, the time the photo was taken, and information on whether the current location is a scenic spot based on GPS information. - Information about the sound level during recording, whether anyone is speaking, and whether there is applause or cheering. Vibration information (acceleration information, camera status), environmental information (temperature, air pressure, illuminance, humidity, amount of UV rays), etc.

[0119] The notification method may be, for example, a sound from the audio output unit 218 or lighting of an LED by the LED control unit 224. By shooting with a notification based on these conditions, an image with a preferred camera angle can be recorded in a scene of high importance. Regarding the notification before shooting, the information of the shot image or various information detected before shooting can be judged based on a neural network to determine the notification method and timing. In addition, in this judgment process, the judgment conditions can also be changed by a learning process described later.

[0120] In S911, editing processing such as processing the image generated in S910 and adding it to the video is executed. Specifically, image processing includes trimming processing based on a person's face or focus position, image rotation processing, HDR (high dynamic range) effect processing, bokeh effect processing, color conversion filter effect processing, and the like. In image processing, a plurality of processed images are generated by a combination of the above processing based on the image data generated in S910. Processing to save the above image data separately from the image data generated in S910 may be performed. In addition, in video processing, processing such as adding a captured video or still image to a generated edited video while applying special effect processing such as slide, zoom, and fade is performed. Regarding the editing processing in S911, the information of the captured image or various information detected before shooting can be judged based on a neural network, and a method of image processing can be determined. In addition, in this judgment processing, the judgment conditions can be changed by a learning processing to be described later.

[0121] In S912, learning information generation processing for the captured image is performed. This processing is processing for generating and recording information used in the learning processing described later. Specifically, for example, the following information is included. The zoom magnification at the time the image was taken, the general object recognition results at the time the image was taken, the face detection results, the number of faces appearing in the image, the degree to which the faces are smiling, the degree to which the eyes are closed, the facial angle, the facial recognition ID number, and the gaze angle of the subject of the image. Scene detection results, time elapsed since the previous image was taken, shooting time, GPS location information, and amount of change from the previous shooting location. Information about the sound levels during recording, who is speaking, and whether there is applause or cheering. Vibration information (acceleration information, camera status), environmental information (temperature, air pressure, illuminance, humidity, UV rays). -Video recording time, information on whether or not the video was recorded manually, etc.

[0122] Furthermore, a score is calculated, which is the output of a neural network that quantifies the user's image preferences. A process is executed to generate this information and record it as tag information in the captured image file. Alternatively, the information may be stored in the non-volatile memory 216 or in the recording medium 221 in a list format of information on each captured image as so-called catalog data.

[0123] In S913, a process for updating past shooting information is performed. Specifically, this is an update process for the number of shots taken for each area described in S908, the number of shots taken for each person registered in personal authentication, the number of shots taken for each subject recognized in general object recognition, and the number of shots taken for each scene in scene determination. In other words, a process is performed to increase the count of the number of images that corresponds to the image shot this time by one. At the same time, a process is performed to store the current shooting time and the evaluation value of automatic shooting and hold them as shooting history information. After S913, the series of processes is terminated.

[0124] Next, learning according to user preferences will be described. In this embodiment, a neural network (NN) as shown in FIG. 13 is used, and a learning processing unit 219 performs learning according to user preferences using a machine learning algorithm. The NN is used to predict an output value from an input value, and by learning the actual values ​​of the input value and the output value in advance, it is possible to estimate an output value for a new input value. By using the NN, it is possible to perform learning according to user preferences for the above-mentioned automatic shooting, automatic editing, and subject search. In addition, registration of subject information (results of face authentication, general object recognition, etc.) which is also feature data to be input to the NN, and changing shooting notification control, low power consumption mode control, and automatic file deletion by learning are also performed.

[0125] In this embodiment, an example of an operation to which the learning process is applied is shown below. (1) Automatic photography (2) Automatic editing (3) Subject search (4)Subject registration (5) Shooting notification control (6) Low power consumption mode control (7) Automatic file deletion (8) Image stabilization (9) Automatic image transfer Among the operations to which the learning process is applied, (2) automatic editing, (7) automatic file deletion, and (9) automatic image transfer are not directly related to the gist of the present invention, so a description thereof will be omitted.

[0126] <Automatic shooting> Learning for automatic photography will be described. In automatic photography, learning is performed to automatically take images that match the user's preferences. As described with reference to FIG. 9, after photography (after S910), a learning information generation process (S912) is performed. This is a process in which an image to be learned is selected using a method described later, and learning is performed by changing the weight of the NN based on the learning information contained in the image. Learning is performed by changing the NN that determines the timing of automatic photography and the NN that determines the photography method (still image photography, video photography, continuous shooting, panoramic photography, etc.).

[0127] <Subject search> Learning for subject search will be described. In subject search, learning is performed to automatically search for a subject that matches the user's preferences. In the subject search process (S904) in FIG. 9, the importance level of each area is calculated, and subject search is performed by driving panning, tilting, and zooming. Learning is performed based on the captured image and detection information during search, and is reflected as the learning result by changing the weight of the NN. By inputting various detection information during the search operation to the NN and determining the importance level, subject search that reflects the learning result can be performed. In addition to calculating the importance level, the search method (speed, frequency of movement) by panning and tilting is controlled.

[0128] <Subject registration> Learning for object registration will be explained. In object registration, learning is performed to automatically register and rank objects that match the user's preferences. As learning, for example, face recognition registration, general object recognition registration, gesture and voice recognition, and sound scene recognition registration are performed. People and objects are authenticated and registered, and ranking is set based on the number and frequency of images captured, the number and frequency of manual shooting, and the frequency of the object appearing during the search. Each piece of information is registered as an input for judgment using a neural network.

[0129] <Shooting notification control> Learning for a shooting notification will be described. As described in S910 of FIG. 9, immediately before shooting, when a predetermined condition is satisfied, the camera notifies the person to be shot that shooting will be performed, and then shooting is performed. For example, a process is executed in which the subject's gaze is visually guided by driving panning and tilting, or the subject's attention is called for using speaker sound emitted from the audio output unit 218 or LED lighting by the LED control unit 224. Immediately after the notification, based on whether or not detection information of the subject (for example, smile level, gaze detection, gesture) has been acquired, it is determined whether or not to use the detection information for learning, and learning is performed by changing the weight of the NN.

[0130] Each piece of detection information immediately before shooting is input to the NN, which decides whether or not to issue an alarm. In the case of an alarm sound, the sound level, type and timing of the sound, as well as the illumination time, speed and camera direction (panning / tilting motion) of the alarm light are determined.

[0131] <Low power consumption mode control> As described with reference to Figures 7 and 8, the power supply to the first control unit 223 (Main CPU) is controlled to be turned on and off. Conditions for returning from the low power consumption mode and conditions for transitioning to the low power consumption state are learned. First, the learning of conditions for releasing the low power consumption mode will be described.

[0132] Sound detection Learning can be performed by manually setting the user's specific voice, the specific sound scene to be detected, and the specific sound level, for example, through communication using a dedicated application of the external device 301. There is also a method in which multiple detection methods are preset in the sound processing unit, and an image to be learned is selected using a method described later. Learning can be performed by learning information about the preceding and following sounds contained in the selected image, and setting the sound judgment to be the activation cause (specific sound command, or sound scenes such as "cheers" and "applause").

[0133] ·Environmental information detection The learning can be performed by manually setting the environmental information changes that the user desires to use as start-up conditions, for example, through communication using a dedicated application of the external device 301. For example, specific conditions such as temperature, air pressure, illuminance, humidity, absolute amount or amount of change in ultraviolet light are set, and the imaging device can be started when the conditions are met. It is also possible to learn a judgment threshold based on each environmental information. If it is determined from the camera detection information after start-up based on the environmental information that the change was not a start-up cause, the parameters of each judgment threshold are set so that the environmental change is difficult to detect.

[0134] The above parameters also change depending on the remaining battery capacity. For example, when the remaining battery capacity is low, it is difficult to proceed to various judgments, and when the remaining battery capacity is high, it is easy to proceed to various judgments. Specifically, even if the shaking state detection result or the sound scene detection result is not a factor when the user intends to start the camera, if the remaining battery capacity is high, it may be determined that the camera should be started.

[0135] Also, the determination of the conditions for releasing the low power consumption mode can be performed based on a neural network from the shaking detection information, sound detection information, time passage detection information, various environmental information, remaining battery power, etc. In this case, an image to be learned is selected by a method described later, and learning is performed by changing the weights of the NN based on the learning information contained in the image.

[0136] Next, learning of the transition conditions to the low power consumption state will be described. As shown in Fig. 7, in the mode setting determination in S704, if it is determined that the mode is not one of "automatic shooting mode", "automatic editing mode", "automatic image transfer mode", "learning mode" and "automatic file deletion mode", the transition to the low power consumption mode occurs. The determination conditions for each mode are as described above, but the conditions for determining each mode also change through learning.

[0137] <Automatic shooting mode> The importance level of each area is determined, and automatic shooting is performed while searching for a subject by panning and tilting. If it is determined that there is no subject to be shot, the automatic shooting mode is released. For example, if the importance levels of all areas or the sum of the importance levels of each area becomes equal to or less than a predetermined threshold, the automatic shooting mode is released. At this time, the predetermined threshold is set to be lowered according to the elapsed time from the time of transition to the automatic shooting mode. As the elapsed time from the time of transition to the automatic shooting mode becomes longer, it becomes easier to transition to the low power consumption mode.

[0138] Also, by changing the predetermined threshold value according to the remaining battery capacity, it is possible to control the low power consumption mode in consideration of the available battery time. For example, when the remaining battery power is low, the threshold value is increased to make it easier to switch to the low power consumption mode, and when the remaining battery power is high, the threshold value is decreased to make it more difficult to switch to the low power consumption mode. Here, a parameter (elapsed time threshold TimeC) for the next low power consumption mode release condition is set for the second control unit 211 according to the elapsed time and the number of shots since the previous transition to the automatic shooting mode. Each of the above threshold values ​​changes through learning. Learning is performed, for example, by manually setting the shooting frequency, startup frequency, etc., through communication using a dedicated application of the external device 301.

[0139] Alternatively, the average value of the time elapsed from when the power button of the camera 101 is turned ON to when the power button is turned OFF and distribution data for each time period may be accumulated to learn each parameter. In this case, for a user for whom the time elapsed from when the power is turned ON to when the power is turned OFF is short, the time interval for returning from the low power consumption mode or transitioning to the low power consumption state is shortened by learning. Conversely, for a user for whom the time elapsed from when the power is turned ON to when the power is turned OFF is long, the time interval is lengthened by learning.

[0140] Learning is also performed based on the detection information during object search. While it is determined that there are many important objects set, the time interval for returning from the low power consumption mode or for transitioning to the low power consumption state is shortened by learning. Conversely, while it is determined that there are few important objects, the time interval is lengthened by learning.

[0141] <Image stabilization> Learning for image blur correction will be described. In S902 of FIG. 9, the image blur correction amount is calculated, and in S905, panning and tilting are driven based on the image blur correction amount. In the image blur correction, learning is performed to perform correction according to the characteristics of the user's shaking. For example, by using a PSF (Point Spread Function) for the captured image, it is possible to estimate the direction and magnitude of blur. In the generation of learning information in S912 of FIG. 9, information on the estimated direction and magnitude of blur is added to the image data.

[0142] 7, the learning mode process in S716 learns the weights of the NN for image blur correction for the predetermined input information and output (estimated direction and magnitude of blur). The predetermined input information is, for example, each piece of detection information during shooting (motion vector information of the image at a predetermined time before shooting, motion information of the detected subject (person or object), vibration information (gyro output, acceleration output, camera state). In addition, environmental information (temperature, air pressure, illuminance, humidity), sound information (sound scene determination, specific sound detection, sound level change), time information (elapsed time from startup, elapsed time from the previous shooting), location information (GPS location information, amount of change in location movement), etc. may be added to the input.

[0143] When calculating the image blur correction amount in S902 of Fig. 9, the above-mentioned detection information is input to a neural network, so that the magnitude of blur at the moment of photographing can be estimated. When the estimated magnitude of blur is larger than a threshold, control such as increasing the shutter speed becomes possible. In addition, when the estimated magnitude of blur is larger than a threshold, there is a possibility that an image blurred image will be captured, so there is a method of prohibiting such photographing.

[0144] In addition, since the driving angle of panning and tilting is limited, after reaching the driving end, further image blur correction cannot be performed. In this embodiment, by estimating the magnitude and direction of the blur during shooting, it is possible to estimate the range required for driving panning and tilting to correct image blur during exposure. Regarding the driving angle of panning and tilting, if there is no margin in the movable range during exposure, a process is executed to increase the cutoff frequency of the filter that calculates the image blur correction amount and set the driving angle so that it does not exceed the movable range. This makes it possible to suppress large blur. Also, if it is predicted that the driving angle will exceed the movable range, the driving angle is changed immediately before exposure, and exposure is started by rotating in the opposite direction to the direction in which the driving angle exceeds the movable range. This makes it possible to perform shooting with suppressed image blur while securing the movable range. By learning about image blur correction according to the user's characteristics and usage during shooting, it is possible to suppress or prevent image blur in the captured image.

[0145] In the determination of the photographing method of this embodiment, a process of determining whether or not to perform panning may be performed. In panning, photographing is performed so that there is no image blurring of a moving subject and the image flows against a non-moving background. In the process of determining whether or not to perform panning, the driving speed of panning and tilting for photographing the subject without blurring is estimated from the detection information before shooting, and image blurring correction of the subject is performed. At this time, the driving speed can be estimated by inputting information to a neural network that has already learned each of the above detection information. The image is divided into a plurality of blocks, and the PSF of each block is estimated, thereby estimating the direction and magnitude of blurring in the block where the main subject is located. Learning is performed based on the information.

[0146] It is also possible to learn the amount of background blur from the information of the image selected by the user. In this case, the amount of blur in the block (image area) where the main subject is not located is estimated, and the user's preference can be learned based on that information. By setting the shutter speed at the time of shooting based on the learned preferred amount of background blur, it is possible to automatically take a picture that produces a panning effect that suits the user's preference.

[0147] Next, the learning method will be described. There are two types of learning methods: "learning within the camera" and "learning through cooperation with an external device such as a communication device." First, the former learning method will be described. There are the following methods for learning within the camera in this embodiment.

[0148] (1) Learning based on detection information during manual shooting As described in S907 to S913 of FIG. 9, the camera 101 can perform manual shooting and automatic shooting. When a manual shooting instruction is given in S907, information indicating that the image is a manually shot image is added to the shot image in S912. Also, when automatic shooting is determined to be ON in S916 and an image is shot, information indicating that the image is an automatically shot image is added to the shot image in S912. In the case of manual shooting, it is highly likely that the image was shot based on the user's favorite subject, favorite scene, favorite place, and time interval. Therefore, learning is performed based on each feature data obtained during manual shooting and learning data of the shot image. Also, learning is performed on the extraction of feature amounts in the shot image, registration of personal authentication, registration of facial expressions for each individual, and registration of combinations of people from the detection information during the manual shooting. Also, learning is performed from the detection information during the subject search, for example, based on the facial expression of a personally registered subject, to change the importance of people and objects nearby.

[0149] (2) Learning from detection information during subject search During the subject search, it is determined what kind of person, object, or scene the subject registered in personal authentication is photographed at the same time, and the time ratio in which the subject is photographed at the same time within the angle of view is calculated. For example, the time ratio in which person A, a subject registered in personal authentication, is photographed at the same time as person B, a subject registered in personal authentication, is calculated. When person A and person B are within the angle of view, various detection information is stored as learning data and learning is performed in the learning mode process (FIG. 7: S716) so that the score for automatic shooting determination is high. In another example, the time ratio in which person A, a subject registered in personal authentication, is photographed at the same time as a "cat," a subject determined by general object recognition, is calculated. When person A and a "cat" are within the angle of view, various detection information is stored as learning data and learning is performed in the learning mode process (FIG. 7: S716) so that the score for automatic shooting determination is high.

[0150] In addition, if a high degree of smile is detected from person A, who is a registered subject for personal authentication, or if an expression such as "happiness" or "surprise" is detected, the subject that appears at the same time is learned to be important. Alternatively, if an expression such as "anger" or "serious face" is detected from person A, the subject that appears at the same time is determined to be unlikely to be important, and learning is not performed.

[0151] Next, the following learning performed in cooperation with an external device in this embodiment will be described. (1) Learning by acquiring images using an external device. (2) Learning by inputting judgment values ​​to images via an external device. (3) Learning by analyzing images stored in an external device. (4) Learning from information uploaded to a Social Networking Service (SNS) server by an external device. (5) Learning by changing camera parameters with an external device. (6) Learning from information obtained by manually editing images using an external device. The explanation will be given in order according to the assigned numbers.

[0152] <Learning by acquiring images using an external device> As described in FIG. 3, the camera 101 and the external device 301 have communication means for performing the first and second communications 302, 303. Image data is mainly transmitted and received by the first communication 302, and images in the camera 101 can be transmitted to the external device 301 via a dedicated application in the external device 301. In addition, thumbnail images of image data stored in the camera 101 can be viewed using a dedicated application in the external device 301. A user can select and confirm an image that he or she likes from among the thumbnail images, or can cause the external device 301 to transmit image data by performing an image acquisition instruction operation. An image selected and acquired by the user is highly likely to be an image that the user likes. Therefore, the acquired image is determined to be an image to be learned. Various learning of the user's preferences can be performed based on the learning information of the acquired image.

[0153] An example of operation will be described with reference to Fig. 14. Fig. 14 is a diagram for explaining an example in which a user browses images in the camera 101 using a dedicated application of the external device 301. Thumbnail images 1604 to 1609 of image data stored in the camera are displayed on the display unit 407. The user can select and acquire an image that he or she likes. Buttons 1601 to 1603 are operated when changing the display method, and constitute a display method changing unit.

[0154] When the first button 1601 is pressed, the display mode is changed to the date and time priority display mode, and images are displayed on the display unit 407 in the order of the date and time of the images captured in the camera 101. For example, an image with a newer date and time is displayed at the position indicated by the thumbnail image 1604, and an image with an older date and time is displayed at the position indicated by the thumbnail image 1609. When the second button 1602 is pressed, the display mode is changed to the recommended image priority display mode. Based on the score that determines the user's preference for each image calculated in S912 of FIG. 9, images in the camera 101 are displayed on the display unit 407 in the order of the scores. For example, an image with a higher score is displayed at the position indicated by the thumbnail image 1604, and an image with a lower score is displayed at the position indicated by the thumbnail image 1609. When the user presses the third button 1603, the user can specify a person or object as a subject, and then specify a specific person or object as a subject to display only the specific subject.

[0155] Buttons 1601 to 1603 can be set to ON at the same time. For example, when all settings are ON, only the specified subject is displayed, and images with more recent shooting dates and times and images with higher scores are displayed with priority. In this way, since the user's preferences are learned for captured images, it is possible to extract only images that the user likes from a large number of captured images with a simple confirmation task.

[0156] <Learning by inputting judgment values ​​to images via an external device> When viewing images stored in the camera 101, the user can assign a score to each image. The user can assign a high score (e.g., 5 points) to an image that the user likes, and a low score (e.g., 1 point) to an image that the user does not like. The camera is configured to learn the judgment value of an image in response to user operations. The score for each image is used for re-learning together with the learning information within the camera. The output of the neural network, which uses feature data from specified image information as input, is trained to approach the score specified by the user.

[0157] In addition to a configuration in which a user inputs a judgment value to a captured image via external device 301, there is also a configuration in which a user operates camera 101 to directly input a judgment value to an image. In this case, for example, camera 101 is equipped with a touch panel display. The user operates a GUI (Graphical User Interface) button displayed on the screen display unit of the touch panel display to set the mode to display captured images. Then, the user can input a judgment value for each image while checking the captured images, thereby performing learning similar to that described above.

[0158] <Learning by analyzing images stored in an external device> The storage unit 404 of the external device 301 also records images other than those captured by the camera 101. Images stored in the external device 301 are easy for users to view, and can also be easily uploaded to a shared server via the public wireless control unit 406, so there is a very high possibility that the images will include many of the user's favorite images.

[0159] The control unit 411 of the external device 301 is assumed to be capable of processing images stored in the storage unit 404 with the same capabilities as the learning processing unit 219 in the camera 101 using a dedicated application. Learning is performed by communicating the processed learning data to the camera 101. Alternatively, the camera 101 may be configured to transmit images and data to be learned to the camera 101 and perform learning within the camera 101. Also, the camera 101 may be configured to select images to be learned from among images stored in the storage unit 404 using a dedicated application, and the images may be learned.

[0160] <Learning from information uploaded to the SNS server by an external device> This section describes a method for using information from social networking services (SNS), which are services and websites that enable the creation of social networks that focus on connections between people, for learning. When uploading an image to an SNS, there is a technique for inputting tags related to the image from an external device 301 and then transmitting the image together with the image. There is also a technique for inputting likes and dislikes for images uploaded by other users. It is also possible to determine whether an image uploaded by another user is a photo that the user who owns the external device 301 likes.

[0161] A dedicated SNS application downloaded into the external device 301 can obtain images uploaded by the user and information about those images. In addition, by inputting data on whether the user likes images uploaded by other users, it is possible to obtain images and tag information that the user likes. These images and tag information are analyzed and learning is performed within the camera 101.

[0162] The control unit 411 of the external device 301 is capable of acquiring images uploaded by the user and images determined to be liked by the user, and processing the images with the same capabilities as the learning processing unit 219 in the camera 101. Learning is performed by communicating the processed learning data to the camera 101. Alternatively, image data to be learned may be transmitted to the camera 101, and learning may be performed within the camera 101.

[0163] It is possible to estimate subject information that the user would like from subject information set in the tag information (for example, object information such as a dog or cat, scene information such as a beach, and facial expression information such as a smile). Learning is performed by registering the subject as a subject to be detected and input to the neural network. In addition, it is also possible to estimate image information that is currently popular in the world from statistics of tag information (image filter information and subject information) on SNS, and configure the camera 101 to be able to learn.

[0164] <Learning by changing camera parameters with an external device> The learning parameters currently set in the camera 101 (such as the weights of the NN and the selection of subjects to be input to the NN) can be transmitted to the external device 301 and stored in the storage unit 404 of the external device 301. Furthermore, using a dedicated application in the external device 301, the learning parameters set in a dedicated server are acquired via the public wireless control unit 406. These can also be set as the learning parameters in the camera 101. The learning parameters can also be restored by saving the parameters at a certain point in time in the external device 301 and setting them in the camera 101. Furthermore, the learning parameters held by other users can be acquired via a dedicated server and set in the owner's own camera 101.

[0165] Also, a dedicated application of the external device 301 may be used to register voice commands, authentication registration, gestures, and important locations registered by the user. Such information is treated as input data for the image capture trigger and automatic image capture determination described in the image capture mode process (FIG. 9). Also, a configuration may be used in which the image capture frequency, start-up interval, ratio of still images to videos, and favorite images can be set, and the start-up interval and other settings described in the low power consumption mode control are performed.

[0166] <Learning from information obtained by manually editing images using an external device> A dedicated application of the external device 301 realizes a function for manual editing according to user operations, and the contents of the editing work can also be fed back to learning. For example, image effect addition (trimming, rotation, slide, zoom, fade, color conversion filter effect, time, still image to video ratio, BGM) can be edited. Learning of an automatic editing neural network is performed so that the addition of manually edited image effects can be judged based on the learning information of the image.

[0167] Next, the learning process sequence will be described. In the mode setting determination of S704 in FIG. 7, it is determined whether or not the learning process should be performed. If it is determined that the learning process should be performed, the learning mode process of S716 is executed. The conditions for determining the learning mode will be described. The determination of whether or not to switch to the learning mode is made based on information such as the elapsed time since the previous learning process was performed, the number of pieces of information available for learning, and whether or not a learning process instruction has been received via a communication device. The learning mode determination process will be described with reference to FIG. 15.

[0168] Fig. 15 is a flowchart for explaining the process of determining whether or not to transition to the learning mode, which is executed in S704 (mode setting determination process) in Fig. 7. When an instruction to start the learning mode determination is given in the mode setting determination process of S704, the process in Fig. 15 starts. In S1401, it is determined whether or not there is a registration instruction from the external device 301. This registration instruction is a registration instruction for learning, such as the above-mentioned <learning by acquiring an image with the external device>, <learning by inputting a determination value into an image via the external device>, and <learning by analyzing an image stored in the external device>.

[0169] If it is determined in S1401 that a registration instruction has been received from the external device 301, the process proceeds to S1408. In S1408, the learning mode determination flag is set to TRUE, and the process of S716 is set to be performed, and then the learning mode determination process ends. Also, if it is determined in S1401 that a registration instruction has not been received from the external device 301, the process proceeds to S1402.

[0170] In S1402, it is determined whether or not a learning instruction has been received from the external device 301. This learning instruction is an instruction to set learning parameters, such as <learning by changing camera parameters with an external device>. If it is determined in S1402 that a learning instruction has been received from the external device 301, the process proceeds to S1408. Also, if it is determined in S1402 that a learning instruction has not been received from the external device 301, the process proceeds to S1403.

[0171] In S1403, the time (denoted as TimeN) that has elapsed since the previous learning process (recalculation of the weights of the NN) is acquired. Then, the process proceeds to S1404, where the number of new data to be learned (denoted as DN) is acquired. The number of data DN corresponds to the number of images that have been specified to be learned during the time (TimeN) that has elapsed since the previous learning process was performed.

[0172] Next, the process proceeds to S1405, where a threshold value DT for determining whether or not to switch to the learning mode is calculated based on the elapsed time TimeN. The threshold value DT is set so that the smaller the value, the easier it is to switch to the learning mode. For example, the value of the threshold value DT when TimeN is smaller than a predetermined value is represented as DTa, and the value of the threshold value DT when TimeN is larger than the predetermined value is represented as DTb. DTa is set larger than DTb, and the threshold value is set to become smaller as time passes. As a result, even if there is little learning data, if the elapsed time is long, it becomes easier to switch to the learning mode, and learning is performed again. In other words, the setting of the ease and difficulty of the camera switching to the learning mode is changed according to the usage time.

[0173] After the process of S1405, the process proceeds to S1406, where it is determined whether the number of data to be learned DN is greater than the threshold value DT. If it is determined that the number of data DN is greater than the threshold value DT, the process proceeds to S1407, and if it is determined that the number of data DN is equal to or less than the threshold value DT, the process proceeds to S1409. In S1407, the number of data DN is set to zero. After that, the process of S1408 is executed, and the learning mode determination process is terminated.

[0174] When proceeding to S1409, since there is no registration instruction or learning instruction from the external device 301 and the number of learning data DN is equal to or smaller than the threshold value DT, the learning mode determination flag is set to FALSE. After it is set so that the process of S716 is not performed, the learning mode determination process is terminated.

[0175] Next, the processing in the learning mode processing (FIG. 7: S716) will be described. FIG. 16 is a flowchart showing an example of the learning mode processing. When it is determined in S715 of FIG. 7 that the learning mode is selected and the process proceeds to S716, the processing in FIG. 16 starts. In S1501, it is determined whether or not there is a registration instruction from the external device 301. If it is determined in S1501 that there is a registration instruction from the external device 301, the process proceeds to S1502. Also, if it is determined in S1501 that there is no registration instruction from the external device 301, the process proceeds to S1504.

[0176] In S1502, various registration processes are executed. Various registrations are registrations of features to be input to the neural network, such as face authentication registration, general object recognition registration, sound information registration, and location information registration. After the registration processes are completed, the process proceeds to S1503. In S1503, a process is performed to change the elements to be input to the neural network from the information registered in S1502. After the process of S1503 is completed, the process proceeds to S1507.

[0177] In S1504, it is determined whether or not there is a learning instruction from the external device 301. If it is determined that there is a learning instruction from the external device 301, the process proceeds to S1505, and if it is determined that there is no learning instruction, the process proceeds to S1506.

[0178] In S1505, the learning parameters communicated from the external device 301 are set in each classifier (such as the weights of the NN), and then the process proceeds to S1507. In addition, in S1506, learning (recalculation of the weights of the NN) is performed. The process proceeds to S1506 when the number of data DN exceeds the threshold value DT and re-learning of each classifier is performed, as described with reference to FIG. 15. The weights of the NN are recalculated by re-learning using the backpropagation method, the gradient descent method, or the like, and the parameters of each classifier are changed. Once the learning parameters are set, the process proceeds to S1507.

[0179] In S1507, a process of rescoring the images in the file is executed. In this embodiment, a score is assigned to all captured images stored in the file of the recording medium 221 based on the learning result, and automatic editing or automatic file deletion is performed according to the assigned score. Therefore, when re-learning or setting of learning parameters from an external device is performed, the scores of captured images must also be updated. In S1507, a recalculation is performed to assign a new score to the captured images stored in the file, and when the process is completed, the learning mode process is terminated.

[0180] The above describes a method for proposing images that match the user's preferences by extracting scenes that are estimated to be preferred by the user, learning their features, and reflecting them in camera operations such as automatic shooting and automatic editing. The embodiments of the present invention are not limited to this application. For example, the present invention can also be applied to applications in which images that are different from the user's own preferences are proposed, as follows.

[0181] <Method using neural networks that have learned preferences> The user's preferences are learned by the method described above. Then, automatic shooting judgment processing is executed in S908 of FIG. 9. Automatic shooting is executed when the output value of the NN is a value indicating that it is different from the user's preferences, which are the teacher data. For example, assume that the image that the user likes is used as the teacher image, and learning is performed so that a high value is output when the image shows characteristics similar to the teacher image. In this case, automatic shooting is executed on the contrary, on the condition that the output value is lower than a predetermined threshold value. Similarly, in the subject search processing and automatic editing processing, processing is executed in which the output value of the NN is a value indicating that it is different from the user's preferences, which are the teacher data.

[0182] <Method of using neural networks that have learned situations different from preferences> At the time of the learning process, a process of learning a situation different from the user's preference as teacher data is executed. In the above example, a learning method was described in which a manually captured image is a scene that the user likes to capture, and this image is used as teacher data. In contrast, a process of not using a manually captured image as teacher data, and adding a scene that has not been manually captured for a predetermined time or more as teacher data is executed. Alternatively, if the teacher data contains data of a scene whose characteristics are similar to those of a manually captured image, a process of deleting this data from the teacher data is executed. In addition, a process of adding an image whose characteristics are different from those of an image acquired by an external device to the teacher data, or a process of deleting an image whose characteristics are similar to those of the acquired image is executed. In this way, data different from the user's preference is accumulated in the teacher data, so that the NN can distinguish a situation different from the user's preference as a result of learning. In automatic shooting, shooting is performed according to the output value of the NN, so a scene different from the user's preference can be shot.

[0183] By deliberately suggesting images that are different from the user's preferences, scenes that the user would not shoot manually can be captured, reducing the number of missed shots. In addition, by suggesting shooting scenes that the user would not have thought of, the effect of encouraging the user to be aware of things and broadening the range of preferences is achieved.

[0184] By combining the above methods, it is easy to propose situations that are somewhat similar to the user's preferences but differ in some areas, and to adjust the degree of conformance to the user's preferences. The degree of conformance to the user's preferences can be changed according to the mode settings, the state of various sensors, and the state of the detection information.

[0185] In this embodiment, a configuration in which learning is performed within the camera 101 has been described. On the other hand, if the external device 301 has a learning function, data required for learning is transmitted to the external device 301, and learning is performed only in the external device 301. Even with such a configuration, it is possible to achieve a learning effect similar to that described above. For example, as described in <Learning by changing camera parameters in an external device>, a configuration in which learning is performed by setting parameters such as the weights of the NN learned by the external device 301 in the camera 101 via communication may be used.

[0186] In another embodiment, the camera 101 and the external device 301 each have a learning function. For example, when the learning mode process (FIG. 7: S716) is performed in the camera 101, learning information held by the external device 301 is transmitted to the camera 101, learning parameters are merged, and learning is performed using the merged parameters.

[0187] According to this embodiment, when a single imaging device is used for automatic photography and automatic authentication and registration, it is possible to perform both photography for automatic photography and photography for automatic authentication and registration. In particular, it is possible to realize control that does not impede opportunities for automatic photography while improving the accuracy of automatic photography through automatic authentication and registration.

[0188] Hereinafter, with reference to Figs. 17 to 35, an embodiment in which a subject person to be photographed is determined and tracking control is performed will be described. In automatic photography, for example, a user can register characteristic information of a main person in a camera and specify that the registered person is to be tracked and photographed preferentially, thereby enabling photography centered on that person (priority person). Even if a priority person is not detected, or if a priority person is detected but not recognized as a priority person, it is desirable to photograph the main person as much as possible. Even if a priority person is detected, if other main people such as family or friends are detected at the same time, it is desirable to control so that these people are also included in the angle of view and unrelated people are not included in the angle of view as much as possible.

[0189] As a subject identification technology, there is a technology that analyzes image data on a frame-by-frame basis to identify detected subjects, extracts the appearance frequency of the identified subjects, and selects the main subject from among the subjects based on the appearance frequency. With this technology, a certain number of subjects are always selected in order of appearance frequency. Therefore, when the absolute number of people is small, there is a possibility that a person who appears much less frequently than the actual main person may be determined to be a main person. In addition, since the distance between the subject and the imaging device is not taken into consideration, even unrelated people who are far from the imaging device may be included as main people.

[0190] In the following, a technique for reducing the frequency with which irrelevant people are included in the shooting angle of view while keeping a main person within the shooting angle of view in an automatic camera that regularly and continuously shoots without a user giving a shooting instruction will be described. Specifically, an example will be shown in which the shooting priority of a person is determined based on the face size, face position, face reliability, and detection frequency of the detected person, and the person to be photographed is determined according to the shooting priority of each person. When a person with a high shooting priority is detected, the person and people with a similar shooting priority are determined as the shooting target, and control is performed to exclude people with a shooting priority that is more than a certain distance away from the user from the shooting target. By selecting the shooting target, it is possible to increase the possibility that the user and people with a shooting priority close to the user are photographed, and to reduce the possibility that irrelevant people are photographed.

[0191] 17 is a block diagram showing an imaging device composed of lens barrel 102, tilt rotation unit 104, pan rotation unit 105, and control box 1100. Control box 1100 includes a microcomputer for controlling the group of photographing lenses included in lens barrel 102, tilt rotation unit 104, and pan rotation unit 105. Control box 1100 is disposed within fixed portion 103 of the imaging device. Control box 1100 remains fixed even when lens barrel 102 is driven for panning or tilting.

[0192] The lens barrel 102 includes a lens unit 1021 that constitutes an imaging optical system, and an imaging section 1022 that has an imaging element. The lens barrel 102 is controlled by a tilt rotation unit 104 and a pan rotation unit 105 so as to rotate in a tilting direction and a panning direction, respectively. The lens unit 1021 is composed of a zoom lens that changes the magnification and a focus lens that adjusts the focus, and is driven and controlled by a lens drive section 1113 in a control box 1100. The zoom mechanism section is composed of a zoom lens and a lens drive section 1113 that drives the lens. The zoom function is realized by moving the zoom lens in the optical axis direction by the lens drive section 1113.

[0193] The imaging section 1022 has an imaging element, receives light incident through each lens group constituting the lens unit 1021, and outputs charge information according to the amount of light as digital image data to the image processing section 1103. The tilt rotation unit 104 and the pan rotation unit 105 rotate the lens barrel 102 according to drive instructions input from the lens barrel rotation drive section 1112 in the control box 1100.

[0194] Next, a description will be given of the internal configuration of the control box 1100. The shooting direction during automatic shooting is controlled by a provisional registration determination unit 1108, a shooting subject determination unit 1110, a drive control unit 1111, and a lens barrel rotation drive unit 1112.

[0195] The image processing unit 1103 acquires the digital image data output from the imaging unit 1022. Image processing such as distortion correction, white balance adjustment, and color interpolation processing is applied to the acquired digital image data. The digital image data after the application is output to the image recording unit 1104 and the subject detection unit 1107. Furthermore, the image processing unit 1103 outputs the digital image data to the feature information extraction unit 1105 in response to an instruction from the provisional registration determination unit 1108.

[0196] The image recording unit 1104 converts the digital image data output from the image processing unit 1103 into a recording format such as JPEG format, and records it on a recording medium (such as a non-volatile memory). The feature information extraction unit 1105 acquires an image of a face located at the center of the digital image data output from the image processing unit 1103. The feature information extraction unit 1105 extracts feature information from the acquired face image, and outputs the face image and feature information to the person information management unit 1106. The feature information is information indicating a plurality of facial feature points located in parts of the face such as the eyes, nose, and mouth, and is used to identify the person of the detected subject. The feature information may be other information indicating facial features such as the face outline, face color information, and face depth information.

[0197] The person information management unit 1106 performs a process of storing and managing the person information linked to each person in the storage unit. An example of the person information will be described with reference to FIG. 18. The person information is composed of a person ID, a face image, feature information, a registration state, a priority setting, and a name. The person ID is an ID (identification information) for identifying multiple person information, and the same ID is not issued and a value of 1 or more is set. The face image data is face image data input from the feature information extraction unit 1105. The feature information is information input from the feature information extraction unit 1105. It is assumed that two registration states, "provisional registration" and "full registration", are defined. "Provisional registration" indicates a state in which a person is determined to be a possible main person by a provisional registration determination. "Full registration" indicates a state in which a person is determined to be a main person by a full registration determination or by the presence or absence of a user operation. The details of the provisional registration determination and full registration determination processes will be described later. The priority setting is a setting indicating whether or not to preferentially shoot by a user operation. The name is a name given to each person by a user operation.

[0198] When the person information management unit 1106 acquires a face image and feature information from the feature information extraction unit 1105, it issues a new person ID, associates the person ID with the input face image and feature information, and adds new person information. When new person information is added, the initial value of the registration state is "provisional registration", the initial value of the priority setting is "none", and the initial value of the name is blank. When the person information management unit 1106 acquires a final registration determination result (person ID to be final registration) from the final registration determination unit 1109, it changes the registration state of the person information corresponding to the person ID to "full registration". When a change instruction for the person information (priority setting or name) is input from the communication unit 1114 by a user operation, the person information management unit 1106 changes the person information according to the instruction. When a change is made to either the priority setting or the name for a person whose registration state is "provisional registration", the person information management unit 1106 determines that the person is a key person and changes the registration state of the person to "full registration". The importance determination unit 1514 will be described later.

[0199] 19 is a schematic diagram showing an example of a screen of a portable terminal device (external device) that communicates with the camera 101. The portable terminal device acquires person information via the communication unit 1114 of the camera 101, and displays a list on the screen. In the example shown in FIG. 19, a face image, name, and priority setting are displayed on the screen. The name and priority setting can be changed by the user. When the name or priority setting is changed, the portable terminal device outputs an instruction to change the name or priority setting linked to the person ID to the communication unit 1114.

[0200] The subject detection unit 1107 (FIG. 17) detects subjects from the digital image data input from the image processing unit 1103, and extracts information on the detected subjects (subject information). An example is shown in which the subject detection unit 1107 detects a person's face as a subject. The subject information is, for example, the number of detected subjects, the position of the face, the size of the face, the direction of the face, and a face reliability indicating the likelihood of detection. The subject detection unit 1107 also calculates a similarity by comparing the characteristic information of each person acquired from the person information management unit 1106 with the characteristic information of the detected subject. If the similarity is equal to or greater than a threshold, a process is executed to add the person ID, registration status, and priority setting of the detected person to the subject information. The subject detection unit 1107 outputs the subject information to the provisional registration determination unit 1108, the main registration determination unit 1109, and the photographing target determination unit 1110. An example of the subject information will be described later with reference to FIG. 20.

[0201] The provisional registration determination unit 1108 determines whether or not the subject detected by the subject detection unit 1107 is likely to be a main person, i.e., whether or not the subject should be provisionally registered. If it is determined that any of the subjects is a person to be provisionally registered, the provisional registration determination unit 1108 calculates a panning drive angle, a tilting drive angle, and a target zoom position required to place the person to be provisionally registered in the center of the screen at a specified size. A command signal based on the calculation result is output to a drive control unit 1111. Details of the provisional registration determination process will be described later with reference to FIG. 22.

[0202] The final registration determination unit 1109 determines a person close to the user, i.e., a person who should be formally registered, based on the subject information acquired from the subject detection unit 1107. If it is determined that any person should be formally registered, the person ID of the person who should be formally registered is output to the person information management unit 1106. Details of the final registration determination process will be described later with reference to Figs. 24 to 26.

[0203] The shooting subject determination unit 1110 determines a subject to be shot based on the subject information acquired from the subject detection unit 1107. Furthermore, based on the determination result of the person to be shot, the shooting subject determination unit 1110 calculates a panning drive angle, a tilting drive angle, and a target zoom position for fitting the person to be shot within the angle of view at a specified size. A command signal based on the calculation result is output to a drive control unit 1111. Details of the shooting subject determination process will be described later with reference to FIG. 27.

[0204] When the drive control unit 1111 acquires a command signal from the provisional registration determination unit 1108 or the shooting subject determination unit 1110, it outputs information on control parameters to the lens drive unit 1113 and the lens barrel rotation drive unit 1112. Parameters based on a target zoom position are output to the lens drive unit 1113. Parameters corresponding to a target position based on a panning drive angle and a tilting drive angle are output to the lens barrel rotation drive unit 1112.

[0205] When there is an input from the provisional registration determination unit 1108, the drive control unit 1111 does not refer to the input from the shooting subject determination unit 1110, but determines each target position (target zoom position, target position based on the drive angle) based on the input value from the provisional registration determination unit 1108. The lens barrel rotation drive unit 1112 outputs drive commands to the tilt rotation unit 104 and the pan rotation unit 105 based on the target position and drive speed from the drive control unit 1111. The lens drive unit 1113 has a motor and driver unit for driving the zoom lens, focus lens, etc. that constitute the lens unit 1021. The lens drive unit 1113 drives each lens based on the target position from the drive control unit 1111.

[0206] The communication unit 1114 transmits the personal information stored in the personal information management unit 1106 to an external device such as a mobile terminal device. Furthermore, when the communication unit 1114 receives an instruction to change the personal information from the external device, it outputs an instruction signal to the personal information management unit 1106. In this embodiment, it is assumed that the change instruction from the external device is an instruction to change the priority setting of the personal information and the name.

[0207] FIG. 20 is a diagram showing an example of image data and an example of subject information acquired by the subject detection unit 1107. FIG. 20(A) is a schematic diagram showing an example of image data input to the subject detection unit 1107. For example, the image data is configured with a horizontal resolution of 960 pixels and a vertical resolution of 540 pixels. FIG. 20(B) is a table showing an example of subject information extracted when the data of the image shown in FIG. 20(A) is input to the subject detection unit 1107. The illustrated subject information is configured by the number of subjects, and the subject ID, face size, face position, face direction, face reliability, person ID, registration status, and priority setting of each subject.

[0208] The number of subjects indicates the number of faces detected. In the example of FIG. 20(B), the number of subjects is 4, which indicates that the face size, face position, face direction, face reliability, person ID, registration status, and priority settings for 4 subjects are included. The subject ID is a number used to identify a subject, and is issued when a new subject is detected. The same subject ID is not issued, and a new value is issued each time a subject is detected. For example, if a specific subject moves out of the field of view and becomes undetectable, then returns to the field of view and is detected again, a new value will be issued even if it is the same subject.

[0209] The face size (w, h) is a numerical value indicating the size of the detected face, and the number of pixels of the width (w) and height (h) of the face are input. In this embodiment, the width and height are assumed to be the same value. The face position (x, y) is a numerical value indicating the relative position of the detected face within the shooting range. When the upper left corner of the image data is defined as the starting point (0, 0) and the lower right corner of the screen is defined as the ending point (960, 540), the number of horizontal pixels and the number of vertical pixels from the starting point to the center coordinates of the face are input. The face direction is information indicating the direction of the detected face, and any one of the following information is input: front, right 45 degrees, right 90 degrees, left 45 degrees, left 90 degrees, and unknown. The face reliability is information indicating the certainty of the detected human face, and any one of values ​​0 to 100 is input. The face reliability is calculated from the similarity with the feature information of a plurality of standard face templates stored in advance.

[0210] The person ID is the same as the person ID managed by the person information management unit 1106. When a subject is detected, the subject detection unit 1107 calculates the similarity between the characteristic information of each person acquired from the person information management unit 1106 and the characteristic information of the subject. The person ID of the person whose similarity is equal to or greater than a threshold is input. If the characteristic information is not similar to any person acquired from the person information management unit 1106, zero is input as the ID value. The registration status and priority setting information are the same as the registration status and priority setting information managed by the person information management unit 1106. If the person ID is not zero, that is, if it is determined to be one of the people managed by the person information management unit 1106, the registration status and priority setting information of the relevant person acquired from the person information management unit 1106 is input.

[0211] The process periodically executed in this embodiment will be described with reference to Fig. 21. Fig. 21 is a flowchart showing the overall flow of photographing and registering and updating person information. When the power of the imaging device is turned on, the imaging unit 1022 of the imaging device starts periodic photographing (video shooting) to obtain image data used for various judgments (photography subject judgment, provisional registration judgment, and final registration judgment). The repetitive process starts at S500.

[0212] Image data acquired by shooting is output to the image processing unit 1103, and in S501, image data that has been subjected to various image processes is acquired. Since the acquired image data is image data for various determinations, this image data is output from the image processing unit 1103 to the subject detection unit 1107. In other words, the image data acquired here corresponds to image data for live view display in an imaging device in which a user adjusts the composition and operates the shutter to shoot, and the periodic shooting to acquire this image data corresponds to live view shooting. Using the acquired image data, the control box 1100 adjusts the composition and determines the timing of automatic shooting.

[0213] Next, in S502, the subject detection unit 1107 detects a subject based on the image data and acquires subject information (see FIG. 20(B)). After the subject is detected and the subject information is acquired, a final registration determination is made in S503. In the final registration determination, a person to be formally registered is determined using information on the detected subject. In this determination, the person information in the person information management unit 1106 is updated, but panning drive, tilting drive, and zoom drive are not executed.

[0214] In S504, a provisional registration determination is performed. In the provisional registration determination, a subject to be provisionally registered is determined from among the detected subjects, and a panning drive angle and a tilting drive angle are obtained based on the position of the face of the subject to be provisionally registered. In addition, a target zoom position is obtained based on the position and size of the face. The provisional registration determination unit 1108 instructs the image processing unit 1103 to output image data to the feature information extraction unit 1105. When the panning drive angle, tilting drive angle, and target zoom position are obtained in the provisional registration determination, panning drive, tilting drive, and zoom drive are performed based on this information, and the composition for provisional registration is adjusted.

[0215] After the process of S504, the process proceeds to S505, where it is determined whether or not the composition adjustment process for temporary registration is being executed. If the composition adjustment process for temporary registration is being executed in S505, the process proceeds to S506, and if the composition adjustment process for temporary registration is not being executed, the process proceeds to S507.

[0216] In S506, the feature information extraction unit 1105 extracts feature information of the subject located at the center of the image data, and outputs the extracted feature information to the person information management unit 1106. In addition, in S507, shooting subject determination is performed. The shooting subject determination unit 1110 determines the subject to be shot from among the detected subjects. A panning drive angle and a tilting drive angle are obtained based on the position of the face of the subject to be shot. In addition, a target zoom position is obtained based on the position and size of the face. When the panning drive angle, tilting drive angle, and target zoom position are obtained by the shooting subject determination, panning drive, tilting drive, and zoom drive are performed based on this information, and the shooting composition is adjusted.

[0217] After S506 and S507, the process proceeds to S508, where it is determined whether the repetitive process is to be ended. If the process is to be continued, the process returns to S500 and continues. The processes shown in S501 to S507 are repeatedly executed in accordance with the imaging cycle of the imaging unit 1022.

[0218] <Temporary registration process> The provisional registration determination process shown in S504 of FIG. 21 will be described with reference to FIG. 22. FIG. 22(A) is a flowchart illustrating the provisional registration determination process performed by the provisional registration determination unit 1108. This process is executed periodically, and a determination is made as to whether or not the subject is likely to be a main person. FIG. 22(B) is a table showing the provisional registration count. The provisional registration count is linked to the subject ID, and when the provisional registration count reaches 50 or more, the corresponding subject is determined to be a person eligible for provisional registration. Since the provisional registration determination is executed over multiple cycles, the current provisional registration count is stored at the time of determination in the current cycle, and in the next cycle, a process is performed in which the provisional registration count added up to the previous cycle is referenced and carried over.

[0219] In S600, a repeat process corresponding to the number of detected subjects is started. When the provisional registration determination unit 1108 acquires subject information from the subject detection unit 1107, it executes the processes of S601 to S609 for each subject, and when any subject is determined to be a provisional registration target, it executes the processes of S610 to S613. In S601, an unregistered determination process is executed. The provisional registration determination unit 1108 refers to the person ID in the subject information, and when it determines that the subject is unregistered (the person ID is zero), it transitions to the process of S602. Also, when it is determined that the value of the person ID is 1 or more, that is, that the subject has already been registered, it transitions to the determination process for the next subject.

[0220] In S602, the provisional registration determination unit 1108 refers to the provisional registration count up to the previous frame that has been saved, and if a provisional registration count for the same subject ID exists, the provisional registration count is inherited. Next, in S603, the provisional registration determination unit 1108 determines whether the face is facing forward. If it is determined that the face is facing forward, the process proceeds to S604, and if it is determined that the face is not facing forward, the process proceeds to S607.

[0221] S604 is a process for judging whether or not the face size at the time of zoom wide is in the range of 100 to 200. If this condition is met, the process proceeds to S605, and if the condition is not met, the process proceeds to S607. S605 is a process for judging whether or not the face reliability is equal to or greater than the threshold value of 80. If this condition is met, the process proceeds to S606, and if the condition is not met, the process proceeds to S607.

[0222] If all the conditions shown in S603 to S605 are met, the process proceeds to S606. In S606, the provisional registration determination unit 1108 determines that the target person is likely to be a main person close to the user, and adds 1 to the provisional registration count (increments). On the other hand, if even one of the conditions shown in S603 to S605 is not met, the process proceeds to S607. In S607, the provisional registration determination unit 1108 determines that the target person is unlikely to be a main person, and sets the provisional registration count to zero.

[0223] After the processes of S606 and S607, in S608, the provisional registration determination unit 1108 compares the provisional registration count value of the subject with a threshold value of 50. If it is determined that the provisional registration count value is less than 50, the process proceeds to S609. On the other hand, if it is determined that the provisional registration count value is 50 or more, the process proceeds to S611.

[0224] In S609, the temporary registration determination unit 1108 determines whether the value of the temporary registration count is greater than zero. If it is determined that the value of the temporary registration count is greater than zero, the process proceeds to S610, and if the condition is not satisfied (the value of the temporary registration count is zero), the process proceeds to S614 without saving the temporary registration count. Also, in S610, the temporary registration determination unit 1108 saves the temporary registration count, and then proceeds to the determination process of S614. If it is determined in S614 that the repetitive process is to be ended and the process is to continue, the process returns to S600 and proceeds to the determination process of the next subject.

[0225] In S611, the temporary registration determination unit 1108 determines that the subject in question may be a main person and sets the subject as a subject for temporary registration. In S612, the temporary registration determination unit 1108 calculates a panning drive angle, a tilting drive angle, and a zoom movement position so that the face of the subject to be temporarily registered is positioned at the center of the screen with an appropriate face size, and outputs a command based on the calculation results to the drive control unit 1111. For example, when the center position of the face is within 5% of the center of the screen and the face size is 100 to 200, it is assumed that the feature information extraction unit 1105 can acquire feature information.

[0226] In this embodiment, in order to obtain the feature information, the subject to be photographed is controlled to be positioned at the center of the screen. However, the present invention is not limited to this, and the feature information may be extracted by performing image processing such as cutting out a part of the image data including the face of the subject without changing the position of the subject.

[0227] In S613, the provisional registration determination unit 1108 instructs the image processing unit 1103 to output image data to the feature information extraction unit 1105. The feature information extraction unit 1105 cuts out a face image located in the center of the input image data, extracts feature information, and outputs it to the personal information management unit 1106. The personal information management unit 1106 adds new personal information based on the input face image and feature information. After the processing of S613, the series of processes ends.

[0228] The zoom position in the imaging device of this embodiment can be set from 0 to 100. The smaller the zoom position value, the wider the image, and the larger the zoom position value, the closer the image is to the telephoto side. That is, the zoom wide shown in S604 means that the zoom position is zero and the angle of view is the widest. In the imaging device, if the face size in the zoom wide mode is 100 to 200, it is determined that the distance between the subject and the imaging device can be predicted to be about 50 cm to 150 cm. In other words, if the subject is located at a distance that is neither too close nor too far from the imaging device, it is determined that there is a possibility that the subject is a main person. In the example of FIG. 22, the process of calculating the distance between the subject and the imaging device from the face size has been described, but the distance to the subject may be measured by another method using a depth sensor, a compound eye lens, or the like.

[0229] Next, a specific example of provisional registration determination when the subject information shown in Fig. 20(B) is input will be described. Note that the zoom position is set to zero here. Since subjects 1 and 2 in Fig. 20(B) have already been registered in S601 in Fig. 22(A) (their person IDs are not zero), the processing from S602 onwards is not executed.

[0230] Since the person ID of subject 3 in FIG. 20B is zero (unregistered) in S601 in FIG. 22A, the processing from S602 onward is executed. As shown in FIG. 22B, the provisional registration count of subject ID 3 up to the previous cycle is 30. In S602 in FIG. 22A, the provisional registration count up to the previous cycle is referenced, and if a provisional registration count of subject ID 3 exists, that information is handed over. Since subject 3 in FIG. 20B faces forward, the processing moves from S603 to S604 in FIG. 22A. Since the face size at the time of zoom wide is 120 in S604, the processing moves to S605, and since the face reliability is 80 in S605, the processing moves to S606. In S606 in FIG. 22, 1 is added to the provisional registration count, making it 31. Since the provisional registration count is less than 50 in S608, the provisional registration count is stored in S609, and then the processing moves to the determination of the next subject.

[0231] For subject 4 in FIG. 20(B), the person ID is zero in S601 in FIG. 22(A), so the processing from S602 onwards is executed. In S602, the provisional registration count up to the previous cycle is referenced, and if a provisional registration count for subject ID 4 exists, that information is carried over. Here, it is assumed that there is no provisional registration count for the subject ID up to the previous cycle. In S603 in FIG. 22(A), the face direction is 90 degrees to the left, so the processing proceeds to S607, and the provisional registration count is set to zero. In S608, the provisional registration count is less than 50, so the processing proceeds to S609, and since the provisional registration count is zero in S609, the provisional registration count is not saved and the processing ends.

[0232] Next, an example will be described in which the provisional registration count becomes 50 or more in S608 in Fig. 22(A), and the subject to be provisionally registered is positioned in the center of the angle of view by panning drive, tilting drive, and zoom drive. When subject 3 in Fig. 20(B) becomes the subject to be provisionally registered, the panning drive angle and tilting drive angle are calculated so that the subject's face position is within a predetermined range. The predetermined range is a range in which the subject's face position is within 5% of the center of the screen, that is, a range of x position coordinate values ​​between 432 and 528 and ay position coordinate values ​​between 513 and 567. Since the face size of subject 3 is within 100 to 200, the zoom position is not changed.

[0233] Fig. 23(A) is a diagram showing an example of image data when the panning position and tilting position are changed with respect to Fig. 20(A). Fig. 23(B) is a table showing an example of subject information extracted when the image data shown in Fig. 23(A) is input to the subject detection unit 1107. In this embodiment, by arranging a face in an appropriate size in the center of the screen, it becomes possible for the feature information extraction unit 1105 to acquire feature information. In the provisional registration determination process, an unregistered person who satisfies a specific condition over multiple periods is determined to be a possible main person and is added to the person information management unit 1106.

[0234] <Registration> Next, the final registration determination process shown in S503 of Fig. 21 will be described with reference to Fig. 24. Fig. 24(A) is a flowchart explaining the final registration determination process performed by the final registration determination unit 1109. Like the provisional registration determination, the final determination process is executed over multiple cycles, and main people are determined from people who have already been provisionally registered.

[0235] FIG. 24(B) is a table showing count A, count B, and official registration count linked to a person ID. Count A and count B are added under different conditions, and when the value of count A is 50 or more, or the value of count B is 50 or more, the official registration count is added. When the official registration count reaches 100, the corresponding subject is determined to be a person eligible for official registration. When determining each cycle, the current count A, count B, and official registration count are stored, and in the next cycle, the various counts added up to the previous cycle are referenced and carried over.

[0236] In S1700, a repeat process corresponding to the number of detected subjects is started. When the final registration determination unit 1109 acquires subject information from the subject detection unit 1107, it executes the processes of S1701 to S1707 in FIG. 24A for each subject. In S1701, the final registration determination unit 1109 performs a "provisional registration" determination. The registration status of the subject information is referenced, and if it is determined to be "provisional registration", the process proceeds to S1702. If it is determined not to be "provisional registration", the process proceeds to the determination process for the next subject.

[0237] In S1702, the main registration determination unit 1109 refers to the various counts stored up to the previous frame, and if there are various counts for the same person ID, the main registration determination unit 1109 takes over the various counts. Then, the main registration determination unit 1109 executes a first main registration count determination (S1703), and further executes a second main registration count determination (S1704). The first main registration count determination is a determination based on the subject information of a single person. A process is executed in which a count A is added according to the distance and reliability between the target person and the imaging device, and the main registration count is added. In addition, the second main registration count determination is a determination based on the degree of association with a person who has already been "mainly registered" and has been determined to be a main person. Specifically, if multiple "main registration people" are detected at the same time, a process is executed in which a count B is added according to whether the distances from the imaging device are equal, and the main registration count is added. The first and second main registration count determination processes will be described in detail later.

[0238] In S1705 following S1704, the final registration determination unit 1109 compares the value of the final registration count of the relevant person with a threshold value of 100. If it is determined that the value of the final registration count is 100 or more, the process proceeds to S1706, and if it is determined that the value of the final registration count is less than 100, the process proceeds to S1707. In S1706, the final registration determination unit 1109 instructs the person information management unit 1106 to change the registration status of the relevant person to "final registration". In addition, in S1707, the final registration determination unit 1109 saves the current various counts. After S1706 and S1707, the process proceeds to S1708, where it is determined whether the repetitive process has ended. If the process is to continue, the process returns to S1700 and continues the process for the next detected subject.

[0239] Next, the process of S1703 in Fig. 24 (first main registration count determination) will be described with reference to the flowchart in Fig. 25. In S1801, the main registration determination unit 1109 determines whether or not the face size at the time of zoom wide is within the range of 100 to 200. If this condition is met, the process proceeds to S1802, and if the condition is not met, the process proceeds to S1804.

[0240] In S1802, the official registration determination unit 1109 determines whether the face reliability is equal to or greater than a threshold value of 80. If this condition is met, the process proceeds to S1803, and if the condition is not met, the process proceeds to S1804. If all of the conditions in S1801 and S1802 are met, the process proceeds to S1803, where a value equivalent to "face size at zoom wide / 10" is added to count A. In addition, in S1804, the official registration determination unit 1109 sets count A to zero, and then ends the process.

[0241] After S1803, in S1805, the main registration determination unit 1109 compares the value of count A with a threshold value of 50. If it is determined that the value of count A is 50 or greater, the process proceeds to S1806, and if it is determined that count A is less than 50, the process ends. In S1806, the main registration determination unit 1109 adds 1 to the main registration count, and in S1807, sets count A to zero. After S1807, the process ends.

[0242] The process of S1704 (second formal registration count determination) in Fig. 24 will be described with reference to the flowchart in Fig. 26. In S1901, the formal registration determination unit 1109 references the subject information and determines whether a person whose registration status is "formal registration", that is, multiple people who have already been determined to be main, have been detected simultaneously. If it is determined that formal registration people have been detected simultaneously, the process proceeds to S1902. If it is determined that formal registration people have not been detected simultaneously, the process proceeds to S1905.

[0243] In S1902, the official registration determination unit 1109 refers to the face size of the subject information and determines whether the face size is close to that of any of the officially registered persons detected at the same time. Specifically, for example, if the face size of the subject information is within the range of "the face size of the officially registered person ±10%" as a determination condition, the face size is considered to be close. If the condition of S1902 is met, the process proceeds to S1903, and if the condition is not met, the process proceeds to S1905.

[0244] In S1903, the final registration determination unit 1109 compares the face reliability with a threshold value of 80. If it is determined that the face reliability is 80 or more, the process proceeds to S1904, and if it is determined that the face reliability is less than 80, the process proceeds to S1905. In S1904, the final registration determination unit 1109 adds a value equivalent to "face size at zoom wide / 10" to count B. In addition, in S1905, the final registration determination unit 1109 sets count B to zero and then ends the process.

[0245] After S1904, in S1906, the main registration determination unit 1109 compares the value of count B with a threshold value of 50. If it is determined that the value of count B is equal to or greater than the threshold value of 50, the process proceeds to S1907. If it is determined that the value of count B is less than the threshold value of 50, the process ends. In S1907, the main registration determination unit 1109 adds 1 to the main registration count, and in S1908 sets count B to zero, and then ends the process.

[0246] Next, a specific example of the final registration determination when the final registration determination unit 1109 acquires the subject information shown in Fig. 20(B) will be described. The zoom position is set to zero. Since the registration status of subjects 1, 3, and 4 in Fig. 20(B) is not "provisional registration" in S1701 in Fig. 24(A), the processes from S1702 onward are not executed. Since the registration status of subject 2 in Fig. 20(B) is "provisional registration" in S1701 in Fig. 24(A), the processes from S1702 onward are executed.

[0247] In S1702 of FIG. 24A, the count A, count B, and main registration count up to the previous cycle are referenced, and if there are various counts with a person ID of 4, the information is handed over. As shown in FIG. 24B, the count A, count B, and main registration count with a person ID of 4 up to the previous cycle are 30, 40, and 70, respectively. The sum of the values ​​of count A and count B is the value of the main registration count. In S1703 of FIG. 24A, the first main registration count determination is executed. In S1801 of FIG. 25, the face size at the time of zoom wide is 110, so the process proceeds to S1802, and in S1802, the face reliability is 90, so the process proceeds to S1803. In S1803 of FIG. 25, since the face size at the time of zoom wide is 110, the count A is incremented by 11 (=110 / 10) to become 41 (=30+11). In S1805 of FIG. 25, since the value of count A is less than the threshold value of 50, the first main registration count determination process ends.

[0248] Next, in S1704 of FIG. 24A, a second formal registration count determination is executed. In S1901 of FIG. 26, the subject information is referenced, and it is found that the registration state of the simultaneously detected subject 1 is "formal registration". It is determined that a formal registration person is simultaneously detected, and the process proceeds to S1902. In S1902 of FIG. 26, the face sizes of the subject 1, who is a formal registration person, and the subject 2 are compared. Since the face size of the subject 1 is 120, if the face size is 120±10%, that is, 108 to 132, it is determined that the face sizes are close. Since the face size of the subject 2 is 110, it is determined that the face size is close to that of the formal registration person, and the process proceeds to S1903. Since the face reliability is 90 in S1903, the process proceeds to S1904.

[0249] In S1904 of FIG. 26, since the face size during zoom wide is 110, count B is incremented by 11 (=110 / 10) to become 51 (=40+11). In S1906 of FIG. 26, since count B is 50 or more, the process proceeds to S1907. In S1907, 1 is added to the value of the main registration count, 70, to become 71. In S1908, count B is set to zero, and the second main registration count determination process is terminated. Next, in S1705 of FIG. 24(A), since the value of the main registration count is less than the threshold value of 100, the process proceeds to S1707. The process of saving various counts is executed with count A of person ID 4 set to 41, count B set to 0, and main registration count set to 71.

[0250] A provisionally registered person who continues to satisfy the conditions that the distance from the image capture device is within a predetermined range or that the distance from a person who has already been determined to be a main person is close over multiple periods is determined to be a main person by the main registration judgment process. Based on this judgment result, the person information management unit 1106 can perform an update.

[0251] <Shooting subject determination> The shooting subject determination process shown in S507 of Fig. 21 will be described in detail with reference to Fig. 27. Fig. 27(A) is a flowchart explaining the process performed by the shooting subject determination unit 1110. This process is executed periodically, and a person to be shot is determined from among detected people. When the shooting subject determination unit 1110 acquires subject information from the subject detection unit 1107, it executes the processes of S1001 to S1008 and determines the subject to be shot. Based on the determination result, the panning drive angle, tilting drive angle, and zoom movement position are calculated in the processes of S1009 and S1010.

[0252] In S1001, the photographing subject determination unit 1110 refers to the subject information and determines whether a person with a priority setting of "Yes" has been detected. If the person is detected, the process proceeds to S1002, and if the person is not detected, the process proceeds to S1005.

[0253] In S1002, the photographing subject determination unit 1110 adds the person with the priority setting of "Yes" to the photographing subject people, and proceeds to S1003. In S1003, the photographing subject determination unit 1110 references the subject information and determines whether a person with a registration status of "Registered" has been detected. If the relevant person has been detected, the process proceeds to S1004, and if the relevant person has not been detected, the process proceeds to S1009. In S1004, the photographing subject determination unit 1110 adds the person with a registration status of "Registered" to the photographing subject people, and proceeds to S1009.

[0254] If a person with priority setting "Yes" has been detected, the processes of S1001 to S1004 determine that the person with priority setting "Yes" and the person with a registration status of "Formal Registration" are the people to be photographed. In S1005, the photographing subject determination unit 1110 refers to the subject information and determines whether a person with a registration status of "Formal Registration" has been detected. If the person is detected, the process proceeds to S1006, and if the person is not detected, the process proceeds to S1009. In S1006, the photographing subject determination unit 1110 adds the person with a registration status of "Formal Registration" to the people to be photographed, and proceeds to S1007.

[0255] In S1007, the photographing subject determination unit 1110 refers to the subject information and determines whether a person whose registration status is "provisionally registered" has been detected. If the relevant person has been detected, the process proceeds to S1008, and if the relevant person has not been detected, the process proceeds to S1009. In S1008, the photographing subject determination unit 1110 adds the person whose registration status is "provisionally registered" to the photographing subject people, and proceeds to S1009.

[0256] If a person with a priority setting of "Yes" is not detected, and a person with a registration status of "Full registration" is detected, the person to be photographed is determined by the processes of S1006 to S1008. That is, the person with a registration status of "Full registration" and the person with a registration status of "Provisional registration" are determined to be the person to be photographed.

[0257] In S1009, the shooting subject determination unit 1110 determines the number of people to be shot. If it is determined that there is one or more people to be shot, the process proceeds to S1010, and if it is determined that the number of people to be shot is zero, the process ends. In S1010, the shooting subject determination unit 1110 calculates a panning drive angle, a tilting drive angle, and a zoom movement position so that the shooting subject falls within the angle of view, and outputs the calculated values ​​to the drive control unit 1111.

[0258] 27B is a table showing an example of the importance of people according to the registration state of the subject information and the priority setting. The photographing priority is expressed by a number from 1 to 4, with 1 being the highest photographing priority and 4 being the lowest photographing priority. A person with a shooting priority of 1 is a person whose registration status is "Registered" and whose priority setting is "Yes." A person with a shooting priority of 2 is a person whose registration status is "Registered" and whose priority setting is "None." A person with a photography priority of 3 is a person whose registration status is "provisional registration." -People with a photography priority of 4 are unregistered people.

[0259] 27(A), when a person with a shooting priority level of 1 is detected, the shooting subject determination unit 1110 will select people with shooting priorities of 1 and 2 as shooting subjects, and will not select people with shooting priorities of 3 and 4 as shooting subjects. Also, when a person with a shooting priority level of 1 is not detected, and a person with a shooting priority level of 2 is detected, the shooting subject determination unit 1110 will select people with shooting priorities of 2 and 3 as shooting subjects, and will not select people with a shooting priority level of 4 as shooting subjects. Furthermore, when a person with a shooting priority level of 1 or 2 is not detected, the determination result is that no subjects will be selected as shooting subjects.

[0260] FIG. 28 is a diagram showing an example of image data and subject information. FIG. 28(A) is a schematic diagram showing an example of image data input to the subject detection unit 1107. FIG. 28(B) is a table showing an example of subject information extracted when the image data shown in FIG. 28(A) is input to the subject detection unit 1107. In the example of FIG. 28(B), the number of subjects is 6, and information on subject ID, face size, face position, face direction, face reliability, person ID, registration state, and priority setting for 6 subjects is shown. A specific example of the shooting subject determination when the shooting subject determination unit 1110 acquires the subject information shown in FIG. 28(B) will be described. Note that the zoom position is zero.

[0261] In S1001 of Fig. 27, the subject information in Fig. 28(B) is referenced, and since the priority setting for subject 2 is "Yes", the process proceeds to S1002, and subject 2 is added as a subject to be photographed. In S1003, the subject information in Fig. 28(B) is referenced, and since the registration status of subject 1 is "Fully Registered", the process proceeds to S1004, and subject 1 is added as a subject to be photographed.

[0262] In S1009 in Fig. 27, since the number of subjects to be photographed is 2, the process proceeds to S1010. In S1010, the panning drive angle, tilting drive angle, and zoom movement position are calculated so that subject 1 and subject 2 fit within the angle of view. Explanation of how to calculate the specific numerical values ​​of the angles and positions is omitted. There are methods such as specifying absolute values, and setting minimum values ​​for the specifiable drive angle and position and gradually changing them to the target angle and position over multiple cycles.

[0263] Fig. 29 is a schematic diagram showing an example of image data resulting from the drive control unit 1111 controlling each drive unit according to input of the calculated panning drive angle, tilting drive angle, and zoom movement position. In the example of Fig. 29, the panning drive, tilting drive, and zoom position movement are controlled so that the centers of gravity of the face positions of subject 1 on the right side and subject 2 on the left side are positioned in the center of the screen, and the face size of each subject falls within the range of 150 to 200.

[0264] The above control allows for shooting in such a way that subjects 1 and 2, which are the subjects of shooting and are determined to have a high shooting priority, are included within the angle of view, while subjects 3 to 6, which are not the subjects of shooting and are determined to have a low shooting priority, are not included in the angle of view. When a person with a certain shooting priority or higher is detected, a process is executed in which a person with a close shooting priority is set as the shooting subject, and a person with a shooting priority farther away from the main person is not set as the shooting subject. As a result, shooting can be performed in which the main person is set as the shooting subject, while people with a low relationship are excluded from the shooting subject as much as possible.

[0265] Next, an embodiment in which an importance determination unit 1514 is added will be described with reference to Fig. 17 and Fig. 30 to Fig. 34. In this embodiment, the person information for determining the shooting priority is further subdivided, and the importance of each person is increased or decreased according to the detection interval, thereby improving the accuracy of distinguishing main people.

[0266] The details of the process by the control box 1100 will be described with reference to Fig. 17, focusing mainly on the differences from the above embodiment. The person information management unit 1106 stores and manages person information linked to each person. The person information will be described below with reference to Fig. 30.

[0267] Fig. 30 is a table showing an example of personal information including importance. Items other than importance are the same as in the previous example, so their explanation will be omitted. Importance is set to a 10-level numerical value from 1 to 10, with 1 being the lowest importance and 10 being the highest importance. The lower limit of importance is "0" when the name is blank, and "5" when a name is entered.

[0268] When the personal information management unit 1106 acquires a face image and feature information from the feature information extraction unit 1105, it issues a new personal ID, associates the personal ID with the input face image and feature information, and adds new personal information. When new personal information is added, the initial value of the registration status is "provisional registration", the importance is "0" (not set), the initial value of the priority setting is "none", and the initial value of the name is blank. When the personal information management unit 1106 acquires a final registration determination result (person ID to be final registration) from the final registration determination unit 1109, it changes the registration status of the personal information corresponding to the person ID of the relevant person to "full registration" and sets the importance to "1". In addition, when an instruction to change the personal information (priority setting information or name) is input from the communication unit 1114 by a user operation, the personal information management unit 1106 changes the personal information according to the instruction. In addition, if there is a change in either the priority setting or the name of a person whose registration status is "provisional registration", the person information management unit 1106 changes the registration status of the person to "full registration", and if there is a name change, sets the importance level to "5".

[0269] When the personal information management unit 1106 receives an instruction to add or subtract importance to a personal ID from the importance determination unit 1514, it adds or subtracts importance to the personal information corresponding to the personal ID of the relevant person. The subject detection unit 1107 detects subjects from the digital image data from the image processing unit 1103, and extracts information on the detected subjects. For example, an example in which the subject detection unit 1107 detects a person's face as a subject will be described. The subject information includes, for example, the number of detected subjects, the face position, the face size, the face direction, and a face reliability indicating the likelihood of detection. An example of the subject information will be described later with reference to FIG. 31.

[0270] The subject detection unit 1107 compares the characteristic information of each person acquired from the person information management unit 1106 with the characteristic information of the detected subject to calculate the similarity. If the similarity is equal to or greater than a threshold, the subject detection unit 1107 adds the person ID, registration status, importance, and priority setting of the detected person to the subject information. The subject detection unit 1107 outputs the subject information to the provisional registration determination unit 1108, the main registration determination unit 1109, the photographing subject determination unit 1110, and the importance determination unit 1514.

[0271] The shooting subject determination unit 1110 determines a subject to be shot based on the subject information acquired from the subject detection unit 1107. The shooting subject determination unit 1110 further calculates a panning drive angle, a tilting drive angle, and a target zoom position for fitting the person to be shot within the angle of view at a specified size based on the determination result of the person to be shot. A command based on the calculation result is output to the drive control unit 1111. Details of the shooting subject determination process will be described later with reference to FIG. 34.

[0272] FIG. 31 is a diagram showing an example of image data and subject information. FIG. 31(A) is a schematic diagram showing an example of image data input to subject detection unit 1107. FIG. 31(B) is a table showing an example of subject information extracted when the image data shown in FIG. 31(A) is input to subject detection unit 1107. An example is shown in which subject information is composed of the number of subjects, subject ID of each subject, face size, face position, face direction, face reliability, person ID, registration status, importance, and priority setting. Items other than importance are the same as in the above example, so their description will be omitted.

[0273] The importance is the same as the importance managed by the person information management unit 1106. If the person ID is not zero, that is, if it is determined to be one of the people managed by the person information management unit 1106, the importance of the corresponding person obtained from the person information management unit 1106 is obtained.

[0274] 32 is a flowchart showing the overall flow of photographing and registering and updating personal information in this embodiment, and the following processing is executed as a periodic process. When the power of the imaging device is turned on, the imaging unit 1022 starts periodic photographing (video shooting) to obtain image data to be used for various judgments. The various judgments are photographic subject judgment, provisional registration judgment, main registration judgment, and importance judgment. The repetitive processing starts at S2800.

[0275] In S2801, image data acquired by shooting is output to the image processing unit 1103, and image data that has been subjected to various image processes is acquired. When a subject is detected in S2802 and subject information is acquired, a final registration determination is made in S2803, an importance determination is made in S2804, and a provisional registration determination is made in S2805. A description of the provisional registration determination process and the final registration determination process is omitted. In S2804, the importance determination unit 1514 determines the importance of the person using the information of the detected subject. In the importance determination, the person information in the person information management unit 1106 is updated, but panning drive, tilting drive, and zoom drive are not executed.

[0276] S2806 is a process for determining whether or not the composition adjustment process for temporary registration is being executed. If it is determined that the composition adjustment process for temporary registration is being executed, the process proceeds to S2807, and if it is determined that the composition adjustment process for temporary registration is not being executed, the process proceeds to S2808. In S2807, the feature information extraction unit 1105 extracts feature information of the subject located at the center of the image data, and outputs it to the person information management unit 1106. Also, in S2807, a subject to be photographed is determined.

[0277] After S2807 and S2808, the process proceeds to S2809, where it is determined whether the repetitive process is completed. If the process is to be continued, the process returns to S2800. The processes of S2801 to S2808 are repeatedly executed in accordance with the imaging cycle of the imaging unit 1022.

[0278] Next, the importance determination process shown in S2804 of FIG. 32 will be described with reference to FIG. 33. FIG. 33(A) is a flowchart explaining the process performed by the importance determination unit 1514. The importance determination process is executed over multiple cycles, and the importance of people who have already been registered is determined and updated. FIG. 33(B) is a table showing the last detection date and time and the last update date and time linked to the person ID. The last detection date and time is the date and time when the person who was registered was last detected. The last update date and time is the date and time when the importance of the person who was registered was last updated. Data on the last detection date and time and the last update date and time for the number of people who are registered is stored in memory, and is referred to when making a determination for each cycle.

[0279] When the importance determination unit 1514 acquires the subject information from the subject detection unit 1107, it executes the process of S2901, then executes the processes of S2902 to S2906 for the detected subject, and executes the processes of S2907 to S2909 for the person to be officially registered. In S2901, the importance determination unit 1514 acquires the current date and time from the system time of the camera 101. Then, a repetitive process corresponding to the number of detected subjects is started in the STA. In S2902, the importance determination unit 1514 refers to the subject information and determines whether the registration status is "official registration". If it is determined to be "official registration", the process proceeds to S2903, and if it is determined to be other than "official registration", the process proceeds to the STB.

[0280] In S2903, the importance determination unit 1514 sets the current date and time as the last detection date and time for the detected person. In S2904, the importance determination unit 1514 determines whether the current date and time is more than 30 minutes and less than 24 hours after the last update date and time. If this condition is met, the process proceeds to S2905, and if the condition is not met, the process proceeds to STB.

[0281] In S2905, the importance determination unit 1514 instructs the person information management unit 1106 to add 1 to the importance, and in S2906, the current date and time is set to the last update date and time. Then, the STB determines whether the repetitive process has ended, and if the process is to be continued, the process returns to the STA and proceeds to processing of the next subject.

[0282] Next, the following process is executed for each person who is registered. In STC, a repetitive process corresponding to the number of registered subjects is started. In S2907, the importance determination unit 1514 refers to the current date and time and determines whether or not the last detection date and time and the last update date and time are both one week or more apart. If it is determined that the person has not been detected or updated for one week or more, the process proceeds to S2908, and if it is determined that the person has been detected or updated within one week, the process proceeds to STD.

[0283] In S2908, the importance determination unit 1514 instructs the person information management unit 1106 to subtract 1 from the importance, and in S2906, the current date and time is set as the last update date and time. Then, in STD, it is determined whether the repetitive process is finished, and if the process is to be continued, the process returns to STC and proceeds to the process for the next registered subject.

[0284] The importance determination process increases the importance of people who are re-detected within a day, and decreases the importance of subjects that are not detected for more than a week. In other words, it is possible to increase the importance of key people who appear frequently, and decrease the importance of unrelated people who are rarely seen or who have been registered.

[0285] The photographing subject determination process shown in S2808 in Fig. 32 will be described with reference to Fig. 34. Fig. 34(A) is a flowchart explaining the process performed by the photographing subject determination unit 1110. This process is executed periodically, and a person to be photographed is determined from among detected people. Fig. 34(B) is a table (photographing priority table) showing the photographing priority of people according to the registration state, importance, and priority setting of the subject information. The photographing priority is expressed by a number from 1 to 13, with 1 being the highest photographing priority and 13 being the lowest photographing priority. A person with a shooting priority of 1 is a person whose registration status is "Registered" and whose priority setting is "Yes." People with a shooting priority of 2 to 11 are people whose registration status is "Registered" and whose priority setting is "None". The higher the importance, the higher the shooting priority. A person with a photography priority of 12 is a person whose registration status is "provisionally registered." -People with a photography priority of 13 are unregistered people.

[0286] When the shooting object determination unit 1110 acquires object information from the object detection unit 1107, it executes the processes of S3001 to S3004 to determine an object to be shot. Based on the determination result, the processes of S3005 and S3006 are performed to calculate a panning drive angle, a tilting drive angle, and a zoom movement position.

[0287] In S3001, the shooting subject determination unit 1110 refers to the subject information and the shooting priority table shown in FIG. 34(B) to obtain the shooting priority of each subject. In S3002, the shooting subject determination unit 1110 determines whether the shooting priority of the subject with the highest shooting priority among all detected subjects is equal to or less than a threshold value of 10. If this condition is met, the process proceeds to STE, and if the condition is not met, it is determined that there is no shooting subject and the process ends. In STE, a repetitive process corresponding to the number of detected subjects is started. In S3003, the shooting subject determination unit 1110 determines whether the shooting priority of each subject is less than the value obtained by adding 2 to the highest shooting priority among all subjects. If this condition is met, the process proceeds to S3004, and if the condition is not met, the process proceeds to STF. If the end of the repetitive process is determined in STF, and the process is to be continued, the process returns to STE and proceeds to the process of the next detected subject.

[0288] In S3004, the shooting subject determination unit 1110 adds the determined detected subject as a shooting subject. For example, if the shooting priority of the subject with the highest shooting priority is "4", then subjects with shooting priorities of "4", "5", and "6" are determined as shooting subjects. If the shooting priority of the subject with the highest shooting priority is "7", then subjects with shooting priorities of "7", "8", and "9" are determined as shooting subjects. After S3004, the process proceeds to STF, where it is determined whether the repetitive process has ended. If the process is to continue, the process returns to STE, and the process proceeds to the process of the next detected subject. When the repetitive process ends, the process proceeds to S3005.

[0289] In S3005, the shooting subject determination unit 1110 determines whether there is one or more people to be shot. If this condition is met, the process proceeds to S3006, and if the condition is not met, the process ends. In S3006, the shooting subject determination unit 1110 calculates the panning drive angle, tilting drive angle, and zoom movement position so that the shooting subject falls within the angle of view, and outputs them to the drive control unit 1111. After that, the series of processes ends.

[0290] The above control allows for shooting in such a way that subjects that are the subject of shooting, i.e., subjects that are determined to have a high shooting priority, are included within the angle of view, while subjects that are not the subject of shooting, i.e., subjects that are determined to have a low shooting priority, are not included in the angle of view. When a person with a relatively high shooting priority is detected, multiple people with close shooting priorities are determined to be shooting subjects, and people with distant shooting priorities are not determined to be shooting subjects. Shooting can be performed in which main people are the shooting subjects, while people with low relationships are excluded from the shooting subjects as much as possible.

[0291] (Modification) A modification of the above embodiment will be described below. In the above embodiment, the subject information is information related to the facial features of a person. In the modification, the subject information can be information related to features of subjects other than a person, such as animals or objects.

[0292] Fig. 35 shows an example in which face information of animals can be detected in addition to people. Fig. 35(A) is a schematic diagram showing an example of image data input to the subject detection unit 1107. Fig. 35(B) is a table showing subject information corresponding to the image data of Fig. 35(A). When photographing an animal or object, provisional registration determination and final registration determination are performed as separate processes from those for people. Alternatively, when animals or objects are mixed with people, a process is performed in which the importance of the subject is weighted according to the type of subject to determine the subject to be photographed.

[0293] In the above embodiment, the lens barrel 102 including the imaging unit 1022 rotates around both the X-axis and the Y-axis, enabling panning and tilting. The present invention is applicable even if the lens barrel 102 is not rotatable around both the X-axis and the Y-axis, as long as the lens barrel 102 is rotatable around one of the axes. For example, in the case of a configuration that allows rotation around the Y-axis, panning is performed based on the orientation of the subject.

[0294] In the above embodiment, an imaging device has been described in which a lens barrel having an imaging optical system and an imaging element and an imaging control device that controls the imaging direction by the lens barrel are integrated. The present invention is not limited to this. For example, the imaging device may have a configuration in which the lens device is replaceable. Also, there is a configuration in which the imaging device is attached to a camera platform having a rotation mechanism that drives in a panning direction and a tilting direction. The imaging device may have an imaging function and other functions. For example, there is a configuration in which a camera platform that can fix a smartphone having an imaging function is combined with a smartphone. Also, the lens barrel and its rotation mechanism (tilt rotation unit and pan rotation unit) do not need to be physically connected to the control box. For example, the rotation mechanism and zoom function may be controlled via wireless communication such as Wi-Fi (registered trademark).

[0295] Although the embodiment has been described in which the image capturing device captures the characteristic information of a person, the present invention is not limited to this, and may be configured to capture a face image and characteristic information in the person information from, for example, another image capturing device for face registration or an external device such as a mobile terminal device, and register or add the face image and characteristic information. Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0296] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0297] 101: Camera, 301, 501: External device

Claims

1. An imaging device capable of automatic photography and automatic authentication registration, An imaging means for imaging a subject; a search means for searching for a subject detected from image data acquired by the imaging means; An authentication and registration means for automatically authenticating and registering the detected subject; a control means for performing an authentication registration determination by the authentication registration means as to whether a first condition for performing the automatic authentication registration is satisfied, and a photographing determination as to whether a second condition for performing the automatic photographing is satisfied, and controlling a timing of the automatic photographing and the automatic authentication registration; the control means controls the search by the search means, and executes the authentication registration determination and the photographing determination for the detected subject, thereby determining a timing for the automatic authentication registration; If the first condition is satisfied as a result of the authentication / registration determination and the photography determination, the authentication / registration means performs automatic authentication / registration of the detected subject, and if the first condition is not satisfied and the second condition is satisfied, the control means controls the automatic photography.

1. An imaging device comprising:

2. An imaging device capable of automatic photography and automatic authentication registration, An imaging means for imaging a subject; a search means for searching for a subject detected from image data acquired by the imaging means; An authentication and registration means for automatically authenticating and registering the detected subject; a control means for performing an authentication registration determination by the authentication registration means as to whether a first condition for performing the automatic authentication registration is satisfied, and a photographing determination as to whether a second condition for performing the automatic photographing is satisfied, and controlling a timing of the automatic photographing and the automatic authentication registration; the control means controls the search by the search means, and executes the authentication registration determination and the photographing determination for the detected subject, thereby determining a timing for the automatic authentication registration; The control means performs control to prioritize the result of the authentication / registration determination over the result of the photography determination when the frequency of photography by the automatic photography is a first frequency, but prioritizes the result of the photography determination over the result of the authentication / registration determination when the frequency of photography by the automatic photography is a second frequency lower than the first frequency.

1. An imaging device comprising:

3. A first change means for changing the photographing direction; a second change means for changing the photographing angle of view, The control means controls the timing of changing the photographing direction or the photographing angle of view by the first or second changing means in the automatic photographing and automatic authentication and registration.

3. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

4. the first change means includes a drive means for rotating the imaging means in a plurality of directions, The second change means changes the angle of view of the automatic photography by driving a lens or by image processing.

4. The imaging device according to claim 3.

5. When it is determined that the first condition is satisfied, the control means performs control by the first change means to position the face of the subject at the center of the photographing angle of view.

5. The imaging device according to claim 3 or 4.

6. When it is determined that the first condition is satisfied, the control means controls the second change means to change the size of the face of the subject to a preset size.

5. The imaging device according to claim 3 or 4.

7. When the control means determines that the second condition is satisfied and that the detected subject is a person, the control means performs control to change the angle of view to a shooting angle of view such that the subject is included within the shooting angle of view by the second change means.

5. The imaging device according to claim 3 or 4.

8. The control means changes the priority of the authentication registration determination based on the number of times photographing is performed within a predetermined time period or on an interval between photographing by the automatic photographing.

3. The imaging device according to claim 2.

9. The control means performs control to interrupt the automatic authentication and registration process when it is determined that the first condition is satisfied and an instruction to take a photograph is issued from an external device.

9. The imaging device according to claim 1, wherein the first and second lenses are arranged in a first direction.

10. The first condition is that face information of a subject is acquired, and the reliability of face detection is higher than a threshold value, or the reliability continues to be higher than the threshold value, or the face of the subject faces forward relative to an imaging device.

10. The imaging device according to claim 1,

11. The control means determines the second condition based on an evaluation value of the automatic photography calculated from a state of a subject and a threshold value thereof.

11. The imaging device according to claim 1,

12. When it is determined that the first condition is satisfied, the control means performs control to adjust the photographing angle of view by the second change means before the automatic authentication and registration.

5. The imaging device according to claim 3 or 4.

13. An acquisition means for acquiring information calculated or changed by machine learning of the image data, The control means performs registration determination of the subject or photography determination based on the second condition using the information acquired by the acquisition means.

13. The imaging device according to claim 1, wherein the imaging device is a single lens.

14. The control means uses the information acquired by the acquisition means to determine whether a condition for transitioning to a low power consumption state or a condition for canceling the low power consumption state is satisfied, and controls the power supply based on the result of the determination.

14. The imaging device according to claim 13.

15. In the automatic photography, the control means acquires information on the distance of the subject and the frequency of detection, determines the priority of each subject for photography, and determines, from among the multiple detected subjects, a subject having a priority within a preset range as a subject to be photographed.

15. The imaging device according to claim 1,

16. The control means determines, as subjects to be photographed, a first subject having a first priority and a second subject having a second priority within a preset range from the first priority.

16. The imaging device according to claim 15.

17. The control means controls the automatic photography without including, as a photography target, a subject having a lower priority than the second priority.

17. The imaging device according to claim 16.

18. The control means determines the priority of photographing each of the first and second subjects using information on the distance from the imaging device to the first and second subjects.

18. The imaging device according to claim 16 or 17.

19. The control means performs a process of storing and managing the characteristic information of the subject in a storage means, and judges whether the characteristic information of the detected subject matches the characteristic information stored in the storage means.

19. The imaging device according to claim 15,

20. The storage means stores the feature information of the subject in association with the priority.

20. The imaging device according to claim 19.

21. When a subject corresponding to the feature information stored in the storage means is detected, the control means performs a process of updating the priority stored in the storage means based on the priority of the detected subject.

21. The imaging device according to claim 20.

22. When the feature information of the detected subject is acquired, the control means performs a process of storing the feature information of the subject whose priority is within a preset value or range in the storage means.

21. The imaging device according to claim 19 or 20.

23. The control means determines the priority of the detected object based on the time elapsed since the last detection date and time of the detected object.

23. The imaging device according to claim 15,

24. A change means for changing the photographing direction is provided, The control means controls the change means to control the automatic photography of the subject of the photography target determined.

24. The imaging device according to claim 15,

25. A changing means for changing a photographing angle of view is provided, The control means controls the change means to control the automatic photography in a state in which the determined subject of the photography target is included within a photography angle of view.

24. The imaging device according to claim 15,

26. The control means determines the priority of the subject using information on the orientation of the face of the subject or a reliability indicating the certainty of the face.

26. The imaging device according to claim 25.

27. The control means controls output of image data of the face of the subject and the priority.

27. The imaging device according to claim 26.

28. A control method executed in an imaging device capable of automatic photography and automatic authentication registration, comprising: a searching step of searching for a subject detected from image data acquired by an imaging means; an authentication and registration process for automatically authenticating and registering the detected subject; a control step of determining whether a first condition for performing the automatic authentication and registration is satisfied or not, and determining whether a second condition for performing the automatic photography is satisfied or not, and controlling timing of the automatic photography and the automatic authentication and registration; In the control step, a process of determining a timing of the automatic authentication and registration is performed by executing the authentication and registration judgment and the photographing judgment for the detected subject while controlling the search for the subject, and if the first condition is satisfied as a result of the authentication and registration judgment and the photographing judgment, the automatic authentication and registration of the detected subject is performed in the authentication and registration step, and if the first condition is not satisfied and the second condition is satisfied, the automatic photographing is controlled in the control step.

23. A method for controlling an imaging apparatus comprising the steps of:

29. A control method executed in an imaging device capable of automatic photography and automatic authentication registration, comprising: a searching step of searching for a subject detected from image data acquired by an imaging means; an authentication and registration process for automatically authenticating and registering the detected subject; a control step of determining whether a first condition for performing the automatic authentication and registration is satisfied or not, and determining whether a second condition for performing the automatic photography is satisfied or not, and controlling timing of the automatic photography and the automatic authentication and registration; In the control step, a process for determining the timing of the automatic authentication and registration is performed by executing the authentication and registration determination and the photographing determination for the detected subject while controlling the search for the subject, and when the photographing frequency by the automatic photographing is a first frequency, the result of the authentication and registration determination is prioritized over the result of the photographing determination, but when the photographing frequency by the automatic photographing is a second frequency lower than the first frequency, the result of the photographing determination is prioritized over the result of the authentication and registration determination.

23. A method for controlling an imaging apparatus comprising the steps of:

30. A method for causing a computer to execute each step according to claim 28 or claim 29. A program characterized by:

Images