Imaging device, control method, and program

Abstract

To facilitate for a user to change a subject that is imaged by an imaging device.SOLUTION: An imaging device includes: imaging means that images a subject; search means that searches for a subject to be imaged by the imaging means; and reception means that receives a change instruction for changing an imaging target without specifying an imaging target to which a change is to be made. When receiving the change instruction, the search means searches for another subject.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an imaging device that automatically captures an image of a subject. [Background technology]

[0002] In recent years, life log cameras that periodically capture images and imaging devices that automatically capture images of the surroundings have been proposed. Such imaging devices aim to capture images of scenes desired by the user without the user being aware of the action by automatically capturing images of the subject. Patent Document 1 discloses an imaging device that can automatically perform pan driving and tilt driving to capture images of a subject. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-117375 Summary of the Invention [Problem to be solved by the invention]

[0004] However, because the imaging device described in Patent Document 1 automatically performs panning and tilting, it does not always capture an image of a subject that matches the user's intention. Therefore, an object of the present invention is to realize automatic imaging that is more in line with the user's intention. [Means for solving the problem]

[0005] In order to achieve the above object, an imaging apparatus of the present invention includes: an imaging unit that captures an image of a subject; a searching unit that searches for a subject to be imaged by the imaging unit; and a receiving unit that receives a change instruction to change the imaging target without specifying a new imaging target. a determining means for determining, in the search started in response to receiving the change instruction, a subject other than the subject that was the subject to be imaged when the change instruction was received, as the subject to be imaged; and the determination means, in the search started in response to receiving the change instruction, calculates the importance of each subject based on information about the face of the detected subject, and determines an imaging target based on the importance; the determination means, in the search started in response to receiving the change instruction, lowers the importance of the subject that was being imaged when the change instruction was received compared to other subjects, thereby determining the other subjects as imaging targets with priority; When the change instruction is received, the search means searches for another subject. [Effects of the Invention]

[0006] According to the present invention, automatic imaging that is more in line with the user's intentions can be realized. [Brief explanation of the drawings]

[0007] [Figure 1] 1A is a diagram showing an example of the appearance of an imaging device according to a first embodiment, and FIG. 1B is a diagram explaining the operation of the imaging device according to the first embodiment. [Figure 2] FIG. 1 is a diagram illustrating a configuration of an imaging apparatus according to a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating a configuration of an imaging device and an external device according to a first embodiment. [Figure 4] FIG. 2 is a diagram illustrating a configuration of an external device according to the first embodiment. [Figure 5] 4 is a flowchart showing automatic photography processing in the first embodiment. [Figure 6] FIG. 2 is a diagram for explaining area division within a captured image in the first embodiment. [Figure 7] 4 is a flowchart showing a voice recognition process in the first embodiment. [Figure 8] 10 is a flowchart showing a subject change process in the first embodiment. [Figure 9] 10 is a flowchart showing a process for determining an initial position of a subject search process in the first embodiment. [Figure 10] 10 is a flowchart showing an imaging process after changing a subject in the first embodiment. [Figure 11] 10 is a flowchart showing a subject change process in the second embodiment. [Figure 12] 13 is a flowchart showing a subject change process in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0009] The embodiment described below is an example of a means for realizing the present invention, and may be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. Furthermore, each embodiment may be appropriately combined.

[0010] [First embodiment] <Configuration of imaging device> FIG. 1 is a diagram showing the configuration of an imaging apparatus according to the first embodiment.

[0011] 1A is provided with operation members including a power switch that can be used to turn the power on and off, etc. The operation members also include a touch panel.

[0012] The lens barrel 102 is a housing containing an optical lens group and an imaging element. The lens barrel 102 is attached to the imaging device 101. The tilt rotation unit 104 and pan rotation unit 105 are rotation mechanisms that can rotate the lens barrel 102 relative to the fixed part 103. The tilt rotation unit 104 is, for example, a motor that can rotate the lens barrel 102 in the pitch direction shown in FIG. 1(b). The pan rotation unit 105 is, for example, a motor that can rotate the lens barrel 102 in the yaw direction shown in FIG. 1(b). The tilt rotation unit 104 and the pan rotation unit 105 can rotate the lens barrel 102 in one or more axial directions. In this embodiment, the Y axis shown in FIG. 1(b) is the rotation axis of the pan rotation unit 105. In this embodiment, the positive direction of the Z axis shown in FIG. 1(b) is the front direction of the imaging device 101.

[0013] The angular velocity meter 106 and the accelerometer 107 are, for example, a gyro sensor and an acceleration sensor, respectively, and are disposed on a fixed portion 103 of the imaging device 101. The imaging device 101 detects vibrations of the imaging device 101 based on the respective velocities measured by the angular velocity meter 106 and the accelerometer 107. The imaging device 101 can generate an image in which the shaking and tilt of the lens barrel 102 have been corrected by rotationally driving the tilt rotation unit 104 and the pan rotation unit 105 based on the detected vibrations of the imaging device 101.

[0014] FIG. 2 is a block diagram showing the configuration of the image capturing apparatus 101 of this embodiment.

[0015] The first control unit 223 includes a processor (e.g., a CPU, a GPU, a microprocessor, an MPU, etc.), a memory (e.g., a DRAM, an SRAM, etc.), etc. The first control unit 223 executes various processes to control each block of the imaging device 101 and controls data transfer between each block. The first control unit 223 is an example of a control means and a determination means.

[0016] The nonvolatile memory 216 is a memory that can record and erase data, and stores constants, programs, etc. for the operation of the first control unit 223.

[0017] The zoom unit 201 is a group of optical lenses that constitute a zoom lens that changes the zoom magnification. The zoom drive control unit 202 is a control unit that drives and controls the optical lenses of the zoom unit 201. The focus unit 203 is a group of optical lenses that adjusts focus. The focus drive control unit 204 drives and controls the optical lenses of the focus unit 203. The imaging unit 206 receives light incident on an imaging element through each optical lens group, and outputs charge information corresponding to the amount of light to the image processing unit 207 as analog image data. The zoom unit 201, zoom drive control unit 202, focus unit 203, focus drive control unit 204, and imaging unit 206 are all included in the lens barrel 102.

[0018] The image processing unit 207 performs image processing such as distortion correction, white balance adjustment, and color interpolation on the image data input from the imaging unit 206, and outputs digital image data. The digital image data output from the image processing unit 207 is converted by the image recording unit 208 into an image file format such as JPEG format or a moving image file format such as MPEG format. The converted digital image data is sent to the memory 215 or a video output unit 217, which will be described later. When recording the digital image data recorded in the memory 215, the first control unit 223 outputs the digital image data to the recording / playback unit 220.

[0019] The lens barrel rotation drive section 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction. Note that the lens barrel rotation drive section 205 is an example of a drive means.

[0020] The device shaking detection unit 209 is equipped with, for example, an angular velocity meter 106 that detects the angular velocity in three axial directions of the imaging device 101, and an accelerometer 107 that detects the acceleration in three axial directions of the device. The device shaking detection unit 209 calculates the rotation angle of the device, the amount of shift of the device, etc., based on the signals detected by the angular velocity meter 106 and the accelerometer 107.

[0021] The audio input unit 213 has a plurality of microphones. The audio input unit 213 also A / D converts audio signals input from the microphones and outputs the converted audio signals to the audio processing unit 214.

[0022] The audio processing unit 214 can detect the direction of sound on the plane on which the multiple microphones are installed. The detected direction of sound can be used for search and automatic imaging, which will be described later. Furthermore, the audio processing unit 214 can detect specific audio commands. In this embodiment, the specific audio commands include two types: trigger words and command words. The trigger word is a command that triggers the start of command word recognition. For example, the trigger word is a command consisting of a specific keyword, such as "OK, camera," spoken by the user. The command word is a command that instructs the imaging device 101 to perform a specific process. For example, the specific process includes still image capture processing, video capture start processing, video capture end processing, sleep processing, subject change processing, and automatic imaging processing. Note that the command word is a command consisting of a different keyword for each specific process, such as "take a still image" for still image capture processing and "take a video" for video capture start processing. These audio commands are pre-recorded in the memory 215 of the imaging device 101. The imaging device 101 may be configured so that, in addition to the pre-recorded voice commands, a voice command for executing any process desired by the user can be registered.

[0023] Furthermore, the audio processing unit 214 performs audio-related processing such as optimization processing and encoding on the input audio signal. The audio signal processed by the audio processing unit 214 is then transmitted to the memory 215 by the first control unit 223. The memory 215 temporarily records the data input from the image recording unit 208 and the audio signal input from the audio processing unit 214. When recording this audio signal, the first control unit 223 outputs the audio signal from the memory 215 to the recording / playback unit 220.

[0024] The recording / playback unit 220 records image data, audio signals, and other control data related to imaging on a recording medium 221. The recording medium 221 may be a recording medium built into the imaging device 101 or a removable recording medium. The recording medium 221 can record various types of data such as image data and audio signals. In this embodiment, the recording medium 221 is a medium with a larger capacity than the nonvolatile memory 216. For example, the recording medium 221 is a recording medium such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a nonvolatile semiconductor memory, or a flash memory.

[0025] Furthermore, the recording and playback unit 220 can read (play back) image data, audio signals, various data, and programs recorded on the recording medium 221. When playing back image data and audio signals recorded on the recording medium 221, the first control unit 223 operates as follows. The first control unit 223 outputs the image data and audio signals read by the recording and playback unit 220 to the image processing unit 207 and audio processing unit 214, respectively. The image processing unit 207 and audio processing unit 214 decode the image data and audio signals, respectively. The image processing unit 207 and audio processing unit 214 output the decoded signals to the video output unit 217 and audio output unit 218, respectively.

[0026] The second control unit 211 controls the power supply to the first control unit 223. For example, the second control unit 211 is made up of a processor (e.g., a CPU, a microprocessor, an MPU, etc.), a memory (e.g., a DRAM, an SRAM, etc.), etc. In this embodiment, the second control unit 211 is provided separately from the first control unit 223 that controls the entire main system of the imaging device 101.

[0027] The first power supply unit 210 and the second power supply unit 212 supply power to operate the first control unit 223 and the second control unit 211, respectively. In this embodiment, the power supplied by the first power supply unit 210 is greater than the power supplied by the second power supply unit 212. In this embodiment, the first power supply unit 210 and the second power supply unit 212 are selected according to the amount of power to be supplied. For example, the first power supply unit 210 is a switch for supplying power to the first control unit 223, and the second power supply unit 212 is a lithium battery or an alkaline battery. When a power switch provided in the imaging device 101 is pressed, power is first supplied to the second control unit 211, and then to the first control unit 223. In addition, the first control unit 223 can control the first power supply unit 210 to turn off the power supply to the first control unit 223. Even while power is not being supplied to the first control unit 223, the second control unit 211 continues to operate and acquires information from the device vibration detection unit 209 and the audio processing unit 214. Based on such input information, the second control unit performs a process of determining whether to activate the first control unit 223. When determining to activate the first control unit 223, the second control unit 211 controls the first power supply unit 210 to supply power to the first control unit 223. In this embodiment, the first power supply unit 210 and the second power supply unit 212 supply power from batteries. In other words, the imaging device 101 is also a mobile terminal.

[0028] For example, when capturing an image, the audio output unit 218 outputs an audio signal such as an electronic shutter sound from a speaker built into the image capturing device 101. For example, when capturing an image, the LED control unit 224 controls an LED provided in the image capturing device 101 to light up or blink in a preset pattern.

[0029] Video output unit 217 is, for example, a video output terminal, and outputs an image signal for displaying a video on a connected external display, etc. Audio output unit 218 and video output unit 217 may be integrated into a single terminal, for example, an interface such as an HDMI (registered trademark) (High-Definition Multimedia Interface (registered trademark)) terminal.

[0030] The communication unit 222 is an interface for communication between the imaging device 101 and an external device. The communication unit 222 transmits and receives data such as audio signals and image data. When the communication unit 222 receives a control signal related to imaging, such as imaging start, imaging end, pan drive, tilt drive, or zoom drive, the first control unit 223 drives the imaging device 101 in accordance with the control signal. The communication unit 222 has a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a Wireless USB, or a GPS receiver.

[0031] The subject detection unit 225 reads the image data output from the image processing unit 207 from the memory 215 and performs subject recognition such as people and objects. For example, when the subject detection unit 225 recognizes a person, it detects the face of the subject. Patterns for determining the face of the subject are registered in advance in the image capture device 101. Note that these patterns are assigned identifiers for distinguishing each subject. In subject face detection processing, the subject detection unit 225 detects the face of the subject by detecting a portion that matches the pattern for determining the face of the subject, which is included in the captured image. Furthermore, the subject detection unit 225 can distinguish between multiple registered people.

[0032] The subject detection unit 225 also simultaneously calculates a reliability indicating the likelihood of the detected subject's face. The reliability is calculated, for example, from the size of the face area in the image, the degree of match with a face pattern, etc. The subject detection unit 225 also performs pattern matching on the subject's face in the image to detect facial information such as whether the detected face is smiling, whether the eyes are open, and the facial orientation. Note that the method for detecting facial information is not limited to pattern matching, and well-known techniques such as methods using deep learning can also be used. Note that the subject detection unit 225 is an example of a detection means.

[0033] In the object recognition process, the subject detection unit 225 can recognize an object by determining whether it matches a pre-registered pattern. In addition, the subject detection unit 225 can recognize an object by extracting the feature amount of the subject using a histogram of the hue, saturation, etc. in the captured image.

[0034] In the above-described manner, the first control unit 223 can detect the subject by the subject detection unit 225 from the captured image data.

[0035] <System configuration with external devices> 3 is a diagram showing an example of the configuration of a wireless communication system between an imaging device 101 and a smart device 301. The imaging device 101 is, for example, a digital camera. The smart device 301 is, for example, a smartphone including a Bluetooth communication module and a wireless LAN communication module.

[0036] In this embodiment, the imaging device 101 and the smart device 301 can communicate via two communication paths. One is a communication path 302 based on a wireless LAN conforming to the IEEE 802.11 standard series, for example. The other is a communication path 303 based on a master-slave relationship, such as a control station and a slave station, such as Bluetooth Low Energy (hereinafter referred to as "BLE"). Note that wireless LAN and BLE are examples of communication methods. Note that other communication methods may be used as long as each communication device has two or more communication functions and, for example, one communication function that communicates in a control station-slave relationship can control the other communication function. However, without loss of generality, it is assumed that the first communication, such as wireless LAN, is capable of faster communication than the second communication, such as BLE, and that the second communication has at least one of lower power consumption and a shorter communication distance than the first communication.

[0037] <External device configuration> The configuration of a smart device 301, which is an example of an external device, will be described with reference to Fig. 4. The smart device 301 is a so-called smartphone or mobile phone. The smart device 301 is, in other words, a portable terminal.

[0038] The smart device 301 includes, for example, a wireless LAN control unit 401 for communication via wireless LAN, a BLE control unit 402 for communication via BLE, and a public line control unit 406 for communication via public wireless. The smart device 301 also includes a packet transmission / reception unit 403.

[0039] The wireless LAN control unit 401 performs RF control of the wireless LAN, communication processing, and various controls of wireless LAN communication compliant with the IEEE802.11 standard series. The wireless LAN control unit 401 also performs protocol processing related to wireless LAN communication. The BLE control unit 402 performs RF control of BLE, communication processing, and various controls of BLE communication. The BLE control unit 402 also performs protocol processing related to BLE communication. The public line control unit 406 performs RF control of public wireless communication, communication processing, and various controls of public wireless communication. The public line control unit 406 also performs protocol processing related to public wireless communication. The public wireless communication is compliant with, for example, the IMT (International Multimedia Telecommunications) standard or the LTE (Long Term Evolution) standard. The packet transmission / reception unit 403 performs processing to execute at least one of transmission and reception of packets related to wireless LAN communication, BLE communication, and public wireless communication. In this embodiment, the smart device 301 is described as performing at least one of sending and receiving packets in communication, but other communication formats such as circuit switching may be used in addition to packet switching.

[0040] The smart device 301 of this embodiment further includes a control unit 411, a recording unit 404, a GPS receiving unit 405, a display unit 407, an operation unit 408, a voice input voice processing unit 409, and a power supply unit 410. The control unit 411 controls the entire smart device 301, for example, by executing a program recorded in the recording unit 404. The recording unit 404 records, for example, the program executed by the control unit 411 and various information such as parameters required for communication. Various operations of the smart device 301, which will be described later, are realized by the control unit 411 executing the program recorded in the recording unit 404.

[0041] The power supply unit 410 supplies power to each unit of the smart device 301. The display unit 407 has a function of outputting visually recognizable information using, for example, an LCD or LED, and a function of outputting sound using, for example, a speaker, and displays and outputs various information. The operation unit 408 is, for example, a button or the like that accepts operations on the smart device 301 by a user. Note that the display unit 407 and the operation unit 408 may be configured using a common component, for example, a touch panel.

[0042] The voice input voice processing unit 409 has, for example, a microphone and performs voice recognition processing on a voice signal input from the microphone. The voice input voice processing unit 409 can voice-recognize a user's operation on the imaging device 101. In this voice recognition processing, the voice input voice processing unit 409 recognizes a voice command issued by the user from a voice signal input from the microphone using a dedicated application. The smart device 301 can register a voice command in the imaging device 101 via, for example, the communication path 302, for causing the voice processing unit 214 of the imaging device 101 to execute a specific process.

[0043] The GPS (Global Positioning System) 405 estimates the current location (longitude and latitude information) of the smart device 301 by analyzing GPS signals received from satellites. If the estimated current location is within a preset range (within a predetermined radius) such as the smart device's home, the smart device 301 notifies the imaging device 101 of the current location information via the BLE control unit 402. The imaging device 101 can use this current location information as parameters for automatic imaging and automatic editing, which will be described later. Furthermore, if there is a change in the current location, such as if the current location moves out of the preset range, the imaging device 101 notifies the imaging device 101 of movement information via the BLE control unit 402 and uses this as parameters for automatic imaging and automatic editing. The smart device 301 may estimate its current location based on information about surrounding wireless networks using a WPS (Wi-Fi Positioning System) or the like.

[0044] As described above, the imaging device 101 and the smart device 301 exchange data with each other through communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, the imaging device 101 and the smart device 301 transmit and receive data such as audio signals and image data. The smart device 301 can also transmit setting information related to imaging by the imaging device 101 to the imaging device 101. The smart device 301 can also transmit control signals related to imaging processing by the imaging device 101 and location information to the imaging device 101.

[0045] <Automatic imaging processing> The automatic imaging process is a process in which the first control unit 223 determines the timing to capture an image and automatically captures an image of a subject. In this automatic imaging process, the first control unit 223 repeats the process of automatically capturing an image of a subject when it is determined that a good image or video can be captured or when a certain amount of time has passed. This allows the user to record good scenes that unexpectedly appear in daily life or casual changes in daily life using the imaging device 101 without having to manually capture images.

[0046] 5 is a flowchart of the automatic imaging process of the imaging device 101 in this embodiment. The process of this flowchart starts when the user turns on the power switch of the imaging device 101. In this embodiment, a wireless connection is established between the imaging device 101 and the smart device 301. The user can also perform various operations on the imaging device 101 using a dedicated application on the smart device 301. The process of each step in the following flowchart is realized by the first control unit 223 controlling each unit of the imaging device 101.

[0047] In step S501, the first control unit 223 determines whether the automatic imaging process is stopped. The stopping of the automatic imaging process will be described later in the explanation of the voice recognition process. If the automatic imaging process is stopped, the first control unit 223 waits until the stop of the automatic imaging process is released. In other words, if the automatic imaging process is stopped, the process of step S501 is repeated until the stop of the automatic imaging process is released. If the automatic imaging process is not stopped, the process of step S502 is executed.

[0048] In step S502, the first control unit 223 causes the image processing unit 207 to perform image processing on the signal captured by the imaging unit 206 and generate an image for object recognition. Furthermore, the first control unit 223 controls the object detection unit 225 to perform object recognition, such as person recognition and animal recognition, from the generated image for object recognition. For example, when performing object recognition, the first control unit 223 stores a pattern for determining the object in advance, and determines the object using the object detection unit 225 based on the degree of match between the stored pattern and a pattern included in the image for object recognition. Furthermore, the first control unit 223 determines the object and detects the position of the object within the angle of view.

[0049] In step S503, the first control unit 223 calculates the amount of image shake correction. Specifically, first, the first control unit 223 calculates the absolute angle of the imaging device based on the angular velocity and acceleration information acquired by the device shake detection unit 209. Then, the first control unit 223 determines the vibration reduction angle at which the tilt rotation unit 104 and the pan rotation unit 105 are moved in an angular direction that cancels out the absolute angle, and sets this as the amount of image shake correction.

[0050] In step S504, the first control unit 223 performs a subject search process. The subject search process is made up of the following processes.

[0051] (1) Area division The area division will be explained using Figures 6(a) to (c). In Figures 6(a) to (c), the position of the imaging device is set as the origin O, and the area on the spherical surface is divided. In the example of Figure 6(a), the area is divided every 22.5 degrees in the tilt direction and the pan direction. When divided as shown in Figure 6(a), as the tilt angle moves away from 0 degrees, the circumference in the horizontal direction becomes smaller, and the area of ​​one area becomes smaller. Therefore, as shown in Figure 6(b), the imaging device of this embodiment sets the horizontal area range to be larger than 22.5 degrees when the tilt angle is 45 degrees or more.

[0052] Next, areas within the angle of view of an image captured by the imaging device 101 will be described with reference to FIGS. 6(c) and 6(d). An axis 1301 is the reference direction of the imaging direction of the imaging device 101, and area division is performed based on this direction. The axis 1301 is, for example, the imaging direction when the imaging device 101 is activated, or a direction that is predetermined as the reference direction for the imaging direction. An area 1302 is the angle of view area of ​​the image captured by the imaging unit 206. FIG. 6(d) is an example of a through image captured by the imaging unit 206 in the area 1302. Within the angle of view of the through image of FIG. 6(d), the image areas are divided into areas 1303 to 1318 based on the area division shown in FIG. 6(c).

[0053] (2) Calculation of importance level for each area For each area divided as described above, an importance level indicating the priority for performing object search is calculated according to the objects present in the area and the scene conditions of the area. The importance level based on the object conditions is calculated based on, for example, the number of objects present in the area, the size of the object's face, the orientation of the object's face, the accuracy of the object's face detection, the object's facial expression, and the object's personal identification result. Furthermore, the importance level according to the scene conditions is, for example, the general object recognition result, the scene determination result (blue sky, backlight, sunset, etc.), the sound level or voice recognition result from the direction of the area, motion detection information within the area, etc.

[0054] Furthermore, for example, when the face of the subject is registered, the first control unit 223 increases the importance level of the area in which the registered face of the subject is detected. For example, the face of the subject is recorded in the nonvolatile memory 216 as a pattern for determining the subject. Note that when increasing the importance level of the area in which the face of the subject is detected, the first control unit 223 returns the importance level of the area to the original importance level after a predetermined time has elapsed or after a predetermined number of images have been captured.

[0055] (3) Determining the search area After determining the importance level of each area as described above, the first control unit 223 determines to focus search on areas with high importance levels, and then calculates the pan angle and tilt angle required to capture an image of one of the areas with high importance levels.

[0056] In step S505, the first control unit 223 performs pan driving and tilt driving. Specifically, the first control unit 223 calculates the pan driving amount and tilt driving amount based on the image shake correction amount and the pan angle and tilt angle calculated in step S504. Then, the first control unit 223 controls the driving of the tilt rotation unit 104 and the pan rotation unit 105 using the lens barrel rotation drive unit 205 based on the calculated pan driving amount and tilt driving amount. In this embodiment, the first control unit 223 detects a subject in an area with a high importance level by the driving in step S505 and starts imaging the subject.

[0057] In step S506, the first control unit 223 controls the zoom unit 201 to perform zoom driving. For example, the zoom is driven according to the state of the subject for which imaging has been started in step S505. For example, if the subject's face is imaged very small within the angle of view, the first control unit 223 controls the zoom to the telephoto side so that the size of the subject's face is imaged appropriately (larger) within the angle of view. On the other hand, if the subject's face is imaged very large within the angle of view, the first control unit 223 controls the zoom to the wide-angle side so that the size of the subject's face is imaged appropriately (smaller) within the angle of view. By controlling the zoom in this way, it is possible to maintain a state suitable for tracking the subject.

[0058] In steps S504 to S506, we have described a method for performing subject search using panning, tilting, and zooming. However, subject search may also be performed using an imaging system that simultaneously captures images in all directions using multiple wide-angle lenses. In this case, performing image processing such as subject detection using all signals obtained by omnidirectional imaging as input images would result in a large overall processing load. Therefore, in this case, a configuration is adopted in which a portion of the image obtained by omnidirectional imaging is cropped, and subject search processing is performed within the cropped image range. In this configuration, the first control unit 223 calculates the importance level for each area, as in the above-described method, changes the crop position based on the importance level, and performs automatic imaging determination, as described below. This enables high-speed subject search while suppressing power consumption due to image processing.

[0059] In step S507, the first control unit 223 determines whether or not to perform automatic imaging.

[0060] Here, the determination of whether to perform automatic imaging will be explained. The determination of whether to perform automatic imaging is made based on whether the imaging score exceeds a predetermined value. The imaging score is a parameter used to determine whether to perform automatic imaging. The imaging score is incremented depending on the subject detection status and the passage of time. For example, consider a design in which automatic imaging is performed when the imaging score exceeds 2000 points. In this case, the imaging score is initially set to 0 points and incremented over time from the time the automatic imaging mode is entered. For example, the imaging score increases at a rate that reaches 2000 points after 120 seconds. In this case, if 120 seconds pass without a subject being detected, the score increments over time and reaches 2000 points, and imaging is performed. Furthermore, if a high-priority subject is detected during this time, 1000 points are incremented. For example, a high-priority subject is a subject that the user has set as a priority subject to be captured among subjects whose faces are registered in the imaging device 101. When a high-priority subject is detected, it is easier to reach 2000 points, which results in an increased frequency of imaging.

[0061] Furthermore, for example, if a smile is recognized from a subject, 800 points are added. Note that this added point based on a smile is added even if the subject is not a high-priority subject. Furthermore, in this embodiment, an example is described in which the added point based on a smile is the same regardless of whether the subject is a high-priority subject, but this is not limited to this. For example, the added point based on detecting a smile from a high-priority subject may be higher than the added point based on detecting a smile from a low-priority subject. This makes it possible to capture images that are more in line with the user's intentions. If the added points based on changes in the facial expressions of these subjects, such as joy, anger, sadness, or happiness, exceed 2000 points, automatic capture is performed. Furthermore, even if the added points based on changes in facial expressions do not exceed 2000 points, the points added over time will reach 2000 points in a shorter time.

[0062] Note that the point increase over time will be described as an example in which, for example, when points are added to reach 2000 points in 120 seconds, 2000 / 120 points are added per second, i.e., linearly with time. However, this is not limited to this. For example, points may not be added until 110 seconds of the 120 seconds, and 200 points may be added per second for the 10 seconds from 110 to 120 seconds to reach 2000 points. This prevents points added due to changes in the subject's facial expression from reaching the point where the image is captured, regardless of the subject's priority. In a point increase method that increases points linearly over time, points are already added over a long period of time, so even points added due to a low-priority subject's change to a smile often reach the point where the image is captured, making it difficult to reflect the priority of the subject. However, lowering the points added due to facial expression changes would miss the timing of facial expression changes, so lowering the points should be avoided. Therefore, points are not added until 110 seconds. In this way, 110 seconds pass without any points being added to low-priority subjects. On the other hand, since high-priority subjects are set to receive 1,000 points upon detection, they will receive 1,000 points even if no points are added as time passes until 110 seconds. This reduces the possibility that low-priority subjects will reach the number of points required for imaging when points are added based on changes in facial expression compared to high-priority subjects, making it easier to determine the level of priority. While the above explanation uses changes in facial expression as an example, other criteria for adding points can also be considered, such as when the voice gets louder or when gestures get larger. In order to make it easier to determine the level of priority for these cases as well, differences in the method of adding points as described above can be used.

[0063] Furthermore, even if the number of points does not exceed 2000 due to the subject's behavior, an image will always be captured every 120 seconds over time, so there will never be a period of time where no images are taken at all.

[0064] Furthermore, if a subject is detected during the process, the time at which the increase begins within the 120 seconds may be brought forward. For example, if a high-priority subject is detected at 60 seconds, even if 1000 points are added, the total will not exceed 2000 points. However, instead of waiting until 110 seconds to increase the points, the linear increase may begin 30 seconds after the subject is detected. Alternatively, the linear increase may begin 20 seconds before 120 seconds, rather than 10 seconds before. This increases the likelihood that a high-priority subject will be captured, making it easier to capture images that better suit the user's intentions.

[0065] When automatic imaging is performed, the imaging score is reset to 0. Automatic imaging will not be performed again until the score exceeds 2000 points.

[0066] The above has described the determination of whether to perform automatic imaging. If the first control unit 223 determines that automatic imaging is to be performed as a result of the above determination, the process of step S508 is executed. If the first control unit 223 determines that imaging is not to be performed, the process of step S501 is executed.

[0067] In step S508, the first control unit 223 executes an imaging process, such as capturing a still image or a moving image.

[0068] The above describes the automatic imaging process of the imaging device 101 in this embodiment. By performing such a process of automatically imaging a subject, the imaging device 101 can capture images or videos of a scene desired by the user without receiving an imaging instruction from the user.

[0069] <Speech recognition processing> 7 is a flowchart of the voice recognition process of the imaging device 101 in this embodiment. The process of this flowchart is started in response to the voice input voice processing unit 409 detecting that a voice signal has been input from the microphone. The process of this flowchart is executed in parallel with the automatic imaging process of FIG. 5. In this embodiment, an example will be described in which the process of this flowchart is started in response to the voice input voice processing unit 409 detecting that a voice signal has been input from the microphone while the automatic imaging process of FIG. 5 is being executed. The process is realized by the first control unit 223 executing a program recorded in the nonvolatile memory 216.

[0070] In step S701, the first control unit 223 determines whether a trigger word has been detected. The trigger word is a startup command for starting voice command recognition, which issues specific instructions to the imaging device 101 by voice. When issuing an instruction to the imaging device 101 by voice, the user needs to utter a command word after the trigger word, and have the imaging device 101 recognize the command word. If the trigger word has been detected, the process of step S702 is executed. If the trigger word has not been detected, the process of step S701 is repeated until the trigger word is detected.

[0071] In step S702, the first control unit 223 stops the automatic imaging process. In this embodiment, when the first control unit 223 detects a trigger word, it transitions to a command word standby state and stops the automatic imaging process. Stopping the automatic imaging means, for example, temporarily suspending subject search using panning, tilting, or zooming, or temporarily suspending the execution of the imaging process. The purpose of stopping the automatic imaging is, for example, to reduce the processing load and quickly respond to the command word issued by the user after the trigger word. Additionally, it is also to enable imaging in the direction desired by the user. For example, if the user is about to issue an imaging instruction via voice command, it is expected that the trigger word will be uttered at the desired timing. However, if the imaging device 101 does not stop the automatic imaging even after detecting the trigger word, it is likely that the imaging direction of the imaging device 101 will have changed since receiving the trigger word by the time the imaging instruction command following the trigger word is received. In other words, if the automatic imaging is not stopped even after detecting the trigger word, it is difficult to capture an image in the direction intended by the user. Therefore, the imaging device 101 stops panning and tilting in response to detecting the trigger word, thereby maintaining a state in which the imaging device 101 faces in the direction in which the user wants to capture an image.

[0072] In step S703, the first control unit 223 outputs a detection sound from the speaker to notify the user that a trigger word has been detected.

[0073] In step S704, the first control unit 223 determines whether a command word has been detected following the trigger word. If a command word has been detected, the process proceeds to step S706, and if not, the process proceeds to step S705.

[0074] In step S705, the first control unit 223 detects the trigger word and determines whether a predetermined time has elapsed since entering a command word standby state. If the predetermined time has elapsed, the process of step S701 is executed, and the first control unit 223 stops the command word standby state and enters a trigger word standby state. In this case, the first control unit 223 resumes the automatic imaging process. If the predetermined time has not elapsed, the first control unit 223 repeats the process of step S704 until a command word is detected.

[0075] In step S706, the first control unit 223 determines whether the detected command word is a still image capturing command. This still image capturing command is a command for causing the imaging device 101 to capture and record one still image. If it is determined to be a still image capturing command, the process of step S707 is executed, and if it is determined not to be a still image capturing command, the process of step S708 is executed.

[0076] In step S707, the first control unit 223 performs a still image capturing process. Specifically, the signal captured by the imaging unit 206 is converted in the image processing unit 207 according to a JPEG format or the like, and the image recording unit 208 records the converted signal on the recording medium 221.

[0077] In step S708, the first control unit 223 determines whether the detected command word is an object change command. An object change command is, for example, a keyword such as "Take a picture of someone else." If it is determined to be an object change command, the process of step S709 is executed, and if it is determined not to be an object change command, the process of step S710 is executed.

[0078] In step S709, the first control unit 223 performs a subject change process. Immediately before this step is executed, the first control unit 223 is in a state of capturing an image of a subject. The subject change process will be described in detail later.

[0079] In step S710, the first control unit 223 determines whether the detected command word is a moving image recording start command. A moving image capturing command is a command that causes the imaging device 101 to capture and record a moving image. If it is determined to be a moving image recording start command, the process of step S712 is executed, and if it is determined not to be a moving image recording start command, the process of step S713 is executed.

[0080] In step S711, the first control unit 223 starts capturing a moving image using the imaging unit 206, and records the captured moving image data on the recording medium 221. While the moving image data is being recorded, the first control unit 223 does not search for a subject, and continues to maintain the automatic imaging stopped state.

[0081] In step S712, the first control unit 223 determines whether the detected command word is a video recording stop command. If it is determined to be a video recording stop command, the process proceeds to step S713, and if it is determined not to be a video recording stop command, the process proceeds to step S714.

[0082] In step S713, the first control unit 223 stops the imaging of the subject by the imaging unit 206 and the recording of the moving image data onto the recording medium 221, thereby completing the recording of the moving image data.

[0083] In step S714, the first control unit 223 executes processing corresponding to other command words. For example, the first control unit 223 executes processing for command words for panning and tilting in a direction specified by the user, and processing for command words for changing various imaging parameters such as exposure compensation of the imaging device 101.

[0084] In steps S715 and S716, the first control unit 223 performs processing to resume the automatic imaging stopped in step S702. For example, the first control unit 223 starts the processing of the flowchart shown in Fig. 5 from the processing of step S502.

[0085] The voice recognition process of the image capture device 101 in this embodiment has been described above.

[0086] Here, the subject change process in step S709 in FIG. 7 will be described.

[0087] <Subject change processing> 8 is a flowchart of the subject change process in this embodiment. Each process in this flowchart is realized by the first control unit 223 executing a program stored in the nonvolatile memory 216. Furthermore, the process of each step in the following flowchart is realized by the first control unit 223 controlling each unit of the imaging device 101. Note that the process in this flowchart corresponds to the process in step S709 in the flowchart in FIG.

[0088] In step S801, first control unit 223 controls the driving of tilt rotation unit 104 and pan rotation unit 105 using lens barrel rotation drive unit 205, and drives the imaging direction of lens barrel 102 toward the initial position for subject search. The process of determining the direction that serves as the initial position for subject search will be described later with reference to Figure 9. In the explanation of this flowchart, the direction that serves as the initial position is assumed to be a direction rotated 180 degrees to the right from the direction of axis 1301 shown in Figure 6(c).

[0089] In step S802, first control unit 223 starts searching for a subject in response to the imaging direction of lens barrel 102 being oriented toward the initial position. In searching for a subject, first control unit 223 controls lens barrel rotation drive unit 205 to start driving tilt rotation unit 104 and pan rotation unit 105 so that the imaging direction of lens barrel 102 is oriented toward the direction that will be the end position of the subject search. In the description of this flowchart, the direction that will be the end position is a direction rotated 180 degrees to the left from the direction of axis 1301 shown in FIG. 6(c). In other words, in the description of this flowchart, the imaging direction of lens barrel 102 is controlled to make one revolution clockwise along a plane parallel to the horizontal plane. The processing of steps S803 to S811 after this step is processing during the subject search.

[0090] In step S803, the first control unit 223 detects a subject using the subject detection unit 225. If it is determined that a subject has been detected, the process proceeds to step S804. If it is determined that a subject has not been detected, the process proceeds to step S811.

[0091] In step S804, the first control unit 223 suspends the search for the subject. For example, the first control unit 223 suspends the driving of the tilt rotation unit 104 and the pan rotation unit 105.

[0092] In step S805, the first control unit 223 captures an image of the subject detected in step S803. The first control unit 223 records the captured image data in the recording medium 221.

[0093] In step S806, the first control unit 223 acquires information about the captured subject. For example, the information about the subject includes whether the face of the subject is registered in the imaging device 101, the priority of the subject, and the face of the subject.

[0094] In step S807, the first control unit 223 determines an importance score for the subject based on the information about the subject acquired in step S806. The importance score is a parameter used by the first control unit 223 to determine the subject to be captured after the search for changing the subject is completed, and is different from the importance level used to determine the search area. For example, if the subject is set by the user as a high-priority subject, it can be assumed that the subject is the subject the user wants to capture, and the importance score will be a relatively high value. Conversely, if the subject is not registered in the imaging device 101, it is not necessarily the subject the user wants to capture, and the importance score will be a relatively low value. Note that the importance score for a subject that is registered in the imaging device 101 but for which no priority is set will be an intermediate value. In other words, the first control unit 223 can determine that a subject with a high importance score is the subject the user wants to capture.

[0095] Furthermore, in this embodiment, the first control unit 223 determines the importance score of a subject with an expression to be a higher value than the importance score of a subject with a serious face. For example, the importance score of a smiling subject is higher than the importance score of a subject with a serious face. Also, for example, a crying face of a child may be an expression that a parent of the child would want to capture, so the first control unit 223 sets the importance score of a crying subject to a higher value than the importance score of a serious subject.

[0096] Furthermore, in this embodiment, the first control unit 223 determines the importance score based on the size of the eyes of the detected subject. For example, the first control unit 223 determines the importance score of a subject whose eyes can be detected to be a higher value than the importance score of a subject whose eyes cannot be recognized because the subject has their eyes closed. Furthermore, for example, the first control unit 223 determines the importance score of a subject whose eyes are open to be a higher value than the importance score of a subject whose detected eyes are relatively small, such as with half-closed eyes. Thus, in this embodiment, the first control unit 223 determines the importance score based on the size of the eyes of the detected subject.

[0097] In this embodiment, the first control unit 223 also determines the importance score based on the orientation of the subject's face. For example, the first control unit 223 determines the importance score of a subject whose entire frontal face is detected to be higher than the importance score of a subject whose face is only partially detected, such as a face facing sideways.

[0098] In this embodiment, the first control unit 223 determines the final importance score of the subject by calculating the sum of the importance scores for the information about these subjects. Note that in this embodiment, the first control unit 223 always determines the importance score of a high-priority subject to be higher than the importance scores of other subjects for which no priority is set. In other words, when both a high-priority subject and another subject for which no priority is set are detected, the first control unit 223 determines the high-priority subject as the subject to be captured after the subject change process.

[0099] In step S808, the first control unit 223 determines whether the importance score of the subject determined in step S807 is the highest among the importance scores determined since the processing of this flowchart began. For example, the first control unit 223 determines whether the importance score is the highest among the importance scores determined since the processing of this flowchart began. If it is determined that the importance score is the highest, the processing of step S809 is executed. If it is determined that the importance score is not the highest, the processing of step S810 is executed.

[0100] In step S809, the first control unit 223 records the imaging direction of the lens barrel 102 when the subject determined to have the highest importance score in step S808 was detected. For example, the first control unit 223 records the angle of the pan direction in the memory 215 as the imaging direction of the lens barrel 102. In this embodiment, the reference direction of the imaging device 101 (the direction of the axis 1301 shown in FIG. 6C) is recorded in the memory 215 as the initial value of the imaging direction of the lens barrel 102. The first control unit 223 also records information about the subject along with the imaging direction of the lens barrel 102 in the memory 215. If the imaging direction of the lens barrel 102 and information about the subject have already been recorded, the first control unit 223 overwrites and records the imaging direction of the lens barrel 102 and information about the subject.

[0101] In step S810, the first control unit 223 resumes the search for the subject. For example, the first control unit 223 causes the tilt rotation unit 104 and the pan rotation unit 105 to resume driving.

[0102] In step S811, the first control unit 223 determines whether the search for the subject is complete. For example, the first control unit 223 determines whether the imaging direction of the lens barrel 102 is pointing in the direction that will be the end position. If it is pointing in the direction that will be the end position, the first control unit 223 determines that the search for the subject is complete. If it is not pointing in the direction that will be the end position, the first control unit 223 determines that the search for the subject is not complete. If it is determined that the search for the subject is complete, the process returns to step S803, and the search for the subject continues. As described in the explanation of step S802, the end position is the same direction as the initial position, so the process of this step is equivalent to determining whether one rotation has been made since the start of the search. Furthermore, as described in step S805, if the first control unit 223 detects a subject during the search for the subject, it captures and records an image of the detected subject. That is, when it is determined in this step that the search for the subject is complete, the first control unit 223 searches the entire circumference to change the subject, and captures and records an image of each surrounding subject once. This allows the imaging device 101 to capture at least once a subject that the user wants captured. Note that the subjects captured include the person who input the subject change command by voice. Furthermore, by repeating the process of step S809, information on the subject determined to have the highest importance score among the subjects captured in the entire circumference is recorded in the memory 215.

[0103] In step S812, first control unit 223 controls lens barrel rotation drive unit 205 to drive tilt rotation unit 104 and pan rotation unit 105 so as to capture an image of the subject determined to have the highest importance score. That is, first control unit 223 controls drive so that the imaging direction of lens barrel 102 becomes the imaging direction when the subject determined to have the highest importance score in step S808 was detected. Furthermore, first control unit 223 determines the subject to be imaged based on the information about the subject last recorded in step S809. After the processing of this step, first control unit 223 executes processing to image the subject. This processing to image the subject will be described later using FIG. 10.

[0104] If a subject is not detected in the subject change processing, the first control unit 223 executes automatic imaging processing. If a subject is not detected in the subject change processing, it is considered that there is no subject around the imaging device 101. On the other hand, since an instruction to change the subject has been issued, it is assumed that the user intends to capture a subject. Therefore, by executing automatic imaging processing, for example, when a new subject comes near the imaging device 101, the first control unit 223 immediately detects and captures the subject. This allows the first control unit 223 to shorten the period during which no subject is captured and automatically capture an image in line with the user's intention. If a subject is not detected in the subject change processing, the processing of step S809 is not executed, and the reference direction of the imaging device 101 recorded in the memory 215 is not overwritten. Therefore, in step S812, the imaging device 101 faces the reference direction of the imaging device 101. Then, after the imaging device 101 faces the reference direction, the first control unit 223 executes automatic imaging processing.

[0105] The subject change processing of the imaging device 101 in this embodiment has been described above.

[0106] As described above, in the subject change process of the imaging device 101 in this embodiment, the imaging device 101 searches for subjects around the imaging device 101. As a result, even if the imaging device 101 ends up tracking a subject that the user does not want to capture through automatic imaging, for example, the user can give the imaging device 101 a chance to stop tracking the subject and search for another subject.

[0107] Furthermore, the imaging device 101 accepts an instruction from the user that does not specify a new imaging target, such as "Take a picture of someone else," and searches for another subject while maintaining automatic imaging. For the user, simply issuing an instruction to change the subject allows the imaging device 101 to capture a desired subject, eliminating the need for the user to specify where (position) or who (name) the desired imaging target is.

[0108] The first control unit 223 may execute the subject change process in response to receiving an instruction to change the subject from the smart device 301. In this case, the user operates a dedicated application on the smart device 301, which causes the smart device 301 to transmit a control signal for changing the subject to the imaging device 101. Then, in response to receiving this control signal, the first control unit 223 starts the subject change process.

[0109] In this embodiment, the first control unit 223 searches the entire periphery of the imaging device 101 to change the subject, but the search range may be narrowed. For example, if the imaging device 101 is installed in a corner of a room, the first control unit 223 may search for the subject in a range excluding the area close to the wall of the room in the subject change process. Also, for example, if the imaging range of the imaging device 101 is limited to 180 degrees, the first control unit 223 searches for the subject in that limited range in the subject change process. In this way, in the subject change process, the first control unit 223 may search for the subject based on the surrounding situation of the imaging device 101 or a preset imaging range.

[0110] 9 is a flowchart of processing for determining a direction that will be the initial position for subject search in subject change processing in this embodiment. Each process in this flowchart is realized by the first control unit 223 executing a program stored in the nonvolatile memory 216. Furthermore, the processing of each step in the following flowchart is realized by the first control unit 223 controlling each unit of the imaging device 101. Note that the processing in this flowchart corresponds to the processing in step S801 in the flowchart in FIG.

[0111] In step S901, the first control unit 223 determines whether the current imaging direction of the lens barrel 102 is to the left or right of the central direction. For example, the central direction is the direction of the axis 1301 shown in FIG. 6(c), with the right side being a range of 180 degrees clockwise from the direction of the axis 1301 and the left side being a range of 180 degrees counterclockwise from the direction of the axis 1301. In this case, the left end direction and the right end direction overlap. Note that the right end may be set to any direction within a range of 180 degrees clockwise from the central direction. Similarly, the left end may be set to any direction within a range of 180 degrees counterclockwise from the central direction.

[0112] If it is determined that the current imaging direction of the lens barrel 102 is the left side, the process of step S902 is executed. In step S902, the first control unit 223 sets the initial position of the subject search to the left end and the end position to the right end. In this case, the first control unit 223 controls the lens barrel rotation drive unit 205 to rotate the subject search clockwise from the left end.

[0113] If it is determined that the current imaging direction of the lens barrel 102 is to the right, the process of step S903 is executed. In step S903, the first control unit 223 sets the initial position of the subject search to the right end and the end position to the left end. In this case, the first control unit 223 controls the lens barrel rotation drive unit 205 to rotate counterclockwise from the right end.

[0114] The above has described the process for determining the direction that will be the initial position for subject search in the subject change process in this embodiment.

[0115] In this way, the first control unit 223 sets the direction that serves as the initial position for subject search in the subject change process to a direction with a small rotation angle in the pan direction.

[0116] In step S801, the imaging device 101 may determine the direction that will be the initial position for the subject change process regardless of the current imaging direction of the lens barrel 102. For example, the imaging device 101 always determines the direction that will be the initial position for the subject change process to be the rightmost direction. In this case, the imaging device 101 always determines the direction that will be the initial position for the subject change process to be the leftmost direction.

[0117] The imaging device 101 may recognize an instruction to change the imaging direction of the imaging device 101, such as "look right" or "look left," as a subject change command. When such a command is adopted, the user needs to understand the position of the subject they wish to capture before giving the instruction, but the search range of the imaging device can be limited to the direction of the desired subject, making it possible to change the subject more efficiently.

[0118] When a change instruction to change the imaging direction of the imaging device 101, such as "look right" or "look left," is recognized as a subject change command, the first control unit 223 sets the direction that serves as the initial position of the subject search to the direction when the subject change command is recognized. The first control unit 223 then executes the subject search process in the direction included in the change instruction. In this case, the end position of the subject search process is the end point in the direction included in the change instruction. For example, if the change instruction is "look right," the end point (end position of the subject search process) is the right end. For example, if the change instruction is "look left," the end point (end position of the subject search process) is the left end.

[0119] <Image capture process after changing the subject> Next, the imaging process after the subject change will be described.

[0120] 10 is a flowchart showing the imaging process after changing the subject in this embodiment. The process of this flowchart is executed after step 812. The process of this flowchart is realized by the first control unit 223 executing a program stored in the nonvolatile memory 216.

[0121] In step S1001, the first control unit 223 determines whether or not a subject has been detected by the above-described subject change process. If it is determined that a subject has been detected, the process of step S1003 is executed. If it is determined that a subject has not been detected, the process of step S1002 is executed.

[0122] In step S1002, first control unit 223 controls lens barrel rotation drive unit 205 to drive the lens barrel to an initial position. For example, this initial position is a direction that serves as an initial position for subject search.

[0123] In step S1003, the first control unit 223 controls the lens barrel rotation drive unit 205 so that the imaging direction of the lens barrel 102 faces the imaging direction recorded in step S809 of Fig. 8. Thereafter, until the processing of step S1007 is started, the first control unit 223 limits the imaging direction to a range that includes the imaging direction recorded in step S809 and is narrower than the subject search range in the subject change processing. This allows the first control unit 223 to continue to include the subject imaged in the subject change processing as the main subject in the angle of view. Furthermore, by setting the range in the imaging direction, the first control unit 223 can capture an image while adjusting the angle of view to achieve a good composition.

[0124] In step S1004, the first control unit 223 determines whether or not the subject determined to have the highest importance score in step S812 has been detected in the imaging direction recorded in step S809 of Fig. 8. If it is determined that the subject determined to have the highest importance score has been detected, the process of step S1005 is executed. If it is determined that the subject determined to have the highest importance score has not been detected, the process of step S1007 is executed.

[0125] In step S1005, the first control unit 223 captures an image of the subject detected in step S1004 and records the captured image data on the recording medium 221.

[0126] In step S1006, the first control unit 223 determines whether a predetermined time has elapsed since the processing of step S1004 was executed. This predetermined time is, for example, 5 minutes or 10 minutes. If it is determined that the predetermined time has not elapsed, the first control unit 223 executes the processing of step S1004 and continues detecting the subject determined to have the highest importance score determined in step S812. If it is determined that the predetermined time has elapsed, the processing of step S1007 is executed.

[0127] In step S1007, the first control unit 223 stops detecting the subject determined to have the highest importance score in the imaging direction recorded in step S809 of Fig. 8 and returns to normal automatic imaging processing. At the same time, the imaging direction recorded in step S809 of Fig. 8 and the limitation on the imaging direction range from the processing of step S1003 are reset. That is, the reference direction is overwritten and recorded in the memory 215, and the subject search range returns to 360 degrees in the automatic imaging processing from the processing of step S1007 onwards. Note that, as a result of executing the automatic imaging processing, the first control unit 223 may continue capturing the subject currently being captured that has been determined to have the highest importance score.

[0128] The imaging process of the imaging device 101 after changing the subject has been described above.

[0129] In this way, the first control unit 223 performs a subject search in response to a subject change instruction from the user, and then captures an image of the subject determined to have the highest importance score, allowing the imaging device 101 to continue capturing images of subjects that are estimated to be the subject the user wants to capture.

[0130] In this flowchart, the first control unit 223 determines whether a predetermined time has elapsed in step S1006, but may instead determine whether images have been captured a predetermined number of times. For example, this predetermined number is 5, 10, or the like. In this case, the process of step S1007 is executed in response to a determination that images have been captured the predetermined number of times. Also, in this case, the process of step S1004 is executed in response to a determination that images have not been captured the predetermined number of times.

[0131] As described above, the imaging device 101 searches for a subject in response to receiving an instruction to change the subject from the user, automatically detects the subject that the user wants to capture, and captures the subject. This allows the imaging device 101 to realize automatic shooting in line with the user's intention.

[0132] In the present embodiment, the first control unit 223 has been described as accepting an instruction to execute a subject change process while the automatic imaging process is being executed. However, the first control unit 223 may also accept an instruction to execute a subject change process even when the automatic imaging process is not being executed. Even in this case, the first control unit 223 starts the automatic imaging process after capturing an image of the subject according to the imaging process after the subject change. This is because it is considered that the user who instructed the subject change process intends for the subject detected by the subject change process to continue being captured. Alternatively, when an instruction to execute a subject change process is received while the automatic imaging process is not being executed, the first control unit 223 may not start the automatic imaging process after capturing an image of the subject according to the imaging process after the subject change. This is because it is considered that the user who instructed the subject change process intends for the subject detected by the subject change process to be captured to a certain extent and does not need to capture images automatically thereafter. Furthermore, when an instruction to execute a subject change process is received while the automatic imaging process is not being executed, it may be possible to set whether or not to start the automatic imaging process after capturing an image of the subject according to the imaging process after the subject change. The user sets whether or not to start the automatic imaging process by instructing the imaging device 101 from the smart device 301. By doing so, it is possible to accommodate whatever the user of the image capturing device 101 desires.

[0133] [Second embodiment] In the second embodiment, the imaging device 101 executes the subject change process with an emphasis on changing the subject to be captured. In this embodiment, the imaging device 101 aims to make it easier to capture the subject that the user wants to capture by prioritizing capturing a subject other than the user who was capturing an image immediately before the start of the subject change process. Note that the configuration of the imaging device 101 and the configuration of the smart device 301 are the same as those in the first embodiment.

[0134] 11 is a flowchart showing the subject change process in the second embodiment. The process of each step in the following flowchart is realized by the first control unit 223 controlling each unit of the imaging device 101. The process of this flowchart corresponds to the process of step S709 in the flowchart of FIG.

[0135] In step S1101, the first control unit 223 records information about the subject currently being imaged. For example, the first control unit 223 records an identifier indicating the subject being imaged when the subject change instruction was received as information about the subject. Hereinafter, the subject being imaged in step S1101 will be referred to as the initial subject. In this embodiment, the first control unit 223 selects only one initial subject. For example, if two subjects are detected within the angle of view, the first control unit 223 selects and records one of them as the initial subject. In this case, the first control unit 223 selects the initial subject based on criteria for selecting the initial subject, such as the size within the angle of view, the proximity to the center of the angle of view, and the proximity to the focal point.

[0136] The processes in steps S1102 to S1107 are similar to the processes in steps S801 to S806 in FIG. 8, respectively, and therefore will not be described further.

[0137] In step S1108, the first control unit 223 determines whether the subject detected in step S1104 is an initial subject. For example, the first control unit 223 detects whether the subject detected in step S1104 is an initial subject by referring to an identifier that is recorded in the information related to the subject and indicates the subject that was being imaged when the subject change instruction was received. If it is determined that the subject detected in step S1104 is an initial subject, the process of step S1109 is executed. If it is determined that the subject detected in step S1104 is not an initial subject, the process of step S1110 is executed.

[0138] In step S1109, the first control unit 223 determines the importance score of the subject detected in step S1104 as the minimum value. That is, in this step, the first control unit 223 determines the importance score of the initial subject so that it is always lower than the importance scores of subjects that are not initial subjects. For example, a negative fixed value that does not occur in normal importance score calculation is set. Alternatively, a fixed value that is smaller than the minimum possible value of each importance calculated from each piece of information such as priority, facial expression, eye size, and face direction is set. That is, in this step, the first control unit 223 does not calculate the importance score. This allows the imaging device 101 to capture an image of a subject other than the initial subject after the change process.

[0139] The processing in steps S1110 to S1115 is similar to the processing in steps S807 to S812 in FIG. 8, respectively, and therefore a description thereof will be omitted.

[0140] The subject change processing of the imaging device 101 in this embodiment has been described above.

[0141] In this way, by determining the importance score of the initial subject as the lowest value, the first control unit 223 can make it easier to capture an image of a subject other than the initial subject in the subject change process.

[0142] In the subject change process, if the first detected subject is the initial subject, the first control unit 223 records the direction of the initial subject in the imaging direction recording process in step S1112. This causes the initial subject to be imaged even if no subject other than the initial subject is found, so the first control unit 223 can avoid a situation where no subject is imaged after the subject change process.

[0143] 11 ends, the first control unit 223 deletes the identifier indicating the subject that was being imaged when the subject change instruction was received, which is recorded as information related to the initial subject. As a result, the next time the first control unit 223 starts the automatic imaging process, the importance score of the initial subject will be determined based on the same criteria as for other subjects.

[0144] In this embodiment, the first control unit 223 determines the importance score of the initial subject to be the lowest value in order to capture images of subjects other than the initial subject, but another method of determining the importance score may be adopted. For example, the first control unit 223 may unconditionally add a predetermined value to the importance scores of subjects other than the initial subject, thereby making it easier to capture images of subjects other than the initial subject.

[0145] [Third embodiment] In the third embodiment, the imaging device 101 captures a moving image during the subject search process in the subject change process. By capturing a moving image during the subject change process, the imaging device 101 aims to capture a subject that the user wants to capture without missing a shot, compared to when capturing a still image during the subject change process. The configurations of the imaging device 101 and the smart device 301 are the same as those of the first embodiment.

[0146] 12 is a flowchart showing the subject search process in the third embodiment. Each process in this flowchart is realized by the first control unit 223 executing a program stored in the nonvolatile memory 216. Furthermore, the process of each step in the following flowchart is realized by the first control unit 223 controlling each unit of the imaging device 101. Note that the process in this flowchart corresponds to the process in step S709 in the flowchart in FIG.

[0147] In step S1201, the first control unit 223 determines whether to capture a moving image or a still image in the subject change process. For example, the first control unit 223 determines whether to capture a moving image or a still image in the subject change process. The user can switch this setting using the smart device 301. If it is determined to capture a moving image, the process of step S1203 is executed. If it is determined to capture a still image, the process of step S1213 is executed. In the process of step S1213, the process of the flowchart shown in FIG. 8 is executed.

[0148] In step S1202, first control unit 223 controls the driving of tilt rotation unit 104 and pan rotation unit 105 by lens barrel rotation drive unit 205, and drives the imaging direction of lens barrel 102 toward the initial position for subject search. For example, first control unit 223 drives tilt rotation unit 104 and pan rotation unit 105 so that the imaging direction of lens barrel 102 is the direction of axis 1301 shown in FIG. 6(c).

[0149] In step S1203, the first control unit 223 starts recording the moving image.

[0150] In step S1204, first control unit 223 starts searching for the subject in response to the imaging direction of lens barrel 102 being oriented toward the initial position (the direction of axis 1301 shown in FIG. 6(c)). In the subject search, first control unit 223 controls lens barrel rotation drive unit 205 to start driving tilt rotation unit 104 and pan rotation unit 105 so that the imaging direction of lens barrel 102 is oriented toward the end position of the subject search. In this embodiment, for example, the imaging direction of lens barrel 102 is controlled to make one revolution clockwise from the direction of axis 1301 shown in FIG. 6(c) along a plane parallel to the horizontal plane. In this case, the orientation of the end position of the subject search will be the same orientation as the initial position (the direction of axis 1301 in FIG. 6(c)). From this step onwards, the processing in steps S1205 to S1210 is processing during the subject search.

[0151] In step S1205, first control unit 223 detects a subject using subject detection unit 225. If it is determined that a subject has been detected, the process proceeds to step S1206. If it is determined that a subject has not been detected, the process proceeds to step S1210.

[0152] In step S1206, the first control unit 223 acquires information about the captured subject. For example, the information about the subject includes whether or not the face of the subject is registered in the imaging device 101, the priority of the subject, the face of the subject, and whether or not the subject was being imaged when the subject change instruction was received.

[0153] In step S1207, the first control unit 223 determines an importance score for the subject based on the information about the subject acquired in step S1206. The importance score is a parameter used by the first control unit 223 to determine the subject to be captured after the search for changing the subject is completed, and is different from the importance level used to determine the search area. For example, if the subject is set by the user as a high-priority subject, it can be assumed that the subject is the subject the user wants to capture, and therefore the importance score will be a high value. Conversely, if the subject is not registered in the imaging device 101, it is not necessarily the subject the user wants to capture, and therefore the importance score will be a low value. Note that the importance score for a subject that is registered in the imaging device 101 but for which no priority is set will be an intermediate value. In other words, the first control unit 223 can determine that a subject with a high importance score is the subject the user wants to capture.

[0154] Furthermore, in this embodiment, the first control unit 223 determines the importance score of a subject with an expression to be higher than the importance score of a subject with a serious expression. For example, the importance score of a smiling subject is higher than the importance score of a subject with a serious expression. Also, for example, because a crying face of a child may be an expression that a parent of the child would want to capture, the first control unit 223 sets the importance score of a crying subject to a higher value than the importance score of a serious subject. Note that in this embodiment, the importance score of a subject with a high priority is set to be higher than the importance score of a subject with an expression such as a smiling face. In this embodiment, the first control unit 223 determines the importance score by calculating the sum of the importance scores for information about these subjects.

[0155] In step S1208, the first control unit 223 determines whether the importance score of the subject determined in step S1207 is the highest among the importance scores determined since the processing of this flowchart began. For example, the first control unit 223 determines whether the importance score is the highest among the importance scores determined since the processing of this flowchart began. If it is determined that the importance score is the highest, the processing of step S1209 is executed. If it is determined that the importance score is not the highest, the processing of step S1210 is executed.

[0156] In step S1209, the first control unit 223 records the imaging direction of the subject determined to have the highest importance score in step S1208. For example, the first control unit 223 records the angle of the pan direction in the memory 215 as the imaging direction of the lens barrel 102. In this embodiment, the initial value of the imaging direction at the start of the processing of this flowchart is the direction that becomes the initial position for subject search. The first control unit 223 also records information about the subject along with the imaging direction of the lens barrel 102 in the memory 215. In addition, if the imaging direction of the lens barrel 102 and information about the subject have already been recorded, the first control unit 223 overwrites and records the imaging direction of the lens barrel 102 and information about the subject.

[0157] In step S1210, the first control unit 223 determines whether or not the search for the subject is complete. For example, the first control unit 223 determines whether or not the imaging direction of the lens barrel 102 is pointing in the direction that will be the end position. If it is pointing in the direction that will be the end position, the first control unit 223 determines that the search for the subject is complete. If it is not pointing in the direction that will be the end position, the first control unit 223 determines that the search for the subject is not complete. If it is determined that the search for the subject is complete, the process of step S1211 is executed. If it is determined that the search for the subject is not complete, the process of step S1205 is executed, and the first control unit 223 continues the search for the subject.

[0158] In step S1211, the first control unit 223 stops recording of the moving image and records the moving image data on the recording medium 221.

[0159] In step S1212, the first control unit 223 controls the lens barrel rotation drive unit 205 to drive and control the tilt rotation unit 104 and the pan rotation unit 105 so as to capture an image of the subject determined to have the highest importance score. That is, the first control unit 223 controls and drives the lens barrel 102 so that the imaging direction of the lens barrel 102 becomes the imaging direction when the subject determined to have the highest importance score in step S808 was detected. Furthermore, the first control unit 223 determines the subject to be imaged based on the information about the subject last recorded in step S1209. After the processing of this step, the first control unit 223 executes processing to image the subject. This processing to image the subject is similar to the processing shown in the flowchart of FIG. 10 described in the first embodiment.

[0160] The subject change processing of the imaging device 101 in this embodiment has been described above.

[0161] In this way, by capturing a moving image in the subject change process, the imaging device 101 can capture an image of a subject that the user wants to capture without missing it, compared to when capturing a still image in the subject change process.

[0162] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a recording medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0163] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

Claims

1. an imaging means for imaging a subject; a search means for searching for a subject to be imaged by the imaging means; a receiving means for receiving a change instruction to change the imaging target without specifying a new imaging target; a determination means for determining, in the search started in response to receiving the change instruction, a subject other than the subject that was the subject to be imaged when the change instruction was received, as the subject to be imaged; the determining means, in the search started in response to receiving the change instruction, calculates the importance of each subject based on information on the face of the detected subject, and determines an imaging target based on the importance; the determining means, in the search started in response to receiving the change instruction, determines the other subjects as the subject to be imaged by giving priority to the other subjects by lowering the importance of the subject that was being imaged when the change instruction was received compared to other subjects; When the change instruction is received, the search means searches for another subject. An imaging device characterized by:

2. The determining means calculates the importance of the subject based on a priority set by a user, the facial expression of the subject, the size of the subject's eyes, and the direction of the subject's face.

2. The imaging device according to claim 1.

3. The present invention further comprises a control means for automatically determining, using the same criteria, whether to change the subject that was the subject of the image capture when the change instruction was received to a subject of the image capture, and whether to change the subject of the image capture, and for controlling an automatic image capture process for automatically capturing an image of the determined subject of the image capture, When the search means does not detect a subject in the search for changing the subject that is started in response to receiving the change instruction, the control means starts the automatic imaging process.

2. The imaging device according to claim 1.

4. An imaging device, an imaging means for imaging a subject; a search means for searching for a subject to be imaged by the imaging means; a receiving means for receiving a change instruction to change the imaging target without specifying a new imaging target, When the change instruction is received, the search means searches for another subject, When the search is started in response to receiving the change instruction, the search means searches the entire periphery of the imaging device. An imaging device characterized by:

5. an imaging means for imaging a subject; a search means for searching for a subject to be imaged by the imaging means; a receiving means for receiving a change instruction to change the imaging target without specifying a new imaging target; a first recording means for recording the imaging direction of the imaging means during a search by the search means, When the change instruction is received, the search means searches for another subject, When the search started in response to the reception of the change instruction is completed, the imaging means faces the direction recorded by the first recording means. An imaging device characterized by:

6. When the subject is detected in the direction recorded by the first recording means, the imaging means repeats imaging of the subject until a predetermined time has elapsed or until the imaging means has imaged the subject a predetermined number of times.

6. The imaging device according to claim 5.

7. The search means automatically starts a search for capturing an image of the subject when the subject is not detected in the imaging direction recorded by the first recording means.

7. The imaging device according to claim 5, wherein the imaging device is a lens.

8. In the search started in response to receiving the change instruction, the imaging means captures a still image of the detected subject in response to the subject being detected.

8. The imaging device according to claim 1, wherein the imaging device is a lens.

9. In the search started in response to receiving the change instruction, the imaging means captures a moving image.

9. The imaging device according to claim 1, wherein the imaging device is a lens.

10. an imaging means for imaging a subject; a search means for searching for a subject to be imaged by the imaging means; a receiving means for receiving a change instruction to change the imaging target without specifying a new imaging target, When the change instruction is received, the search means searches for another subject, In the search started in response to the reception of the change instruction, the search means starts the search after directing the imaging direction of the imaging means to a predetermined direction different from the position where the change instruction was received. An imaging device characterized by:

11. The search means determines the predetermined direction according to the imaging direction of the imaging means when the change instruction is received.

11. The imaging device according to claim 10.

12. The search means determines a range in which to search for the subject in accordance with a direction included in the change instruction from the position where the change instruction is accepted.

12. The imaging device according to claim 1, wherein the imaging device is a lens.

13. Further having a microphone, The receiving means receives the change instruction by analyzing the audio signal input from the microphone.

13. The imaging device according to claim 1, wherein the imaging device is a lens.

14. an imaging means for imaging a subject; a search means for searching for a subject to be imaged by the imaging means; a receiving means for receiving a change instruction to change the imaging target without specifying a new imaging target, When the change instruction is received, the search means searches for another subject, The apparatus further includes a driving means for driving the imaging direction of the imaging means in the search process by the searching means. An imaging device characterized by:

15. A control method for an imaging device having an imaging means, comprising: an imaging step of imaging a subject by the imaging means; a step of searching for a subject to be imaged in the imaging step; receiving a change instruction to change the imaging target without specifying a new imaging target; a step of searching for another subject when the change instruction is received; and A control method characterized in that the searching step drives the imaging direction of the imaging means to search for another subject.

16. A computer-readable program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 14.

Images