Imaging device, control method thereof, and storage medium

Abstract

The present invention relates to an imaging device, a control method thereof, and a storage medium. The imaging device includes: an imaging unit, a control unit configured to control the imaging unit to automatically capture images, a registration unit configured to register a subject, and an association unit configured to associate the subject registered by the registration unit with a specific date. The control unit performs control such that, compared with a subject registered by the registration unit for which the date associated with the association unit is not the current day or for which no date is associated by the association unit, a subject for which the date associated with the association unit is the current day is preferentially captured.

Description

Technical Field

[0001] The present invention relates to an imaging device capable of automatically capturing images. Background Art

[0002] In recent years, automatic imaging cameras that regularly and continuously capture images without using a capture instruction from a user have been known. For example, Japanese Unexamined Patent Application Publication No. 2020-92354 discusses a device having a pan / tilt mechanism and capable of automatically searching for a subject and automatically capturing an image of the subject. The conventional technique discussed in Japanese Unexamined Patent Application Publication No. 2020-92354 determines which person or scene to capture based on the facial expressions and voices of each person through importance settings. However, the conventional technique does not at all consider important days such as the birthdays of each person. It is generally considered that many users hope that the camera preferentially captures images of people associated with important days such as birthdays and wedding anniversaries. However, the conventional technique does not notice the intentions of such users, and thus may not necessarily be able to perform automatic imaging based on the intentions of the users. Summary of the Invention

[0003] According to an aspect of the present invention, an imaging device includes: an imaging unit; a control unit configured to control the imaging unit to automatically capture an image; a registration unit configured to register a subject; and an association unit configured to associate the subject registered by the registration unit with a specific date. The control unit performs control such that, compared with a subject registered by the registration unit for which the specific date associated with the association unit is not the current day or for which no date is associated by the association unit, the subject for which the specific date associated with the association unit is the current day is preferentially captured.

[0004] A control method for an imaging device having an imaging unit, the control method including: controlling the imaging unit to automatically capture an image; registering a subject; and associating the registered subject with a specific date, wherein, compared with a registered subject for which the associated date is not the current day or for which no date is associated, the subject for which the associated date is the current day is preferentially captured.

[0005] A non-transitory computer-readable storage medium storing instructions for causing a computer to execute a control method for an imaging device having an imaging unit, the control method including: controlling the imaging unit to automatically capture an image; registering a subject; and associating the registered subject with a specific date, wherein, compared with a registered subject for which the associated date is not the current day or for which no date is associated, the subject for which the associated date is the current day is preferentially captured.

[0006] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0007] Figure 1A Shows an example of the appearance of a camera device according to one or more aspects of the present invention, and Figure 1B shows the operation of a camera device according to one or more aspects of the present invention.

[0008] Figure 2 is a block diagram showing the configuration of a camera device according to one or more aspects of the present invention.

[0009] Figure 3 Shows the configuration of a camera device and an external device according to one or more aspects of the present invention.

[0010] Figure 4 is a block diagram showing the configuration of an external device according to one or more aspects of the present invention.

[0011] Figure 5 is a flowchart showing an automatic shooting process according to one or more aspects of the present invention.

[0012] Figure 6 is a flowchart showing a voice recognition process according to one or more aspects of the present invention.

[0013] Figure 7 is a flowchart showing a frequency setting process according to one or more aspects of the present invention.

[0014] Figure 8A , 8B , 8C and 8D show the area division in the captured image according to one or more aspects of the present invention.

[0015] Figure 9 Shows an example screen displayed on an external device according to one or more aspects of the present invention.

[0016] Figure 10A and 10B show an example screen displayed on an external device according to one or more aspects of the present invention.

[0017] Figure 11 is a table showing subject information according to one or more aspects of the present invention.

[0018] Figure 12 is a flowchart showing a process for setting priorities according to one or more aspects of the present invention.

[0019] Figure 13 is a diagram showing control related to the ratio of the number of captured images according to one or more aspects of the present invention.

[0020] Figure 14Shows an example screen displayed on an external device according to one or more aspects of the present invention.

[0021] Figure 15 Shows an example screen displayed on an external device according to one or more aspects of the present invention.

[0022] Figure 16A And 16B Shows an example screen displayed on an external device according to one or more aspects of the present invention.

[0023] Figure 17A And 17B Shows an example screen displayed on an external device according to one or more aspects of the present invention. Detailed Description of the Invention

[0024] Exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] The following exemplary embodiments will be regarded as exemplary examples for implementing the present invention and can be corrected, modified, and combined as needed based on the configuration of the device according to the present invention and other various conditions.

[0026] Figure 1A And 1B Schematically shows a camera device according to a first exemplary embodiment.

[0027] Reference Figure 1A , the camera device 101 is provided with an operation member for operating a power switch (hereinafter referred to as a power button). The power button can be replaced by operations such as touch, tap, or swipe. The camera device 101 is provided with a lens barrel 102 as a housing including a lens group and an image sensor for taking images. The lens barrel 102 is provided with a rotation mechanism for driving the lens barrel 102 to rotate relative to a fixed portion 103. The pitch rotation unit 104 is a motor drive mechanism capable of rotating the lens barrel 102 in the pitch direction shown in Figure 1B . The yaw rotation unit 105 is a motor drive mechanism capable of rotating the lens barrel 102 in the yaw direction shown in Figure 1B . This means that the lens barrel 102 can rotate in more than one direction. Figure 1B Shows the axis definition at the position of the fixed portion 103. Both an angular velocity meter 106 and an accelerometer 107 are mounted at the fixed portion 103 of the camera device 101. The camera device 101 detects the vibration of the camera device 101 by using the angular velocity meter 106 and the accelerometer 107, and drives the pitch rotation unit 104 and the yaw rotation unit 105 to rotate based on the detected shake angle. The camera device 101 is configured to correct the shake and tilt of the lens barrel 102 as a moving unit.

[0028] Figure 2 is a block diagram showing the configuration of the imaging device 101 according to the first exemplary embodiment.

[0029] Reference Figure 2 , the first control unit 223 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, and a microprocessing unit (MPU)) and a memory (e.g., a dynamic random access memory (DRAM) and a static random access memory (SRAM)). These devices perform various processes to control the respective blocks of the imaging device 101 and to control data transmission between the blocks. The non-volatile memory (electrically erasable programmable read-only memory (EEPROM)) 216 stores constants and programs for the operation of the first control unit 223.

[0030] Reference Figure 2 , the zoom unit 201 includes a zoom lens for magnification. The zoom drive control unit 202 drives and controls the zoom unit 201. The focus adjustment unit 203 includes a lens for focus adjustment. The focus drive control unit 204 drives and controls the focus adjustment unit 203.

[0031] In the imaging unit 206, the image sensor receives incident light passing through the respective lens groups and outputs charge information corresponding to the light amount as analog image data to the image processing unit 207. The image processing unit 207 applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to the digital image data output by analog-to-digital (A / D) conversion, and outputs the digital image data after the application. The digital image data output from the image processing unit 207 is converted into a recording format such as the Joint Photographic Experts Group (JPEG) format by the image recording unit 208, and then is sent to the memory 215 and the video output unit 217 (described below).

[0032] The lens barrel rotation drive unit 205 drives the pitch rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the pitch and pan directions, respectively.

[0033] The device shake detection unit 209 is equipped with, for example, an angular velocity meter (gyro sensor) 106 for detecting the angular velocity in the three-axis directions of the imaging device 101 and an accelerometer (acceleration sensor) 107 for detecting the acceleration in the three-axis directions of the imaging device 101. The device shake detection unit 209 calculates the rotation angle and offset amount of the imaging device 101 based on the detected signals.

[0034] The audio input unit 213 acquires an audio signal collected from around the imaging device 101 by using a microphone provided on the imaging device 101, performs analog-to-digital conversion on the audio signal, and sends the digitized signal to the audio processing unit 214. The audio processing unit 214 performs optimization processing and other audio processing on the input digital audio signal. The first control unit 223 sends the audio signal processed by the audio processing unit 214 to the memory 215. The memory 215 temporarily stores the image signal and the audio signal obtained by the image processing unit 207 and the audio processing unit 214, respectively.

[0035] The image processing unit 207 and the audio processing unit 214 respectively read the image signal and the audio signal temporarily stored in the memory 215, and encode the image signal and the audio signal to generate a compressed image signal and a compressed audio signal. The first control unit 223 sends the compressed image signal and the compressed audio signal to the recording / reproducing unit 220.

[0036] The recording / reproducing unit 220 records the compressed image signal and the compressed audio signal generated by the image processing unit 207 and the audio processing unit 214, respectively, and other imaging control data in the recording medium 221. In the case where the audio signal is not compressed or encoded, the first control unit 223 sends the audio signal generated by the audio processing unit 214 and the compressed image signal generated by the image processing unit 207 to the recording / reproducing unit 220, and causes the recording / reproducing unit 220 to record these signals in the recording medium 221.

[0037] The recording medium 221 may be a recording medium built in the imaging device 101 or a removable recording medium. The recording medium 221 can record various types of data such as a compressed image signal, a compressed audio signal, and an audio signal generated by the imaging device 101. Therefore, a medium having a larger capacity than the non-volatile memory 216 is usually used. For example, the recording medium 221 includes a hard disk, an optical disk, a magneto-optical disk, a recordable optical disk (CD-R), a recordable digital versatile disk (DVD-R), a magnetic tape, a non-volatile semiconductor memory, a flash memory, and all other types of recording media.

[0038] The recording / reproducing unit 220 reads out (reproduces) the compressed image signal, the compressed audio signal, the audio signal, various types of data, and a program recorded in the recording medium 221. The first control unit 223 sends the read compressed image signal and the read compressed audio signal to the image processing unit 207 and the audio processing unit 214, respectively. The image processing unit 207 and the audio processing unit 214 respectively temporarily store the compressed image signal and the compressed audio signal in the memory 215, decode these signals according to a predetermined process, and send the decoded signals to the video output unit 217 and the audio output unit 218.

[0039] The audio input unit 213 is configured to obtain an audio signal through a plurality of microphones on the imaging device 101. The audio processing unit 214 detects the direction of sound on the plane where the microphone is installed and is used for searching and automatic imaging (as described below). The audio processing unit 214 further detects specific audio commands. Some audio commands are pre-registered commands. The user may be able to register a specific sound as an audio command in the imaging device 101. The audio processing unit 214 also performs sound scene recognition. In sound scene recognition, the audio processing unit 214 determines the sound scene by using a learning model that has completed machine learning based on a large amount of audio data. Specific machine learning algorithms include the nearest neighbor method, the naive Bayes method, decision trees, and support vector machines. Specific machine learning algorithms also include deep learning that automatically generates feature quantities and binds weight coefficients for learning by using neural networks. Any of the above available algorithms can be applied to this exemplary embodiment as needed.

[0040] According to this exemplary embodiment, a neural network for detecting specific scenes such as "cheering", "applauding", and "uttering" is provided in the audio processing unit 214. The audio processing unit 214 is configured to output a detection trigger signal to the first control unit 223 and the second control unit 211 when a specific sound scene or a specific audio command is detected.

[0041] More specifically, the neural network of the audio processing unit 214 prepares audio information for the "cheering", "applauding", and "uttering" scenes and performs learning by using the audio information as an input and the detection trigger signal as an output.

[0042] The second control unit 211, which is separately provided from the first control unit 223 that controls the entire main system of the imaging device 101, controls the power supply to the first control unit 223.

[0043] The first power supply unit 210 and the second power supply unit 212 are respectively powered to operate the first control unit 223 and the second control unit 211. When the power button on the imaging device 101 is pressed, power is supplied to both the first control unit 223 and the second control unit 211. As described below, the first control unit 223 is controlled to turn off the power supply from the first control unit 223 itself to the first power supply unit 210. Even when the operation of the first control unit 223 does not work, the second control unit 211 is still working to receive information from the device shake detection unit 209 and the audio processing unit 214. The second control unit 211 performs a process for determining whether to start the first control unit 223 based on various input information. When the second control unit 211 determines to start the first control unit 223, the second control unit 211 issues a power supply instruction to the first power supply unit 210. According to this exemplary embodiment, the first power supply unit 210 and the second power supply unit 212 are powered by a battery. More specifically, the imaging device 101 is also a portable terminal.

[0044] The audio output unit 218 outputs a preset audio mode from the built-in speaker of the imaging device 101 during imaging, for example.

[0045] The light-emitting diode (LED) control unit 224 controls a preset illumination / flashing mode of the LED provided on the imaging device 101 during imaging, for example.

[0046] The video output unit 217 includes, for example, a video output terminal and sends an image signal to display a video image on a connected external display. The audio output unit 218 and the video output unit 217 can be a combined terminal such as a terminal such as a high-definition multimedia interface.

[0047] The communication unit 222 performs communication transmission and reception of data such as audio signals, image signals, compressed audio signals, and compressed image signals between the imaging device 101 and an external device. The communication unit 222 also receives imaging control signals including imaging start and end commands and pan drive, tilt drive, and zoom drive, etc., and drives the imaging device 101 based on instructions from an external device capable of communicating with the imaging device 101. Information such as various parameters related to learning to be processed by the learning processing unit 219 (described below) is sent and received between the imaging device 101 and the external device. Examples of the communication unit 222 include an infrared communication module, a communication module, a wireless local area network (LAN) communication module, a wireless universal serial bus (USB), a global positioning system (GPS) receiver, and other wireless communication modules.

[0048] <System configuration with an external communication device>

[0049] Figure 3 An example configuration of a wireless communication system between a camera device 101 and an external device, a smart device 301, is shown. The camera device 101 is a digital camera having a photographing function, and the external device 301 (hereinafter referred to as the smart device 301) is a smart device including a Bluetooth communication module and a wireless LAN communication module.

[0050] The camera device 101 and the smart device 301 can communicate with each other through communication 302 based on a wireless LAN conforming to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard series, and communication 303 such as low power consumption (hereinafter referred to as BLE). The wireless LAN and BLE are examples of communication methods. Each communication device has at least two different communication functions. In communication based on the relationship between a control station and a slave station, other communication methods can be used as long as one communication function can control the other communication function. Without loss of generality, it is assumed that, compared with the second communication such as BLE, the first communication such as a wireless LAN achieves communication at a higher communication rate, and that, compared with the first communication, the second communication provides at least one of lower power consumption and a shorter communication distance.

[0051] <Configuration of External Communication Device>

[0052] Reference will be made to Figure 4 the configuration of the smart device 301 as an example of an external communication device. The smart device 301 is a so-called mobile phone, i.e., a portable terminal.

[0053] In addition to including a wireless LAN control unit 401 for wireless LAN and a BLE control unit 402 for BLE, the intelligent device 301 further includes, for example, a public communication line control unit 406 for public wireless communication. The intelligent device 301 also includes a packet transmission / reception unit 403. The wireless LAN control unit 401 performs RF control for the wireless LAN, communication processing, a driver for performing various controls on wireless LAN communication conforming to the IEEE 802.11 standard series, and protocol processing related to wireless LAN communication. The BLE control unit 402 performs RF control for BLE, communication processing, a driver for performing various controls in BLE communication, and protocol processing related to BLE communication. The public communication line control unit 406 performs RF control for public wireless communication, communication processing, a driver for performing various controls on public wireless communication, and protocol processing related to public wireless communication. The public wireless communication conforms to, for example, the International Multimedia Telecommunications (IMT) standard and the Long Term Evolution (LTE) standard. The packet transmission / reception unit 403 processes at least one of packet transmission and reception related to wireless LAN, BLE communication, and public wireless communication. This exemplary embodiment will be described on the premise that the intelligent device 301 performs at least one of packet transmission and reception during communication. However, communication formats such as circuit switching other than packet switching are also applicable.

[0054] The intelligent device 301 also includes, for example, a control unit 411, a storage unit 404, a Global Positioning System (GPS) reception unit 405, a display unit 407, an operation unit 408, an audio input / audio processing unit 409, and a power supply unit 410. The control unit 411 controls the entire intelligent device 301 by executing a control program stored in the storage unit 404. The storage unit 404 stores, for example, a control program executed by the control unit 411 and various information such as parameters required for communication. When the control unit 411 executes the control program stored in the storage unit 404, various operations (described below) are realized.

[0055] The power supply unit 410 supplies power to the intelligent device 301. The display unit 407 has a function of outputting information that can be visually recognized by using a liquid crystal display (LCD) and an LED or outputting sound by using a speaker, and displays various information. The operation unit 408 includes, for example, buttons for receiving operations of the user on the intelligent device 301. The display unit 407 and the operation unit 408 can be configured by a common component such as a touch panel.

[0056] The audio input / audio processing unit 409 may be configured to obtain the user's voice from a general microphone built in the smart device 301, and obtain the user's operation instruction through speech recognition processing.

[0057] The audio input / audio processing unit 409 also obtains an audio command through the user's voice via a dedicated application in the smart device 301. The audio input / audio processing unit 409 may register the audio command as a specific audio command for instructing the audio processing unit 214 of the imaging device 101 to identify a specific audio command through communication 302 via a wireless LAN.

[0058] The GPS receiving unit 405 receives a GPS signal transmitted from a satellite, and analyzes the GPS signal to estimate the current position (longitude and latitude information) of the smart device 301. Alternatively, the GPS receiving unit 405 may estimate the current position of the smart device 301 based on information obtained from surrounding wireless networks by using a Wi-Fi positioning system (WPS). In a case where the acquired current GPS position information indicates a position within a preset position range (within a predetermined radial range), the smart device 301 notifies the imaging device 101 of motion information via the BLE control unit 402 for use as a parameter (as described below) for automatic imaging and automatic editing. On the other hand, in a case where the GPS position information indicates a predetermined or greater position change, the smart device 301 notifies the imaging device 101 of motion information via the BLE control unit 402 for use as a parameter (as described below) for automatic imaging and automatic editing.

[0059] As described above, the smart device 301 exchanges data with the imaging device 101 in communication via the wireless LAN control unit 401 and the BLE control unit 402. For example, the smart device 301 transmits and receives data such as audio signals, image signals, compressed audio signals, and compressed image signals. The smart device 301 also issues a shooting instruction and other operation instructions, transmits audio command registration data, and issues a predetermined position detection notification and a location movement notification based on the GPS position information. The smart device 301 also transmits and receives data for learning via a dedicated application in the smart device 301.

[0060] <Shooting operation sequence>

[0061] Figure 5 is a flowchart showing the automatic shooting process of the imaging device 101 according to the present exemplary embodiment.

[0062] When the user operates the power button provided on the imaging device 101, the processing of the flowchart starts. According to the present exemplary embodiment, a wireless communication connection is constantly established between the imaging device 101 and the intelligent device 301, and various operations can be performed through a dedicated application on the intelligent device 301. The processing of each step in the flowchart is implemented by the first control unit 223 that controls each unit of the imaging device 101.

[0063] In step S501, the first control unit 223 determines whether the automatic imaging is currently stopped. The stopping of the automatic imaging will be described below with reference to the flowchart of the speech recognition processing. In the case where the automatic imaging is stopped (Yes in step S501), the first control unit 223 waits without performing any processing until the stopping of the automatic imaging is released. On the other hand, in the case where the automatic imaging is not stopped (No in step S501), the processing proceeds to step S502. In step S502, the first control unit 223 performs image recognition processing.

[0064] In step S502, the first control unit 223 causes the image processing unit 207 to perform image processing on the signal captured by the imaging unit 206 to generate an image for subject recognition.

[0065] The image processing unit 207 performs subject recognition such as human and object recognition based on the generated image.

[0066] In the case of recognizing a person, the image processing unit 207 detects the face and body of the subject. In the face detection processing, a predefined pattern for judging a human face enables the position consistent with the pattern included in the captured image to be detected as a human face image.

[0067] The image processing unit 207 also simultaneously calculates the reliability indicating the probability of the face of the subject. For example, the reliability is calculated based on the size of the face area in the image and the degree of consistency with the face pattern.

[0068] Similarly in object recognition, an object consistent with a pre-registered pattern can be recognized.

[0069] There is also a method of extracting a feature subject by using the histogram of the hue or saturation in the captured image. In this case, for the image of the subject captured in the imaging perspective, the first control unit 223 divides the distribution derived from the histogram of the hue or saturation into a plurality of intervals, and classifies the captured images for each interval.

[0070] For example, a histogram having a plurality of color components is generated for the captured image. Then, the histogram is divided into mound-shaped distribution ranges, and the captured images are classified by the regions belonging to the same interval combination. Then, the image region of the subject is recognized.

[0071] By calculating the evaluation value for each image region of the recognized subject, the image region of the subject with the highest evaluation value can be determined as the main subject region.

[0072] The above method can obtain each subject information from the imaging information.

[0073] In step S503, the first control unit 223 calculates the amount of image shake correction. More specifically, the first control unit 223 calculates the absolute angle of the imaging device 101 based on the angular velocity and acceleration information acquired by the device shake detection unit 209. Then, the first control unit 223 obtains the image stabilization angle used to move the pitch rotation unit 104 and the pan rotation unit 105 in the angular direction for canceling the absolute angle, and determines this image stabilization angle as the amount of image shake correction. For this process of calculating the amount of image shake correction, the calculation method can be changed by a learning process (described below).

[0074] In step S504, the first control unit 223 determines the state of the imaging device 101. The first control unit 223 determines the current vibration and movement state of the imaging device 101 based on the angle and movement amount detected based on the angular velocity information, acceleration information, and GPS position information.

[0075] For example, when the imaging device 101 is attached to a motor vehicle to capture an image, the subject information such as the surrounding scenery changes significantly due to the moving distance.

[0076] This enables the first control unit 223 to determine whether the current state is a "vehicle movement state" where the imaging device 101 is attached to a motor vehicle and moving at a high speed. This state can be used for automatic subject search (described below).

[0077] The first control unit 223 determines whether the angle changes significantly to determine whether the current state is a "still image shooting state" where the imaging device 101 has almost no shake angle.

[0078] In the "still image shooting state", since it is considered that the imaging device 101 itself has no angle change, subject search can be performed for still image shooting.

[0079] In the case where the angle changes relatively largely, the first control unit 223 determines it as a "handheld image shooting state", and subject search can be performed for handheld image shooting.

[0080] In step S505, the first control unit 223 performs a subject search process. The subject search includes the following processes:

[0081] (1) Region division

[0082] will refer to Figure 8A 、 8B 、8C, and 8D to describe the area division. As Figure 8A shown, the area division is performed on the entire circumference centered on the position of the imaging device 101 (assuming the origin O is the position of the imaging device 101). In Figure 8A the example of, the area division is performed at intervals of 22.5 degrees in each pitch and pan direction. In Figure 8A the area division of, as the angle in the pitch direction increases from 0 degrees, the horizontal circumference decreases to reduce the area region. Therefore, in the imaging device 101 according to this exemplary embodiment, as Figure 8B shown, when the pitch angle is 45 degrees or greater, the area range is set to be greater than 22.5 degrees. Figure 8C and 8D show examples of the area division in the imaging view. Axis 1301 is the orientation of the imaging device 101 at initialization. The area division is performed using this direction angle as the reference position. Figure 8D shows an example of an image captured in the view area 1302. In the image captured from this view, the image division is performed based on the area division shown in the divided areas 1303 to 1318 as Figure 8D shown.

[0083] (2) Calculate the importance level of each area

[0084] For each of the divided regions described above, an importance level indicating the priority order during search is calculated based on the subject present in the region and the scene state in the region. For example, the importance level based on the subject state is calculated based on the number of people in the region, the face size and orientation of each person, the probability of detecting a face, the facial expression of each person, and the personal authentication result. Examples of the importance level based on the scene state include general object recognition results, scene judgment results (e.g., blue sky, backlight, and night scene), the sound level from the region direction and speech recognition results, and information on motion detection in the region. In the determination of the state of the imaging device 101 in step S504, the vibration state of the imaging device 101 is detected, and the importance level can be made changeable based on the vibration state. For example, in the case where the current state is determined to be the "still image shooting state", the importance level is determined to be high when a specific person's face authentication is detected, so as to perform subject search among the subjects with high priority registered in the face authentication (e.g., the user of the imaging device 101). In addition, automatic imaging is also performed in the priority order of the faces (as described below). Even if the user of the imaging device 101 holds the imaging device 101 for a long time and performs imaging, the user can record many images including the user by removing the imaging device 101 and placing it on a table or the like. Since subject search can be performed by panning / tilting operations in this case, the user can simply record images including the user and group photos including many faces by appropriately setting the imaging device 101 regardless of the setting angle of the imaging device 101. Only under the above conditions, as long as each region remains unchanged, the region with the highest importance level remains unchanged. As a result, the searched region will remain unchanged. Therefore, the importance level is changed according to past imaging information. More specifically, for a region that has been continuously designated as a search region within a predetermined period, the importance level can be lowered, or for a region where imaging is performed in step S513 (described below), the importance level can be lowered within a predetermined period.

[0085] (3) Determine the search target region

[0086] In the case of calculating the importance level for each region as described above, the region with a high importance level is determined as the search target region, and then the panning / tilting search target angle required to grasp the search target region in the viewing angle is calculated.

[0087] In step S506, the first control unit 223 performs panning / tilting driving. More specifically, the first control unit 223 increases the driving angle in the control sampling based on the image shake correction amount and the panning / tilting search target angle to calculate the panning / tilting driving amount. Then, the lens barrel rotation driving unit 205 controls the driving of the tilt rotation unit 104 and the panning rotation unit 105.

[0088] In step S507, the first control unit 223 controls the zoom unit 201 to perform zoom driving. More specifically, the first control unit 223 performs zoom driving based on the state of the search target subject determined in step S505. For example, if the face of a person as the search target subject is too small (i.e., smaller than the minimum detectable size), the face cannot be detected or may be lost. In this case, the first control unit 223 controls to increase the size of the face in the image by zooming in the telephoto side. On the other hand, if the face of a person is too large in the image, the subject is likely to be out of the viewing angle due to the movement of the subject or the imaging device 101 itself. In this case, the first control unit 223 controls to reduce the size of the face on the screen by zooming in the wide-angle direction. Performing zoom control in this way can keep the subject in a state suitable for tracking.

[0089] Although in steps S505 to S507, the first control unit 223 performs subject search by panning / tilting and zoom driving, subject search can also be performed by an imaging system that uses multiple wide-angle lenses to capture images in all directions at once. For an omnidirectional camera, image processing such as subject detection that uses all signals obtained during imaging as input images requires an extremely large amount of processing. Therefore, the first control unit 223 is configured to crop a part of the image, and then perform subject search processing within the range of the cropped image. The first control unit 223 calculates the importance level for each region as described above, changes the cropping position based on the importance level, and makes a determination for automatic imaging (described below). This makes it possible to reduce the power consumption of image processing and perform high-speed subject search.

[0090] In step S508, the first control unit 223 reads a frequency parameter as a setting value indicating the ease of automatic imaging. The user can set the frequency parameter to any frequency, that is, select a frequency from the options of "low", "medium", and "high" via a dedicated application of the intelligent device 301. Compared with the number of images captured when the frequency parameter is set to "low", setting the frequency parameter to "high" can capture a larger number of images within a predetermined time period. The number of images captured when the frequency parameter is set to "medium" is between the number of images captured when the frequency parameter is set to "low" and the number of images captured when the frequency parameter is set to "high". This setting can be automatically changed by frequency setting processing (described below).

[0091] In step S509, the first control unit 223 determines whether the read frequency parameter is a predetermined value. For example, when "highest" is set as the frequency of automatic imaging (yes in step S509), the process proceeds to step S510. On the other hand, when "highest" is not set (no in step S509), the process proceeds to step S512. The frequency setting process (described below) automatically selects the frequency setting "highest". In the normal frequency setting by the user using the dedicated application of the intelligent device 301, as described above, the frequency is selected from the options of "low", "medium", and "high". In other words, in the setting by the user operation, the frequency is not set to "highest".

[0092] In step S510, the first control unit 223 determines whether the frequency increase time has ended since the frequency parameter was set to "highest" in step S705 (described below) until the setting returns to the original setting. When the frequency increase time has ended (yes in step S510), the process proceeds to step S511. On the other hand, when the frequency increase time has not ended (no in step S510), the process proceeds to step S512.

[0093] In step S511, since the frequency increase time has ended, the first control unit 223 restores the frequency parameter to the frequency setting before "highest" was selected. When a predetermined number or more of images are captured by automatic imaging during the frequency increase time, the first control unit 223 determines the current scene as the target scene, and thus the frequency increase time can be extended. This process enables continuous shooting of the scene desired by the user.

[0094] In step S512, the first control unit 223 determines whether to perform automatic imaging.

[0095] Now, the determination of automatic imaging will be described. The first control unit 223 determines whether to perform automatic imaging by judging whether the importance score exceeds a predetermined value. The importance score refers to a parameter used to judge whether to perform automatic imaging and is different from the importance level used to determine the search area. Based on the subject detection state and the passage of time, scores are added to the importance score. Assume an example case where automatic imaging is designed to be performed when the importance score exceeds 2000 points. In this case, the importance score is initially 0 points, and scores are added to the importance score based on the passage of time since entering the automatic imaging mode. For example, if there is no subject with high priority, the importance score increases at the following increase rate, that is, the importance score reaches 2000 points within 120 seconds. If no subject with high priority is detected after 120 seconds, the importance score reaches 2000 points through the addition of scores over time. Then, imaging is performed. When a subject with high priority is detected during the passage of time, 1000 points are added. Therefore, in the state where a subject with high priority is detected, the importance score may reach 2000 points. As a result, the imaging frequency may increase.

[0096] For example, when a smile of the subject is recognized, 800 points are added. This smile-based score addition is performed even for a subject without high priority. Although this exemplary embodiment will be described with an example case where the smile-based score addition is the same regardless of whether the subject has high priority, the present invention is not limited thereto. For example, the smile-based score addition for detecting a subject with high priority may be higher than that for detecting a subject without high priority. This enables imaging to be performed more in accordance with the user's intention to a greater extent. Through the addition of scores accompanying the facial expression changes of these subjects, automatic imaging is performed when the importance score exceeds 2000 points. Even if the importance score does not exceed 2000 points through the addition of scores accompanying facial expression changes, the importance score reaches 2000 points within a shorter time period through the addition of scores over the subsequent time.

[0097] An exemplary embodiment will be described with an example case of adding points over time, where points are added such that the importance score reaches 2000 points within 120 seconds (adding 2000 / 120 points per second), that is, adding points linearly with respect to time. However, it is not limited thereto. In a possible example of the point-adding method, points are not added until 110 seconds out of 120 seconds, and 200 points can be added per second in the subsequent 10 seconds (from 110 seconds to 120 seconds). By increasing the importance score in this way, it is possible to prevent the importance score from reaching the target score regardless of the priority level due to the addition of points for the facial expression change of the subject. In the case of the point-adding method where the importance score increases linearly over time, points are added over a long period of time. Therefore, in this case, even if points are added for the facial expression change accompanied by a low-priority subject, the importance score generally reaches the target score, so it is relatively difficult to reflect the priority level. On the other hand, in the case of low point addition accompanied by facial expression changes, the moment of facial expression change may be missed. Since this point-adding method needs to be avoided, points are not added until 110 seconds. Using this point-adding method, points are not added to the subject with low priority until 110 seconds. On the other hand, since 1000 points are added when a subject with high priority is detected, even if point addition over time is not performed until 110 seconds, 1000 points are added. Therefore, in the case of adding points accompanied by facial expression changes, compared with the subject with high priority, the subject with low priority makes it possible to limit the possibility of the score reaching the score for imaging, thus allowing the level of priority to take effect. Although the addition of points accompanied by facial expression changes has been described above, the criteria for adding points are not limited thereto. Examples of the criteria for adding points include the case where the voice volume increases and the case where the amount of postures and gestures increases. In order to allow the level of priority to take effect for these criteria, it is only necessary to provide the differences in the above-described point-adding methods.

[0098] Even if the importance score does not exceed 2000 points through the actions of the subject, imaging is performed over time until 120 seconds, so it is impossible not to perform imaging for a certain period of time.

[0099] If a subject is detected midway, the time for increasing the importance score can start earlier within 120 seconds. More specifically, for example, if a subject with high priority is detected at 60 seconds, 1000 points are added, but the importance score does not exceed 2000 points. However, in this case, the first control unit 223 can start linearly increasing the importance score 30 seconds after the subject is detected, instead of adding points until 110 seconds. Alternatively, the first control unit 223 can start linearly increasing the importance score 20 seconds before 120 seconds instead of 10 seconds before 120 seconds. Increasing the importance score in this way increases the likelihood of capturing a subject with high priority, making it easier to achieve imaging according to the user's intention.

[0100] When automatic imaging is performed, the importance score is reset to 0 points. Automatic imaging will not be performed until the importance score exceeds 2000 points again.

[0101] The frequency parameter is used to control the method of increasing the importance score over time. The frequency parameter is set such that it takes 120 seconds until automatic imaging in the case where no subject is detected in the above example. Although the frequency parameter is "medium" in this example, the first control unit 223 changes the method of increasing the importance score such that automatic imaging is performed at 60 seconds in the frequency boost state (frequency parameter is "highest"). For example, in this case, the first control unit 223 can add 2000 / 60 points per second, or add no points until 55 seconds and then add 400 points per second from 55 seconds to 60 seconds. The advantages of the latter case are as described above. Example designs for other frequency parameters are as follows. In the case where the frequency parameter is "high", the first control unit 223 increases the importance score such that the importance score reaches 2000 points within 100 seconds. In the case where the frequency parameter is "low", the first control unit 223 increases the importance score such that the importance score reaches 2000 points within 240 seconds. In the case where the frequency parameter is "highest", at least one image is captured in the shortest time (e.g., 60 seconds according to this exemplary embodiment). Therefore, increasing the imaging frequency means increasing the number of images to be captured per unit time by changing the point-adding method, while decreasing the imaging frequency means decreasing the number of images to be captured per unit time by changing the point-adding method.

[0102] This completes the description of the determination of whether to perform automatic imaging. When the first control unit 223 determines to perform automatic imaging (Yes in step S512), the process proceeds to step S513. On the other hand, when the first control unit 223 determines not to perform automatic imaging (No in step S512), the process returns to step S501.

[0103] In step S513, the first control unit 223 performs imaging processing. The imaging processing includes still image capture and moving image capture.

[0104] Figure 6 FIG. is a flowchart showing the voice recognition processing of the imaging device 101 according to the present exemplary embodiment. When the sound emitted by the user is input to the built-in microphone of the imaging device 101, the audio input / audio processing unit 409 performs voice recognition processing to obtain a user operation instruction.

[0105] In step S601, the first control unit 223 determines whether a wake word is detected. The wake word refers to a start command for enabling audio command recognition, and the audio command recognition is used to issue a specific audio instruction to the imaging device 101. When an audio instruction is issued, the first control unit 223 needs to receive a command word after wake word recognition to successfully recognize. In the case where a wake word is detected (Yes in step S601), the process proceeds to step S602. On the other hand, in the case where a wake word is not detected (No in step S601), the process repeats step S601 until a wake word is detected.

[0106] In step S602, the first control unit 223 stops the automatic imaging processing. When the first control unit 223 recognizes the wake word, the imaging device 101 enters the command word waiting state and stops the automatic imaging processing. Stopping the automatic imaging means performing subject search and imaging processing using pan / tilt and zoom operations. The purpose of stopping the automatic imaging includes stopping the automatic imaging and entering the command word waiting state to quickly respond to the command word instruction that will be given immediately after the wake word. In the case where the user intends to issue an imaging instruction by using an audio instruction, the purpose of stopping the automatic imaging also includes stopping the pan / tilt operation to enable imaging in the direction desired by the user.

[0107] In step S603, the first control unit 223 generates a recognition sound to notify the user of the successful recognition of the wake word.

[0108] In step S604, the first control unit 223 determines whether a command word is detected. In the case where a command word is detected (Yes in step S604), the process proceeds to step S606. On the other hand, in the case where a command word is not detected (No in step S604), the process proceeds to step S605.

[0109] In step S605, the first control unit 223 determines whether a predetermined period of time has elapsed since the wake-up word was detected and the command word waiting state was entered. If the predetermined period of time has elapsed (Yes in step S605), the process returns to step S601. In step S601, the first control unit 223 stops the command word waiting state and enters the wake-up word waiting state. On the other hand, if the predetermined period of time has not elapsed (No in step S605), the process repeats step S604 until a command word is detected.

[0110] In step S606, the first control unit 223 determines whether the detected command word is a still image capture command. The still image capture command is a command for requesting the imaging device 101 to capture and record a still image. If the first control unit 223 determines that the command word is a still image capture command (Yes in step S606), the process proceeds to step S607. On the other hand, if the first control unit 223 determines that the command word is not a still image capture command (No in step S606), the process proceeds to step S608.

[0111] In step S607, the first control unit 223 performs still image capture processing. More specifically, the image processing unit 207 converts the signal captured by the imaging unit 206 into a JPEG file, and then the image recording unit 208 records the JPEG file in the recording medium 221.

[0112] In step S608, the first control unit 223 determines whether the detected command word is a subject search command. If the first control unit 223 determines that the command word is a subject search command (Yes in step S608), the process proceeds to step S609. On the other hand, if the first control unit 223 determines that the command word is not a subject search command (No in step S608), the process proceeds to step S611.

[0113] In step S609, the first control unit 223 performs subject search processing. If the search target area has been determined through the subject search processing in step S505, and the subject has been captured through the pan / tilt drive in step S506 and the zoom drive in step S507, the first control unit 223 stops tracking the subject and performs subject search processing to search for other subjects. If the user issues a subject search instruction while a subject is being captured, this instruction means that there is another subject in addition to the currently captured subject.

[0114] After the processing in steps S607 to S609 is completed, then in step S610, the first control unit 223 performs a frequency setting process. The frequency setting process refers to a process for setting a frequency parameter indicating the number of images to be captured within a predetermined period. The content of the process will be described in detail below. The frequency setting process in step S610 sets the imaging frequency to a higher value.

[0115] In step S611, the first control unit 223 determines whether the detected command word is a moving image recording start command. The moving image recording start command is a command that requests the imaging device 101 to capture and record a moving image. When the first control unit 223 determines that the command word is a moving image recording start command (Yes in step S611), the process proceeds to step S612. On the other hand, when the first control unit 223 determines that the command word is not a moving image recording start command (No in step S611), the process proceeds to step S613.

[0116] In step S612, the first control unit 223 starts capturing a moving image by using the imaging unit 206 and records the moving image in the recording medium 221. While recording the moving image, the first control unit 223 maintains the automatic imaging stop state without performing pan / tilt drive, zoom drive, and subject search.

[0117] In step S613, the first control unit 223 determines whether the detected command word is a moving image recording stop command. When the first control unit 223 determines that the command word is a moving image recording stop command (Yes in step S613), the process proceeds to step S614. On the other hand, when the first control unit 223 determines that the command word is not a moving image recording stop command (No in step S613), the process proceeds to step S615.

[0118] In step S614, the first control unit 223 stops capturing and recording the moving image by using the imaging unit 206 and completes recording the moving image file in the recording medium 221.

[0119] In step S615, the first control unit 223 performs other processing through an audio command. Examples of other processing include processing of a command for performing a pan / tilt operation in a user-specified direction and processing of a command for changing various imaging parameters including exposure correction.

[0120] In steps S616 and S617, the first control unit 223 performs a restart process for the automatic imaging stopped in step S602. By this operation, the processing in steps S502 to S510 is enabled and the automatic imaging is restarted.

[0121] In this case, in the case of a moving image recording start and a moving image recording stop instruction, the frequency setting process is not performed. This is because, since the signal from the imaging unit 206 is continuously recorded after the start of moving image recording, it is meaningless to set a high frequency. After stopping the moving image recording, the user issues a recording stop instruction indicating that the scene to be recorded has ended. Therefore, the frequency setting process is not performed to avoid useless imaging by unnecessarily setting a high frequency.

[0122] If the battery included in the imaging device 101 has a low remaining capacity, or if the temperature of the imaging device 101 reaches a predetermined temperature or higher due to heat generation, it is preferable to prevent the imaging unit 206 from operating frequently. In this state, the first control unit 223 may not set the frequency parameter to "highest" (as described below) in Figure 7 step S704.

[0123] Figure 7 is a flowchart showing the frequency setting process of the imaging device 101 according to the present exemplary embodiment. As a method of setting the frequency for automatically taking pictures for the user, the user can use a dedicated application in the smart device 301. The process of this flowchart also starts when Figure 6 step S610 is executed. The process also starts when the user issues a frequency change instruction via the dedicated application on the smart device 301.

[0124] In step S701, the first control unit 223 determines whether frequency setting is performed by using the dedicated application on the smart device 301. In the case where frequency setting is performed by using the dedicated application (Yes in step S701), the process proceeds to step S702. On the other hand, in the case where frequency setting is not performed by using the dedicated application (for example, in the case of performing frequency setting in step S610) (No in step S701), the process proceeds to step S703.

[0125] In step S702, the first control unit 223 sets the frequency parameter specified by the user. For example, the frequency parameter can be set by selecting "low", "medium", or "high" from the item of the automatic shooting frequency on the screen of the dedicated application on the smart device 301 as shown in Figure 9 .

[0126] Now, the application screen in Figure 9 will be described.

[0127] Through the dedicated application of the smart device 301, still images and moving images are prepared as the content to be automatically shot. In addition, as the content to be automatically shot, the user can set whether still images or moving images are prioritized through the dedicated application. AsFigure 9 As shown, this setting can be changed by touching (flicking) the knob of the slider. When the still image is set to be prioritized, the first control unit 223 captures more still images than moving images. When the moving image is set to be prioritized, the first control unit 223 captures more moving images than still images.

[0128] The user can also set the range for searching for the scene to be captured by the imaging device 101 as the degree of the angle relative to the front direction. In Figure 9 the example of, the user can set three different ranges: a range of 60 degrees (30 degrees in each of the right and left directions relative to the front direction), a range of 180 degrees (90 degrees in each of the right and left directions relative to the front direction), and a range of 360 degrees. A numerical value can be input to enable a finer range setting.

[0129] When performing automatic imaging, there is a concern of capturing too much content. Therefore, a function for automatically deleting images is provided. This function can be turned on or off by the smart device 301. For example, the images to be automatically deleted can be deleted in ascending order of the imaging date or ascending order of importance. For example, for still images, the importance refers to a parameter for numerically estimating whether the user will preferably record the image, such as whether the image has undergone little jitter or whether the image includes a person. For example, for moving images, the importance is numerically calculated based on whether the image includes a person or whether a human voice such as a dialogue is recorded in the image. The first control unit 223 processes a larger total numerical value as a higher importance.

[0130] This completes the Figure 9 description of the screen in. The will be described again Figure 7 .

[0131] In step S703, the first control unit 223 determines whether the frequency setting is called by voice recognition processing. When the first control unit 223 determines that the frequency setting is called by voice recognition processing (Yes in step S703), the process proceeds to step S704. On the other hand, when the first control unit 223 determines that the frequency setting is not called by voice recognition processing (No in step S703), the first control unit 223 ends the frequency setting process.

[0132] In step S704, the first control unit 223 sets the frequency parameter to a frequency higher than the frequency that can be set in step S702. This is because the timing when the user issues a shooting instruction is at least the timing of the shooting desired by the user. More specifically, the scene when the user issues a shooting instruction is the one desired by the user, so the scene desired by the user may occur during a close time period. Considering this, the imaging device 101 according to the present exemplary embodiment estimates a specific time period since the input of the audio command (triggered by the audio instruction of the user's audio command) as the scene to be shot, and increases the imaging frequency. This enables the user-desired image to be shot without losing the image. Although the present exemplary embodiment has been described above with respect to setting the "highest" frequency parameter, it is not limited thereto. Each time the frequency is set in the case of issuing an audio command instruction, the first control unit 223 can increase the frequency in a stepwise manner. In this case, the upper limit frequency is determined by the fastest frame rate of continuous imaging provided by the imaging device 101.

[0133] In step S705, the first control unit 223 sets the frequency boost time to the time period until the frequency parameter set to "highest" in step S704 returns to the original parameter, and then starts counting down. Assume an example case where the frequency is set to "highest" by an audio command instruction when the frequency is set to "medium". If the frequency boost time is 60 seconds, the frequency once set to "highest" returns to "medium" within 60 seconds (actual processing is performed in step S511). The frequency boost time in this case refers to the time period during which the "highest" frequency is maintained. Although the frequency boost time is automatically set, the user can set the frequency boost time to any set time.

[0134] In this case, instead of restoring the frequency boost time setting as time elapses for a predetermined time period, the first control unit 223 can restore the setting to the original setting according to whether a predetermined number of images are to be shot by automatic imaging.

[0135] In the case where the frequency is set to "highest" again by an audio command before the countdown of the frequency boost time ends, the first control unit 223 extends the predetermined time period until the frequency setting returns to the original setting, or increases the predetermined number of images.

[0136] In addition, the first control unit 223 can judge whether to restore the frequency setting to the original setting according to whether the subject search process has been performed omnidirectionally in the panning direction.

[0137] The above exemplary embodiments are based on the use of an audio command as a camera instruction from a user. Additionally, even when a camera instruction is issued from the smart device 301 or the BLE remote controller via the communication unit, the first control unit 223 can set the frequency setting to "highest" after executing the instruction. Further, even when an instruction for performing a process corresponding to a specific vibration pattern using the acceleration sensor in the camera device 101 is detected, the first control unit 223 can set the frequency setting to "highest" after executing the instruction. Moreover, even when a gesture instruction based on a gesture is received as a result of analyzing the movement of the user's hand via the camera unit, the first control unit 223 can set the frequency setting to "highest" after executing the instruction.

[0138] This exemplary embodiment tracks a subject by panning / tilting and zoom driving to capture an image desired by the user. Assume a possible implementation where an omnidirectional camera as the camera unit constantly captures images in all directions and crops a required image range from the captured images to obtain an image of the subject. In this case, the first control unit 223 constantly performs moving image recording, and when a crop instruction is input, records the moving image in a still image format and then increases the frame frequency of the moving image. Further, in this case, similar to the camera frequency according to the above exemplary embodiments, the first control unit 223 can set the highest settable frame frequency or set a value exceeding the settable value. Similar to the above exemplary embodiments, the condition for restoring the increased frame frequency can be based on the elapse of a predetermined period of time. This enables image recording at a higher frequency around the timing desired by the user, and accordingly makes it easier to capture an image of a moving object, for example, without focus jitter.

[0139] If the camera timing does not arrive within the frequency increase time, there may be a case where an image is not captured. To avoid this, when a still image capture command is received, the first control unit 223 captures an image without performing panning / tilting driving, zoom driving, and subject search. Subsequently, the first control unit 223 continuously captures three different images while performing subject search. Then, the first control unit 223 enters a frequency increase state and performs automatic camera shooting within a predetermined period of time. Operating in this way enables at least four different images to be captured when the user intends to issue a still image capture instruction by using the still image capture command, thus avoiding the case where an image is not captured.

[0140] <Set Priority>

[0141] Next, the setting of the subject priority will be described. The priority according to the present exemplary embodiment refers to a parameter for preferentially capturing an image of a specific person by pre-registering information about the specific person in the imaging device 101 as compared to other people.

[0142] First, the person registration screen in the dedicated application of the smart device 301 will be described. Figure 10A and 10B An example of the person registration screen displayed on the dedicated application of the smart device 301 is shown.

[0143] When the imaging device 101 according to the present exemplary embodiment recognizes a person, the first control unit 223 crops the facial part of the person and then sends the facial part to the smart device 301. If multiple people are recognized, the first control unit 223 crops the facial parts of these people and then sends the facial parts to the smart device 301. Whether the recognized person is pre-registered or not, the dedicated application of the smart device 301 lists the images of the facial parts in a graphical user interface (GUI) (not shown). In the case where the user selects one of the listed facial images, a person registration screen as shown in Figure 10A and 10B is displayed.

[0144] Referring to Figure 10A in the person registration screen, the facial image 1001 is the facial image of the person recognized by the imaging device 101. When the smart device 301 displays the transmitted data, the screen in Figure 10A is realized. A person registration screen is provided for each recognized person. The setting item 1002 is used to turn on or off the priority setting and is shown as a star mark in the example in Figure 10A . The default setting is non-priority setting, and the star mark is shown in gray. When the user touches the setting item 1002, the priority setting is performed, and the star mark changes to yellow indicating that the priority setting is turned on.

[0145] The name display box 1003 is an area for displaying the name of the recognized person. The default display of the name display box 1003 is blank. When the user selects this area, the name of the person in the facial image 1001 can be input. When the user inputs a name, the person is registered as a registered subject. The data of the subject registered by inputting a name in the name display box 1003 is not automatically deleted. On the other hand, the subject for which no name is input into the name display box 1003 before a predetermined period of time is automatically deleted.

[0146] The birthday frame 1004 is an area for displaying the birthday of the recognized person. The default display of the birthday frame 1004 is blank. When the user selects this area, the birthday of the person in the facial image 1001 can be input. For birthday input, a calendar for date selection can be displayed. When the user selects any date from the calendar, that date is input. Alternatively, for example, the user can specify the date by inputting a string. Although the date is specified in the format of month-day-year in Figure 10A , the format of month-day is also applicable.

[0147] Although the present exemplary embodiment has been described above with respect to the method of registering the birthday of the subject as a specific date associated with the registered subject, the specific date associated with the registered subject is not limited to the birthday. In addition, for one subject, multiple specific dates associated with the registered subject can be selected. Further, in addition to dates that come every year (such as birthdays, etc.), a single event such as an entrance ceremony and a wedding can also be set as a specific date associated with the registered subject.

[0148] The subject information input in this way is sent from the smart device 301 to the imaging device 101 via the wireless LAN control unit 401 or the BLE control unit 402. The imaging device 101 records the received subject information in the non-volatile memory 216.

[0149] Figure 11 Schematically shows the subject information stored in the imaging device 101. "Subject number" is a number that uniquely identifies the subject. "Priority setting" records the priority setting of the subject specified by the user in the Figure 10A screen. "Name" records the string specified by the user in the Figure 10A screen. "Birthday" records the date specified by the user in the Figure 10A screen.

[0150] Will refer to Figure 12 Describe the method for judging the priority of each subject performed by the imaging device 101. When the imaging device 101 recognizes a subject in the image captured during sequential imaging, the flowchart in Figure 12 is started.

[0151] In step S1201, the first control unit 223 obtains the recognized subject number from the non-volatile memory 216.

[0152] In step S1202, the first control unit 223 confirms the priority setting for each recognized subject number. When the priority setting is on (Yes in step S1202), the process proceeds to step S1203.

[0153] On the other hand, when the priority setting is set to off (No in step S1202), the process proceeds to step S1204.

[0154] In step S1203, the first control unit 223 sets the priority of the subject to priority level 1. In step S1204, the first control unit 223 sets the priority of the subject to priority level 0.

[0155] In step S1205, the first control unit 223 determines whether the date is a specific date for each recognized subject number. More specifically, the imaging device 101 according to the present exemplary embodiment compares the date of the birthday corresponding to each subject number recorded in the non-volatile memory 216 with the current date to determine whether the two dates are the same. If the two dates are the same, the first control unit 223 determines that the current date is a birthday. More specifically, when the first control unit 223 determines that the current date is a specific date (Yes in step S1205), the process proceeds to step S1206. On the other hand, when the first control unit 223 determines that the current date is not a specific date (No in step S1205), the process ends.

[0156] In step S1206, the first control unit 223 adds 0.5 to the priority of the subject.

[0157] In the above description, the priority value during priority setting is 1, and the increase amount of the priority caused by the birthday is 0.5. When the increase amount of the priority caused by the birthday is lower than the priority setting value caused by the priority setting, the first control unit 223 maintains a state in which the subject under the priority setting by the user is given a higher priority than the priority of the subject having a birthday today. However, the priority value to be set is not limited to this. For example, when the priority setting value caused by the priority setting is equal to the increase amount of the priority caused by the birthday, the first control unit 223 captures the same number of images of the subject under the priority setting by the user and the subject not under the priority setting (today is the birthday). When the increase amount of the priority caused by the birthday is higher than the priority setting value caused by the priority setting, the first control unit 223 captures an image of the subject having a birthday today prior to the subject under the priority setting by the user. The increase amount of the priority may not be a fixed value. For example, the first control unit 223 may use a value calculated based on other information including the number of registered subjects, the number of subjects in the viewing angle, the smiling degree and face orientation of the subject, and the volume of the voice of the subject. The first control unit 223 may also use a value that changes over time.

[0158] Although the present exemplary embodiment has been described above in terms of the method of adding priority, the method of relatively subtracting the priority of other subjects is also applicable.

[0159] No special setting of the priority value is required in terms of processing. As a result of executing a predetermined algorithm, the system can be configured to operate based on a priority setting similar to the priority setting according to the present exemplary embodiment.

[0160] Next, the process of determining the target positions of the lens drive and the lens barrel rotation drive by using the priorities calculated in the above process will be described.

[0161] The purpose of using priorities in automatic imaging is to capture more images of more important subjects in various ways and with a preferred composition. One possible method of achieving this purpose is to use priorities in subject tracking, controlling the number of images to be captured for each subject, subject search, composition adjustment, and determining the imaging timing.

[0162] First, the use of priorities in subject tracking will be described. Subject tracking refers to the operation of constantly accommodating a specific subject within the viewing angle. More specifically, the first control unit 223 controls the pitch rotation unit 104 and the pan rotation unit 105 so that the lens barrel 102 maintains the direction toward the subject. According to the present exemplary embodiment, the first control unit 223 uses subject priorities when determining the tracking target subject. More specifically, the first control unit 223 determines the subject with the highest priority among the subjects detected in the viewing angle as the tracking target.

[0163] The first control unit 223 can use the information about controlling the number of images to be captured (described below) to determine the tracking target. In this case, the first control unit 223 considers the subject priority and the previous number of images captured for each subject to select the tracking target.

[0164] Next, reference will be made to Figure 13Describe the use of priority in controlling the number of images to be captured. The first control unit 223 determines the imaging target subject by using the ratio of the priorities of the respective subjects as the target value for the ratio of the number of images to be captured. The first control unit 223 compares the ratio of each subject included in the previously captured image stored in the recording medium 221 with the priority of each subject, and identifies the subject with the smallest imaging ratio relative to the target value as the imaging target. When the determined imaging target subject exists in the viewing angle, the first control unit 223 captures an image after performing composition adjustment (described below). Although this example uses the ratio of the subject relative to all past images, the present invention is not limited thereto. Examples of other methods include a method using the ratio of the subjects captured in the most recent 100 imaging operations, a method using the latest information, and a method using the ratio of the imaging operations on the day.

[0165] In addition to the method using the ratio of the subjects captured in the past, other applicable methods also include a method of sequentially capturing images for each priority. For example, in the case where there is a subject A with a priority of 1 and a subject B with a priority of 0.5, a specific method changes the imaging target subject such that the subject A is captured twice and then the subject B is captured once.

[0166] Next, the use of priority in composition adjustment will be described. In one method, the first control unit 223 adjusts the viewing angle during the period from tracking the imaging target subject until imaging to obtain a better composition. First, in order to mainly capture the subject with a high priority, the first control unit 223 performs pan control and tilt control to hold the subject with the highest priority among the subjects in the viewing angle at the center of the image. In another method, the first control unit 223 may calculate the center of gravity of the priorities of the subjects in the viewing angle, and then hold the position of the center of gravity at the center of the image. The first control unit 223 may switch between these two methods. Then, in focus adjustment, the first control unit 223 focuses on the subject with the highest priority among the subjects in the viewing angle. Then, in order to adjust the size of the subject in the viewing angle, the first control unit 223 adjusts the zoom position to achieve an optimal size of the subject with the highest priority among the subjects in the viewing angle. Examples of applicable methods for adjusting the size of the subject include a method using the face size, a method using the body size, and a method using the size of a part of the body. In the case where many subjects exist inside and outside the viewing angle, examples of applicable methods for adjusting the composition include a method of adjusting the direction and zoom position of the camera to maximize the total priority value of the subjects included in the viewing angle.

[0167] Next, the use of priority in imaging timing will be described. The imaging timing is determined based on the importance score. In Figure 5The description of the determination of whether to perform automatic imaging in step S512 above describes an algorithm for determining the importance score. When there is a subject with a high priority in the viewing angle, this algorithm adds points. This exemplary embodiment assumes three different priority modes: 0, 0.5, and 1. For example, this algorithm adds 0 points for priority 0, 500 points for priority 0.5, 1000 points for priority 1, and 1500 points for priority 1.5. In this way, the first control unit 223 differentiates the possibility that the importance score reaches 2000 points according to the priority. As a result, the imaging frequency increases in the order of the priority.

[0168] In the above example, if a birthday is set in the Figure 10A screen, the priority based on the birthday is automatically changed. For example, in addition to the birthday input, the user can also select whether to change the priority based on the birthday. For example, display Figure 10B the person registration screen shown in. In addition to the Figure 10A screen, this screen also includes a birthday priority setting item 1005 for turning on or off the birthday priority setting. The default setting is non-birthday priority setting, and the star mark is displayed in gray. In this state, even if a birthday is input, the priority remains unchanged. When the user touches the birthday priority setting item 1005, the birthday priority setting is performed, and the star mark changes to yellow indicating that the birthday priority setting is turned on. To implement this process, a step for determining whether to perform the birthday priority setting is provided before step S1205 in the Figure 12 process. When the first control unit 223 determines that the birthday priority setting is to be performed (yes in step S1205), the process proceeds to step S1206. On the other hand, when the first control unit 223 determines that the birthday priority setting is not performed (no in step S1205), the process ends.

[0169] According to this exemplary embodiment, in a system including the imaging device 101 and the intelligent device 301, the first control unit 223 automatically changes the priority of the subject based on the date and time information associated with the subject. This enables automatic imaging to be achieved to a greater extent according to the user's intention.

[0170] A second exemplary embodiment will be described with respect to an imaging device that can select whether to add a priority for a specific date to each subject, and an imaging device that can set the increase amount of the subject priority on a specific date in a multi-level manner. The system configuration according to the second exemplary embodiment is similar to the system configuration according to the first exemplary embodiment, and its redundant description will be omitted. The second exemplary embodiment will be described around the differences from the first exemplary embodiment.

[0171] Figure 14Shows a person registration screen according to this exemplary embodiment. This screen is displayed in a manner similar to the screen according to the first exemplary embodiment. Regions identical to the regions in the Figure 10A and 10B screens are assigned the same reference numerals. Different from the screens in Figure 10A and 10B , the screen in Figure 14 displays item 1401 for changing the priority setting in a stepwise manner and item 1402 for changing the birthday priority setting in a stepwise manner. Items 1401 and 1402 are represented by five star marks arranged in a row. Each time the user touches each item, the priority setting changes in a stepwise manner. The default setting is that the priority setting is off. Touching each item raises the priority setting by one level. When the user touches each item when the upper limit is reached, the setting returns to the default setting. Alternatively, the setting can be changed to the level corresponding to the position of the touched item. In the example of Figure 14 , the priority setting changes to level 3, and the birthday priority setting changes to level 5. In this way, the current stage setting is displayed by changing the color of the star marks set for each stage.

[0172] According to this exemplary embodiment, priority settings 0.2, 0.4, 0.6, 0.8, and 1.0 are assigned to the star marks from left to right. More specifically, in step S1203 in Figure 12 , instead of setting the priority to 1, the first control unit 223 sets the priority corresponding to the set level. In step S1206 in Figure 12 , instead of adding 0.5 to the priority, the first control unit 223 adds the priority corresponding to the set level.

[0173] Although five levels are used in the above example, the present invention is not limited thereto. The number of levels between the priority setting and the birthday priority setting can be different.

[0174] Examples of other applicable methods besides the method of using multiple star marks include methods of setting values by using a search bar, a drop-down menu, or inputting a numerical value.

[0175] Although this exemplary embodiment has been described above around the method of registering the birthday of a subject as a specific date associated with the registered subject, the specific date associated with the registered subject does not have to be limited to the birthday. In addition, for one subject, multiple specific dates associated with the registered subject can be selected. Moreover, besides dates that come every year (such as birthdays, etc.), a single event such as an enrollment ceremony and a wedding can be set as the specific date associated with the registered subject.

[0176] By setting the increment of the priority in a multi-level manner as described above, the user can make more refined settings for the photographed subject.

[0177] A third exemplary embodiment will be described with respect to an imaging device that can set an increase amount of priority for each event using an event including a plurality of subjects as a specific date. The system configuration according to the third exemplary embodiment is similar to the system configuration according to the first exemplary embodiment, and redundant descriptions thereof will be omitted. The third exemplary embodiment will be described with respect to the differences from the first exemplary embodiment.

[0178] Figure 15 An event setting screen that is a screen of a dedicated application of the intelligent device 301 according to the present exemplary embodiment is shown. The dedicated application of the intelligent device 301 displays an item for displaying the event setting screen through a top screen (not shown). The user can display the event setting screen by touching the item.

[0179] In Figure 15 the event setting screen, a date setting area 1501 enables the user to set a specific date common to a plurality of registered subjects. The default setting is blank. When the user selects this area, a calendar for date selection will be displayed. When the user selects an arbitrary date from the calendar, that date will be input. Although the above calendar method is provided as an example of a date selection method, a calendar is not necessarily used. For example, date specification by string input can be used. Although the date is specified in the format of month-day-year in Figure 15 , the format of month-day is also applicable. In addition, the specific date is not limited to one day, and multiple days can be set. For example, a user interface for setting start and end dates can be provided.

[0180] The registered subject selection area 1502 displays a list of registered subjects using icons. This area provides the user with a function of setting a subject as a participant in an event when the subject icon is pressed. The selection status display icon 1503 is displayed as an icon of a subject registered as a participant in an event to indicate that the subject is a participant in the event. When the user presses the subject icon again while the selection status display icon 1503 is displayed, the subject is removed from the participants in the event. By deleting the selection status display icon 1503, the first control unit 223 notifies the user that the subject is not a participant in the event.

[0181] The specific date name setting area 1504 provides the user with a function of arbitrarily setting the name of the event set in the event setting screen. When the user presses the specific date name setting area 1504, an arbitrary string can be input.

[0182] The priority setting area 1505 for the subject associated with a specific date provides the user with a function of presetting the priority of the subject for the event registered in the participation event screen on the event date. In addition to the set priority being applied to all subjects participating in the event, the priority setting area 1105 for the subject associated with a specific date is similar to the priority setting item and the birthday priority setting item according to the second exemplary embodiment.

[0183] The specific date repeat setting area 1506 provides the user with a function of repeating the application of the event. When the user presses the specific date repeat setting area 1506, the repeat mode can be selected. Examples of selectable setting items include annually, monthly, weekly, daily, and biweekly. In addition, no other repeats can be selected.

[0184] In the case of registering multiple subjects as participants in the event setting screen according to the third exemplary embodiment, imaging can be performed only when multiple subjects are included in the view. This process allows for preferentially recording the communication state among the event participants.

[0185] Examples of methods for determining the application method of the subject priority on a specific date include a method of enabling the addition of priority only when multiple subjects are included in the view. This process makes it possible to prevent the search and composition adjustment from being affected when photographing the state of a single event participant.

[0186] A user interface for grouping one or more subjects can be provided so that a group can be specified when selecting the participant subject during event registration. In addition, an arbitrary group name can be assigned to each group.

[0187] Examples of events to be registered include birthdays, wedding anniversaries, group formation anniversaries, entrance ceremonies, graduation ceremonies, weddings, family gatherings, drinking parties, sports meets, trips, etc. However, the events to be registered are not limited to these, and any event can be registered.

[0188] As described above, this exemplary embodiment enables the provision of a method to the user: in the function of automatically changing the priority of the subject according to the date and time information associated with the subject, setting the increase amount of the priority for each specific date. This exemplary embodiment also enables the provision of a method to the user: in the function of automatically changing the priority of the subject according to the date and time information associated with the subject, setting a specific date common to multiple registered subjects.

[0189] A fourth exemplary embodiment will be described with respect to an imaging device that increases the priority of a subject not only on a specific date but also on several days including before and after the specific date. The system configuration according to the fourth exemplary embodiment is similar to the system configuration according to the first exemplary embodiment, and redundant descriptions thereof will be omitted. The fourth exemplary embodiment will be described with respect to the differences from the first exemplary embodiment.

[0190] The above exemplary embodiments are based on a method for adding the priority of a subject on a specific date. A fourth exemplary embodiment will be described with respect to a method for adding the priority of a subject not only on a specific date but also on several days including before and after the specific date.

[0191] For example, if the birthday of a certain subject is a weekday, the birthday party may usually be held on a neighboring weekend. Therefore, the first control unit 223 can provide a function of preferentially capturing an image of the subject even if the birthday party is held outside the specific date by adding the priority of the subject on three days before and after the specific date. The number of days including before and after the specific date is not limited to three days. The number of days may be different between before and after the specific date. For example, the first control unit 223 can add priority on two days before the specific date and five days after the specific date.

[0192] The amount of increase in the priority of the specific date is not necessarily the same as the amount of increase in the priority between before and after the specific date. Examples of applicable methods include a method for reducing the amount of increase in priority as the interval from the specific date increases, and a method for adding priority only on the nearest holiday.

[0193] As described above, the present exemplary embodiment enables the user to be provided with a method of adding the priority of a subject not only on a specific date but also on several days before and after the specific date in a function of automatically changing the priority of a subject according to the date and time information associated with the subject.

[0194] A fifth exemplary embodiment will be described with respect to an imaging device that notifies a user that the priority of a subject for a specific date has changed. The system configuration according to the fifth exemplary embodiment is similar to the system configuration according to the first exemplary embodiment, and redundant descriptions thereof will be omitted. The fifth exemplary embodiment will be described with respect to the differences from the first exemplary embodiment.

[0195] Figure 16A A notification 1601 to be displayed when the priority has changed according to the above exemplary embodiment is shown.

[0196] The notification 1601 is displayed on the screen of the smart device 301. The message includes an image 1602 of the subject whose priority has changed and a message 1603 notifying that the priority has changed.

[0197] When a dedicated application is launched on the intelligent device 301, the notification 1601 is displayed within the application. On the other hand, when the dedicated application is not launched, the notification 1601 is displayed in the notification area provided by the operating system of the intelligent device 301. For example, if the intelligent device 301 is a smart phone installed with an Android or iOS operating system, the notification 1601 is displayed in the notification area provided by the operating system.

[0198] According to this exemplary embodiment, after changing the priority of the subject, the first control unit 223 issues the notification 1601 at the timing of first photographing the subject.

[0199] The first control unit 223 may change the display content of the notification 1601 based on the number of subject notification objects. Figure 16B The case where there are multiple subject notification objects (in this case, three subjects) is shown. The message 1603 includes the name of the subject notification object recorded in the subject information ( Figure 16A "Airi" in Figure 16A and the event name on a specific date ( Figure 16A "birthday" in

[0200] The notification 1601 is provided as a push button that has the function of displaying a screen including a list of photos of the subject notification object when pressed.

[0201] Although in this exemplary embodiment, the image for subject detection stored in the subject information is used as the subject image 1602, it is not necessarily required to use the image for subject detection. Examples of applicable methods include a method of displaying an image of the subject stored in a recording medium and a method of not displaying the subject image 1602. In the case of multiple subject notification objects as shown in Figure 16B the first control unit 223 may display one image of the subject notification object or display multiple images of the subject notification object.

[0202] Although in the present exemplary embodiment, the name of the notified object subject is included in the message 1603, it is not necessarily required to include the name of the subject therein. However, in a system that registers and photographs multiple subjects as in the present exemplary embodiment, it is desirable to display the message 1603 so that the user can identify which subject the notification is for. This does not apply to a system that registers and photographs only one subject. In the case of a notification for a group with a given group name according to the third exemplary embodiment, the group name can be used instead of the subject name.

[0203] Although the notification 1601 is issued after changing the priority and once imaging is performed, the present invention is not limited thereto. Examples of applicable methods include a method of issuing the notification 1601 immediately after the subject information detection unit detects a subject, a method of issuing the notification 1601 even if no subject is detected when the determined day is the event date, and a method of issuing the notification 1601 when a specific time on the event date arrives.

[0204] This exemplary embodiment enables the user to easily identify that the priority of the subject has changed through the function of automatically changing the subject priority based on the date and time information associated with the subject.

[0205] The sixth exemplary embodiment will be described with reference to an imaging device that asks the user whether to change the priority when a subject is detected on a specific date. The system configuration according to the sixth exemplary embodiment is similar to the system configuration according to the first exemplary embodiment, and its redundant description will be omitted. The sixth exemplary embodiment will be described with reference to the differences from the first exemplary embodiment.

[0206] Figure 17A and 17B Fig. 15 shows a notification 1701 for asking the user whether to change the priority when changing the priority of a subject according to the above exemplary embodiment.

[0207] The notification 1701 includes an image 1602 of the subject, a message 1703, an approval (yes) button 1704 for approving the content of the message 1703, and a rejection (no) button 1705 for rejecting the content of the message 1703.

[0208] The image 1602 of the subject is similar to the image according to the fifth exemplary embodiment, and its redundant description will be omitted.

[0209] The information 1703 includes characters for confirming whether to prioritize the subject for shooting. The user can press the approval button 1704 and the rejection button 1705. When the user presses the approval button 1704, the first control unit 223 changes the priority of the subject based on the method according to the above exemplary embodiment to prioritize shooting a photo of the subject. When the user presses the rejection button 1705, the first control unit 223 keeps the priority of the subject unchanged and continues shooting based on the preset shooting conditions.

[0210] Although in this exemplary embodiment, the notification 1701 is displayed by a mechanism similar to the notification 1601 according to the fifth exemplary embodiment, the notification 1701 can also be displayed, for example, through a dialog box.

[0211] In addition, when issuing the notification, the imaging device 101 or the smart device 301 can generate sound, light, or vibration to notify the user of the issuance of the notification.

[0212] When the user presses the approval button 1704 or the rejection button 1705, a new notification can be issued in response to the button press. For example, when the user presses the approval button 1704, a notification such as "Okay. Many photos of Airi will be taken" can be issued. When the user presses the rejection button 1705, a notification such as "Okay. Photos of Airi will be taken as usual" can be issued.

[0213] This exemplary embodiment enables the user to select whether to change the priority of the subject before the priority of the subject is automatically changed based on the date and time information associated with the subject.

[0214] Other exemplary embodiments

[0215] In the case where a program for implementing at least one function according to the above exemplary embodiment is provided to a system or device via a network or a storage medium, and at least one processor in the computer of the system or device reads and executes the program, the present invention can also be implemented. In addition, the present invention can also be implemented by a circuit (for example, an application specific integrated circuit (ASIC)) for implementing at least one function.

[0216] Other embodiments

[0217] Embodiments of the present invention can also be implemented by the following method, that is, software (program) for performing the functions of the above embodiments is provided to a system or device via a network or various storage media, and the computer or the central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0218] Although the present invention has been described with reference to exemplary embodiments, the scope of the appended claims will be given the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An imaging device, comprising: an imaging unit; a control unit configured to control the imaging unit to automatically capture an image; a registration unit configured to register a subject; and an association unit configured to associate the subject registered by the registration unit with a specific date, wherein the control unit controls such that, compared with a subject registered by the registration unit for which the specific date associated with the association unit is not the current day or for which no date is associated by the association unit, a subject for which the specific date associated with the association unit is the current day is preferentially captured, the imaging device further includes a setting unit configured to set the subject registered by the registration unit to be preferentially captured, wherein the control unit controls such that, compared with the subject set by the setting unit to be preferentially captured, a subject for which the date associated with the association unit is the current day is preferentially captured.

2. The imaging device according to claim 1, further comprising a notification unit configured to notify a user, wherein when a subject for which the date associated with the association unit is the current day is detected, the notification unit notifies the user of the detection.

3. A control method for an imaging device, the imaging device having an imaging unit, the control method comprising: controlling the imaging unit to automatically capture an image; registering a subject; and associating the registered subject with a specific date, wherein, compared with a registered subject for which the associated date is not the current day or for which no date is associated, a subject for which the associated date is the current day is preferentially captured, the control method further includes setting the registered subject to be preferentially captured, wherein, compared with the subject set to be preferentially captured, a subject for which the associated date is the current day is preferentially captured.

4. The control method for an imaging device according to claim 3, wherein when a subject for which the associated date is the current day is detected, the detection is notified to the user.

5. A non-transitory computer-readable storage medium storing instructions for causing a computer to execute a control method for an imaging device, the imaging device having an imaging unit, the control method comprising: controlling the imaging unit to automatically capture an image; registering a subject; and associating the registered subject with a specific date, wherein, compared with a registered subject for which the associated date is not the current day or for which no date is associated, a subject for which the associated date is the current day is preferentially captured, the control method further includes setting the registered subject to be preferentially captured, wherein, compared with the subject set to be preferentially captured, a subject for which the associated date is the current day is preferentially captured.

6. A computer program product comprising instructions for causing a computer to execute the control method according to claim 3 or 4.

Images