Imaging device, control method for imaging device, program

Abstract

To solve the problem in which there is a possibility of missing a shot since automatic shooting means provided by an imaging apparatus alone cannot take a picture at a timing desired by a user.SOLUTION: The imaging apparatus includes sound-collecting means for collecting sound, analysis means for analyzing the sound collected by the sound collecting means, automatic shooting means for automatically shooting, and setting means for setting a shooting frequency of the automatic shooting means. If a result of the analysis by the analysis means is a specific voice instruction, the imaging apparatus sets the shooting frequency higher by the setting means after the operation according to the instruction.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an imaging device capable of receiving an instruction using voice.

Background Art

[0002] In recent years, there have been proposed a life log camera that automatically repeats shooting periodically, and an imaging device that automatically shoots by the camera itself judging the shooting situation. With these devices, the purpose is to shoot an image of a scene desired by the user without the user being aware by automatically shooting.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the device described in Patent Document 1, the shooting timing is judged using the information obtained by detecting the face of the subject, or from the number of past shots, the target number of shots, etc., and shooting is automatically performed.

[0005] However, since it is only automatic, the user's intention is not always reflected. Therefore, shooting cannot be performed at the timing desired by the user only by this, and there is a possibility that a shot may be missed. Therefore, the purpose of the present invention is to reduce the occurrence of missed shots.

Means for Solving the Problems

[0006] To achieve the above object, the imaging device of the present invention includes a sound collection means for collecting sound, an analysis means for analyzing the sound collected by the sound collection means, an automatic shooting means for automatically performing shooting, and a setting means for setting the shooting frequency of the automatic shooting means. When the result analyzed by the analysis means is a specific voice instruction, after performing the operation according to the instruction, the setting means For automatic shooting sets the shooting frequency to at a frequency higher than the maximum frequency that can be arbitrarily set by the user and is characterized in that. and, when a predetermined time has elapsed after the shooting frequency is set higher by the setting means, return the shooting frequency to its original value

Advantages of the Invention

[0007] According to the present invention, the occurrence of missed shots can be reduced.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.

[0010] ​Furthermore, the embodiments described below are merely examples of means for implementing the present invention, and may be appropriately modified or changed according to the configuration of the apparatus to which the present invention is applied and various conditions. Also, the embodiments may be combined as appropriate.

[0011] <Configuration of Imaging Device> FIG. 1 is a diagram schematically showing an imaging device according to the first embodiment.

[0012] The imaging device 101 shown in FIG. 1(a) is provided with an operation member capable of operating a power switch (hereinafter referred to as a power button, but it may also be an operation such as a tap, flick, or swipe on a touch panel). A lens barrel 102, which is a housing including a photographic lens group and an imaging element for imaging, is attached to the imaging device 101, and a rotation mechanism is provided that can rotationally drive the lens barrel 102 with respect to a fixing portion 103. The tilt rotation unit 104 is a motor drive mechanism that can rotate the lens barrel 102 in the pitch direction shown in FIG. 1(b), and the pan rotation unit 105 is a motor drive mechanism that can rotate the lens barrel 102 in the yaw direction. Therefore, the lens barrel 102 can rotate in one or more directions. Note that FIG. 1(b) shows the axis definition at the position of the fixing portion 103. Both the angular velocity meter 106 and the acceleration meter 107 are mounted on the fixing portion 103 of the imaging device 101. Then, based on the angular velocity meter 106 and the acceleration meter 107, the vibration of the imaging device 101 is detected, and the tilt rotation unit and the pan rotation unit are rotationally driven based on the detected swing angle. As a result, a configuration is provided to correct the shake or inclination of the lens barrel 102, which is a movable part.

[0013] FIG. 2 is a block diagram showing the configuration of the imaging device according to the present embodiment.

[0014] In FIG. 2, the first control unit 223 consists of a processor (e.g., CPU, GPU, microprocessor, MPU, etc.) and a memory (e.g., DRAM, SRAM, etc.). These execute various processes to control each block of the imaging device 101 and control data transfer between the blocks. The non-volatile memory (EEPROM) 216 is an electrically erasable and recordable memory, and stores constants, programs, etc. for the operation of the first control unit 223.

[0015] In FIG. 2, the zoom unit 201 includes a zoom lens that performs zooming. The zoom drive control unit 202 drives and controls the zoom unit 201. The focus unit 203 includes a lens that performs focus adjustment. The focus drive control unit 204 drives and controls the focus unit 203.

[0016] In the imaging unit 206, the imaging element receives light incident through each lens group and outputs information on electric charges corresponding to the amount of the light as analog image data to the image processing unit 207. The image processing unit 207 applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to the digital image data output by A / D conversion, and outputs the digital image data after the application. The digital image data output from the image processing unit 207 is converted into a recording format such as the JPEG format by the image recording unit 208 and transmitted to the memory 215 and the video output unit 217 described later.

[0017] The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction.

[0018] The device shake detection unit 209 is equipped with, for example, an angular velocity meter (gyro sensor) 106 that detects the angular velocity in the three-axis directions of the imaging device 101 and an accelerometer (acceleration sensor) 107 that detects the acceleration in the three-axis directions of the device. The device shake detection unit 209 calculates the rotation angle of the device, the shift amount of the device, etc. based on the detected signals.

[0019] The audio input unit 213 acquires an audio signal collected from the periphery of the imaging device 101 using a microphone provided in the imaging device 101, performs analog-to-digital conversion, and transmits it to the audio processing unit 214. The audio processing unit 214 performs processing related to audio such as normalization processing of the input digital audio signal. Then, the audio signal processed by the audio processing unit 214 is transmitted to the memory 215 by the first control unit 223. The memory 215 temporarily stores the image signal and the audio signal obtained by the image processing unit 207 and the audio processing unit 214.

[0020] The image processing unit 207 and the audio processing unit 214 read out the image signal and the audio signal temporarily stored in the memory 215, perform encoding of the image signal, encoding of the audio signal, etc., and generate a compressed image signal and a compressed audio signal. The first control unit 223 transmits these compressed image signals and compressed audio signals to the recording and reproducing unit 220.

[0021] The recording and reproducing unit 220 records the compressed image signal, the compressed audio signal, and other control data related to shooting generated by the image processing unit 207 and the audio processing unit 214 on the recording medium 221. Also, when the audio signal is not compressed and encoded, the first control unit 223 transmits the audio signal generated by the audio processing unit 214 and the compressed image signal generated by the image processing unit 207 to the recording and reproducing unit 220 and records them on the recording medium 221.

[0022] The recording medium 221 may be a recording medium built into the imaging device 101 or a removable recording medium. The recording medium 221 can record various data such as the compressed image signal, the compressed audio signal, and the audio signal generated by the imaging device 101, and a medium with a larger capacity than the non-volatile memory 216 is generally used. For example, the recording medium 221 includes all types of recording media such as hard disks, optical disks, magneto-optical disks, CD-Rs, DVD-Rs, magnetic tapes, non-volatile semiconductor memories, and flash memories.

[0023] The recording and playback unit 220 reads (plays back) the compressed image signal, compressed audio signal, audio signal, various data, and program recorded on the recording medium 221. Then, the first control unit 223 transmits the read compressed image signal and compressed audio signal to the image processing unit 207 and the audio processing unit 214. The image processing unit 207 and the audio processing unit 214 temporarily store the compressed image signal and compressed audio signal in the memory 215, decode them according to a predetermined procedure, and transmit the decoded signals to the video output unit 217 and the audio output unit 218.

[0024] A plurality of microphones are mounted on the imaging device 101 in the audio input unit 213, and the audio processing unit 214 can detect the direction of sound on a plane where a plurality of microphones are installed, which is used for search and automatic shooting described later. Furthermore, the audio processing unit 214 detects a specific voice command. The voice command may be configured such that in addition to several pre-registered commands, the user can register a specific voice with the imaging device. Also, audio scene recognition is performed. In audio scene recognition, audio scene determination is performed by a learned model that has been learned by machine learning based on a large amount of audio data in advance. Specific algorithms for machine learning include the nearest neighbor method, naive Bayes method, decision tree, support vector machine, etc. Also, deep learning (deep learning) that uses a neural network to generate its own feature quantities and connection weight coefficients for learning can be mentioned. Appropriately, those available among the above algorithms can be used and applied to this embodiment.

[0025] In this embodiment, for example, a neural network for detecting specific scenes such as "cheers are rising", "clapping hands", and "making a sound" is set in the audio processing unit 214. When a specific audio scene or specific voice command is detected, a detection trigger signal is output to the first control unit 223 and the second control unit 211.

[0026] That is, the neural network of the audio processing unit 214 prepares the audio information of the scenes of "cheers are rising", "clapping hands", and "making a sound" in advance, uses the audio information as input, and learns with the detection trigger signal as output.

[0027] A second control unit 211, which is provided separately from a first control unit 223 that controls the entire main system of the imaging device 101, controls the power supply to the first control unit 223.

[0028] A first power supply unit 210 and a second power supply unit 212 respectively supply power for operating the first control unit 223 and the second control unit 211. When a power button provided on the imaging device 101 is pressed, power is first supplied to both the first control unit 223 and the second control unit 211. However, as will be described later, the first control unit 223 is controlled to turn off its own power supply to the first power supply unit 210. Even while the first control unit 223 is not operating, the second control unit 211 is operating, and information from the device shake detection unit 209 and the audio processing unit 214 is input. The second control unit performs a determination process as to whether or not to start the first control unit 223 based on various input information, and is configured to give a power supply instruction to the first power supply unit when the start determination is made. In the present embodiment, the power supply unit supplies power from a battery. That is, the imaging device 101 is also a portable terminal.

[0029] The audio output unit 218 outputs a preset audio pattern from a speaker built in the imaging device 101, for example, at the time of shooting.

[0030] The LED control unit 224 controls an LED provided on the imaging device 101, for example, at the time of shooting, with a preset lighting and blinking pattern.

[0031] The video output unit 217 is composed of, for example, a video output terminal, and transmits an image signal to display a video on a connected external display or the like. Also, the audio output unit 218 and the video output unit 217 may be a combined single terminal, for example, a terminal such as an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.

[0032] The communication unit 222 communicates between the imaging device 101 and an external device, and transmits and receives data such as voice signals, image signals, compressed voice signals, and compressed image signals. Further, it receives control signals related to shooting, such as a shooting start or end command, panning, tilting, and zoom driving, and drives the imaging device 101 according to instructions from an external device capable of mutual communication with the imaging device 101. Also, information such as various parameters related to learning processed by the learning processing unit 219 described later is transmitted and received between the imaging device 101 and the external device. The communication unit 222 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, WirelessUSB, or a GPS receiver.

[0033] <System configuration with external communication device> FIG. 3 is a diagram showing a configuration example of a wireless communication system between the imaging device 101 and an external device 301. The imaging device 101 is a digital camera having a shooting function, and the external device 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

[0034] The imaging device 101 and the smart device 301 can communicate by, for example, communication 302 via a wireless LAN conforming to the IEEE802.11 standard series, and communication 303 having a master-slave relationship such as a control station and a subordinate station, such as Bluetooth Low Energy (hereinafter referred to as "BLE"). Note that wireless LAN and BLE are examples of communication methods. Each communication device has two or more communication functions, and for example, if it is possible to control one communication function while communicating in the relationship between a control station and a subordinate station, another communication method may be used. However, without loss of generality, the first communication such as wireless LAN enables faster communication than the second communication such as BLE, and the second communication consumes less power or has a shorter communication range than the first communication.

[0035] <Configuration of external communication device> The configuration of the smart device 301 as an example of an external communication device will be described with reference to FIG. 4. The smart device 301 is a so-called mobile phone, i.e., a mobile terminal.

[0036] The smart device 301 includes, for example, a wireless LAN control unit 401 for wireless LAN, a BLE control unit 402 for BLE, and a public line control unit 406 for public wireless communication. The smart device 301 further includes a packet transmission / reception unit 403. The wireless LAN control unit 401 performs RF control, communication processing, and various controls for communication by wireless LAN conforming to the IEEE 802.11 standard series, and protocol processing related to communication by wireless LAN. The BLE control unit 402 performs RF control, communication processing, and various controls for communication by BLE, and protocol processing related to communication by BLE. The public line control unit 406 performs RF control, communication processing, and various controls for public wireless communication, and protocol processing related to public wireless communication. The public wireless communication conforms to, for example, the IMT (International Multimedia Telecommunications) standard or the LTE (Long Term Evolution) standard. The packet transmission / reception unit 403 performs processing for executing at least one of transmission and reception of packets related to communication by wireless LAN, BLE, and public wireless communication. In this example, the smart device 301 is described as performing at least one of packet transmission and reception in communication, but other communication formats such as circuit switching may be used in addition to packet switching.

[0037] The smart device 301 further includes, for example, a control unit 411, a storage unit 404, a GPS reception unit 405, a display unit 407, an operation unit 408, an audio input and audio processing unit 409, and a power supply unit 410. The control unit 411 controls the entire smart device 301 by executing, for example, a control program stored in the storage unit 404. The storage unit 404 stores, for example, the control program executed by the control unit 411 and various types of information such as parameters necessary for communication. The various operations described later are realized by the control unit 411 executing the control program stored in the storage unit 404.

[0038] The power supply unit 410 supplies power to the smart device 301. The display unit 407 has a function capable of outputting information that can be visually recognized, such as an LCD or an LED, or outputting sound such as a speaker, and displays various types of information. The operation unit 408 is, for example, a button or the like that receives an operation of the smart device 301 by the user. Note that the display unit 407 and the operation unit 408 may be configured by a common member such as a touch panel.

[0039] The audio input and audio processing unit 409 may be configured to acquire the voice uttered by the user from a general-purpose microphone built into the smart device 301 and obtain the operation command of the user through voice recognition processing.

[0040] Also, a voice command is obtained by the pronunciation of the user through a dedicated application in the smart device. Then, it can also be registered as a specific voice command for causing the voice processing unit 214 of the imaging device 101 to recognize the specific voice command via the communication 302 using the wireless LAN.

[0041] The GPS (Global Positioning System) 405 receives GPS signals notified from satellites, analyzes the GPS signals, and estimates the current position (longitude and latitude information) of the smart device 301. Alternatively, the position estimation may be performed by using a WPS (Wi-Fi Positioning System) or the like to estimate the current position of the smart device 301 based on information on wireless networks existing in the surroundings. When the acquired current GPS position information is located within a preset position range (within a range of a predetermined radius), movement information is notified to the imaging device 101 via the BLE control unit 402 and used as parameters for automatic shooting and automatic editing described later. Also, when there is a position change of a predetermined amount or more in the GPS position information, movement information is notified to the imaging device 101 via the BLE control unit 402 and used as parameters for automatic shooting and automatic editing described later.

[0042] As described above, the imaging device 101 and the smart device 301 exchange data through communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, data such as audio signals, image signals, compressed audio signals, and compressed image signals are transmitted and received. Also, operation instructions such as shooting of the imaging device 101 are sent from the smart device, voice command registration data is transmitted, and a predetermined position detection notification and a location movement notification based on GPS position information are performed. Also, transmission and reception of learning data are performed via a dedicated application in the smart device.

[0043] <Sequence of Imaging Operation> FIG. 5 is a flowchart of the automatic shooting process of the imaging device 101 in the present embodiment.

[0044] When the user operates the power button provided on the imaging device 101, the processing of this flowchart starts. In this embodiment, the imaging device 101 and the smart device 301 are always connected by wireless communication, and various operations are possible from the dedicated application on the smart device 301. Also, the processing of each step in the following flowchart is realized by the first control unit 223 controlling each part of the imaging device 101.

[0045] In S501, the first control unit 223 determines whether it is in a state where automatic shooting is stopped. The stop of automatic shooting will be described in the flowchart of the voice recognition process described later. If it is in a state where automatic shooting is stopped, it waits without doing anything until the stop of automatic shooting is released. If automatic shooting is not stopped, it proceeds to S502 and performs image recognition processing.

[0046] In S502, the first control unit 223 causes the image processing unit 207 to perform image processing on the signal captured by the imaging unit 206 and generate an image for subject recognition.

[0047] Subject recognition such as person or object recognition is performed on the generated image.

[0048] When recognizing a person, the face or body of the subject is detected. In the face detection process, a pattern for determining a person's face is predefined, and a location in the captured image that matches the pattern can be detected as the person's face image.

[0049] Also, a reliability indicating the likelihood of the subject being a face is calculated simultaneously. The reliability is calculated, for example, from the size of the face area in the image and the degree of match with the face pattern.

[0050] Similarly for object recognition, an object that matches a pre-registered pattern can be recognized.

[0051] There are also methods of extracting a feature subject by using a histogram such as hue and saturation in the captured image. In this case, regarding the image of the subject captured within the shooting angle of view, the distribution derived from the histogram such as the hue and saturation of the image is divided into a plurality of sections, and a process of classifying the captured image for each section is executed.

[0052] For example, histograms of a plurality of color components are created for the captured image, divided in the peak distribution range, and the images captured in the regions belonging to the same combination of sections are classified, and the image region of the subject is recognized.

[0053] By calculating an evaluation value for each recognized image region of the subject, it is possible to determine the image region of the subject with the highest evaluation value as the main subject region.

[0054] By the above method, each subject information can be obtained from the imaging information.

[0055] In S503, the first control unit 223 calculates the image blur correction amount. Specifically, first, the absolute angle of the imaging device is calculated based on the angular velocity and acceleration information acquired by the device shake detection unit 209. Then, the anti-vibration angle for moving the tilt rotation unit 104 and the pan rotation unit 105 in the angular direction that cancels the absolute angle is obtained and used as the image blur correction amount. Note that the image blur correction amount calculation process here can change the calculation method by the learning process described later.

[0056] In S504, the first control unit 223 determines the state of the imaging device. Based on the angles and movement amounts detected by the angular velocity information, acceleration information, GPS position information, etc., it is determined what kind of vibration / movement state the imaging device is in currently.

[0057] For example, when the imaging device 101 is mounted on a vehicle for shooting, the subject information such as the surrounding scenery changes greatly depending on the distance traveled.

[0058] Therefore, it is possible to determine whether it is in a "vehicle moving state" where it is mounted on a vehicle or the like and moving at a high speed, and use it for automatic subject search which will be described later.

[0059] Also, it is determined whether the change in angle is large, and it is determined whether the imaging device 101 is in a "fixed shooting state" with almost no sway angle.

[0060] In the case of the "fixed shooting state", since it can be considered that there is no change in the angle of the imaging device 101 itself, subject search for fixed shooting can be performed.

[0061] Also, when the change in angle is relatively large, it is determined as the "handheld state", and subject search for handheld use can be performed.

[0062] In S505, the first control unit 223 performs subject search processing. The subject search is composed of the following processing.

[0063] (1) Area division Using FIG. 8, area division will be described. As shown in FIG. 8(a), area division is performed over the entire circumference with the position of the imaging device (assuming the origin O is the position of the imaging device) as the center. In the example of FIG. 8(a), division is performed at 22.5 degrees in each of the tilt direction and the pan direction. When divided as shown in FIG. 8(a), as the angle in the tilt direction deviates from 0 degrees, the horizontal circumference becomes smaller and the area region becomes smaller. Therefore, in the imaging device of this embodiment, as shown in FIG. 8(b), when the tilt angle is 45 degrees or more, the horizontal area range is set larger than 22.5 degrees. FIGS. 8(c) and (d) show examples of area division within the shooting angle of view. Axis 1301 is the direction of the imaging device 101 at the time of initialization, and area division is performed with this direction angle as the reference position. 1302 indicates the angle of view area of the captured image, and an image example at that time is shown in FIG. 8(d). Within the image projected onto the angle of view, the image is divided as shown in 1303 to 1318 of FIG. 8(d) based on the area division.

[0064] (2) Calculation of importance level for each area For each of the areas divided as described above, an importance level indicating the priority order of search is calculated according to the subject existing in the area and the scene situation of the area. The importance level based on the subject situation is calculated, for example, based on the number of people existing in the area, the size of the face of the person, the face orientation, the probability of face detection, the expression of the person, and the personal authentication result of the person. Further, the importance level according to the scene situation is, for example, the general object recognition result, the scene discrimination result (blue sky, backlight, sunset, etc.), the sound level or voice recognition result from the direction of the area, the motion detection information within the area, etc. Also, in the state determination of the imaging device (S504), the vibration state of the imaging device is detected, and the importance level can also be changed according to the vibration state. For example, when it is determined to be in the "set-up shooting state", when the face authentication of a specific person is detected, the importance level is determined to increase so that subject search is performed centering on the subject with a high priority among those registered in the face authentication (for example, the user of the imaging device). Also, the automatic shooting described later will also prioritize the above face, and even if the user of the imaging device spends a lot of time walking around with the imaging device on, by removing the imaging device and placing it on a desk etc., many images of the user can be left. At this time, since it is possible to search by panning and tilting, without considering the placement angle of the imaging device etc., just by installing it appropriately, images of the user and group photos with many faces can be left. Note that only with the above conditions, as long as there is no change in each area, the area with the highest importance level will be the same, and as a result, the area to be searched will never change. Therefore, the importance level is changed according to the past shooting information. Specifically, the importance level of an area that has been continuously designated as the search area for a predetermined time may be decreased, or in the area where shooting is performed in S513 described later, the importance level may be decreased for a predetermined time.

[0065] (3) Determination of the search target area After the importance level of each area is calculated as described above, the area with the highest importance level is determined as the search target area. Then, the pan-tilt search target angle required to capture the search target area within the angle of view is calculated.

[0066] In S506, the first control unit 223 performs pan-tilt driving. Specifically, the pan-tilt driving amount is calculated by adding the driving angles in control sampling based on the image blur correction amount and the pan-tilt search target angle, and the tilt rotation unit 104 and the pan rotation unit 105 are respectively driven and controlled by the lens barrel rotation driving unit 205.

[0067] In S507, the first control unit 223 controls the zoom unit 201 to perform zoom driving. Specifically, the zoom is driven according to the state of the search target subject determined in S505. For example, when the search target subject is a person's face, if the face on the image is too small and cannot be detected because it is smaller than the minimum detectable size, there is a risk of losing sight of it. In such a case, control is performed so that the size of the face on the image becomes larger by zooming toward the telephoto side. On the other hand, when the face on the image is too large, the subject is likely to deviate from the angle of view due to the movement of the subject or the imaging device itself. In such a case, control is performed so that the size of the face on the screen becomes smaller by zooming toward the wide-angle side. By performing zoom control in this way, a state suitable for tracking the subject can be maintained.

[0068] In S505 to S507, a method of performing subject search by pan-tilt and zoom driving has been described, but subject search may also be performed in an imaging system that uses a plurality of wide-angle lenses to capture the entire scene at once. In the case of an omnidirectional camera, if all the signals obtained by imaging are used as input images and image processing such as subject detection is performed, enormous processing is required. Therefore, a configuration is adopted in which a part of the image is cut out and subject search processing is performed within the cut-out image range. Similar to the method described above, the importance level for each area is calculated, the cut-out position is changed based on the importance level, and the determination for automatic shooting described later is made. As a result, it is possible to reduce the power consumption due to image processing and perform high-speed subject search.

[0069] In S508, the first control unit 223 reads the frequency parameter. The frequency parameter is a setting value indicating the ease of automatic shooting. Through the dedicated application of the smart device 301, the user can set an arbitrary frequency from options such as "low", "medium", and "high". When the frequency is set to "high", more images will be taken per predetermined time compared to when it is set to "low". For the "medium" setting, the number of images taken is between the "low" and "high" settings. Also, it can be automatically changed by the frequency setting process described later.

[0070] In S509, the first control unit 223 determines whether the read frequency parameter is a predetermined value. For example, if "highest" is set as the frequency for automatic shooting, it proceeds to S510; otherwise, it proceeds to S512. Note that the setting of "highest" frequency is a setting automatically changed by the frequency setting process described later, and in the normal user's frequency setting using the dedicated application of the smart device 301, it is set from the options of "low", "medium", and "high" as described above. That is, in the setting by user operation, the frequency "highest" is not set.

[0071] In S510, the first control unit 223 determines whether the frequency boost time from the start of the frequency parameter setting in S705 (described later) to returning to the original frequency before being set to "highest" has ended. If it has ended, it proceeds to S511; otherwise, it proceeds to S512.

[0072] In S511, since the frequency boost time has ended, the first control unit 223 returns the frequency parameter to the frequency setting before it was set to "highest". At this time, if more than a predetermined number of images have been taken by automatic shooting during the frequency boost time, it can be determined that the current scene is the scene to be photographed, so the frequency boost time may be extended. By doing so, the user can continue to take the scenes they want to take.

[0073] In S512, the first control unit 223 determines whether to perform automatic shooting.

[0074] Here, the determination of whether to perform automatic shooting will be described. The determination of whether to perform automatic shooting is made based on whether the importance score exceeds a predetermined value. The importance score is a parameter used for determining whether to perform automatic shooting, and is different from the importance level for determining the search area. The importance score is scored according to the detection status of the subject and the passage of time. For example, consider a case where it is designed to perform automatic shooting when the importance score exceeds 2000 points. In this case, first, the initial value of the importance score is 0 points, and it is scored as time elapses from the time when the automatic shooting mode is entered. If there is no subject with high priority, it increases at a rate such that it reaches 2000 points after, for example, 120 seconds. If 120 seconds have passed without detecting a subject with high priority, it reaches 2000 points due to the scoring based on the passage of time, and shooting is performed. Also, when a subject with high priority is detected during the passage of time, 1000 points are added. Therefore, in a state where a subject with high priority is detected, it is easier to reach 2000 points, and as a result, the shooting frequency is likely to increase.

[0075] Also, for example, when the smiling face of the subject is recognized, 800 points are added. Note that the scoring based on this smiling face is added even if the subject is not a subject with high priority. Also, in this embodiment, the case where the score for the scoring based on the smiling face is the same regardless of whether the subject is a subject with high priority will be described as an example, but it is not limited to this. For example, the score for adding points in response to detecting the smiling face of a subject with high priority may be made higher than the score for adding points in response to detecting the smiling face of a subject with low priority. By doing so, it becomes possible to perform shooting more in line with the user's intention. If the total score exceeds 2000 points due to the scoring associated with the expression change of these subjects, automatic shooting is performed. Also, even if the total score does not exceed 2000 points due to the scoring associated with the expression change, it reaches 2000 points in a shorter time due to the subsequent scoring based on the passage of time.

[0076] Note that, regarding the point addition based on the passage of time, for example, when adding points so that it reaches 2,000 points in 120 seconds, it is explained by taking as an example the case of adding 2000 / 120 points every second, that is, adding points linearly with respect to time, but it is not limited to this. For example, no points are added until 110 seconds out of 120 seconds, and within the 10 seconds from 110 seconds to 120 seconds, points can be added at 200 points per second so as to reach 2,000 points. By doing so, it is possible to prevent reaching the number of points to be photographed regardless of the high or low priority due to the change in the expression of the subject. In the case of the point addition method that increases linearly with the passage of time, since the state of having already been added points due to the passage of time is long, even in the case of point addition accompanying the change of a subject with low priority to a smiling face, it often reaches the number of points to be photographed, and it is difficult to reflect the high or low priority so much. On the other hand, if the number of points added accompanying the change in expression is lowered, the timing of the change in expression will be missed, so it is desirable to avoid the countermeasure of lowering the number of points added. Therefore, no points are added until 110 seconds. By doing so, the subject with low priority will pass 110 seconds without being added points. On the other hand, since 1,000 points are added to the subject with high priority when it is detected, even if there is no point addition due to the passage of time until 110 seconds, it will be in a state where 1,000 points have been added. As a result, when point addition accompanying the change in expression is performed, the possibility of the subject with low priority reaching the number of points for taking a photo can be suppressed compared to the subject with high priority, and the high or low priority can function more easily. In the above explanation, the change in expression is taken as an example, but other criteria for adding points can be considered, such as when the voice becomes louder or the body movements and hand gestures become larger. For these as well, in order to make it easier for the high or low priority to function, the difference in the point addition method as described above may be provided.

[0077] Also, even if the subject's actions do not exceed 2,000 points, it will definitely be photographed in 120 seconds due to the passage of time, so it is not the case that it will not be photographed at all for a certain period.

[0078] Also, if a subject is detected during the process, the time to start the increase within 120 seconds may be advanced. That is, for example, if a subject with a high priority is detected at the 60 - second mark, 1000 points are added thereby, but it still does not exceed 2000 points. Instead of not increasing until 110 seconds, linear increase may start 30 seconds after the subject is detected. Alternatively, linear increase may start 20 seconds before instead of 10 seconds before the 120 - second mark. By doing so, the possibility of photographing a subject with a high priority increases, making it easier to achieve photographing in line with the user's intention.

[0079] When automatic shooting is performed, the importance score is reset to 0 points. Automatic shooting will not be performed again until it exceeds 2000 points.

[0080] Here, the frequency parameter is used to control the way the importance score increases over time. In the above example, when no subject is detected, it is set to take 120 seconds until automatic shooting. This is explained by taking the case where the frequency parameter is "medium" as an example. In the state of frequency boost (frequency parameter "highest"), the way of increasing the importance score is changed so that automatic shooting is performed in 60 seconds. In this case, for the way of increase, 2000 / 60 points may be added per second, or for example, no points are added until 55 seconds, and in the remaining 5 seconds until 60 seconds, 400 points are added per second. The advantage of the latter case is as described above. For other frequency examples, for example, when the frequency parameter is "high", it is increased to reach 2000 points in 100 seconds, and when the frequency parameter is "low", it is increased to reach 2000 points in 240 seconds, etc. are designed. As described above, in the case of the frequency parameter "highest", the frequency is such that at least one shot is taken in the shortest time (60 - second example in the description of this embodiment). Therefore, increasing the shooting frequency means increasing the number of shots taken per unit time by changing the method of adding points, and decreasing the shooting frequency means decreasing the number of shots taken per unit time by changing the method of adding points.

[0081] The above is an explanation of the determination of whether to perform automatic shooting. Based on the above judgment, if it is determined to perform automatic shooting, the process proceeds to S513, and if it is determined not to shoot, the process proceeds to S501.

[0082] In S513, the first control unit 223 executes a shooting process. The shooting process mentioned here includes still image shooting and video shooting.

[0083] FIG. 6 is a flowchart of the voice recognition process of the imaging device 101 in the present embodiment. When voice uttered by a user is input to a microphone built in the imaging device 101, the voice input voice processing unit 409 performs a voice recognition process to obtain an operation command of the user.

[0084] In S601, the first control unit 223 determines whether a wake word has been detected. The wake word is an activation command for starting voice command recognition for giving a specific instruction to the imaging device 101 by voice. When giving an instruction by voice, it is necessary to generate a command word after wake word recognition and succeed in recognition. If a wake word is detected, the process proceeds to S602, and if not, the process of S601 is repeated until it is detected.

[0085] In S602, the first control unit 223 sets the automatic shooting process to a stopped state. After recognizing the wake word, since it enters the waiting state for the command word, the automatic shooting process is stopped. The stop of automatic shooting refers to the subject search using the pan / tilt operation and zoom operation and the execution of the shooting process. The purpose of stopping automatic shooting is to quickly respond to the instruction of the command word to be issued next to the wake word, so the automatic shooting process is stopped to enter the command word waiting state. Also, when trying to give a shooting instruction by voice instruction, stopping the pan / tilt enables shooting in the direction the user intends to shoot.

[0086] In S603, the first control unit 223 sounds a recognition sound to indicate to the user that the wake word has been successfully recognized.

[0087] In S604, the first control unit 223 determines whether a command word has been detected. If a command word is detected, the process proceeds to S606; if not, the process proceeds to S605.

[0088] In S605, the first control unit 223 detects the wake word and determines whether a predetermined time has elapsed since entering the command word waiting state. If the predetermined time has elapsed, the process proceeds to S601, the waiting state for the command word is stopped, and the waiting state for the wake word is entered. If the predetermined time has not elapsed, S604 is repeated until a command word is detected.

[0089] In S606, the first control unit 223 determines whether the detected command word is a still image shooting command. This still image shooting command is a command for requesting the imaging device 101 to execute shooting and recording of one still image. If it is determined to be a still image shooting command, the process proceeds to S607; otherwise, the process proceeds to S608.

[0090] In S607, the first control unit 223 performs still image shooting processing. Specifically, the signal captured by the imaging unit 206 is converted into, for example, a JPEG file by the image processing unit 207 and recorded on the recording medium 221 by the image recording unit 208.

[0091] In S608, the first control unit 223 determines whether the detected command word is a subject search command. If it is determined to be a subject search command, the process proceeds to S609; otherwise, the process proceeds to S610.

[0092] In S609, the first control unit 223 performs subject search processing. If the search target area has already been determined by the subject search processing in S505 and the subject is being captured by the pan / tilt drive in S506 and the zoom drive in S507, the tracking of that subject is stopped, and the subject search processing is executed to search for other subjects. This is because if the user instructs subject search while a subject is being captured, it means that there is a subject that the user wants to photograph separately from the currently captured subject.

[0093] After the processing from S607 to S609 is completed, in S610, frequency setting processing is performed. The frequency setting processing is a process of setting a frequency parameter indicating how many images are to be taken within a predetermined time. Although the details of the processing content will be described later, in the frequency setting processing executed in S610, the shooting frequency is set to be higher.

[0094] In S611, the first control unit 223 determines whether the detected command word is a moving image recording start command. The moving image shooting command is a command that requests the imaging device 101 to capture and record a moving image. If it is determined to be a moving image recording start command, the process proceeds to S612; otherwise, it proceeds to S613.

[0095] In S612, the first control unit 223 starts shooting a moving image using the imaging unit 206 and records it on the recording medium 221. During the recording of the moving image, pan / tilt and zoom drives are not performed, subject search is not performed, and the automatic shooting remains in a stopped state.

[0096] In S613, the first control unit 223 determines whether the detected command word is a moving image recording stop command. If it is determined to be a moving image recording stop command, the process proceeds to S614; otherwise, it proceeds to S615.

[0097] In S614, the first control unit 223 stops shooting and recording the moving image using the imaging unit 206 and completes the recording of the moving image file on the recording medium 221.

[0098] In S615, the first control unit 223 executes other processes in the voice command. For example, it performs processes for commands to perform pan-tilt in the direction specified by the user, and processes for commands to change various shooting parameters such as exposure correction.

[0099] In S616 and S617, the first control unit 223 performs a restart process for the automatic shooting stopped in S602. As a result, the processes of S502 to S510 become operable and the automatic shooting is restarted.

[0100] At this time, in the case of instructions for starting and stopping video recording, the frequency setting process is not executed. This is because after starting video recording, signals from the imaging unit 206 are continuously recorded, so there is no meaning in setting a high frequency. Also, after stopping video recording, the fact that the user has instructed to stop recording indicates that the scene to be left in the recording has ended. Therefore, it is to prevent taking unnecessary images by setting a high frequency without permission.

[0101] Also, when the remaining battery level of the imaging device 101 is low or when the imaging device 101 has reached a temperature equal to or higher than a predetermined temperature due to heat generation, it is preferable not to operate the imaging unit 206 frequently. In such a situation, the frequency parameter according to S704 in FIG. 7 described later may not be set to "highest".

[0102] FIG. 7 is a flowchart of the frequency setting process of the imaging device 101 in the present embodiment. As a means for the user to set the frequency of automatic shooting, there is a method of performing it via a dedicated application in the smart device 301. The process of this flowchart is also started in response to the execution of S610 in FIG. 6. Further, it is also started in response to the user instructing a change in frequency via a dedicated application in the smart device 301.

[0103] In S701, the first control unit 223 determines whether it is a frequency setting via a dedicated application in the smart device 301. If it is a frequency setting via a dedicated application, the process proceeds to S702; otherwise (for example, when executed in S610), the process proceeds to S703.

[0104] In S702, the first control unit 223 makes a setting according to the frequency parameter instructed by the user. For example, as shown in FIG. 9, on the screen of the dedicated application in the smart device 301, it is possible to make a setting by selecting "low", "medium", or "high" from the item of the automatic shooting frequency.

[0105] Here, the application screen of FIG. 9 will be described.

[0106] In the dedicated application of the smart device 301, still images and moving images are prepared as contents to be automatically shot. Furthermore, as the content to be automatically shot, it is possible to set whether to prioritize still images or moving images from the dedicated application. This setting can be changed by touching (flicking) the knob of the slider bar as shown in FIG. 9. When set to prioritize still images, more still images are shot than moving images. Also, when set to prioritize moving images, more moving images are shot than still images.

[0107] Also, it is possible to set the range within which the imaging device searches for the scene to be imaged from the front direction. In the example of FIG. 9, three patterns can be set: a range of 60 degrees in total (30 degrees to the left and right from the front), a range of 180 degrees in total (90 degrees to the left and right from the front), and a full circle. Note that it may be possible to input numerical values so that a finer range setting is possible.

[0108] Also, when taking pictures automatically, there is a concern that too much content may be captured. Therefore, a function to automatically delete images is provided, and this function can be turned on and off from the smart device 301. Note that the images to be automatically deleted may be deleted in order, for example, starting from the oldest shooting date and time, or may be deleted in ascending order of low importance. The importance mentioned here is, for example, in the case of still images, a parameter obtained by quantifying parameters that are predicted to be images that the user would like to keep, such as whether there is little blur or whether a person is depicted. Also, in the case of videos, for example, whether a person is depicted or whether a person's voice such as a conversation is recorded is quantified to calculate the importance. Then, those with a higher total value are treated as having a higher importance.

[0109] The above is the description of FIG. 9. Return to the description of FIG. 7.

[0110] In S703, the first control unit 223 determines whether it is the frequency setting called from the speech recognition process. If it is the frequency setting called from the speech recognition process, the process proceeds to S704; otherwise, the frequency setting process ends.

[0111] In S704, the first control unit 223 sets a frequency higher than the frequency that can be set in S702 for the frequency parameter. The reason for this is that the timing when the user gives a shooting instruction is at least the timing when shooting is desired. That is, at the timing when the user gives a shooting instruction, since it is a situation where shooting is desired, it is considered that a scene where shooting is desired is likely to occur in a period close in time. Focusing on this point, the imaging device of the present embodiment uses the voice instruction by the user's voice command as a trigger, estimates that a scene to be shot should be captured for a certain period after the voice command is input, and increases the shooting frequency. As a result, it is possible to capture an image that the user wants to capture without missing it. In the present embodiment, it is described as setting the frequency parameter to "highest", but each time the frequency setting is performed according to the voice command instruction, the frequency may be increased step by step. In this case, the upper limit of the frequency is the maximum shooting speed of the continuous shooting provided in the imaging device 101.

[0112] In S705, the first control unit 223 sets the frequency boost time until the frequency parameter set to "highest" in S704 is returned to the original parameter, and starts counting down. For example, when the frequency setting is set to "medium" and the frequency setting is set to "highest" by a voice command instruction, if the frequency boost time is 60 seconds, the frequency setting will return to "medium" after 60 seconds have elapsed (the actual process is performed in S511). Here, the frequency boost time refers to the time during which the frequency remains at the highest state. Although this frequency boost time is automatically set, the user may be able to set an arbitrary time.

[0113] At this time, in addition to resetting the setting when a predetermined time has elapsed, the frequency boost time may be reset based on whether a predetermined number of shots are taken by continuous shooting.

[0114] Also, if the frequency setting is set to "highest" again by a voice command before the countdown of the frequency boost time ends, the predetermined time or the predetermined number of shots until the frequency setting is reset is extended.

[0115] Furthermore, as a determination to restore the frequency setting, it may be determined based on whether the subject search process has been performed in all directions in the pan direction.

[0116] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0117] For example, in the above embodiment, an example of a voice command is used as a means for a shooting instruction from the user. In addition to this, even when shooting is instructed by an instruction via a communication means from a smart device or a BLE remote control, the frequency setting may be set to "highest" after executing the instruction. Also, when an instruction to execute processing according to a specific vibration pattern using an acceleration sensor in the imaging device is detected, the frequency setting may be set to "highest" after executing the instruction. Furthermore, even when a gesture instruction by a gesture is received by analyzing the movement of the user's hand through the imaging unit, the frequency setting may be set to "highest" after executing the instruction.

[0118] In addition, in the present embodiment, it has been characterized in that an image desired by the user is captured by tracking a subject by means of pan-tilt driving and zoom driving. In this regard, for example, an implementation may be considered in which a 360° camera is adopted as the imaging means to constantly capture all directions, and an image of the subject is obtained by cutting out an image of a necessary range from the captured image. In such a case, video recording is constantly executed, and in response to an input of a cut-out instruction, recording is performed in the format of a still image, and then the frame rate of the video is increased. Even in this case, similar to the shooting frequency in the above-described embodiment, the highest rate at which the frame rate can be set may be used, or a value exceeding the settable value may be used. Also, as a condition for returning the increased frame rate to the original value, the elapse of a certain period of time may be adopted as in the above-described embodiment. As a result, recording is performed more frequently around the timing when the user desires to record an image, and as a result, for example, an effect is obtained that it becomes easier to acquire an image in which no blurring of the focus occurs for a moving object.

[0119] Note that if the shooting timing does not come within the frequency boost time, it is conceivable that not even a single shot is taken. Therefore, first, when a still image shooting command is received, pan-tilt or zoom driving is not performed, and a single shot is taken without searching for the subject. Subsequently, three consecutive shots are taken while searching for the subject. Thereafter, for a predetermined period of time, the frequency boost state is entered and automatic shooting is performed. By doing so, when the user intentionally instructs still image shooting by the still image shooting command, it is ensured that at least four shots are taken and no situation occurs where not even a single shot is taken.

[0120] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and having one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Claims

1. A sound collection means for collecting sound, An analysis means for analyzing the sound collected by the sound collection means, An automatic shooting means for automatically performing shooting, A setting means for setting the shooting frequency of the automatic shooting means, having, As a result of the analysis by the analysis means, when it is a specific voice instruction, after performing an operation according to the instruction, the shooting frequency of the automatic shooting is set by the setting means to a frequency higher than the highest value that can be arbitrarily set by the user, An imaging device, characterized in that when a predetermined time has elapsed after the shooting frequency is set higher by the setting means, the shooting frequency is restored to its original value.

2. The imaging device according to claim 1, wherein the automatic shooting means automatically pans, tilts, and zooms the imaging device, and automatically performs shooting of a still image or a moving image while tracking a subject.

3. The imaging device according to claim 1, characterized in that when the automatic shooting means performs shooting of a predetermined number of shots or more in a state where the shooting frequency is set higher by the setting means, the predetermined time is extended.

4. The imaging device according to claim 1, characterized in that when a specific voice instruction is recognized by the analysis means in a state where the shooting frequency is set higher by the setting means, the predetermined time is extended.

5. The imaging device according to claim 1, further comprising a rotating means for changing the orientation of the imaging device, and when the rotating means searches for a subject in all directions after the shooting frequency is set higher by the setting means, the shooting frequency is restored to its original value.

6. The imaging device according to claim 1, characterized in that when the specific voice instruction analyzed by the analysis means is a shooting instruction, the shooting frequency is set higher by the setting means.

7. The imaging device according to claim 1, wherein when the specific voice instruction analyzed by the analysis means is an instruction to search for a subject, the shooting frequency is set higher by the setting means.

8. The imaging device according to claim 1, wherein when the specific voice instruction analyzed by the analysis means is an instruction to start recording a moving image, the frequency setting is not set higher by the setting means.

9. The imaging device according to claim 1, wherein when the specific voice instruction analyzed by the analysis means is an instruction to stop recording a moving image, the frequency setting is not set higher by the setting means.

10. The imaging device according to claim 1, wherein when the remaining battery level of the imaging device is less than a predetermined amount, even if the voice analyzed by the analysis means is the specific voice instruction, the frequency setting is not set higher by the setting means.

11. The imaging device according to claim 1, wherein when the temperature of the imaging device is higher than a predetermined temperature, even if the voice analyzed by the analysis means is the specific voice instruction, the frequency setting is not set higher by the setting means.

12. The imaging device according to claim 1, wherein even when a specific instruction via communication means from a mobile terminal, a specific vibration pattern using an acceleration sensor of the imaging device is detected, or a specific instruction by a gesture instruction that realizes an instruction by the movement of the user's hand is given, the frequency is set higher by the setting means.

13. A control method for an imaging device having a sound collection means for collecting sound, an analysis step of analyzing the sound collected by the sound collection means, an automatic shooting step of automatically performing shooting, and a setting step of setting the shooting frequency in the automatic shooting step. As a result of the analysis in the above analysis step, if it is a specific voice instruction, after performing the operation according to the instruction, the above setting step is executed, and the shooting frequency of the automatic shooting is set to a frequency higher than the highest value of the frequency that can be arbitrarily set by the user. A control method for an imaging device, characterized in that when a predetermined time has elapsed after the shooting frequency is set higher in the above setting step, the shooting frequency is restored to the original value.

14. A computer-readable program for causing a computer to function as each means of the imaging device according to any one of claims 1 to 12.

Images