Camera device, control method, and recording medium capable of recognizing voice commands

Abstract

The present invention provides an imaging device, control method, and recording medium capable of recognizing voice commands. The imaging device includes: an imaging unit configured to capture an image of a subject; a driving unit configured to drive a driving member configured to change the imaging direction of the imaging device; and a detection unit configured to detect a first voice command for performing specific processing and a second voice command for initiating detection of the first voice command. The driving unit restricts driving of the driving member in response to the detection unit detecting the second voice command.

Description

Technical Field

[0001] The present invention relates to an imaging device capable of recognizing voice commands. Background Art

[0002] In recent years, there have been discussions about imaging devices that can capture images of their surroundings. Such imaging devices aim to capture images of a scene desired by a user by automatically capturing an image of a subject in response to recognizing a voice command issued by the user.

[0003] Japanese Patent Application Laid-Open No. 2019-106694 discusses a camera device having an automatic camera mode that performs pan / tilt drive or zoom drive to automatically search for a subject and capture an image of the subject. In the automatic camera mode, the camera device enables manual camera shooting based on voice commands.

[0004] In a voice command-based process, a device configured to detect a voice command typically detects a voice command (command word) for performing a predetermined process after detecting a voice command (trigger word) for starting detection of a command word, in order to prevent misoperation. However, Japanese Patent Laid-Open No. 2019-106694 does not consider the use of trigger words by a camera device. In the case where the aforementioned voice command-based process is applied to the camera device discussed in Japanese Patent Laid-Open No. 2019-106694, it is assumed that the camera device discussed in Japanese Patent Laid-Open No. 2019-106694 detects trigger words and command words while automatically searching for a subject in an automatic camera mode. When the trigger word is detected, since the subsequent voice command-based process has not yet been identified, the camera device continues to automatically take pictures. In the case where the command word spoken by the user is a voice command instructing the camera device to take an image of a subject, the subject that the user desires to photograph is expected to be the subject that the camera device is photographing when the user starts speaking the trigger word. However, if the automatic search is continued after the trigger word is detected, the subject that the user desires to photograph may fall out of the angle of view until the command word is detected, and the image of the subject that the user desires may be missed. Summary of the Invention

[0005] According to an aspect of the present invention, an imaging apparatus includes: an imaging unit configured to capture an image of a subject; a driving unit configured to drive a driving member configured to change an imaging direction of the imaging unit; and a detection unit configured to detect a first voice command for performing specific processing and a second voice command for starting detection of the first voice command. The driving unit restricts driving of the driving member in response to the detection unit detecting the second voice command.

[0006] A control method for an imaging device, the control method comprising: capturing an image of a subject; driving a driving member configured to change an imaging direction during the capturing; detecting a first voice command for performing a specific process and a second voice command for starting detection of the first voice command; and limiting driving of the driving member in response to detecting the second voice command.

[0007] A non-transitory computer-readable recording medium having recorded thereon a program for causing an imaging device to perform a control method, the control method comprising: capturing an image of a subject; driving a driving member configured to change an imaging direction during the capturing; detecting a first voice command for performing a specific process and a second voice command for starting detection of the first voice command; and limiting driving of the driving member in response to detection of the second voice command.

[0008] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1A is a diagram illustrating an example of an imaging apparatus according to the first exemplary embodiment. Figure 1B are diagrams illustrating the operation of the imaging apparatus according to the first exemplary embodiment.

[0010] Figure 2 is a diagram illustrating a configuration of an imaging apparatus according to a first exemplary embodiment.

[0011] Figure 3 is a diagram illustrating a system configuration including an imaging apparatus and external devices according to a first exemplary embodiment.

[0012] Figure 4 is a diagram illustrating a configuration of an external device according to the first exemplary embodiment.

[0013] Figure 5 is a flowchart illustrating automatic imaging processing according to the first exemplary embodiment.

[0014] Figure 6A 、 Figure 6B 、 Figure 6C and Figure 6D is a diagram illustrating region division of a captured image according to the first exemplary embodiment.

[0015] Figure 7 is a flowchart illustrating speech recognition processing according to the first exemplary embodiment. DETAILED DESCRIPTION

[0016] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0017] The exemplary embodiments described below are examples of means for realizing the present invention, and can be appropriately corrected or modified based on the configuration and various conditions of the device to which any exemplary embodiment is applied. The exemplary embodiments can also be appropriately combined.

[0018] <Camera Equipment Configuration>

[0019] Figure 1A : is a diagram illustrating a configuration of an imaging apparatus 101 according to a first exemplary embodiment of the present invention.

[0020] Figure 1A The illustrated imaging apparatus 101 is provided with operating members including a power switch that enables a user to perform an operation of switching the power on and off, and further includes a touch panel.

[0021] The lens barrel 102 is a housing including an optical lens group and an image sensor. The lens barrel 102 is attached to the imaging device 101. The tilt rotation unit 104 and the pan rotation unit 105 are components for rotating the lens barrel 102 relative to the fixing portion 103. For example, the tilt rotation unit 104 can be Figure 1B The motor for rotating the lens barrel 102 in the tilt direction (pitch direction) shown in FIG. Figure 1B The motor for rotating the lens barrel 102 in the yaw direction (pan direction) shown in FIG. The pitch rotation unit 104 and the pan rotation unit 105 can drive the lens barrel 102 to rotate in one or more axis directions. In this exemplary embodiment, Figure 1B The Y axis shown represents the rotation axis of the pan rotation unit 105. In this exemplary embodiment, Figure 1B The positive Z-axis direction shown represents the front direction of the imaging apparatus 101 .

[0022] The angular velocity meter 106 and the accelerometer 107 are, for example, a gyro sensor and an acceleration sensor, respectively, and are arranged on the fixed portion 103 of the imaging apparatus 101. The imaging apparatus 101 detects vibration of the imaging apparatus 101 based on the angular velocity and acceleration measured by the angular velocity meter 106 and the accelerometer 107, respectively. By rotationally driving the pitch rotation unit 104 and the pan rotation unit 105 based on the detected vibration of the imaging apparatus 101, the imaging apparatus 101 can generate an image corrected for shake and tilt.

[0023] Figure 2 : is a block diagram illustrating the configuration of the imaging apparatus 101 according to the present exemplary embodiment.

[0024] The first control unit 223 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, or a microprocessing unit (MPU)) and a memory (e.g., a dynamic random access memory (DRAM) or a static random access memory (SRAM)). The first control unit 223 performs various processes to control the blocks in the imaging apparatus 101 and controls data transfer between the blocks. The first control unit 223 is an example of a control unit and a determination unit.

[0025] The nonvolatile memory 216 can record and erase data. Operation constants and programs of the first control unit 223 are recorded in the nonvolatile memory 216.

[0026] The zoom unit 201 is an optical lens group that forms a zoom lens for changing a zoom magnification. The zoom drive control unit 202 controls the drive of the optical lens in the zoom unit 201. The focus adjustment unit 203 is an optical lens group for focus adjustment.

[0027] The focus drive control unit 204 controls the drive of the optical lens in the focus unit 203. The imaging unit 206 includes an image sensor that receives light incident through the optical lens group. The imaging unit 206 outputs information about the charge corresponding to the amount of received light as analog image data to the image processing unit 207. The lens barrel 102 includes the zoom unit 201, the zoom drive control unit 202, the focus unit 203, the focus drive control unit 204, and the imaging unit 206.

[0028] The image processing unit 207 performs image processing on the analog image data input from the imaging unit 206 and outputs digital image data. Examples of image processing include distortion correction, white balance adjustment, and color interpolation processing. The image recording unit 208 converts the digital image data output from the image processing unit 207 into an image file format such as the Joint Photographic Experts Group (JPEG) format or a moving picture file format such as the Moving Picture Experts Group (MPEG) format. The converted digital image data is sent to the memory 215 and the video output unit 217, which will be described below. If the digital image data stored in the memory 215 is to be recorded, the first control unit 223 outputs the digital image data to the recording / reproducing unit 220.

[0029] The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction. The lens barrel rotation drive unit 205 changes (drives) the imaging direction by driving the lens barrel 102 in the tilt direction and the pan direction. The lens barrel rotation drive unit 205 is an example of a drive unit.

[0030] The device shake detection unit 209 includes, for example, the angular velocity meter 106 that detects the angular velocity of the imaging device 101 in the three-axis direction, and the accelerometer 107 that detects the acceleration of the imaging device 101 in the three-axis direction. Based on the signals detected by the angular velocity meter 106 and the accelerometer 107, the device shake detection unit 209 calculates the rotation angle of the imaging device 101 and the displacement amount of the imaging device 101.

[0031] The audio input unit 213 includes a plurality of microphones. The audio input unit 213 performs analog-to-digital (A / D) conversion on audio signals input from the microphones and outputs the resulting audio signals to the audio processing unit 214.

[0032] The audio processing unit 214 is capable of detecting the direction of sound on a plane on which multiple microphones are installed. The direction of sound detected by the audio processing unit 214 can be used for searching and automatic imaging, which will be described below. In addition, the audio processing unit 214 can recognize specific voice commands. In this exemplary embodiment, there are two types of specific voice commands: trigger words and command words. A trigger word is a command that serves as a trigger for starting the recognition of a command word. For example, a trigger word is a command that includes specific keywords spoken by the user, such as "OK" and "camera." A command word is a command that instructs the imaging device 101 to perform a predetermined process. Examples of predetermined processes include still image shooting, moving image shooting start processing, moving image shooting end processing, sleep processing, subject change processing, and automatic imaging processing. A command word is a command that includes keywords that vary depending on the predetermined process. Examples of keywords include "shoot still image" for still image shooting and "shoot moving image" for moving image shooting start processing. These voice commands are pre-recorded in the memory 215 of the imaging device 101. The imaging apparatus 101 may be configured to register a voice command for performing a user's desired process in addition to pre-recorded voice commands.

[0033] The audio processing unit 214 also performs audio-related processing such as optimization processing and encoding on the input audio signal. The first control unit 223 sends the audio signal processed by the audio processing unit 214 to the memory 215. The memory 215 temporarily stores the data input from the image recording unit 208 and the audio signal input from the audio processing unit 214. When the audio signal is to be recorded, the first control unit 223 outputs the audio signal from the memory 215 to the recording / reproducing unit 220.

[0034] The recording / reproducing unit 220 records image data, audio signals, and other data such as control data related to imaging on a recording medium 221. The recording medium 221 may be built into the imaging apparatus 101 or may be detachable. The recording medium 221 can record various types of data, such as image data and audio signals. In this exemplary embodiment, the capacity of the recording medium 221 is greater than the capacity of the nonvolatile memory 216. Examples of recording media include a hard disk, an optical disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a recordable digital versatile disc (DVD-R), a magnetic tape, a nonvolatile semiconductor memory, and a flash memory.

[0035] The recording / reproducing unit 220 can record (reproduce) image data, audio signals, various types of data, and programs recorded on the recording medium 221. The image data and audio signals recorded on the recording medium 221 are reproduced as follows. The first control unit 223 outputs the image data and audio signals read by the recording / reproducing unit 220 to the image processing unit 207 and the audio processing unit 214, respectively. The image processing unit 207 and the audio processing unit 214 then decode the image data and the audio signals, respectively. The image processing unit 207 and the audio processing unit 214 output the decoded signals to the video output unit 217 and the audio output unit 218, respectively.

[0036] The second control unit 211 controls the power supply to the first control unit 223. For example, the second control unit 211 includes a processor (such as a CPU, a microprocessor, or an MPU) and a memory (such as a DRAM or SRAM). In the present exemplary embodiment, the second control unit 211 is provided separately from the first control unit 223 for controlling the entire main system of the imaging apparatus 101.

[0037] The first power supply unit 210 and the second power supply unit 212 supply power for operating the first control unit 223 and the second control unit 211, respectively. In this exemplary embodiment, the power supplied by the first power supply unit 210 is greater than the power supplied by the second power supply unit 212. In this exemplary embodiment, the first power supply unit 210 and the second power supply unit 212 are selected based on the amount of power to be supplied. For example, the first power supply unit 210 is a switch for supplying power to the first control unit 223. The second power supply unit 212 is a lithium battery or an alkaline battery. When the power switch of the imaging apparatus 101 is pressed, power is first supplied to the second control unit 211 and then to the first control unit 223.

[0038] The imaging device 101 has a sleep state. In the sleep state, the first control unit 223 controls the first power supply unit 210 to shut down the power supply to the first control unit 223. In the sleep state, although the first control unit 223 is not powered, the second control unit 211 still operates and obtains information from the device shake detection unit 209 and the audio processing unit 214. The second control unit 211 performs processing for determining whether to activate the first control unit 223 based on such input information.

[0039] When the second control unit 211 determines to activate the first control unit 223 (canceling the sleep state), the second control unit 211 controls the first power supply unit 210 to supply power to the first control unit 223 .

[0040] For example, during imaging, the audio output unit 218 outputs an audio signal such as an electronic shutter sound from a speaker built into the imaging apparatus 101. For example, during imaging, the light emitting diode (LED) control unit 224 controls an LED provided on the imaging apparatus 101 to turn on or blink in a preset pattern.

[0041] The video output unit 217 includes, for example, a video output terminal, and outputs an image signal for displaying a video image on an external display connected to the imaging apparatus 101. The audio output unit 218 and the video output unit 217 may be configured as an interface having one integrated terminal, such as a High-Definition Multimedia Interface (registered trademark) (HDMI (registered trademark)) terminal.

[0042] The communication unit 222 is an interface for communicating between the camera 101 and an external device. For example, the communication unit 222 sends and receives data such as audio signals and image data to and from the external device. When the communication unit 222 receives a control signal related to camera shooting (such as starting camera shooting, ending camera shooting, pan driving, tilt driving, and zoom driving), the first control unit 223 drives the camera 101 based on the control signal. The communication unit 222 includes a wireless communication module such as an infrared communication module, a Bluetooth communication module, and a control module. communication module, wireless local area network (LAN) communication module, wireless universal serial bus (USB) module and global positioning system (GPS) receiver.

[0043] The subject detection unit 225 reads the image data output from the image processing unit 207 from the memory 215 and performs subject recognition including person recognition and object recognition. For example, in the case of person recognition, the subject detection unit 225 detects the face of the subject. The pattern used to determine the face of the subject is pre-registered in the camera 101. The pattern is provided with an identifier for identifying different subjects. In the process for detecting the face of the subject, the subject detection unit 225 detects an area that matches the pattern used to determine the face of the subject in the captured image, thereby detecting the face of the subject. The subject detection unit 225 is also capable of distinguishing a plurality of persons registered in the camera 101 from one another.

[0044] At the same time, the subject detection unit 225 calculates a degree of reliability that indicates the certainty of the detected subject's face. For example, the degree of reliability is calculated based on the size of the facial region in the image and the degree of matching with the facial pattern. Furthermore, by performing pattern matching on the subject's face in the image, the subject detection unit 225 can detect facial information such as whether the detected face is smiling, whether the face's eyes are open, and the direction of the face. The method used to detect facial information is not limited to pattern matching, and known deep learning-based methods can also be used.

[0045] In the object recognition process, the subject detection unit 225 can recognize the object by determining whether the object matches a pre-registered pattern. Alternatively, the subject detection unit 225 can recognize the object by extracting the feature quantity of the object using the hue, saturation, and other histograms of the captured image.

[0046] With the above-described method, the first control unit 223 can detect a subject from a captured image by using the subject detection unit 225 .

[0047] <System Configuration Including External Devices>

[0048] Figure 3 The present invention shows an example of a configuration of a wireless communication system including the imaging apparatus 101 and the smart device 301. For example, the imaging apparatus 101 is a digital camera. For example, the smart device 301 is a smartphone including a Bluetooth communication module and a wireless LAN communication module.

[0049] In the present exemplary embodiment, the imaging apparatus 101 and the smart device 301 can communicate with each other via two communication paths. For example, one is a communication path 302 based on a wireless LAN conforming to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards. The other is a communication path 303 having a master-slave relationship between a control station and a slave station, such as Bluetooth Low Energy (BLE). Low Energy). Wireless LAN and Bluetooth low energy are examples of communication methods. For example, other communication methods may be used as long as each communication device has two or more communication functions, and the communication function that performs communication in a relationship between a control station and a slave station can control the other communication functions. However, without loss of generality, it is assumed here that the first communication, such as wireless LAN communication, can communicate faster than the second communication, such as Bluetooth low energy, and the second communication has at least one of lower power consumption and a shorter communication range than the first communication.

[0050] <Configuration of External Devices>

[0051] Will refer to Figure 4 The configuration of the smart device 301 as an example of an external device is described. The smart device 301 is a smartphone or a mobile phone. In other words, the smart device 301 is a portable terminal.

[0052] The smart device 301 includes, for example, a wireless LAN control unit 401 for wireless LAN communication, a Bluetooth low energy control unit 402 for Bluetooth low energy communication, and a public line control unit 406 for public wireless communication. The smart device 301 also includes a packet transmission / reception unit 403 .

[0053] The wireless LAN control unit 401 performs wireless LAN radio frequency (RF) control, communication processing, and various other types of control for communications using a wireless LAN that complies with the IEEE 802.11 standard series. The wireless LAN control unit 401 also performs protocol processing related to wireless LAN communications. The Bluetooth Low Energy control unit 402 performs Bluetooth Low Energy RF control, communication processing, and various other types of control for Bluetooth Low Energy communications. The Bluetooth Low Energy control unit 402 also performs protocol processing related to Bluetooth Low Energy communications. The public line control unit 406 performs public wireless communication RF control, communication control, and various other types of control for public wireless communications. The public line 406 also performs protocol processing related to public wireless communications. Public wireless communications comply with, for example, the International Multimedia Telecommunications (IMT) standard or the Long Term Evolution (LTE) standard. The packet transmission / reception unit 403 performs processing for at least one of transmitting and receiving packets related to wireless LAN communications, Bluetooth Low Energy communications, and public wireless communications. Although in this exemplary embodiment, the smart device 301 is described as performing at least one of packet transmission and packet reception during communications, any other communication method (such as circuit switching) may be used instead of packet switching.

[0054] The smart device 301 according to the present exemplary embodiment further includes a control unit 411, a recording unit 404, a GPS receiving unit 405, a display unit 407, an operation unit 408, an audio input / audio processing unit 409, and a power supply unit 410. The control unit 411 controls the entire smart device 301 by, for example, executing a program recorded in the recording unit 404. The recording unit 404 records, for example, the program to be executed by the control unit 411 and various types of information including parameters used for communication. Various operations of the smart device 301, which will be described below, are realized by the control unit 411 executing the program recorded in the recording unit 404.

[0055] The power supply unit 410 supplies power to each unit of the smart device 301. For example, the display unit 407 has a function capable of outputting visually recognizable information (such as a liquid crystal display (LCD) or LEDs) and a function capable of outputting sound (such as a speaker), and displays and outputs various types of information. For example, the operation unit 408 includes buttons for receiving user operations on the smart device 301. The display unit 407 and the operation unit 408 can be formed of a common component such as a touch panel.

[0056] The audio input / audio processing unit 409 includes, for example, a microphone and performs voice recognition processing on audio signals input from the microphone. The audio input / audio processing unit 409 can recognize user operations on the information processing device 101 through voice recognition. During the voice recognition process, the audio input / audio processing unit 409 uses a dedicated application to identify user voice commands from the audio signals input from the microphone. For example, the smart device 301 can register a voice command with the camera 101, which causes the audio processing unit 214 of the camera 101 to perform specific processing via the communication path 302.

[0057] The GPS receiving unit 405 analyzes the GPS signals received from the satellite to estimate the current location (longitude and latitude information) of the smart device 301. If the estimated current location falls within a preset range (predetermined radius range) such as the user's home, the smart device 301 notifies the camera 101 of the information about the current location via the Bluetooth low energy control unit 402. The camera 101 can use the current location as a parameter for automatic shooting (described below) and automatic editing.

[0058] If the current location of the smart device 301 changes and moves outside the preset range, the smart device 301 notifies the camera 101 of the information regarding the movement via the Bluetooth Low Energy control unit 402. The camera 101 uses this information as a parameter for automatic shooting and automatic editing. The smart device 301 can use the Wi-Fi Positioning System (WPS) to estimate the current location of the smart device 301 based on information about nearby wireless networks.

[0059] As described above, the smart device 301 uses the wireless LAN control unit 401 and the Bluetooth Low Energy control unit 402 to communicate with the camera 101 to exchange data. For example, the camera 101 and the smart device 301 can transmit and receive audio signals and data such as image data to and from the camera 101. The smart device 301 can also transmit setting information related to the camera 101's image capture to the camera 101. The smart device 301 can also transmit control signals related to the camera 101's image capture processing and position information to the camera 101.

[0060] <Automatic Camera Processing>

[0061] In automatic image capture, the first control unit 223 determines the capture timing and automatically captures an image of a subject. In automatic image capture, if the first control unit 223 determines that a good image or moving image can be captured, or if a specific time period has elapsed, the first control unit 223 repeats a cycle of automatically determining a subject as the capture target and capturing an image of the subject. This allows the imaging apparatus 101 to record unexpectedly good scenes or subtle changes in daily life without requiring the user to manually capture images.

[0062] Figure 5 : is a flowchart illustrating automatic imaging processing performed by the imaging apparatus 101 according to the present exemplary embodiment.

[0063] The processing in this flowchart begins when the user turns on the power switch of the imaging apparatus 101. In this exemplary embodiment, a wireless connection is established between the imaging apparatus 101 and the smart device 301. The user can use a dedicated application on the smart device 301 to perform various operations on the imaging apparatus 101. The processing in the steps in the flowchart is implemented by the first control unit 223 for controlling each unit of the imaging apparatus 101.

[0064] In step S501, the first control unit 223 determines whether the automatic camera process is suspended. The suspension of the automatic camera process will be described in the description of the voice recognition process below. In the case where the automatic camera process is suspended ("Yes" in step S501), the first control unit 223 waits until the suspension of the automatic camera process is canceled. In other words, in the case where the automatic camera process is suspended, the processing of step S501 is repeated until the suspension of the automatic camera process is canceled. In the case where the automatic camera process is not suspended ("No" in step S501), the processing enters step S502.

[0065] In step S502, the first control signal 223 causes the image processing unit 207 to perform image processing on the signal captured by the imaging unit 206 and generate an image for subject recognition. The first control unit 223 also controls the subject detection unit 225 to perform subject recognition, including human and animal recognition, on the generated image for subject recognition. For example, the first control unit 223 pre-stores a pattern for subject identification. During subject recognition, the first control unit 223 causes the subject detection unit 225 to determine the subject based on the degree of match between the stored pattern and the pattern included in the image for subject recognition. In this case, the subject detection unit 225 performs human recognition on the subject.

[0066] In addition to determining the subject, the first control unit 223 also detects the position of the subject within the angle of view.

[0067] In step S503, the first control unit 223 calculates an image shake correction amount. More specifically, the first control unit 223 first calculates the absolute angle of the imaging device 101 based on the information about the angular velocity and acceleration obtained by the device shake detection unit 209. The first control unit 223 then determines a stabilization angle for moving the tilt rotation unit 104 and pan rotation unit 105 in an angular direction to cancel out the absolute angle as the image shake correction amount.

[0068] In step S504, the first control unit 223 performs a subject search process. The subject search process includes the following processes:

[0069] (1) Regional division

[0070] Will refer to Figure 6A 、 Figure 6B 、 Figure 6C and Figure 6D To describe the regional division. Figure 6A 、 Figure 6B and Figure 6C In , the spherical area is divided with the position of the imaging device 101 as the origin O. Figure 6AIn the example shown in FIG, the spherical area is divided into units of 22.5° in the pitch direction and the pan direction.

[0071] According to Figure 6A As shown in the division, as the angle in the pitch direction moves away from 0°, the horizontal perimeter decreases and each area becomes smaller. Figure 6B As shown, the imaging apparatus 101 according to the present exemplary embodiment sets the horizontal range of each area having a pitch angle of 45° or greater to be greater than 22.5°.

[0072] Next, we will refer to Figure 6C and Figure 6D The image capture device 101 captures an area within the angle of view of the image. Axis 1301 represents a reference direction for the image capture direction of the image capture device 101. Area division is performed based on this direction. For example, axis 1301 represents the image capture direction of the image capture device 101 when powered on, or a predetermined reference direction for the image capture direction. Area 1302 represents the angle of view of the image captured by the image capture unit 206. Figure 6D An example of a through image captured by the imaging unit 206 in the area 1302 is shown. Figure 6D In, based on Figure 6C As shown in the area division, the image area within the viewing angle of the through image is divided into areas 1303 to 1318.

[0073] (2) Calculation of the importance level of each region

[0074] Based on the subjects in the area and the scene conditions of the area, the first control unit 223 calculates the importance level of each area obtained by the above division. The importance level represents the order of priority when performing the subject search. The importance level based on the condition of the subject is calculated, for example, based on the number of subjects in the area, the size of the subject's face, the direction of the subject's face, the certainty of the subject's face detection, the subject's facial expression and the subject's character recognition result. For example, the importance level based on the scene condition is calculated based on general object recognition results, scene judgment results (such as blue sky, backlight and night scene), the level of sound coming from the area and the voice recognition result, and motion detection information in the area. At this time, the first control unit drives the camera 101 to search the entire surrounding environment.

[0075] For example, when the face of the subject is registered in the imaging device 101, the first control unit 223 increases the importance level of the region where the face of the subject registered in the imaging device 101 is detected. For example, the face of the subject is recorded as a pattern for determining the subject in the non-volatile memory 216. When the importance level of the region where the face of the subject is detected is increased, the first control unit 223 restores the importance level of the region to the original level after a predetermined time has passed or a predetermined number of imaging operations have been performed.

[0076] (3) Determination of search area

[0077] After determining the importance level of each area as described above, the first control unit 223 determines to intensively search for areas with high importance levels.The first control unit 223 then calculates the pan and tilt angles for capturing images of areas with high importance levels.

[0078] In step S505, the first control unit 223 performs panning and tilting. More specifically, based on the image shake correction amount and the pan and tilt angles calculated in step S504, the first control unit 223 calculates the panning and tilting drive amounts. Based on the calculated panning and tilting drive amounts, the first control unit 223 controls the tilting rotation unit 104 and the panning rotation unit 105, respectively, via the lens barrel rotation drive unit 205. In this exemplary embodiment, it is assumed that the first control unit 223 detects a subject in an area with a high level of importance by performing the panning and tilting drives in step S505 and begins capturing an image of the subject. The first control unit 223 then controls the lens barrel rotation drive unit 205 to track the subject (to keep the subject within the angle of view).

[0079] In step S506, the first control unit 223 controls the zoom unit 201 to perform zoom drive. For example, based on the state of the subject whose image capture has started in step S505, the first control unit 223 drives the zoom unit 201. For example, if the face of the subject within the angle of view is very small, the first control unit 223 controls the zoom unit 201 to zoom toward the telephoto side so that the image of the subject's face is captured at an appropriate (larger) size within the angle of view. On the other hand, if the face of the subject within the angle of view is very large, the first control unit 223 controls the zoom unit 201 to zoom toward the wide-angle side so that the image of the subject's face is captured at an appropriate (smaller) size within the angle of view. This zoom control can maintain a state suitable for tracking the subject.

[0080] In steps S504, S505, and S506, a method for performing a subject search using pan drive, tilt drive, and zoom drive has been described. Alternatively, a camera system that uses multiple wide-angle lenses to capture images in all directions at once can perform a subject search. In this case, the overall processing load for image processing (such as subject detection) using all signals obtained by omnidirectional imaging as input images is high. Therefore, in this case, the first control unit 223 is configured to cut out a portion of the image obtained by omnidirectional imaging and perform subject search processing in the cut-out image area.

[0081] With this configuration, the first control unit 223 calculates the importance level of each area in a manner similar to the above-described method, changes the cut-out position based on the importance level, and performs automatic image capturing judgment to be described below.

[0082] This enables subject search to be performed at high speed while suppressing power consumption of image pickup processing.

[0083] In step S507 , the first control unit 223 determines whether to perform automatic imaging.

[0084] The determination of whether to perform automatic image capture will now be described. The determination of whether to perform automatic image capture is made based on whether the image capture score exceeds a predetermined value. The image capture score is a parameter used to determine whether to perform automatic image capture. The image capture score increases based on the subject detection status and the elapsed time. For example, assume that the first control unit 223 is designed to perform automatic image capture when the image capture score exceeds 2000 points. In this case, the image capture score has an initial value of 0 points and increases based on the time that has elapsed since entering the automatic image capture mode. For example, the image capture score increases at a rate that reaches 2000 points within 120 seconds. If 120 seconds have passed without a subject being detected, the image capture score reaches 2000 points based on the increase in elapsed time, and automatic image capture is performed. If a high-priority subject is detected during the elapsed time, 1000 points are added to the image capture score. For example, among subjects whose faces are registered in the image capture device 101, a high-priority subject is a subject that the user has set as a high-priority image capture target. In the case where a high-priority subject is detected, the imaging score can more easily reach 2000 points, and thus the imaging frequency may be increased.

[0085] In addition, for example, in the case where a smiling face of a subject is recognized, 800 points are added to the video recording score. Scores based on smiling faces are added not only for high-priority subjects but also for other subjects. In this typical embodiment, an example will be described in which the same score is added based on smiling faces regardless of whether the subject is high priority or not. However, this is not restrictive. For example, a higher score can be added in response to detecting a smiling face of a high-priority subject than in response to detecting a smiling face of a low-priority subject. This enables video recording to be closer to the user's intention. In the case where the video recording score exceeds 2000 points due to the addition of a score based on such emotional changes in the subject's facial expression, automatic video recording is performed. Even in the case where the video recording score does not exceed 2000 points after adding the score based on the change in facial expression, the video recording score reaches 2000 points faster by adding points based on the passage of time.

[0086] As an example, a linear increase in the score over time will be described. For example, if 2000 points are added within 120 seconds, 2000 / 120 points are added every second. However, this is not restrictive. For example, the camera score can be increased in such a way that no points are added until 110 seconds out of 120 seconds, and 200 points are added every second to reach 2000 points within the 10 seconds from 110 to 120 seconds. This prevents the camera score from reaching a value that would initiate camera recording regardless of the priority level due to the addition of points based on changes in the subject's facial expression. Using a method that linearly increases the camera score over time is less likely to reflect the priority level. This is because, if the camera score has already increased over a long period of time, the score may frequently reach a value that initiates camera recording due to the addition of points based on the change of a low-priority subject to a smiling face. On the other hand, it is not desirable to reduce the points added based on changes in facial expression, because if the score is reduced, the moment of the facial expression change may be missed. In view of this, the first control unit 223 is configured not to increase the imaging score until 110 seconds have passed. If a low-priority subject is present, 110 seconds have passed without adding any points. Conversely, if a high-priority subject is present, 1000 points are added upon detection of the high-priority subject. Therefore, even if the time-based score is not added until 110 seconds have passed, 1000 points have already been added before 110 seconds.

[0087] Therefore, when adding points based on changes in the subject's facial expression, the likelihood of the camera score reaching a value sufficient to initiate filming can be lower when a low-priority subject is present than when a high-priority subject is present. This makes it more likely that the priority level will be reflected. While the above description uses changes in the subject's facial expression as an example, other criteria for adding points can be used. Examples of criteria include whether speech becomes louder and whether gestures become larger. Similarly, using such criteria, the scoring method can be changed as described above to make it more likely that the priority level will be reflected.

[0088] Even if the image capture score does not exceed 2000 points based on the subject's behavior, an image is always captured within 120 seconds based on the passage of time. This prevents a period of time when no image is captured.

[0089] If a subject is detected midway through the 120-second time period, the incrementing time can be started earlier. More specifically, for example, if a high-priority subject is detected after 60 seconds, 1000 points are added, but the camera score does not exceed 2000 points. Then, the linear increment can begin 30 seconds after the subject is detected, rather than waiting until 110 seconds. Alternatively, the linear increment can begin 20 seconds earlier than 120 seconds, rather than 10 seconds earlier. This increases the likelihood of capturing images of high-priority subjects, making it easier to capture images that are more closely aligned with the user's intent.

[0090] After automatic recording is performed, the recording score is reset to 0. Automatic recording will not be performed until the recording score exceeds 2000 points again.

[0091] The above is a description of determining whether to perform automatic imaging. If the control unit 223 performs the above determination and determines to perform automatic imaging ("Yes" in step S507), the process proceeds to step S508. If the first control unit 223 determines not to perform automatic imaging ("Yes" in step S507), the process proceeds to step S501.

[0092] In step S508, the first control unit 223 performs an image capturing process. Examples of the image capturing process include a still image capturing process and a moving image capturing process.

[0093] The above is a description of the automatic imaging process performed by the imaging apparatus 101 according to the present exemplary embodiment. By performing the above-described process for automatically capturing an image of a subject, the imaging apparatus 101 can capture an image or moving image desired by the user without an imaging instruction from the user.

[0094] The first control unit 223 may repeatedly perform the subject search process and the image capture process for a predetermined period of time, and repeatedly perform the sleep process for entering the sleep state for a predetermined period of time. In the sleep process, the first control unit 223 controls the first power supply unit 210 to turn off the power supply to the first control unit 223 while the second control unit 211 is operating.

[0095] This enables the first control unit 223 to automatically capture an image or moving image of a scene desired by the user while suppressing power consumption. In order to distinguish it from the aforementioned continuous imaging process, this automatic imaging process will also be referred to as intermittent imaging process.

[0096] <Speech Recognition Processing>

[0097] Figure 7 : is a flowchart illustrating voice recognition processing performed by the imaging apparatus 101 according to the present exemplary embodiment.

[0098] The processing of this flowchart starts when the audio input / audio processing unit 409 detects the input of the audio signal from the microphone. Figure 5 In this exemplary embodiment, the automatic image capture processing of Figure 5 The flowchart is started when the audio input / audio processing unit 409 detects input of an audio signal from a microphone during the automatic image pickup process. This process is realized by the first control unit 223 executing a program recorded in the nonvolatile memory 216.

[0099] In step S701, the first control unit 223 determines whether a trigger word is detected. The trigger word is a start command for starting to recognize a voice command for giving a specific instruction to the camera 101 by voice. In the case where the user gives an instruction to the camera 101 by voice, the user speaks a command word after the trigger word and causes the camera 101 to recognize the command word. If the trigger word is detected ("Yes" in step S701), the process proceeds to step S702. If the trigger word is not detected ("No" in step S701), the process returns to step S701. The process of step S701 is repeated until the trigger word is detected.

[0100] In step S702, the first control unit 223 suspends the automatic camera processing. In this exemplary embodiment, when the trigger word is detected, the first control unit 223 transfers to the command word waiting state. More specifically, the first control unit 223 suspends the reference Figure 5The flowchart of describes the subject search process (in step S504), the drive process (in step S505 and step S506), and the camera process (in step S508) in the automatic camera process. Examples of drive processes include pan drive, tilt drive, and zoom drive. Examples of camera processes include still image shooting and moving image shooting. On the other hand, in step S702, the first control unit 223 does not pause, and continues the image recognition process for subject detection (in step S502). The reason for pausing the automatic camera process when the trigger word is detected as described above is as follows. It is considered that the user recognizes the state that the camera device 101 is facing the subject that the user desires to shoot, and speaks the trigger word at the moment when the user desires to shoot the subject. However, based on the detection of the trigger word, the first control unit 223 cannot determine whether to perform the camera process because no command word indication is detected. At the moment of recognizing the command word after the trigger word, the first control unit 223 determines to perform the camera process. In other words, in the case where the first control unit 223 continues the automatic video recording process after detecting the trigger word, the direction of the video recording device 101 when recognizing the command word may be different from the direction in which the user desires to capture the image. In order to solve this problem, in response to detecting the trigger word, the first control unit 223 suspends the automatic video recording process, thereby maintaining the video recording direction in which the video recording device 101 is facing at the moment when the user desires to capture the image. Then, in the case where the user speaks the command word for giving a video recording instruction after the trigger word, the first control unit 223 is able to perform video recording processing to capture the image of the subject that the user desires to capture. In other words, the first control unit 223 is able to capture the image of the subject that the user desires to capture by suspending the drive process.

[0101] In step S703 , the first control unit 223 outputs a detection sound from the speaker to notify the user that the trigger word has been detected.

[0102] In step S704, after detecting the trigger word, the first control unit 223 determines whether a command word is detected. If a command word is detected ("Yes" in step S704), the process proceeds to step S706. If a command word is not detected ("No" in step S704), the process proceeds to step S705.

[0103] In step S705, the first control unit determines whether a predetermined time has passed since the trigger word was detected and the command word waiting state was entered. If the predetermined time has passed ("yes" in step S705), the process proceeds to step S701. In step S701, the first control unit 223 cancels the command word waiting state and enters the trigger word waiting state. At this point, the first control unit 223 resumes the automatic camera processing. If the predetermined time has not yet passed ("no" in step S705), the process proceeds to step S704. The first control unit 223 repeats the processes of steps S704 and S705 until the command word is detected.

[0104] In step S706, the first control unit 223 determines whether the detected command word is a still image shooting command. A still image shooting command is a command for causing the imaging apparatus 101 to perform imaging and recording processing for a single still image. If the detected command is determined to be a still image shooting command ("YES" in step S706), the process proceeds to step S707. If the detected command is determined not to be a still image shooting command ("NO" in step S706), the process proceeds to step S708.

[0105] In step S707 , the first control unit 223 performs still image capturing processing. More specifically, the first control unit 223 causes the image processing unit 207 to convert the signal captured by the image capturing unit 206 into a JPEG format, and causes the image recording unit 208 to record the image file on the recording medium 221 .

[0106] In step S708, the first control unit 223 determines whether the detected command word is a subject change command. An example of a subject change command is the key phrase "photograph another person." If the detected command word is determined to be a subject change command ("Yes" in step S708), the process proceeds to step S709. If the detected command word is determined not to be a subject change command ("No" in step S708), the process proceeds to step S710.

[0107] In step S709, the first control unit 223 performs a subject change process. In the subject change process, the first control unit 223 controls the drive of the lens barrel 102 to capture a subject other than the subject currently being captured. If no subject is detected when the command word is detected, this step is skipped.

[0108] In step S710, the first control unit 223 determines whether the detected command word is a moving image recording start command. A moving image recording start command is a command that causes the imaging device 101 to perform moving image capture and recording processing. If the detected command word is determined to be a moving image recording start command ("Yes" in step S710), the process proceeds to step S711. If the detected command word is determined not to be a moving image recording start command ("No" in step S710), the process proceeds to step S712.

[0109] In step S711, the first control unit 223 starts capturing a moving image by using the image capturing unit 206, and records the captured moving image data on the recording medium 221. During recording of the moving image data, the first control unit 223 does not perform subject search and maintains pause of the automatic imaging process.

[0110] In step S712, the first control unit 223 determines whether the detected command word is a moving image recording end command. If the detected command word is determined to be a moving image recording end command ("Yes" in step S712), the process proceeds to step S713. If the detected command word is determined not to be a moving image recording end command ("No" in step S712), the process proceeds to step S714.

[0111] In step S713 , the first control unit 223 stops capturing an image of the subject using the imaging unit 206 and recording moving image data on the recording medium 221 , and ends recording the moving image data.

[0112] In step S714, the first control unit 223 performs processing corresponding to any other command words. For example, the first control unit 223 performs processing corresponding to a command word for performing panning and tilting in the directions specified by the user, or a processing corresponding to a command word for changing various imaging parameters of the imaging apparatus 101, including exposure correction parameters.

[0113] In steps S715 and S716, the first control unit 223 performs processing for resuming the automatic image pickup processing that was suspended in step S702. For example, the first control unit 223 performs processing from the processing of step S502. Figure 5 The process of the flowchart shown.

[0114] The above describes the voice recognition processing performed by the imaging apparatus 101 according to the present exemplary embodiment.

[0115] As described above, in the voice recognition processing of the imaging apparatus 101 according to the present exemplary embodiment, the imaging apparatus 101 suspends the automatic imaging processing in response to detection of a trigger word. More specifically, Figure 7 In step S702 of the flowchart shown, the first control unit 223 suspends the automatic image capture process in response to detecting the trigger word, thereby maintaining the image capture direction that the image capture device 101 was facing at the moment the user desired to capture an image. Therefore, if the user speaks a command word for instructing the camera to capture an image after the trigger word, the first control unit 223 performs the image capture process at the same viewing angle as when the trigger word was received. As a result, the image of the subject desired by the user can be captured.

[0116] In step S702, the first control unit 223 may control the driving speed of the lens barrel 102 in the imaging direction to be lower than the driving speed immediately before the trigger word is detected, rather than suspending the automatic imaging process. In the present exemplary embodiment, the first control unit 223 controls the driving speed of the lens barrel 102 in the imaging direction to be lower than the driving speed of the lens barrel 102 in the imaging direction during the automatic imaging process, taking into account the processing time for detecting the command word.

[0117] If the instruction of the command word is not a shooting instruction, the first control unit 223 then returns the driving speed to the original speed and continues the automatic shooting process. This makes it easier for the camera 101 to bring the shooting direction desired by the user into the angle of view and to continue changing the angle of view during the automatic shooting process.

[0118] If a subject is being detected at the moment the trigger word is detected, then in step S702, the first control unit 223 can control the drive of the lens barrel 102 to track the subject instead of pausing the drive process. This is because it is assumed that the subject being detected by the camera device 101 at the moment the user speaks the trigger word is the subject the user desires to capture, and by continuing to drive the lens barrel 102 to track the subject, it is more likely that the user's desired image will be obtained. In this case, the first control unit 223 controls the drive of the lens barrel 102 to track the subject being captured at the moment the trigger word is detected (to keep the subject within the field of view). As described above, by driving the camera direction to track the subject being detected at the moment the trigger word is detected, the first control unit 223 can capture an image of the subject the user desires to capture. In the case where a subject is being detected at the moment the trigger word is detected, in step S702, the first control unit 223 can similarly control the drive of the lens barrel 102 to track the subject instead of pausing the drive process.

[0119] If no subject is detected at the time the trigger word is detected, the first control unit 223 may control the drive of the lens barrel 102 to perform subject search processing in step S702. For example, the first control unit 223 may perform subject search processing within a range that includes the current camera direction and is narrower than the search range of the automatic camera process. As described above, the first control unit 223 can capture an image of the subject desired by the user by searching the vicinity of the current camera direction for the subject in response to the detection of the trigger word.

[0120] As described above, the imaging device 101 not only suspends automatic imaging processing in response to detection of a trigger word, but also tracks the subject being imaged at the moment the trigger word is detected, or limits the driving range in response to detection of a trigger word.

[0121] This enables the imaging apparatus 101 to capture an image of a subject that the user desires to capture.

[0122] In step S702, the first control unit 223 may stop executing the still image capturing process or the moving image capturing process. This reduces the load of the image capturing process, so that the first control unit 223 can quickly respond to the instruction of the command word issued by the user after the trigger word.

[0123] If it is determined in step S705 that the predetermined time has elapsed, the first control unit 223 may output a sound indicating that the predetermined time has elapsed from the audio output unit 218. Since the user can recognize that the camera 101 has failed to detect the command word, the user can more easily know when to give a voice instruction again.

[0124] When the first control unit 223 is performing intermittent imaging processing as automatic imaging processing, in response to detecting the trigger word, the first control unit 223 may postpone execution of the sleep processing until the processing indicated by the command word is completed. In this case, in response to completion of the processing indicated by the command word, the first control unit 223 executes the sleep processing in the intermittent imaging processing.

[0125] The exemplary embodiments of the present invention may also be implemented by supplying a program for implementing one or more functions according to the aforementioned exemplary embodiments to a system or device via a network or storage medium, and causing one or more processors in a computer in the system or device to read and execute the program. The exemplary embodiments of the present invention may also be implemented by a circuit (such as an application-specific integrated circuit (ASIC)) for implementing one or more functions according to the aforementioned exemplary embodiments.

[0126] The present invention is not limited to the aforementioned exemplary embodiments and can be implemented by modifying the components without departing from the main purpose thereof. In addition, various inventions can be formed by appropriately combining the multiple components described in the aforementioned exemplary embodiments. For example, some of the components described in the exemplary embodiments may be omitted. Components according to different exemplary embodiments may be appropriately combined.

[0127] Other embodiments

[0128] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0129] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A camera device comprising: an imaging unit configured to capture an image of a subject; a driving unit configured to drive a driving member configured to change an imaging direction of the imaging unit; as well as a detection unit configured to detect a first voice command for performing a specific process and a second voice command for starting detection of the first voice command, wherein the driving unit limits driving of the driving member in response to the detection unit detecting the second voice command, and Wherein, in a case where the driving unit restricts driving of the driving member, the driving unit drives the driving member at a speed lower than a driving speed of the driving member immediately before the detection unit detects the second voice command.

2. The imaging device according to claim 1, wherein When the detection unit detects the first voice command for performing image pickup processing while the drive unit is restricting the driving of the drive member, the image pickup unit captures an image of a subject.

3. The imaging device according to claim 1 , further comprising: a determination unit configured to automatically determine an imaging target; as well as A control unit is configured to control automatic imaging processing, wherein the automatic imaging processing is processing for causing the drive unit to drive the drive member so that the imaging unit captures an image of the imaging target determined by the determination unit.

4. The imaging device according to claim 3, wherein The determination unit calculates an importance level of each detected subject based on information about the face of the detected subject, and determines the imaging target based on the importance level.

5. The imaging device according to claim 4, wherein The determination unit calculates the importance level of the subject based on a priority level set by a user, a facial expression of the subject, an eye size of the subject, and a facial direction of the subject. The imaging device according to claim 3 , wherein: During execution of the automatic image capturing process, the detection unit detects the first voice command and the second voice command.

7. The imaging device according to claim 3, wherein When the detection unit detects the second voice command during execution of the automatic image capturing process, the automatic image capturing process is suspended.

8. The imaging device according to claim 3, wherein The driving unit restricts driving of the driving member in a case where the detection unit detects the second voice command during execution of the automatic image capturing process.

9. The imaging device according to claim 3, wherein In the automatic image pickup process, the drive unit drives the drive member with the entire surrounding environment of the image pickup apparatus as a driving range.

10. The imaging device according to claim 3, wherein In a case where the drive unit restricts driving of the drive member, the drive unit drives the drive member within a drive range narrower than a drive range in the automatic image capturing process.

11. The imaging device according to claim 1, wherein In a case where the driving unit restricts driving of the driving member, the driving unit drives the driving member so that the subject being imaged when the second voice command is detected falls within a viewing angle.

12. The imaging device according to claim 1, wherein In a case where the driving unit limits the driving of the driving member, the driving unit drives the driving member to search for an object during the period in which the driving of the driving member is limited.

13. The imaging device according to claim 1, wherein In a case where the driving unit restricts the driving of the driving member, the driving unit stops the driving of the driving member.

14. The imaging device according to claim 1, further comprising a microphone, in, The detection unit detects the first and second voice commands by analyzing an audio signal input from the microphone.

15. A control method for a camera device, the control method comprising: capturing an image of a subject; driving a driving member configured to change a shooting direction during the shooting; detecting a first voice command for performing a specific process and a second voice command for starting detection of the first voice command; as well as limiting actuation of the actuation member in response to detecting the second voice command, Wherein, in the case where the driving of the driving member is restricted, the driving member is driven at a speed lower than a driving speed of the driving member immediately before the second voice command is detected.

16. A non-transitory computer-readable recording medium having recorded thereon a program for causing an imaging apparatus to perform a control method, the control method comprising: capturing an image of a subject; driving a driving member configured to change a shooting direction during the shooting; detecting a first voice command for performing a specific process and a second voice command for starting detection of the first voice command; as well as limiting actuation of the actuation member in response to detecting the second voice command, Wherein, in the case where the driving of the driving member is restricted, the driving member is driven at a speed lower than a driving speed of the driving member immediately before the second voice command is detected.

17. A computer program product comprising a program for causing an imaging device to perform a control method, the control method comprising: capturing an image of a subject; driving a driving member configured to change a shooting direction during the shooting; detecting a first voice command for performing a specific process and a second voice command for starting detection of the first voice command; as well as limiting actuation of the actuation member in response to detecting the second voice command, Wherein, in the case where the driving of the driving member is restricted, the driving member is driven at a speed lower than a driving speed of the driving member immediately before the second voice command is detected.

Images