Voice input device, control method thereof and storage medium

Abstract

The present invention relates to a voice input device, a control method thereof, and a storage medium. The voice input device inputs voice and performs control so that, when a second voice instruction for operating the voice input device is input within a fixed time period after a first voice instruction for enabling operation of the voice input device using voice is input, processing corresponding to the second voice instruction is performed. In a case where it is estimated that a predetermined user has issued the second voice instruction, when the second voice instruction is input, the voice input device performs processing corresponding to the second voice instruction even if the first voice instruction is not input.

Description

Technical Field

[0001] The present invention relates to a voice input device for executing a process corresponding to a voice instruction, a control method thereof, and a storage medium. Background Art

[0002] In recent years, devices provided with a voice operation function capable of recognizing a spoken voice input and performing an operation corresponding to the voice input are known. For example, an operation for displaying a desired page for viewing, an operation for changing a current setting value of a device, and an operation for calling a specific person or inputting the text of an email message can be performed using voice through the voice operation function. Although the voice operation function has the advantage of being able to operate using natural speech, these devices may also react to unexpected voices, resulting in erroneous operations.

[0003] Japanese Patent Laid-Open No. 2019-86535 proposes a technology that reduces the occurrence of erroneous operations by enabling voice operation by receiving a specific voice command called a wake-up word for a device capable of voice operation, and receiving voice operation only within a predetermined time period after the wake-up word. In addition, Japanese Patent Laid-Open No. 2017-54065 proposes a technology for recognizing the state of a user's mouth from a subject image, and receiving voice operation when the mouth is recognized as open.

[0004] The problem with the technology proposed in Japanese Patent Laid-Open No. 2019-86535 is that in order to perform voice operation, it is necessary to always input the wake-up word, which is time-consuming and laborious. When operating a camera, it is sometimes desirable to quickly change settings to avoid missing photo opportunities. In the case where it takes time and effort to always input the wake-up word, not only will simple operations become cumbersome, but also when performing voice operation after giving the wake-up word, it is possible to miss the desired photo opportunity.

[0005] In addition, using the technology proposed in Japanese Patent Laid-Open No. 2017-54065, in situations such as when a user's hand covers his or her mouth (for example, when making a peace sign in front of his or her mouth), a situation may occur where voice operation cannot be performed because the user's mouth cannot be recognized from the image. Summary of the invention

[0006] The present invention has been made in view of the above-mentioned problems, and realizes a technology for suppressing the occurrence of erroneous operations while enabling quick operations when performing voice operations.

[0007] In order to solve the above-mentioned problem, one aspect of the present invention provides a voice input device, comprising: a voice input unit, which is configured to input voice; and a control unit, which is configured to control so that when a second voice instruction for operating the voice input device is input within a fixed time period after a first voice instruction for enabling operation of the voice input device using voice is input, processing corresponding to the second voice instruction is performed, wherein, in a case where it is estimated that the second voice instruction is issued by a predetermined user, when the second voice instruction is input, the control unit also performs processing corresponding to the second voice instruction even if the first voice instruction is not input.

[0008] Another aspect of the present invention provides a control method for a voice input device, the voice input device including a voice input device configured to input voice, the control method including: performing control so that when a second voice instruction for operating the voice input device is input within a fixed time period after a first voice instruction for enabling operation of the voice input device using voice is input, processing corresponding to the second voice instruction is performed, wherein in the control, when the second voice instruction is estimated to be issued by a predetermined user, processing corresponding to the second voice instruction is performed when the second voice instruction is input even if the first voice instruction is not input.

[0009] Another aspect of the present invention provides a non-transitory computer-readable storage medium, which includes instructions for performing a control method for a voice input device, the voice input device including a voice input device configured to input voice, the control method including: performing control to execute processing corresponding to a second voice instruction for operating the voice input device when the second voice instruction is input within a fixed time period after a first voice instruction for enabling operation of the voice input device using voice is input, wherein, in the control, when the second voice instruction is estimated to be issued by a predetermined user, when the second voice instruction is input, processing corresponding to the second voice instruction is executed even if the first voice instruction is not input.

[0010] According to the present invention, when voice operation is performed, the occurrence of erroneous operation can be suppressed while enabling quick operation.

[0011] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A and Figure 1B is a diagram showing the appearance of a digital camera serving as an example of the voice input device according to the present invention.

[0013] Figure 2 is a block diagram showing an example functional configuration of a digital camera according to the first embodiment.

[0014] Figure 3 is a flowchart showing a series of operations of the main process according to the first embodiment.

[0015] Figure 4A and Figure 4B : is a flowchart showing a series of operations of the wake-up word omission processing according to the first embodiment.

[0016] FIG. 5A to FIG. 5C is a diagram showing the state of the digital camera in the first embodiment.

[0017] Figure 6 is a block diagram showing an example functional configuration of a digital camera according to the second embodiment.

[0018] Figure 7 is a block diagram showing an example functional configuration of a mobile phone serving as an example of a communication device according to the second embodiment.

[0019] Figure 8 : is a flowchart showing a series of operations of the wake-up word omission processing according to the second embodiment.

[0020] Fig. 9 is a flowchart showing a series of operations of the main process according to the second embodiment. DETAILED DESCRIPTION

[0021] First embodiment

[0022] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but are not limited to the invention requiring all such features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are assigned to the same or similar configurations, and redundant descriptions thereof are omitted.

[0023] Hereinafter, an example of using a digital camera that can be operated using voice recognition as an example of a voice input device will be described. However, the present embodiment is applicable not only to digital cameras but also to other devices that can be operated using voice recognition. For example, these devices may include smartphones, personal computers, game consoles, tablet terminals, music players, and medical equipment.

[0024] Digital camera configuration

[0025] Figure 1A and Figure 1BAn external view of a digital camera 100 serving as an example of a voice input device according to the present embodiment is shown. Figure 1A is a front perspective view of the digital camera 100, and Figure 1B 1 is a perspective view of the back of the digital camera 100. The display unit 28 is a display unit provided on the back of the camera and displays images and various information. The touch panel 70a is capable of detecting touch operations on the display surface (operation surface) of the display unit 28. The viewfinder external display unit 43 is a display unit provided on the top of the camera and displays various setting values ​​of the camera (including shutter speed and aperture). The shutter button 61 is an operation unit for giving a shooting instruction. The mode conversion switch 60 is an operation unit for switching between various modes. The terminal cover 40 is a cover for protecting a connector (not shown) for connecting a connection cable of an external device to the digital camera 100.

[0026] The main electronic dial 71 is a rotating operating member included in the operation unit 70, and by rotating the main electronic dial 71, processing such as changing the shutter speed, aperture, and other setting values ​​can be performed. The power switch 72 is an operating member for switching between turning on (ON) and turning off (OFF) the power of the digital camera 100. The sub-electronic dial 73 is a rotating operating member included in the operation unit 70, and can move the selected frame, and perform image jump, etc. The cross key 74 is an operating member included in the operation unit 70, and has a push button that can be pressed in four directions. Operations can be performed according to the direction in which the cross key 74 is pressed. The set (SET) button 75 is a push button included in the operation unit 70, and is mainly used to set the selected item, etc. The moving image button 76 is used to instruct the start and stop of moving image shooting (recording). The voice input button 77 is a button that can limit the operation using voice to be enabled only when the button is pressed according to the setting of the digital camera 100. However, in the following description, the following case will be shown as an example: a setting value that enables input of a voice regardless of whether the voice input button 77 is pressed is set, and various types of processing are operated in response to the voice operation. The zoom button 78 is an operation button included in the operation unit 70, and is used to switch the zoom mode on and off in the live view display of the shooting mode. By operating the main electronic dial 71 after setting the zoom mode to on, zooming in and out of the live view image can be performed. In the playback mode, the zoom button 78 is used as a zoom button for enlarging the playback image and increasing the magnification. The play button 79 is an operation button included in the operation unit 70, and is used to switch between the shooting mode and the playback mode. By pressing the play button 79 during the shooting mode, the mode can be turned into the playback mode, and the latest image among the images recorded on the recording medium 200 can be displayed on the display unit 28. The menu button 81 is included in the operation unit 70, and by pressing the menu button 81, a menu screen in which various settings can be performed is displayed on the display unit 28. The user can configure various settings using the menu screen displayed on the display unit 28 as well as the cross key 74 and the setting button 75 .

[0027] The communication terminal 10 is a communication terminal for enabling the digital camera 100 to communicate with a lens unit 150 (detachable) described later. The eyepiece unit 16 is an eyepiece unit of an eyepiece viewfinder (a viewfinder of a user-peep-through type), and the user can see a video displayed on an internal EVF (Electronic Viewfinder) 29 through the eyepiece unit 16. The eye approach detection unit 57 is an eye approach detection sensor for detecting whether the eye of the image shooter has approached the eyepiece unit 16.

[0028] The cover 202 is a cover of a slot for accommodating the recording medium 200. The grip portion 90 is a holding portion shaped so that it can be easily held with the right hand when the user holds the digital camera 100 to prepare for shooting. The shutter button 61 and the main electronic dial 71 are arranged at positions operable with the index finger of the right hand in a state where the digital camera 100 is held with the little finger, ring finger, and middle finger of the right hand around the grip portion 90. In addition, the sub-electronic dial 73 is arranged at a position operable with the thumb of the right hand in the same state.

[0029] Next, refer to Figure 2 An example functional structure of the digital camera 100 according to the present embodiment is described. Figure 2 , the lens unit 150 is a lens unit equipped with an interchangeable photographic lens. The lens 103 is generally composed of a plurality of lenses, but only one lens is shown here for simplicity. The communication terminal 6 is a communication terminal for the lens unit 150 to communicate with the digital camera 100. The lens unit 150 communicates with the system control unit 50 via the communication terminal 6 and the above-mentioned communication terminal 10, and controls the aperture 1 via the aperture drive circuit 2 using the internal lens system control circuit 4. Thereafter, focusing is performed by shifting the lens 103 via the AF drive circuit 3.

[0030] The shutter 101 is a focal plane shutter capable of freely controlling the exposure time of the imaging unit 22 through the control of the system control unit 50 .

[0031] The imaging unit 22 is an image sensor composed of a CCD or CMOS device or the like that converts an optical image into an electric signal. The A / D converter 23 is used to convert an analog signal output by the imaging unit 22 into a digital signal.

[0032] The image processing unit 24 performs color conversion processing and predetermined resizing processing such as pixel interpolation and reduction on the data from the A / D converter 23 or the data from the memory control unit 15 described later. In addition, the image processing unit 24 performs predetermined calculation processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the calculation results obtained by the image processing unit 24. TTL (through the lens) AF (auto focus) processing, AE (auto exposure) processing, and EF (electronic flash) processing are thereby performed. The image processing unit 24 also performs predetermined calculation processing using the captured image data, and performs TTL AWB (auto white balance) processing based on the obtained calculation results.

[0033] The memory control unit 15 controls data transmission and reception between the A / D converter 23, the image processing unit 24, and the memory 32. The output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the camera unit 22 and converted into digital data by the A / D converter 23, and image data for display on the display unit 28 and the EVF 29. The memory 32 is provided with a sufficient storage capacity to store a predetermined number of still images or a predetermined time length of moving images and audio.

[0034] In addition, the memory 32 also serves as a memory (video memory) for image display. The image data for display written in the memory 32 is displayed by the display unit 28 or the EVF 29 via the memory control unit 15. The display unit 28 and the EVF 29 perform display depending on the signal from the memory control unit 15 on a display device such as an LCD or an organic electroluminescent display. Live view display (LV display) can be performed by sequentially transferring the data A / D converted by the A / D converter 23 and stored in the memory 32 to the display unit 28 or the EVF 29 and displaying the data. Hereinafter, the image displayed in live view is referred to as a live view image (LV image).

[0035] The infrared light emitting diode 166 is a light emitting element for detecting the user's line of sight position on the viewfinder screen, and irradiates the eyeball (eye) 161 of the user whose eye has approached the eyepiece unit 16 with infrared light. The infrared light emitted by the infrared light emitting diode 166 is reflected by the eyeball (eye) 161, and the infrared reflected light reaches the dichroic mirror 162. The dichroic mirror 162 reflects only infrared light and transmits visible light. The infrared reflected light whose optical path has been changed forms an image on the imaging surface of the line of sight detection sensor 164 via the imaging lens 163. The imaging lens 163 is an optical component that constitutes the line of sight detection optical system. The line of sight detection sensor 164 is composed of a camera device such as a CCD image sensor.

[0036] The sight line detection sensor 164 photoelectrically converts the incident infrared reflected light into an electric signal, and outputs the electric signal to the sight line detection circuit 165. The sight line detection circuit 165 includes at least one processor, and detects the sight line position of the user according to the image or movement of the eyeball (eye) 161 of the user based on the output signal of the sight line detection sensor 164, and outputs the detection information to the system control unit 50. In this way, the sight line detection block 160 is composed of a dichroic mirror 162, an imaging lens 163, a sight line detection sensor 164, an infrared light emitting diode 166, and a sight line detection circuit 165. The sight line detection block 160 detects the sight line, for example, by the so-called corneal reflection method. The corneal reflection method is a method involving detecting the orientation and position of the sight line according to the positional relationship between the pupil of the eyeball (eye) 161 and the reflected light generated by the infrared light emitted by the infrared light emitting diode 166 being reflected by the cornea of ​​the eyeball (eye) 161. There are also various other methods for detecting the orientation and position of the sight line, such as the so-called scleral reflection method, which utilizes the difference in light reflection between the iris and the white of the eye. Note that the visual line detection unit may use any other method than the above that can detect the visual line orientation and position.

[0037] Various setting values ​​(including shutter speed and aperture) of the digital camera 100 are displayed on the out-of-viewfinder display unit 43 via the out-of-viewfinder display unit drive circuit 44 .

[0038] The nonvolatile memory 56 is an electrically erasable / recordable memory and uses, for example, a flash ROM. Constants and programs used for the operation of the system control unit 50 are stored in the nonvolatile memory 56. The program referred to here is a computer program for executing various flowcharts described later in this embodiment.

[0039] The system control unit 50 includes, for example, one or more processors such as a CPU, and controls the entire digital camera 100. Each process of the present embodiment described later is realized by extracting a program recorded on the nonvolatile memory 56 to the system memory 52 and executing it with the system control unit 50. For example, a RAM is used for the system memory 52, and constants, variables, programs, and the like read from the nonvolatile memory 56 used in the operation of the system control unit 50 are extracted to the RAM. The system control unit 50 also performs display control by controlling the memory 32, the display unit 28, and the like.

[0040] The system timer 53 is a timing unit for measuring the time used in various controls and the time of the built-in clock. The mode conversion switch 60, the first shutter switch 62, the second shutter switch 64, and the operation unit 70 are operation units for inputting various operation instructions to the system control unit 50. The mode conversion switch 60 switches the operation mode of the system control unit 50 to one of a still image shooting mode, a moving image shooting mode, and the like. The modes in the still image shooting mode include an automatic shooting mode, an automatic scene discrimination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, there are various scene modes configured to shoot specific scene shooting settings, and a custom mode, and the like. Using the mode conversion switch 60, the user can directly switch to any one of these modes. Alternatively, the following structure can be adopted: after initially switching to the shooting mode list screen using the mode conversion switch 60, one of the multiple displayed modes is selected, and switching is performed using another operation member. Similarly, the moving image shooting mode may also include multiple modes.

[0041] The first shutter switch 62 is turned on when the shutter button 61 provided in the digital camera 100 is partially operated with a so-called half-press (shooting preparation instruction), and generates a first shutter switch signal SW1. Shooting preparation operations such as AF (auto focus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (electronic flash) processing are started with the first shutter switch signal SW1.

[0042] The second shutter switch 64 is turned on when the shutter button 61 is fully operated in a so-called full press (shooting instruction), and generates a second shutter switch signal SW2. The system control unit 50 uses the second shutter switch signal SW2 to start a series of shooting processing operations from reading out a signal from the image pickup unit 22 to writing a shot image as an image file to the recording medium 200.

[0043] The operation unit 70 is various types of operation members used as an input unit for receiving operations from the user. As described above, the operation unit 70 includes the shutter button 61, the touch panel 70a, the main electronic dial 71, the sub-electronic dial 73, the cross key 74, and the setting button 75. The operation unit 70 also includes a moving image button 76, a voice input button 77, a zoom button 78, a play button 79, and a menu button 81.

[0044] The power control unit 80 is composed of a battery detection circuit, a DC-DC converter, and a switch circuit for switching the power supply block, etc., and detects whether the battery is installed, the type of battery, and the remaining battery. In addition, the power control unit 80 controls the DC-DC converter based on the detection result and the instruction of the system control unit 50, and supplies the required voltage to various components including the recording medium 200 within the required time period. The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, and an AC adapter.

[0045] The recording medium I / F 18 is an interface with a recording medium 200 which is a memory card, a hard disk, or the like. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0046] The communication unit 54 includes, for example, a communication circuit or a communication module, which is connected wirelessly or with a cable, and performs transmission and reception of video signals and audio signals. The communication unit 54 can also be connected to wireless communication for mobile phones, a wireless LAN (local area network), and the Internet. The communication unit 54 can transmit images (including live view images) captured by the camera unit 22 and images recorded on the recording medium 200, and can receive images and various other information from external devices.

[0047] The posture detection unit 55 detects the posture of the digital camera 100 relative to the gravity direction. Based on the posture detected by the posture detection unit 55, the image captured by the camera unit 22 can be distinguished as an image captured by the digital camera 100 held horizontally or an image captured by the digital camera 100 held vertically. For the system control unit 50, orientation information depending on the posture detected by the posture detection unit 55 can be added to the image file of the image captured by the camera unit 22, and the image can be rotated and recorded. As for the posture detection unit 55, an acceleration sensor, a gyro sensor, etc. can be used. It is also possible to use an acceleration sensor, a gyro sensor, etc. as the posture detection unit 55 to detect the movement (pan, pitch, rise, whether it is still, etc.) of the digital camera 100.

[0048] The eye approach detection unit 57 is an eye approach detection sensor for detecting that the eye (object) 161 has moved closer (approaching) or has moved away (moving away) from the eyepiece unit 16 of the viewfinder (approach detection). The system control unit 50 switches between display (display state) / non-display (non-display state) of the display unit 28 and the EVF 29 according to the state detected by the eye approach detection unit 57. More specifically, at least in the case where the digital camera 100 is in the shooting standby state and the setting for switching the display destination of the live view image captured by the imaging unit 22 is set to automatic switching, when the eye has not yet approached the eyepiece unit 16, the display setting is turned on with the display unit 28 as the display destination, and the EVF 29 is set to non-display. In addition, when the eye has approached the eyepiece unit 16, the display setting is turned on with the EVF 29 as the display destination, and the display unit 28 is set to non-display. For example, an infrared proximity sensor may be used for the eye proximity detection unit 57, and the eye proximity detection unit 57 is capable of detecting the proximity of a certain object to the eyepiece unit 16 of the viewfinder including the EVF 29. In the case where the object has approached the eyepiece unit 16, infrared light projected from a projection unit (not shown) of the eye proximity detection unit 57 is reflected and received by a light receiving unit (not shown) of the infrared proximity sensor. The eye proximity detection unit 57 is also capable of distinguishing the degree to which the object has approached the eyepiece unit 16 (eye proximity distance) according to the amount of infrared light received. In this way, the eye proximity detection unit 57 performs eye proximity detection for detecting the proximity distance of the object to the eyepiece unit 16. Note that in the present embodiment, the projection unit and the light receiving unit of the eye proximity detection unit 57 are devices separate from the above-mentioned infrared light emitting diode 166 and the line of sight detection sensor 164. However, the infrared light emitting diode 166 may also be used as the projection unit of the eye proximity detection unit 57. In addition, the line of sight detection sensor 164 may also be used as the light receiving unit. The case where an object is detected to be approaching the eyepiece unit 16 within a predetermined distance from a non-eye approach state (non-approach state) will be regarded as detecting that the user's eyes have approached the eyepiece unit 16. The case where the object detected to be approaching moves away from the eye approach state (approach state) by a predetermined distance or more will be regarded as detecting that the user's eyes have moved away from the eyepiece unit 16. For example, due to setting a hysteresis, the threshold for detecting the user's eyes approaching and the threshold for detecting the user's eyes moving away may be different. In addition, from after detecting that the user's eyes have approached until detecting that the user's eyes have moved away, it will be regarded as the eye approach state. From after detecting that the user's eyes have moved away until detecting that the user's eyes have approached, it will be regarded as the non-eye approach state. Note that an infrared proximity sensor is given as an example, and any sensor capable of detecting that the eye or an object regarded as the user's eye has approached the eyepiece unit 16 can be used for the eye approach detection unit 57.

[0049] The system control unit 50 is capable of detecting the following operation or status based on the output of the visual line detection block 160 .

[0050] - The line of sight of the user whose eyes have approached the eyepiece unit 16 is newly input (detected). That is, the start of line of sight input.

[0051] - There is a state where the user's line of sight input is close to the eyepiece unit 16 .

[0052] - A state in which the user whose eyes have approached the eyepiece unit 16 is looking steadily.

[0053] - The line of sight of the user whose eyes have approached the eyepiece unit 16 has shifted. That is, the line of sight input ends.

[0054] A state in which there is no visual line input of the user whose eyes have approached the eyepiece unit 16 .

[0055] The stable gaze described here refers to a situation where the user's line of sight does not move more than a predetermined amount within a predetermined time.

[0056] The touch panel 70a and the display unit 28 may be integrally configured. For example, the touch panel 70a is configured so that the transmittance of light does not interfere with the display on the display unit 28, and the touch panel 70a is attached to the upper layer of the display surface of the display unit 28. The input coordinates of the touch panel 70a are associated with the display coordinates on the display screen of the display unit 28. Therefore, a GUI (Graphical User Interface) that looks as if the user directly operates the screen displayed on the display unit 28 can be provided. The system control unit 50 is capable of detecting the following operations on the touch panel 70a or the state of the touch panel 70a.

[0057] - A finger or pen that was not touching the touch panel 70a newly touches the touch panel 70a. That is, the start of touch (hereinafter referred to as touchdown).

[0058] - A state in which a finger or a pen is touching the touch panel 70a (hereinafter referred to as touch-on).

[0059] - The finger or pen moves away while touching the touch panel 70a (hereinafter referred to as touch movement).

[0060] - The finger or pen touching the touch panel 70a is lifted up. That is, the touch ends (hereinafter referred to as touch stop).

[0061] - A state in which nothing is touching the touch panel 70a (hereinafter referred to as untouched).

[0062] When a touch is detected, touch continuation is also detected at the same time. Touch continuation is usually detected after a touch as long as touch stop is not detected. Touch movement is also detected in a state where touch continuation is detected. Even when touch continuation is detected, touch movement is not detected unless the touch position moves. Untouched occurs after touch stop has been detected for all touching fingers and pens.

[0063] These operations and states, as well as the position coordinates when the finger or pen is touching the touch panel 70a, are notified to the system control unit 50 via the internal bus. The system control unit 50 determines what kind of operation (touch operation) is performed on the touch panel 70a based on the notified information. With regard to touch movement, the moving direction of the finger or pen moving on the touch panel 70a can also be determined for each vertical component and horizontal component on the touch panel 70a based on the change in the position coordinates. The case where a touch movement of a predetermined distance or longer is detected will be regarded as being determined to have been performed. An operation involving quickly moving a certain distance while the finger is touching the touch panel and then directly lifting the finger is called a flick. In other words, a flick is an operation involving quickly tracking on the touch panel 70a as if flicking the touch panel 70a with a finger. When a touch movement of a predetermined distance or more is detected at a predetermined speed or more, and then the touch stops, it can be determined that a flick has been performed (it can be determined that there is a flick after the sliding operation). In addition, a touch operation involving touching multiple locations (e.g., two points) at the same time and moving the touch locations closer together is called pinching, and a touch operation involving moving multiple touch locations further apart is called splitting. Splitting and pinching are collectively referred to as a pinching operation (or simply pinching). Any touch panel using various methods can be used for the touch panel 70a, such as a resistive touch panel, a capacitive touch panel, a surface acoustic wave touch panel, an infrared touch panel, an electromagnetic induction touch panel, an image recognition touch panel, and an optical sensor touch panel. Depending on the method, contact with the touch panel is detected as a touch, or a finger or a pen approaching the touch panel is detected as a touch, and any method can be used.

[0064] The voice input unit 85 includes a microphone not shown, and performs processing for acquiring environmental sounds around the digital camera and voice instructions from the user described later. The voice input unit 85 can input voices from a plurality of microphones not shown arranged at different positions on the housing of the voice input device. In this case, the system control unit 50 can estimate the direction from which the voice is emitted based on the time lag of the input voice, etc.

[0065] Furthermore, the present invention is applicable not only to a digital camera itself but also to a control device that communicates with a digital camera (including a network camera) via wired or wireless communication and remotely controls the digital camera. As for the device that remotely controls the digital camera, for example, there are devices such as a smartphone, a tablet PC, and a desktop PC. By notifying commands for causing the digital camera to perform various operations and configure various settings from the control device side based on operations performed on the control device side and processing performed on the control device side, the digital camera can be controlled from a remote location. Furthermore, a structure can be adopted in which a live view image captured by a digital camera can be received via wired or wireless communication and displayed on the control device side.

[0066] A series of operations handled by the main

[0067] Next, we will refer to Figure 3 The main process according to the present embodiment is described. The operation of the main process is an operation from recognizing the user's voice to completing the voice operation. Note that each process of the main process is implemented by extracting the program stored in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50. In addition, the wake-up word is a specific voice instruction for enabling the operation of the digital camera 100 using voice.

[0068] In step S301, the system control unit 50 determines whether a voice instruction from the user has been received from the voice input unit 85. Based on the signal from the voice input unit 85, the system control unit 50 proceeds to step S302 if it is determined that a voice instruction from the user has been received from the voice input unit 85, and proceeds to step S309 if it is determined that a voice instruction from the user has not been received.

[0069] In step S302, the system control unit 50 performs a wake-up word omission process. Figure 4A and 4B The wake-up word omission processing is described in detail.

[0070] In step S303, the system control unit 50 determines whether the digital camera 100 is in the wake-up word omitted state. For example, the system control unit 50 determines whether the digital camera 100 is in the wake-up word omitted state based on the state information of the digital camera 100 stored in the memory 32 (may be stored in the system memory 52; this also applies hereinafter). If it is determined that the state of the digital camera 100 is the wake-up word omitted state, the system control unit 50 proceeds to step S304, and if it is determined that the state of the digital camera 100 is not the wake-up word omitted state, the system control unit 50 proceeds to step S305. As described below, normally, in the case where the input voice instruction is a wake-up word, the digital camera 100 enters the voice instruction enabled state, and performs processing corresponding to the voice instruction received thereafter. However, in the wake-up word omitted state, the digital camera 100 can perform processing corresponding to the voice instruction due to the first reception of the voice instruction without receiving the wake-up word (i.e., by omitting the reception of the wake-up word). In other words, by providing the wake-up word omitted state, it is possible to perform a quick operation when performing a voice operation.

[0071] In step S304, the system control unit 50 performs voice instruction processing according to the voice instruction input by the user. The voice instruction processing includes, for example, processing for executing shooting of still images and moving images according to the voice instruction from the user. Alternatively, the voice instruction processing may include processing for displaying a screen of "confirming the setting value of the shutter speed (Tv value) or the aperture value (Av value)" corresponding to the voice instruction or "paging through help / guide or photos" corresponding to the voice instruction according to the voice instruction.

[0072] In step S305 (i.e., in the case where the state of the digital camera 100 is the wake-up word non-omitted state), the system control unit 50 determines whether operation by voice command is enabled (voice command enabled state). For example, if it is determined that the state of the digital camera 100 is the voice command enabled state based on the state information of the digital camera 100 stored in the memory 32, the system control unit 50 proceeds to step S304. On the other hand, if it is determined that the digital camera 100 is not in the voice command enabled state, the system control unit 50 proceeds to step S306.

[0073] In step S306, the system control unit 50 determines whether the received voice instruction is a predetermined wake-up word. If the received voice instruction is determined to be a predetermined wake-up word (i.e., a specific voice instruction for enabling voice operation) because, for example, the voice instruction matches the wake-up word, the system control unit 50 proceeds to step S307. On the other hand, if it is determined that the received voice instruction is not a wake-up word, the system control unit 50 proceeds to step S312.

[0074] In step S307, the system control unit 50 sets the digital camera 100 to a state in which an operation using a voice command is enabled (i.e., a voice command enabled state) for a fixed period of time. As described above with respect to the processing of steps S303 to S305, when a voice command is received while the digital camera 100 is in the voice command enabled state, processing corresponding to the voice command is performed. At this time, the system control unit 50 stores the state in the state information of the digital camera 100 in the memory 32, and measures the duration of the voice command enabled state using a timer.

[0075] In step S308, the system control unit 50 displays, for example, on the display unit 28 Figure 5B The voice command activation status screen is shown. Figure 5B In the illustrated example, a guide 502 indicating that the state of the digital camera 100 is a voice instruction enabled state is displayed on the voice operable screen. Figure 5B In the illustrated example, the case where a guide is displayed is shown as an example, but the same state may be indicated using a predetermined icon.

[0076] In step S309, the system control unit 50 determines whether a fixed period of time has elapsed after the state of the digital camera 100 has been changed to the voice command enabled state. Based on the time measured by the timer, the system control unit 50 proceeds to step S310 if it is determined that the fixed period of time has elapsed after the change to the voice command enabled state, and proceeds to step S312 if it is determined that the fixed period of time has not elapsed after the change to the voice command enabled state.

[0077] In step S310, the system control unit 50 sets the state of the digital camera 100 to a state in which operation by voice command is disabled (voice command disabled state), and stores the state in the state information of the digital camera 100 in the memory 32. In step S311, the system control unit 50 displays, for example, a display on the display unit 28. Figure 5C The Voice Command Disable status screen is shown. Figure 5C In the example shown, a guide 503 indicating that the state of the digital camera 100 is a voice instruction disabled state is displayed on the voice operable screen. Figure 5C In the illustrated example, the case where a guide is displayed is shown as an example, but the same state may be indicated using a predetermined icon.

[0078] In this manner, in the case where a voice command is received during a fixed period of time after the state of the digital camera 100 is changed to the voice command enabled state, the system control unit 50 executes a process corresponding to the voice command. On the other hand, in the case where a voice command is not received during a fixed period of time after the state of the digital camera 100 is changed to the voice command enabled state, the system control unit 50 returns the state of the digital camera 100 to the voice command disabled state again to ensure that the voice command is not processed, thereby avoiding an erroneous operation based on the received voice command.

[0079] In step S312, the system control unit 50 determines whether there is a main process end instruction such as turning off the power switch 72. If it is determined that the main process end instruction is input based on a signal from the power switch 72 or the operation unit 70, for example, the system control unit 50 ends the main process. On the other hand, if it is determined that the main process end instruction has not been input, the system control unit 50 returns the process to step S301 to repeat the above-mentioned process. Note that in the event of an abnormal operation such as turning the mode dial during the above-mentioned operation, the system control unit 50 may interrupt a series of operations of the main process in progress.

[0080] A series of operations for wake-up word omission processing

[0081] Next, a series of operations of the wake-up word omission processing performed in step S302 will be described with reference to 4A and 4B. The operation of the wake-up word omission processing is an operation for controlling the state of the digital camera 100 to the wake-up word omission state based on the position information of the subject recognized from the image and the voice direction of the recognized voice. Note that, similar to the main processing, the wake-up word omission processing is implemented by extracting the program stored in the nonvolatile memory 56 into the system memory 52 and executing it by the system control unit 50.

[0082] In step S401, the system control unit 50 determines whether a subject has been recognized in the captured image output by the imaging unit 22. As a result of the image recognition processing on the captured image, if it is determined that a certain type of subject has been recognized, the system control unit 50 proceeds to step S403, and if it is determined that a subject has not been recognized, the system control unit 50 proceeds to step S402.

[0083] In step S402, the system control unit 50 changes the state of the digital camera 100 to the wake-up word non-omitted state, and stores the state in the state information of the digital camera in the memory 32. Note that the wake-up word non-omitted state is a state in which voice operation cannot be performed by voice instruction until the user inputs the wake-up word. The wake-up word non-omitted state has sub-states of a voice instruction enabled state in which voice operation is enabled after the wake-up word is received and a voice instruction disabled state in which voice operation is prohibited because the wake-up word is not received.

[0084] In step S403, the system control unit 50 acquires facial information including features such as eyes, nose, and mouth of the recognized subject, and stores the acquired facial information in the memory 32. In step S404, the system control unit 50 determines whether the facial information of the recognized subject matches the facial information of a user registered in advance. If it is determined that the facial information of the recognized subject matches the facial information of the registered user, the system control unit 50 proceeds to step S405, and if it is determined that the facial information of the recognized subject does not match the facial information of the registered user, the system control unit 50 proceeds to step S402.

[0085] In step S405, the system control unit 50 acquires voice information, and stores the acquired voice information in the memory 32, wherein the voice information includes features such as the volume of the voice received by the voice input unit 85 and the voice print of the voice. In step S406, the system control unit 50 determines whether the voice information received by the voice input unit 85 matches the voice information of the registered user. If it is determined that the voice information of the voice received by the voice input unit 85 matches the voice information of the registered user, the system control unit 50 proceeds to step S407. On the other hand, if it is determined that the voice information of the voice received by the voice input unit 85 does not match the voice information of the registered user, the system control unit 50 proceeds to step S402.

[0086] In step S407, the system control unit 50 determines whether the voice volume of the voice received by the voice input unit 85 is greater than or equal to the fixed voice volume. If it is determined that the voice volume of the voice received by the voice input unit 85 is greater than or equal to the fixed voice volume, the system control unit 50 proceeds to step S408, and if it is determined that the voice volume of the voice received by the voice input unit 85 is not greater than or equal to the fixed voice volume, the system control unit 50 proceeds to step S402. By determining voice information and voice volume in addition to the result of face recognition using an image in this manner, erroneous operations in which words inadvertently spoken by the user while facing the digital camera 100 are manipulated as voice instructions can be reduced.

[0087] In step S408, the system control unit 50 acquires information about the operation mode currently set in the digital camera 100 from the nonvolatile memory 56, and stores the acquired operation mode information in the memory 32. In step S409, the system control unit 50 determines whether the current operation mode of the digital camera 100 is the shooting mode. If it is determined that the current operation mode of the digital camera 100 is the shooting mode, the system control unit 50 proceeds to step S410, and if it is determined that the current operation mode is not the shooting mode, the system control unit 50 proceeds to step S402. In this way, the wake-up word can be omitted according to whether the current operation mode is the shooting mode (i.e., the operation mode requiring immediacy), so that quick operation can be performed when performing voice operation, and the risk of erroneous operation can be reduced.

[0088] In step S410, the system control unit 50 determines whether the voice instruction received by the voice input unit 85 is a predetermined shooting instruction. If it is determined that the voice instruction received by the voice input unit 85 is a predetermined shooting instruction, the system control unit 50 proceeds to step S411, and if it is determined that the voice instruction received by the voice input unit 85 is not a predetermined shooting instruction, the system control unit 50 proceeds to step S402. In this way, if a configuration is adopted that enables the omission of the wake-up word only when a specific voice instruction is input, even when the operating mode is a shooting mode, the wake-up word can be omitted only for voice instructions (shooting instructions) that require immediacy. That is, quick operation of the required voice operation becomes possible, and the risk of erroneous operation is further reduced.

[0089] In step S411, the system control unit 50 acquires the direction of the recognized subject and the direction from which the speech is emanated, and stores the acquired subject direction information and speech emanation direction information in the memory 32. For example, the system control unit 50 estimates the position (e.g., direction and distance) from the digital camera 100 to the subject based on the size and position of the face region of the subject in the captured image. In addition, the system control unit 50 calculates the speech emanation direction based on, for example, the time lag of the speech input from the above-mentioned plurality of microphones.

[0090] In step S412, the system control unit 50 determines whether the direction of the subject recognized by the face matches the direction of the voice emission. If it is determined that the direction of the recognized subject matches the direction of the voice emission, the system control unit 50 proceeds to step S413, and if it is determined that the direction of the recognized subject does not match the direction of the voice emission, it proceeds to step S402. At this time, if the direction of the subject recognized by the face matches the direction of the voice emission, it can be estimated that the user as the subject has issued a voice instruction.

[0091] In step S413, the system control unit 50 changes the state of the digital camera 100 to the wake-up word omitted state, and stores the state information of the digital camera 100 in the memory 32. By adopting this configuration, in a case where it can be estimated that a voice instruction is issued by the user who is the subject based on the image recognition of the subject and the recognition of the utterance direction of the voice, the wake-up word can be omitted.

[0092] In step S414 , the system control unit 50 displays a display indicating that the digital camera is in the wake-up word omitted state on the display unit 28 . Figure 5A An example is shown in which a guide 501 indicating a wake-up word omitted state is shown on a voice operable screen. Note that in this example, a case where a guide is displayed is shown as an example, but the same state can be indicated using an icon.

[0093] Note that in the above description, an example in which the present embodiment is performed using an image captured by the front side of the digital camera 100 is explained. However, an image capture unit not shown may be further arranged on the back side of the digital camera 100 (i.e., the image capture unit may be provided on both the front side and the back side like a smartphone), and the above embodiment may be applied to a user on the back side of the digital camera 100. Alternatively, in a device having image capture units on both the front side and the back side, it may be determined whether the direction of the subject to be facially recognized matches the direction in which the voice is uttered, further taking into account whether the image is captured by the front side or the back side camera. Of course, the above embodiment may also be applied to a device in which the image capture unit is arranged in a cylindrical housing.

[0094] In addition, in the above description, the case where the device for recognizing the subject is integrated with the device for inputting the voice is explained as an example. However, for example, the present embodiment can also be applied to a configuration where these devices are separated, such as a configuration where an image is captured by a camera and a voice is input by a smartphone. In addition, in the present embodiment, the case where a digital camera is used is explained as an example, but the present embodiment can also be applied to a configuration where a shooting function and a voice input function are provided to a smartphone.

[0095] As described above, according to the present embodiment, since the voice operation is started by transmitting the wake-up word when the direction of the received voice is different from the direction in which the subject exists, erroneous operation can be prevented. In the case where the direction of the received voice is the same as the direction in which the subject exists, the voice operation can be started immediately by omitting the wake-up word, so that a user-friendly voice operation that is not susceptible to erroneous operation can be provided.

[0096] Second embodiment

[0097] Next, a second embodiment will be described. In the second embodiment, near field communication is used to measure the relative direction and distance of the user (another communication device), and the change of the state to the wake-up word omitted state or the voice instruction enabled state is controlled in consideration of the relative direction and distance. Therefore, the configuration of the digital camera 600 of the present embodiment further includes a near field communication unit in the configuration of the first embodiment, and the remaining configuration is the same or substantially the same as the first embodiment. In addition, in the present embodiment, part of the main processing and the wake-up word omission processing of the second embodiment are different from those of the first embodiment, and the remaining processing is the same or substantially the same. Therefore, the description of the same configuration and processing will be omitted, and the description will focus on the differences.

[0098] Configuration of Digital Camera 600

[0099] Figure 6 An example functional configuration of a digital camera 600 serving as an example of a voice input device according to the present embodiment is shown.

[0100] For example, the near field communication unit 601 includes an antenna for wireless communication and a modulation / demodulation circuit, a communication controller, etc. for processing wireless signals. The near field communication unit 601 implements near field communication that complies with the IEEE 802.15 standard (so-called Bluetooth (registered trademark)) by outputting a modulated wireless signal from the antenna and demodulating the wireless signal received by the antenna. In the present embodiment, for example, the Bluetooth (registered trademark) communication adopts the 5.1 version of Bluetooth (registered trademark) low energy consumption with low power consumption. Compared with wireless LAN communication, the Bluetooth (registered trademark) communication has a narrow communication range (i.e., a short communication distance). In addition, compared with wireless LAN communication, the communication speed of the Bluetooth (registered trademark) communication is slow. On the other hand, compared with wireless LAN communication, the power consumption of the Bluetooth (registered trademark) communication is small.

[0101] Note that the digital camera 600 in the present embodiment has a function of measuring the relative direction and distance of another Bluetooth (registered trademark) device as a communication device through Bluetooth (registered trademark) communication via the near field communication unit 601. The near field communication unit 601 transmits a direction detection signal from an antenna. The other Bluetooth (registered trademark) device calculates the relative direction and distance of the digital camera 600 based on Bluetooth (registered trademark) 5.1, and transmits the relative direction and distance to the digital camera 600. The system control unit 50 of the digital camera 600 calculates the relative position of the other Bluetooth (registered trademark) device based on the relative direction and distance of the other Bluetooth (registered trademark) device received via the near field communication unit 601, and holds the calculated relative position in the memory 32.

[0102] In the present embodiment, the communication speed of the communication realized by the communication unit 54 is faster than that of the communication realized by the near field communication unit 601. Also, the communication realized by the communication unit 54 has a wider communicable range than that of the communication performed by the near field communication unit 601.

[0103] The communication unit 54 of the digital camera 600 in the present embodiment has an AP mode in which the digital camera 600 operates as an access point in the infrastructure mode and a CL mode in which the digital camera 600 operates as a client in the infrastructure mode. By operating the communication unit 54 in the CL mode, the digital camera 600 in the present embodiment can operate as a CL device in the infrastructure mode. In the case where the digital camera 600 operates as a CL device, it is possible to participate in a network formed by nearby AP devices by connecting to the AP device. In addition, by operating the communication unit 54 in the AP mode, the digital camera 600 in the present embodiment can also operate as a simplified AP (hereinafter referred to as a simple AP), which is a type of AP but has limited functions. When the digital camera 600 operates as a simple AP, the digital camera 600 forms a network itself. It is possible that a device near the digital camera 600 recognizes the digital camera 600 as an AP device and participates in the network formed by the digital camera 600. As described above, it is assumed that a program for operating the digital camera 600 is retained in the non-volatile memory 56.

[0104] Note that the digital camera 600 in the present embodiment is a simple AP, and although it is an AP, it does not have a gateway function for transmitting data received from a CL device to an Internet provider, etc. Therefore, even when data is received from other devices participating in the network formed by the digital camera 600, the digital camera 600 cannot transmit the received data to a network such as the Internet, etc.

[0105] Configuration of mobile phone 700

[0106] As mentioned above, the measurement of relative direction and distance using near field communication requires a counterpart communication device. Figure 7 An example functional configuration of a mobile phone 700 as an example of a communication device of the present embodiment is described. Note that in the following description, an example in which a mobile phone is given as a communication device will be shown, but the communication device is not limited thereto. For example, the communication device may be a digital camera, a tablet device, or a personal computer capable of wireless communication.

[0107] The control unit 701 includes one or more processors such as a CPU, and controls each unit of the mobile phone 700 by executing a program described later and inputting signals. Note that instead of the control unit 701 controlling the entire device, the entire device may be controlled by a plurality of hardware sharing processes.

[0108] The imaging unit 702 converts the subject light formed by the lens included in the imaging unit 702 into an electric signal, performs noise reduction processing, etc., and outputs digital data as image data. After the captured image data is stored in the buffer memory, a predetermined calculation operation is performed on it by the control unit 701, and it is recorded in the recording medium 710.

[0109] The nonvolatile memory 703 is an electrically erasable / recordable memory. The nonvolatile memory 703 stores a program executed by the control unit 701 (i.e., an OS (operating system) as basic software), an application program that cooperates with the OS to realize an application function, and the like. In addition, in the present embodiment, the nonvolatile memory 703 stores an application program (also referred to as an app for short) for communicating with the digital camera 600. The working memory 704 is used as an image display memory of the display unit 706, a working area of ​​the control unit 701, and the like.

[0110] The operation unit 705 is used to receive instructions from the user for the mobile phone 700. The operation unit 705 includes operation members such as a power button for the user to instruct power on / off of the mobile phone 700 and a touch panel formed on the display unit 706.

[0111] The display unit 706 performs display of image data, text display for interactive operation, etc. Note that the mobile phone 700 does not necessarily have to be provided with the display unit 706. The mobile phone 700 only needs to be connectable to the display unit 706 and have at least a display control function for controlling the display of the display unit 706.

[0112] The recording medium 710 can record image data output by the camera unit 702. The recording medium 710 may be configured to be removable from the mobile phone 700 or may be built in the mobile phone 700. That is, the mobile phone 700 only needs to have at least a function of accessing the recording medium 710.

[0113] The communication unit 711 is a communication interface for connecting to an external device, and includes a communication circuit or a communication module. The mobile phone 700 of the present embodiment can exchange data with the digital camera 600 via the communication unit 711. In the present embodiment, the communication unit 711 includes an antenna, and the control unit 701 can be connected to the digital camera 600 via the antenna. Note that the control unit 701 can be directly connected to the digital camera 600 or can be connected via an access point. As for the protocol for communicating data, for example, PTP / IP (Picture Transfer Protocol based on Internet Protocol) via a wireless LAN can be used. Note that the communication with the digital camera 600 is not limited to this. For example, the communication unit 711 may include an infrared communication module, a Bluetooth (registered trademark) communication module, or a wireless communication module such as a wireless USB. In addition, the communication unit 711 may adopt a wired connection such as a USB cable, HDMI (registered trademark) or IEEE 1394.

[0114] For example, the near field communication unit 712 is composed of an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller. The near field communication unit 712 implements near field communication in accordance with IEEE 802.15 by outputting a modulated wireless signal from the antenna and demodulating the wireless signal received by the antenna. In the present embodiment, the near field communication unit 712 communicates with other devices according to IEEE802.15.1 (so-called Bluetooth (registered trademark)). In addition, in the present embodiment, the Bluetooth (registered trademark) communication adopts, for example, Bluetooth (registered trademark) low energy 5.1 (BLE) with low power consumption.

[0115] Note that the mobile phone 700 in this embodiment has a function of detecting the direction of a signal sent from another Bluetooth (registered trademark) device through Bluetooth (registered trademark) communication via a near field communication unit 712, and a function of measuring the distance to the device that sends the signal. In the case where a plurality of antennas for wireless communication are provided and a direction detection signal is received, the near field communication unit 712 calculates the relative direction of the device that sends the signal based on the phase difference of each antenna. The near field communication unit 712 measures the distance to the device that sends the signal based on the field strength of the received signal. In addition, the control unit 701 of the mobile phone 700 sends the detected signal direction and the measured distance to another Bluetooth (registered trademark) device through Bluetooth (registered trademark) communication via the near field communication unit 712.

[0116] The public network communication unit 713 is a communication interface used when performing public wireless communication, and includes a communication circuit or a communication module. The mobile phone 700 can call other devices via the public network communication unit 713. At this time, the control unit 701 implements the call by inputting and outputting voice signals via the microphone 714 and the speaker 715. In the present embodiment, the public network communication unit 713 is an antenna, and the control unit 701 can be connected to the public network via the antenna. Note that one antenna can be used as both the communication unit 711 and the public network communication unit 713.

[0117] A series of operations for wake-up word omission processing

[0118] Next, we will refer to the attached Figure 8 A series of operations of the wake-up word omission processing for setting the state of the digital camera to the wake-up word omission state when the digital camera 600 and the user face each other at a close distance is described. Note that this series of operations is realized by extracting the program stored in the nonvolatile memory 56 to the system memory 52 and executing it by the system control unit 50. In addition, in the case where, for example, the digital camera 600 and the user face each other at a close distance, the wake-up word omission processing is performed. Fig. 9 In the main process of the present embodiment shown in FIG. 1 , if it is determined that a voice command has been received (“Yes” in step S301), this series of operations (corresponding to Fig. 9 , or step S901 in the above procedure.

[0119] In step S801, the system control unit 50 calculates information about the position Oz of the user and the direction Od that the user faces, and stores the calculated information in the memory 32. For example, the system control unit 50 recognizes the face area of ​​the user in the image from the image data acquired by the camera unit 22, and calculates the position Oz of the user based on the size of the face and the coordinates of the face. Alternatively, the following method may be adopted, which involves calculating the position Oz of the user based on the relative direction and distance of the user obtained through the communication signal of the near field communication unit 601 and the mobile phone 700 carried by the user. The system control unit 50 recognizes the face area of ​​the user in the image from the image data acquired by the camera unit 22, and calculates the direction Od that the user faces based on the orientation of the face (for example, the vertical / horizontal tilt angle relative to the front). At this time, the system control unit 50 may store the face area of ​​the user in the image as the coordinates Ov in the image in the memory 32. In the case where multiple persons are detected in the image, the system control unit 50 may store the position Oz, the direction Od, and the coordinates Ov in the image for the number of detected persons in the memory 32. When the near field communication unit 601 receives the user's position Oz, the system control unit 50 can check the person (user) holding the mobile phone 700 with the person in the image by comparing the coordinates Ov in the image, and calculate the position Oz and direction Od.

[0120] In step S802, the system control unit 50 reads the position Oz and the direction Od from the memory 32, and determines whether the user is observed according to whether the position Oz and the direction Od exist. If it is determined that the user is observed, the system control unit 50 proceeds to step S803, and if this is not the case, proceeds to step S807.

[0121] In step S803, the system control unit 50 reads out the orientation of the optical axis of the imaging unit 22 and the direction Od in which the user faces from the memory 32, calculates the angle difference θ of the orientation (the angle θ at which the user and the digital camera face each other), and stores the calculated angle difference θ in the memory 32. The angle θ at which the user and the digital camera 600 face each other takes a value of 0 to 90 degrees, and takes a value of 0 degrees when the user and the digital camera 600 face each other directly.

[0122] In step S804, the system control unit 50 reads out the position Oz of the user and the distance threshold Zth by which it can be determined that the user is at a close distance from the memory 32, and determines whether the distance to the user is less than or equal to a predetermined distance threshold (less than or equal to the distance threshold Zth). If it is determined that the distance to the user is less than or equal to the distance threshold, the system control unit 50 proceeds to step S805, and if it is determined that the distance to the user is greater than the distance threshold, the system control unit 50 proceeds to step S807. In the case where the positions Oz and directions Od of a plurality of persons are read out from the memory 32, the system control unit 50 can change the distance threshold Zth to distinguish a normal conversation between persons from the voice input to the digital camera 600.

[0123] In step S805, the system control unit 50 reads out the angle θ at which the user and the digital camera 600 face each other, and the angle threshold θth that can be used to determine that the user and the digital camera face each other from the memory 32, and determines whether the angle θ is less than or equal to a predetermined angle threshold (less than or equal to the angle threshold θth). If it is determined that the angle θ is less than or equal to the angle threshold θth, the system control unit 50 proceeds to step S806, and if it is determined that the angle θ is greater than the threshold, the system control unit 50 proceeds to step S807. Note that in the case where the positions Oz and directions Od of a plurality of persons are read out from the memory 32, the system control unit 50 may change the angle threshold θth to distinguish a normal conversation between persons from the voice input to the digital camera 600.

[0124] In this way, if the user is at a close distance and the angular difference between the direction of the user's face and the orientation of the optical axis of the imaging unit 22 is less than or equal to the angular threshold, the system control unit 50 is able to estimate that the user and the digital camera 600 are facing each other and that the user as the subject has issued a voice instruction.

[0125] In step S806, the system control unit 50 reads the facing time T from the memory 32, counts the facing time T, and stores the counted time T in the memory 32. In step S807, the system control unit 50 reads the facing time T from the memory 32, clears the facing time T, and stores the cleared facing time T in the memory 32. The system control unit 50 thereafter returns the processing to step S801.

[0126] In step S808, the system control unit 50 reads out the facing time T and the time threshold Tth from the memory 32, and determines whether the facing time T is greater than or equal to a predetermined time threshold (greater than or equal to the time threshold Tth). If it is determined that the facing time T is greater than or equal to the time threshold Tth, the system control unit 50 proceeds to step S809, and if it is determined that the facing time T is less than the time threshold Tth, the system control unit 50 proceeds to step S801. The time threshold Tth is set to a predetermined time with which it can be determined that the camera and the user face each other, such as in an eye contact manner. In this way, the system control unit 50 can determine that the user and the digital camera 600 face each other by further considering the facing time. In the case where the positions Oz and directions Od of multiple people are read from the memory 32, the system control unit 50 can change the time threshold Tth to distinguish a normal conversation between people from the voice input to the digital camera 600. The time threshold Tth can be 0.

[0127] In step S809, the system control unit 50 sets the state of the digital camera 600 to the wake-up word omitted state, and stores the set state information in the memory 32. Thereafter, the system control unit 50 ends the wake-up word omitted processing.

[0128] A series of operations handled by the main

[0129] Fig. 9 1 shows a series of operations of the main process in this embodiment. In addition to the above-mentioned wake-up word omission process, the main process is different from the main process in the first embodiment in that the state of the digital camera 600 is set to the voice instruction enabled state using the wake-up word and the user's position. That is, the main process is different from the main process in the first embodiment in that the process of step S902 is performed after the judgment process in step S306.

[0130] In step S301 and steps S303 to S306, the system control unit 50 performs processing similar to the first embodiment. Furthermore, in step S901, the above-described wake-up word omission processing in the second embodiment is performed.

[0131] In step S902, the system control unit 50 reads out the user's position Oz and the distance threshold Zth2 from the memory 32, and the distance threshold Zth2 can be used to determine that the user is at a long distance. If it is determined that the user's position Oz is less than or equal to the predetermined distance threshold (less than or equal to the distance threshold Zth2), the system control unit 50 proceeds to step S307. On the other hand, if it is determined that the user's position Oz is a position farther than the distance threshold, the system control unit 50 returns to step S312. Thereafter, the system control unit 50 performs the processing starting from step S307 similarly to the first embodiment, and ends the main processing.

[0132] As described above, according to this embodiment, if the digital camera and the user face each other at a close distance, a voice command can be received without receiving a wake-up word. Therefore, it is possible to enable fast operation while suppressing the occurrence of erroneous operation when performing voice operation, and it is possible to provide the user with a more natural and intuitive UI.

[0133] Other embodiments

[0134] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0135] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A voice input device, comprising: A speech input unit configured to input speech; as well as a control unit configured to control, in a case where a second voice instruction for operating the voice input device is input within a fixed time period after a first voice instruction for enabling operation of the voice input device using voice is input, to execute processing corresponding to the second voice instruction, wherein, in a case where it is estimated that the predetermined user issued the second voice instruction, when the second voice instruction is input, the control unit also performs processing corresponding to the second voice instruction even if the first voice instruction is not input, and Among them, when it is estimated that the second voice instruction is issued by a predetermined user, if the voice input device is not set to a shooting mode, or if the voice input device is set to a shooting mode but the second voice instruction is not a shooting instruction, then when the first voice instruction is not input, the control unit does not perform processing corresponding to the second voice instruction.

2. The voice input device according to claim 1, further comprising: an imaging unit configured to output a photographed image including the predetermined user as a subject, Wherein, the control unit: calculating a first direction indicating a direction from the voice input device to the predetermined user based on the captured image, and calculating a direction in which the second voice instruction was issued as a second direction based on the second voice instruction, and When the first direction matches the second direction, it is estimated that the second voice instruction is issued by the predetermined user.

3. The voice input device according to claim 1, further comprising: an imaging unit configured to output a photographed image including the predetermined user as a subject, Wherein, the control unit: calculating a distance from the voice input device to the predetermined user based on the second voice instruction, and calculating a direction of a face of the subject in the captured image as a third direction, and When the distance to the predetermined user is less than or equal to a threshold value and it is determined that the fourth direction as the orientation of the voice input device and the third direction face each other, it is estimated that the predetermined user issued the second voice instruction.

4. The voice input device according to claim 1, further comprising: an imaging unit configured to output a photographed image including the predetermined user as a subject; as well as a communication unit configured to perform near field communication with a communication device of the predetermined user, Wherein, the control unit: calculating a distance from the voice input device to the communication device based on near field communication with the communication device, and calculating a direction of the face of the subject in the captured image as a third direction, and When the distance to the communication device is less than or equal to a threshold value and it is determined that the fourth direction as the orientation of the voice input device and the third direction face each other, it is estimated that the second voice instruction is issued by the predetermined user.

5. The voice input device according to claim 3, wherein: When the angle difference between the fourth direction determined by the orientation of the optical axis of the camera unit and the third direction is less than or equal to a predetermined angle threshold, the control unit determines that the fourth direction as the orientation of the voice input device and the third direction face each other.

6. The voice input device according to claim 5, wherein: When the angle difference between the fourth direction determined by the orientation of the optical axis of the camera unit and the third direction is less than or equal to a predetermined angle threshold, and the time during which the angle difference is less than or equal to the predetermined angle threshold is greater than or equal to a predetermined time threshold, the control unit determines that the fourth direction as the orientation of the voice input device and the third direction face each other.

7. The voice input device according to claim 3, wherein: In a case where the distance from the voice input device to the predetermined user is not less than or equal to the threshold value, the control unit does not perform a process corresponding to the second voice instruction even if the second voice instruction is input.

8. The voice input device according to claim 2, further comprising: a determination unit configured to determine whether the predetermined user is a registered user based on the face of the subject in the captured image or the voice of the second voice instruction, In which, when it is determined that the predetermined user is a registered user and it is estimated that the predetermined user issued the second voice instruction, when the second voice instruction is input, even if the first voice instruction is not input, the control unit performs processing corresponding to the second voice instruction.

9. The voice input device according to claim 2, wherein: The camera unit is arranged on at least one of a front side and a rear side of the voice input device.

10. The voice input device according to claim 1, wherein: If the volume of the second voice instruction is not greater than or equal to the fixed volume, the control unit does not perform a process corresponding to the second voice instruction.

11. The voice input device according to claim 1, wherein: The voice input unit is configured to acquire, from an external device, a voice input by the external device.

12. A control method for a voice input device, the voice input device comprising a voice input unit configured to input voice, the control method comprising: controlling to execute processing corresponding to a second voice instruction when a second voice instruction for operating the voice input device is input within a fixed time period after a first voice instruction for enabling operation of the voice input device using voice is input, wherein, in the control, in a case where it is estimated that the second voice instruction is issued by a predetermined user, when the second voice instruction is input, processing corresponding to the second voice instruction is performed even if the first voice instruction is not input, and Among them, when it is estimated that the second voice instruction is issued by a predetermined user, if the voice input device is not set to a shooting mode, or if the voice input device is set to a shooting mode but the second voice instruction is not a shooting instruction, then when the first voice instruction is not input, the processing corresponding to the second voice instruction is not performed.

13. A non-transitory computer-readable storage medium comprising instructions for performing a control method of a voice input device, the voice input device comprising a voice input unit configured to input a voice, the control method comprising: controlling to execute processing corresponding to a second voice instruction when a second voice instruction for operating the voice input device is input within a fixed time period after a first voice instruction for enabling operation of the voice input device using voice is input, wherein, in the control, in a case where it is estimated that the second voice instruction is issued by a predetermined user, when the second voice instruction is input, processing corresponding to the second voice instruction is performed even if the first voice instruction is not input, and Among them, when it is estimated that the second voice instruction is issued by a predetermined user, if the voice input device is not set to a shooting mode, or if the voice input device is set to a shooting mode but the second voice instruction is not a shooting instruction, then when the first voice instruction is not input, the processing corresponding to the second voice instruction is not performed.

Images