Electronic device and control method and program thereof

Abstract

To provide an electronic apparatus, a control method thereof, and a program for preventing operation on items not intended by a user and improving operability.SOLUTION: An electronic apparatus (digital camera 100) includes an operation unit 70 that accepts operations by a user, a display unit 28 that displays items, a line-of-sight detection unit 110 that detects a position of a user's line of sight with respect to the display unit, and a system control unit 50 that executes an operation corresponding to a first item assigned to an operation member or a second item assigned to an item displayed at the detected line-of-sight position. When the operation unit accepts the operation by the user and the item is not displayed at the detected line-of-sight position, the system control unit executes the operation corresponding to the first item assigned to the operation unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an electronic device capable of accepting operations based on detection of a user's line of sight, and a control method and program thereof. [Background technology]

[0002] Conventionally, electronic devices such as digital cameras are provided with operation members such as dials and buttons. For example, when a user is looking through a viewfinder, the user cannot see these operation members, making it difficult to accurately select an operation member and perform a desired operation. Therefore, if the user's line of sight is detected and the item related to the operation is determined according to the line of sight, the user can perform the operation while looking through the viewfinder.

[0003] For example, Patent Document 1 discloses a system that detects the gaze of a user observing a screen, selects a menu item corresponding to the identified gaze position, and then, when it detects a blink of the user's eye, determines the selected menu item. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-251658 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, specific items are also assigned in advance to operation members such as dials and buttons, so if a user is trying to operate a specific item by operating an operation member and another item is operated based on the result of detection of the user's line of sight, an operation will be performed on an item different from the user's intention, resulting in a problem of reduced operability.

[0006] In view of the above problems, the present invention aims to prevent an operation of an item other than an item pre-assigned to an operating member from being performed by operating the operating member when the user is not looking at a specific item, thereby preventing an operation from being performed on an item not intended by the user and improving operability. [Means for solving the problem]

[0007] In order to solve the above problems, one aspect of the present invention is an electronic device, comprising: an operating member that accepts an operation by a user; and a screen including a live view image different from the item. Display Department and the display On the predetermined screen displayed in the A gaze detection means for detecting a gaze position of a user; While the predetermined screen is being displayed By the user When an operation is performed on the operating member, a first process or a second process different from the first process is performed depending on whether or not the item is displayed at a gaze position detected by the gaze detection means on the predetermined screen. Control to execute control means for detecting the presence of the control The means are: On the predetermined screen, the line of sight detection means When the item is not displayed at the detected gaze position, assigned to the operating member Corresponding to the first item above The first process Control to execute and when the item is displayed at a gaze position detected by the gaze detection means on the predetermined screen, the second process corresponding to the second item assigned to the item is executed. The present invention is characterized by the above. Effect of the Invention

[0008] According to the present invention, in an electronic device capable of accepting operations based on detection of a user's line of sight, when the user is not directing his / her line of sight to a specific item, an operation member is prevented from being operated to operate items other than those previously assigned to the operation member, thereby preventing operations on items not intended by the user and improving operability. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing an external configuration of the electronic device. [Diagram 2] 1 is a block diagram showing an example of an internal configuration of an electronic device. [Diagram 3]13 is a flowchart illustrating an example of a target item changing process by operating an operating member. [Figure 4] FIG. 13 is a diagram showing a display example of a setting item screen. [Diagram 5] FIG. 2 is a screen display diagram showing the display of the EVF. [Figure 6] 11 is a diagram showing the external appearance and the internal configuration of an electronic device according to a second embodiment. FIG. [Figure 7] 11 is a flowchart showing an example of an IoT control process performed by operating an operating member. [Figure 8] 10 is a screen transition diagram showing the transition of screens displayed on a display unit of the electronic device of the second embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In each drawing, the same reference numerals are used for the same configurations as a rule, and duplicated explanations will be omitted. Furthermore, numerical values ​​and the like exemplified to make the explanation concrete are not limited to these unless otherwise specified.

[0011] Furthermore, the present invention is not limited to the following examples, and can be modified as appropriate without departing from the spirit of the invention. For example, the configurations of the following examples may be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions.

[0012] <Example 1> 1(a) and 1(b) show external views of a digital camera 100 as an example of an electronic device to which the present invention can be applied. Fig. 1(a) is a front perspective view of the digital camera 100, and Fig. 1(b) is a rear perspective view of the digital camera 100.

[0013] The display unit 28 is provided on the rear surface of the digital camera 100 and displays images and various information. The touch panel 70a can detect touch operations on the display surface (touch operation surface) of the display unit 28. The outside-finder display unit 43 is provided on the top surface of the digital camera 100 and displays various settings of the digital camera 100, including the shutter speed and aperture. The shutter button 61 is an operation member for issuing shooting instructions. The mode change switch 60 is an operation member for switching between various modes. The terminal cover 40 is a cover for protecting a connector (not shown) for a connection cable or the like that connects the digital camera 100 to an external device.

[0014] The main electronic dial 71 is a rotary operation member, and by turning the main electronic dial 71, settings such as the shutter speed and aperture can be changed. The power switch 72 is an operation member that switches the power of the digital camera 100 between ON and OFF. The sub electronic dial 73 is a rotary operation member, and by turning the sub electronic dial 73, the selection frame can be moved, images can be forwarded, and so on. The four-way key 74 is configured so that the up, down, left, and right parts can each be pressed, and processing can be performed according to the part of the four-way key 74 that is pressed. The SET button 75 is a push button, and is mainly used to confirm a selection item, and so on.

[0015] The video button 76 is used to start or stop video shooting (recording). The AE lock button 77 is a push button, and by pressing the AE lock button 77 in a shooting standby state, the exposure state can be fixed. The enlargement button 78 is an operation button for switching the enlargement mode ON and OFF in the live view display (LV display) in the shooting mode. By turning the enlargement mode ON and then operating the main electronic dial 71, the live view image (LV image) can be enlarged or reduced. In the playback mode, the enlargement button 78 functions as an operation button for enlarging the playback image or increasing its magnification ratio. The playback button 79 is an operation button for switching between the shooting mode and the playback mode. By pressing the playback button 79 during the shooting mode, the mode is switched to the playback mode, and the latest image among the images recorded in the recording medium 200 (described later) can be displayed on the display unit 28. The menu button 81 is a push button used to perform an instruction operation to display a menu screen, and when the menu button 81 is pressed, a menu screen on which various settings can be made is displayed on the display unit 28. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the four-way key 74, and the SET button 75.

[0016] The touch bar 82 (multifunction bar: M-Fn bar) is a line-shaped touch operation member (line touch sensor) capable of receiving touch operations. The touch bar 82 is disposed at a position where it can be touched (touched) by the thumb of the right hand when gripping the grip part 90 in a normal grip (a grip recommended by the manufacturer). The touch bar 82 is a reception section that can receive tap operations (operations in which the user touches and then releases the touch bar without moving within a predetermined period of time) and slide operations to the left and right (operations in which the user touches and then moves the touch position while still touching) on ​​the touch bar 82. The touch bar 82 is an operation member that is different from the touch panel 70a, and does not have a display function.

[0017] The communication terminal 10 is a communication terminal through which the digital camera 100 communicates with the lens unit 150 (described below; detachable) side. The eyepiece 16 is the eyepiece of the eyepiece finder 17 (a peer-in type finder), and the user can view the image displayed on the internal EVF 29 through the eyepiece 16. The eyepiece detection unit 57 is an eyepiece detection sensor that detects whether or not the user (photographer) has placed his / her eye close to the eyepiece 16.

[0018] Line-of-sight detection units 110a (not shown) and 110b are disposed above and below eyepiece viewfinder 17. When a user looks into eyepiece viewfinder 17, line-of-sight detection units 110a and 110b detect the user's line of sight and determine which location the user is looking at.

[0019] The lid 202 is a lid for a slot that stores a recording medium 200 (described later). The grip section 90 is a holding section shaped to be easily gripped by the right hand when the user holds the digital camera 100. The shutter button 61 and the main electronic dial 71 are arranged at positions that can be operated by the index finger of the right hand when the digital camera 100 is held by gripping the grip section 90 with the little finger, ring finger, and middle finger of the right hand. In the same state, the sub electronic dial 73 and the touch bar 82 are arranged at positions that can be operated by the thumb of the right hand. The thumb rest section 91 (thumb standby position) is a grip member provided on the rear side of the digital camera 100 at a position where it is easy to place the thumb of the right hand that is gripping the grip section 90 when none of the operation members are being operated. The thumb rest section 91 is made of a rubber member or the like for increasing the holding force (grip feeling).

[0020] FIG. 2 is a block diagram showing an example of the configuration of the digital camera 100. The lens unit 150 is a lens unit equipped with an interchangeable photographing lens. The lens 103 is usually composed of multiple lenses, but FIG. 2 shows only one lens for simplicity. The communication terminal 6 is a communication terminal through which the lens unit 150 communicates with the digital camera 100, and the communication terminal 10 is a communication terminal through which the digital camera 100 communicates with the lens unit 150. The lens unit 150 communicates with the system control unit 50 via these communication terminals 6 and 10. The lens unit 150 controls the aperture 1 via the aperture drive circuit 2 by the internal lens system control circuit 4. The lens unit 150 adjusts the focus by displacing the position of the lens 103 via the AF drive circuit 3 by the lens system control circuit 4.

[0021] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50 .

[0022] The imaging unit 22 is an imaging element configured with a CCD, a CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 22 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 23 converts the analog signal output from the imaging unit 22 into a digital signal.

[0023] The image processing unit 24 performs predetermined processing (pixel interpolation, resizing such as reduction, color conversion, etc.) on the data from the A / D converter 23 or the data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This allows TTL (through-the-lens) type AF (autofocus) processing, AE (autoexposure) processing, EF (flash pre-flash) processing, etc. to be performed. The image processing unit 24 further performs predetermined arithmetic processing using the captured image data, and performs TTL type AWB (auto white balance) processing based on the obtained arithmetic results.

[0024] The output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15. Alternatively, the output data from the A / D converter 23 is written to the memory 32 via the memory control unit 15 without going through the image processing unit 24. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, and image data to be displayed on the display unit 28 and EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0025] The memory 32 also serves as a memory (video memory) for image display. The D / A converter 19 converts the image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28 or the EVF 29. In this way, the image data for display written in the memory 32 is displayed by the display unit 28 or the EVF 29 via the D / A converter 19. The display unit 28 and the EVF 29 each perform display according to the analog signal from the D / A converter 19 on a display device such as an LCD or an organic EL. The digital signal that has been A / D converted by the A / D converter 23 and stored in the memory 32 is converted into an analog signal by the D / A converter 19, and the analog signal is sequentially transferred to the display unit 28 or the EVF 29 for display, thereby performing a live view display (LV). Hereinafter, an image displayed in the live view display is referred to as a live view image (LV image).

[0026] Various settings of the camera, including the shutter speed and aperture, are displayed on the outside viewfinder display section 43 via an outside viewfinder display section drive circuit 44 .

[0027] The non-volatile memory 56 is an electrically erasable and recordable memory, such as an EEPROM. Constants, programs, and the like for operating the system control unit 50 are recorded in the non-volatile memory 56. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0028] The system control unit 50 is a control unit consisting of at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 executes the programs recorded in the nonvolatile memory 56 described above to realize each process of this embodiment described below. The system memory 52 is, for example, a RAM, and the system control unit 50 loads constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like into the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 19, the display unit 28, and the like.

[0029] The system timer 53 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0030] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, and the like, and detects whether a battery is attached, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage for the necessary period to each unit including the recording medium 200. The power supply unit 30 is composed of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries, an AC adapter, and the like.

[0031] The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0032] The communication unit 54 transmits and receives video signals and audio signals to and from an external device connected wirelessly or via a wired cable. The communication unit 54 can also connect to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can receive image data and various other information from the external device.

[0033] The attitude detection unit 55 detects the attitude of the digital camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 55, it is possible to determine whether the image captured by the imaging unit 22 was captured with the digital camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. An acceleration sensor, a gyro sensor, or the like can be used as the attitude detection unit 55. It is also possible to detect the movement of the digital camera 100 (panning, tilting, lifting, whether it is stationary, etc.) using the acceleration sensor or gyro sensor that is the attitude detection unit 55.

[0034] The eye-contact detection unit 57 is an eye-contact detection sensor that detects (approach detection) the approach (eye-contact) and departure (eye-away) of the eye (object) to the eyepiece unit 16 of the eyepiece finder 17 (hereinafter simply referred to as the "finder"). The system control unit 50 switches between display (display state) / non-display (non-display state) of the display unit 28 and the EVF 29 according to the state detected by the eye-contact detection unit 57. More specifically, at least in the shooting standby state and when the display destination switching is automatic switching, when the eye is not placed in contact with the camera, the display is turned on with the display unit 28 as the display destination and the EVF 29 is not displayed. Also, when the eye is placed in contact with the camera, the display is turned on with the EVF 29 as the display destination and the display unit 28 is not displayed. For example, an infrared proximity sensor can be used as the eye-contact detection unit 57, and it can detect the approach of some object to the eyepiece unit 16 of the finder 17 incorporating the EVF 29. When an object approaches, infrared light projected from a light projecting section (not shown) of the eyepiece detection section 57 is reflected by the object and received by a light receiving section (not shown) of the infrared proximity sensor. The amount of infrared light received can also determine how close the object is to the eyepiece section 16 (eyepiece distance). In this way, the eyepiece detection section 57 performs eyepiece detection to detect the proximity of the object to the eyepiece section 16. When an object is detected approaching within a predetermined distance from the non-eyepiece state (non-approaching state), it is detected that the object has been placed in eye contact. When an object whose approach was detected moves away from the eyepiece state (approaching state) by a predetermined distance or more, it is detected that the object has been removed. The threshold for detecting eye contact and the threshold for detecting removal of the eye may be different, for example, by providing a hysteresis. In addition, after detecting the eye contact, the eyepiece remains in the eye contact state until removal of the eye is detected. After detecting the removal of the eye, the eyepiece remains in the non-eye contact state until removal of the eye is detected. The infrared proximity sensor is just one example, and other sensors may be used for eye proximity detection unit 57 as long as they can detect the approach of an eye or an object that can be regarded as an eye.

[0035] The gaze detection unit 110 is a gaze position detection sensor that determines at which part of the EVF 29 the user is gazing.

[0036] The operation unit 70 is an input unit that accepts operations from the user (user operations) and is used to input various operational instructions to the system control unit 50. As shown in Fig. 2, the operation unit 70 includes a mode changeover switch 60, a shutter button 61, a power switch 72, a touch panel 70a, a touch bar 82, etc. The operation unit 70 also includes, as other operation members 70b, a main electronic dial 71, a sub electronic dial 73, a four-way key 74, a SET button 75, a video button 76, an AE lock button 77, a magnification button 78, a playback button 79, a menu button 81, etc.

[0037] The mode changeover switch 60 changes the operation mode of the system control unit 50 to one of a still image shooting mode, a video shooting mode, a playback mode, etc. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, there are various scene modes and custom modes that are shooting settings according to shooting scenes. The user can directly switch to one of these modes using the mode changeover switch 60. Alternatively, after switching to a list screen of shooting modes with the mode changeover switch 60, the user may selectively switch to one of the displayed modes using another operating member. Similarly, the video shooting mode may also include multiple modes.

[0038] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. The first shutter switch 62 is turned on when the shutter button 61 is pressed halfway (instruction to prepare for shooting) and generates a first shutter switch signal SW1. The system control unit 50 starts preparation operations for shooting such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing in response to the first shutter switch signal SW1. The second shutter switch 64 is turned on when the shutter button 61 is pressed fully (instruction to shoot) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of operations for shooting processing, from reading out a signal from the imaging unit 22 to writing a captured image to the recording medium 200 as an image file, in response to the second shutter switch signal SW2.

[0039] The touch panel 70a and the display unit 28 can be integrally configured. For example, the touch panel 70a is configured so that the light transmittance does not interfere with the display of the display unit 28, and is attached to the upper layer of the display surface of the display unit 28. Then, input coordinates on the touch panel 70a are associated with display coordinates on the display surface of the display unit 28. This makes it possible to provide a GUI (Graphical User Interface) that makes it seem as if the user can directly operate the screen displayed on the display unit 28. The GUI is an example of a predetermined item in this embodiment. The system control unit 50 can detect the following operations or states on the touch panel 70a. A finger or pen that has not been touching the touch panel 70a touches the touch panel 70a again, that is, the start of touching (hereinafter referred to as Touch-Down). A state in which the touch panel 70a is touched with a finger or a pen (hereinafter referred to as Touch-On) A finger or a pen is moved while touching the touch panel 70a (hereinafter referred to as Touch-Move). The finger or pen that had been touching the touch panel 70a is released from the touch panel 70a, that is, the end of touch (hereinafter referred to as touch-up). A state in which nothing is touching the touch panel 70a (hereinafter referred to as Touch-Off)

[0040] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on will also be detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. After it is detected that all fingers or pens that were touching have touched up, touch-off will occur.

[0041] These operations and states, as well as the position coordinates of the touch panel 70a touched by a finger or pen, are notified to the system control unit 50 via the internal bus. The system control unit 50 then determines what kind of operation (touch operation) was performed on the touch panel 70a based on the notified information. For touch moves, the direction of movement of the finger or pen moving on the touch panel 70a can also be determined for each vertical and horizontal component on the touch panel 70a based on the change in position coordinates. If a touch move of a predetermined distance or more is detected, it is determined that a slide operation has been performed. An operation in which a finger is touched on the touch panel 70a, quickly moved a certain distance, and then released is called a flick. In other words, a flick is an operation in which a finger is quickly traced on the touch panel 70a as if flicking it. If a touch move of a predetermined distance or more at a predetermined speed or more is detected and a touch up is detected immediately, it can be determined that a flick has been performed (it can be determined that a flick has occurred following a slide operation). Furthermore, a touch operation in which multiple points (for example, two points) are touched together (multi-touch) and the touch positions are brought closer together is called pinch in, and a touch operation in which the touch positions are moved away from each other is called pinch out. Pinch out and pinch in are collectively called pinch operation (or simply pinch). The touch panel 70a may be of any of various touch panel types, such as a resistive film type, a capacitive type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type. There are types that detect a touch by contact with the touch panel, and types that detect a touch by the approach of a finger or a pen to the touch panel, and either type may be used.

[0042] In addition, the system control unit 50 can detect the following operations or states on the touch bar 82. A finger that has not been touching the touch bar 82 touches the touch bar 82 again, that is, the start of touching (hereinafter referred to as touch-down). A state in which the touch bar 82 is touched with a finger (hereinafter referred to as Touch-On) The finger is moved while touching the touch bar 82 (hereinafter referred to as Touch-Move). The finger that had been touching the touch bar 82 is released from the touch bar 82, that is, the end of touch (hereinafter referred to as touch-up). A state in which nothing is touching the touch bar 82 (hereinafter referred to as Touch-Off)

[0043] When touch-down is detected, touch-on is also detected at the same time. After touch-down, touch-on will usually continue to be detected unless touch-up is detected. If touch-move is detected, touch-on will also be detected at the same time. Even if touch-on is detected, touch-move will not be detected if the touch position does not move. After it is detected that all fingers or pens that were touching have touched up, touch-off will occur.

[0044] These operation states and the position coordinates where the finger touches the touch bar 82 are notified to the system control unit 50 through the internal bus, and the system control unit 50 determines what kind of operation (touch operation) has been performed on the touch bar 82 based on the notified information. For touch move, horizontal (left and right) movement on the touch bar 82 is detected. If it is detected that the touch position has moved a predetermined distance or more (the amount of movement of the touch operation is a predetermined amount or more), it is determined that a slide operation has been performed. If an operation is performed in which a finger touches the touch bar 82 and the touch is released within a predetermined time without performing a slide operation, it is determined that a tap operation has been performed. In this embodiment, the touch bar 82 is a capacitive touch sensor. However, it may be a touch sensor of another type, such as a resistive film type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, or a light sensor type.

[0045] Next, a setting value changing process performed by operating the operation members of the digital camera 100 of this embodiment will be described with reference to the flowchart of FIG.

[0046] The flow in FIG. 3 is started, for example, when eye proximity detection unit 57 detects that the user has placed the eye on viewfinder 17.

[0047] In S301, the system control unit 50 detects the user's gaze position via the gaze detection unit 110, and in S302, displays a gaze position display 503 (described later) at the user's gaze position on the EVF 29.

[0048] Then, in S303, it is determined whether the main electronic dial 71 has been rotated by the user, and if it is determined that the main electronic dial 71 has been rotated (YES in S303), it is further determined whether a predetermined item is displayed at the user's line of sight (S304). That is, it is determined whether the user is looking at the predetermined item. If it is determined that the predetermined item is displayed at the line of sight (YES in S304), the value of the item corresponding to the predetermined item (for example, the exposure compensation value when it is determined that the exposure level display is being viewed) is changed in response to the dial rotation (S305).

[0049] On the other hand, when it is determined that a predetermined item is not displayed at the gaze position (NO in S304), the photographing mode of the digital camera 100 is determined in sequence. Specifically, it is determined in sequence whether the still image photographing mode is set to an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), or the like. That is, first, it is determined whether it is the aperture priority mode (Av mode) (S306), and if it is the aperture priority mode (Av mode) (YES in S306), the aperture value is changed according to the dial rotation (S307). Next, it is determined whether it is the shutter speed priority mode (Tv mode) (S308), and if it is the shutter speed priority mode (Tv mode) (YES in S308), the shutter speed is changed according to the dial rotation (S309). If it is neither the Av mode nor the Tv mode (NO in S308), the value of the function setting previously assigned to the dial operation is changed according to the dial rotation (S310).

[0050] When it is determined in S303 that the main electronic dial 71 has not been rotated by the user (NO in S303), it is further determined whether the user has performed an operation such as shooting (S311). If the user has performed an operation (YES in S311), a process corresponding to the operation is executed (S312), and the flow ends. On the other hand, if the user has not performed an operation (NO in S311), the flow returns to the state immediately after the start of the flow.

[0051] Next, an operation for the user to assign function settings to the main electronic dial 71 will be described with reference to the screen transition diagram of FIG.

[0052] In any still image shooting mode, when the menu button 81 included in the operation unit 70 is pressed, a menu screen as shown in Fig. 4(a) is displayed on the display unit 28. When the "Function registration" item is selected on the menu screen and the set button is pressed, a function registration screen as shown in Fig. 4(b) is displayed on the display unit 28, which allows the user to simultaneously identify the function allocation status in multiple shooting modes. In Fig. 4(b), a preset switching row 401 is a display area for the user to set which operating member is to be assigned as the priority operating member. By pressing the left or right button with the selection frame placed on the row 401, the user can select which operating member is to be assigned as the priority operating member.

[0053] FIG. 5 is a screen display diagram showing the display of the EVF 29.

[0054] In FIG. 5, reference numeral 502 denotes an AF frame that indicates a range for performing autofocus.

[0055] Reference numeral 503 denotes a gaze position display showing the gaze position of the user detected by the gaze detection unit 110, and in the illustrated example, is composed of a cross and a circle C surrounding it.

[0056] Reference numerals 504, 505, 506, 507, and 508 denote GUIs for displaying parameters relating to shooting. Reference numeral 504 denotes a shooting mode setting, 505 denotes an ISO sensitivity setting, 506 denotes a shutter speed setting, 507 denotes a white balance setting, and 508 denotes an exposure compensation setting.

[0057] Fig. 5(a) is an example of a screen display when the user is directing his / her gaze at the ISO sensitivity setting 505, and a gaze position display 503 is semi-transparently superimposed on the position of the ISO sensitivity setting 505. When the user rotates the main electronic dial 71 at this time, the screen transitions to a screen having a GUI 509 for changing the ISO sensitivity value as shown in Fig. 5(b), and the ISO sensitivity value is changed according to the amount of rotation.

[0058] Fig. 5(c) is an example of a screen display in a state where the user directs his / her gaze to a live view image (LV image) other than the GUI that displays the parameters related to shooting, and a gaze position display 503 is semi-transparently superimposed on the position of the live view image (LV image). At this time, the shutter speed priority mode (Tv mode) is set. At this time, when the user rotates the main electronic dial 71, the screen transitions to a screen having a GUI 510 for changing the shutter speed as shown in Fig. 5(d), and the shutter speed is changed according to the amount of rotation.

[0059] Fig. 5(e) is an example of a screen display in a state where the user directs his / her gaze to a live view image (LV image) other than the GUI that displays parameters related to shooting, and a gaze position display 503 is semi-transparently superimposed on the position of the live view image (LV image). At this time, the manual shooting mode (M mode) is set. In this case, when the user rotates the main electronic dial 71, the setting value related to the function assigned to the main electronic dial 71 as the priority operation member is changed on the function registration screen shown in Fig. 4(b).

[0060] In this way, when the user is directing his / her gaze to the GUI, the function corresponding to the GUI is determined as an item related to the operation of one operation member (main electronic dial 71 in the embodiment) according to the user's gaze position. Therefore, it is possible to operate a plurality of items with one operation member (for example, main electronic dial 71). Also, when the user is not directing his / her gaze to the GUI, the operation member can operate the function prioritized in each shooting mode. That is, in Av priority mode (aperture priority mode), the Av value (aperture value) can be changed by operating one operation member (main electronic dial 71 in the embodiment). Also, in Tv priority mode (shutter speed priority mode), the Tv value (shutter speed) can be changed by operating one operation member (main electronic dial 71 in the embodiment), and in other shooting modes, the setting value related to the pre-assigned function can be changed by operating one operation member (main electronic dial 71 in the embodiment). It is obvious to a user of a camera capable of setting the shooting mode that the function prioritized in Av priority mode (aperture priority mode) is the Av value (aperture value), and the function prioritized in Tv priority mode (shutter speed priority mode) is the Tv value (shutter speed). Therefore, there is no risk of the user making an unintended (unintended) change in settings through the operation of the operation member (for example, the main electronic dial 71), and an intuitive and easy-to-understand operation can be realized.

[0061] Furthermore, when a rotating member or a touch member is applied to the operating member, the user can operate the operating member in either the direction of increasing or decreasing the set value, relying solely on the sense of touch, without having to see the operating member itself, making it even more suitable for operating multiple items.

[0062] Moreover, by configuring the gaze position display 503 with a dot or a cross and a circle surrounding it and displaying it semi-transparently, it becomes easy to see which GUI is recognized as the visual target of the user in the electronic device according to the embodiment of the present invention. Therefore, function selection by gaze can be performed more preferably.

[0063] In addition, when the gaze position display 503 overlaps with a GUI, the transparency of the gaze position display 503 can be increased or the display can be made invisible. In this case, it becomes easier to understand which GUI is recognized as the visual target of the user.

[0064] In the present invention, the input of the display position by the user's line of sight is used to determine the operation target item of the operation member, but it may also be used for other purposes. For example, it may also be used to set the area where autofocus is performed (so-called "AF frame").

[0065] Furthermore, in the present invention, parameters related to imaging cannot be changed by input from the user's line of sight alone, and changing these parameters requires the operation of an operating member such as a rotating member or a touch member, which further reduces the risk of unintended (unintended) setting changes for the user.

[0066] The present invention can also be applied to devices other than imaging devices. For example, the present invention may be applied to a computer (PC) so that the target for operating the scroll wheel of a mouse is determined according to the line of sight on the display (when viewing an area other than the GUI, a pre-assigned operation is performed).

[0067] <Example 2> Next, as a second embodiment, an example in which the present invention is applied to a head-mounted display in which digital graphics items are superimposed and displayed as if they existed in a live view (LV) will be described. This is a so-called MRHMD (Mixed Reality Head Mounted Display).

[0068] The Internet of Things (IoT) is an emerging technology that allows electrical appliances to be identified, located, addressed, and controlled through a network (Internet or ad-hoc systems). Examples of electrical appliances include everyday "things" such as TV systems, lighting fixtures, air conditioners, music systems, room and garage doors, home security systems, fans, sprinkler systems, microwave ovens, ovens, dishwashers, washing machines, dryers, etc.

[0069] In addition, WS-Discovery and multicast DNS (mDNS) are known as service discovery protocols for searching for services on a local network or nearby devices.

[0070] Therefore, image recognition processing is performed on the live view image, and the setting value of the device function to be changed or controlled for the subject displayed in the live view is determined by the user's gaze input, and the operation can be performed with a single operating member. This realizes simple, intuitive, and easy-to-understand operation.

[0071] Fig. 6(a) shows an external view of the MRHMD600 according to the second embodiment of the present invention as seen from the front side. Fig. 6(b) shows an external view as seen from the rear side, and Fig. 6(c) shows a side view of the MRHMD600 when worn on the user's head. Fig. 6(d) shows a schematic block diagram of the MRHMD600.

[0072] The head mounting part 601 is for mounting the MRHMD 600 on the user's head, and is composed of components 621 to 625. To mount the MRHMD 600 on the head, first, the length adjustment part 623 is loosened with the adjuster 622, and the MRHMD 600 is put on the head. Then, the forehead mounting part 625 is brought into close contact with the forehead, and the length adjustment part 623 is tightened with the adjuster 622 so that the side mounting part 621 and the back mounting part 624 are brought into close contact with the side and back of the head, respectively. In addition to the goggle type shown here, various types such as a spectacle frame type and a helmet type are available.

[0073] The imaging unit 602 is a so-called digital camera, and is disposed in the MRHMD 600 so as to capture an image in approximately the same direction as the face of a user wearing the MRHMD 600 on his / her head. Specifically, light incident from outside the MRHMD 600 through an entrance window 612 is guided by an optical prism 617 into the MRHMD 600, and is received by an imaging sensor 616.

[0074] The display unit 604 (604L, 604R) is composed of a screen 610 (610L, 610R), a color liquid crystal display 614 (614L, 614R), and an optical prism 615 (615L, 614R). The L and R added to the reference symbols indicate that the display unit 604 is for the left eye and the right eye, respectively. The display unit 604 is disposed at a position corresponding to the eyeglass lenses of eyeglasses so as to face the position of the user's eyes. Specifically, the image displayed by the color liquid crystal display 614 is guided to the screen 610 by the optical prism 615 and displayed on the screen 610.

[0075] The output light of the optical prism 615 of the display unit 604 and the input light of the optical prism 617 of the imaging unit 602 are aligned with the optical axis of the user's pupil when the MRHMD 600 is worn. The imaging sensor 616 captures an image of a subject in real space that can be visually recognized depending on the position and direction of the user.

[0076] Line-of-sight detection units 605La, 605Lb, 605Ra, and 605Rb are disposed above and below display units 604L and 604R, respectively. When a user looks into display unit 604, line-of-sight detection unit 605 detects the user's line of sight and determines which part the user is looking at.

[0077] The color liquid crystal display 614 displays an image in which a predetermined item is superimposed (image synthesis) on an image (live view) of real space captured by the imaging unit 602.

[0078] As shown in FIG. 6(d), the MRHMD 600 includes a display control unit 630, an imaging control unit 640, a CPU 650, a memory 660, a power supply unit 670, a communication unit 680, and an operation unit 690 inside the main body (including the head mounting unit 601).

[0079] The display control unit 630 controls the display of the display unit 604. Specifically, the display control unit 630 controls the size, position, orientation, color, and transparency of an item to be superimposed (image synthesis) on a real space image (live view), as well as the movement and brightness change associated with a change in the real space image (live view).

[0080] The imaging control unit 640 controls exposure, distance measurement, etc. based on the results of a predetermined calculation process using the captured image. This performs AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, etc. If the imaging unit 602 is equipped with a mechanism for inserting and removing an optical filter from the optical path and an anti-shake mechanism, etc., the imaging control unit 640 inserts and removes the optical filter and performs anti-shake operation depending on the operating situation.

[0081] The CPU 650 controls the entire MRHMD 600 and performs various arithmetic processing. The CPU 650 executes programs recorded in the memory 660 to realize the control and processing described below.

[0082] The memory 660 consists of a work area and a non-volatile area. Programs read from the non-volatile area and constants and variables for system control are expanded in the work area. Also, image data of virtual objects to be displayed superimposed on real space is held in the memory 660 for display. Also, image data captured by the imaging unit 602 and A / D converted is expanded in the memory 660 for the purpose of image analysis, image processing, etc.

[0083] The power supply unit 670 is made up of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery or a Li battery, or an AC adapter, and supplies power to the entire MRHMD 600. The power supply unit 670 includes a power switch that switches the power on and off according to a user operation or other conditions.

[0084] Under the control of the CPU 650, the communication unit 680 exchanges data with other devices, servers, etc. via the Internet, a LAN (Local Area Network), Bluetooth (registered trademark), WiFi Direct, etc.

[0085] The operation unit 690 is an input unit that accepts operations from a user (user operations) and is used to input various operation instructions to the CPU 650. The operation unit 690 is, for example, an electronic dial or a touch bar, etc. Also, it may be provided separately as a remote controller.

[0086] FIG. 7 is a flowchart showing a flow of determination and processing in which an electronic device according to a second embodiment of the present invention controls functions of an IoT device in response to a user's eye-gaze input and an operation on an operation member.

[0087] When the user turns on the power of an electronic device according to an embodiment of the present invention, first, the electronic device transmits a service search packet according to a service search protocol (S701).

[0088] Next, in S702, the system waits for a response from the device to the search packet transmitted in S701. Next, in S703, it is determined whether a predetermined waiting time has elapsed. If the waiting time has elapsed (YES in S703), the system ends the device search process (S704) and starts imaging by the imaging unit 602 (S705). Then, for each device found, it is determined whether the device's appearance characteristic data has been received (S706). If the data has been received (YES in S706), the data is compared with the image of the subject in the live view (S707), and a GUI for controlling the subject (device) is displayed near the matched subject (device) (S708). If the data of the appearance characteristics has not been received (NO in S706), it is determined whether the device's appearance characteristic data is stored in the memory 660 (S709). If the appearance characteristic data is held (YES in S709), the process proceeds to S707, where the data is compared with the image of the subject in the live view (S707), and a GUI for controlling the subject (device) is displayed near the matched subject (device) (S708). If the appearance characteristic data is not held (NO in S709), a message is displayed to the user prompting the user to associate the searched device with the subject in the live view (S710). If the user has associated the device (YES in S711), the image of the subject is recorded and stored in memory 660 as the appearance characteristic data of the device (S712).

[0089] Then, it is determined whether a predetermined waiting time has elapsed (S713). If the waiting time has elapsed (YES in S713), the process proceeds to S714, where the gaze detection unit 605 detects the user's gaze position (S714), and a "gaze position display" is displayed at the user's gaze position on the display unit 604 (S715).

[0090] Then, in S716, it is determined whether the operation unit 690 has been operated by the user, and if it is determined that the operation has been performed (YES in S716), it is further determined whether the GUI of the device function is displayed at the user's line of sight (S717).If it is determined that the user is looking at a specific item (YES in S717), the value of the item corresponding to that GUI (for example, the air conditioner temperature setting when it is determined that the air conditioner temperature GUI is being viewed) is changed according to the amount of operation (S718).

[0091] On the other hand, when it is determined that the user is not looking at the GUI of the device function (NO in S717), the function setting values ​​preset in the operation unit 690, such as the brightness of the lighting, are changed according to the amount of operation (S719).

[0092] When it is determined in S715 that the operation unit 690 has not been operated by the user (NO in S715), it is further determined whether the user has performed a termination operation such as turning off the power (S720). Then, when the user has performed a termination operation (YES in S720), the flow ends. On the other hand, when there is no operation by the user (NO in S720), the state returns to the state immediately after the start of imaging (S706).

[0093] FIG. 8 is a screen transition diagram showing the transition of screens displayed on the display unit 604 of the electronic device according to the second embodiment of the present invention.

[0094] In FIG. 8, an air conditioner 801, a curtain 802, and a TV system 803 are each subjects in the live view.

[0095] Further, a temperature GUI 8011 and a wind power GUI 8012 are displayed near the air conditioner 801, an open / close GUI 8021 is displayed near the curtains 802, and a volume GUI 8031 ​​and a channel GUI 8032 are displayed near the TV system.

[0096] Fig. 8(a) is an example of a display when the user is directing his / her gaze at the temperature GUI 8011, and the gaze position display 804 is semi-transparently superimposed on the position of the temperature GUI 8011. When the user operates the operation unit 690 at this time, the display transitions to a screen having a GUI for changing the air conditioner temperature as shown in Fig. 8(b), and the temperature setting of the air conditioner is changed according to the amount of operation.

[0097] 8(c) is an example of a screen display when the user directs his / her gaze to a live view image (LV image) other than the GUI, and a gaze position display 804 is semi-transparently superimposed on the position of the live view image (LV image). In addition, at this time, if the brightness of the lighting is preset as a priority operation item of the operation unit 690, the brightness of the lighting is changed according to the amount of operation of the operation unit 690 by the user.

[0098] As described above, the electronic device (MRHMD) of this embodiment is convenient because it allows multiple network-controllable device functions to be operated with a single operating member. In addition, there is no risk of unintended (unintended) setting changes for the user, and intuitive and easy-to-understand operations can be realized.

[0099] In addition, although a user wearing the MRHMD on his / her head cannot directly see the operation unit 690, when a rotating member or a touch member is applied to the operation unit 690, it is possible to operate the operation unit 690 in the direction of increasing or decreasing the set value by relying only on the sense of touch. Therefore, it is more suitable for operating multiple items.

[0100] <Other Examples> The control by the CPU of the present invention may be performed by a single piece of hardware, or a separate hardware resource may be responsible for the calculation processing as so-called cloud computing, and the electronic device may be controlled based on the results.

[0101] In addition, although the preferred embodiment of the present invention has been described, the present invention is not limited to the above-mentioned specific embodiment, and various forms within the scope of the gist of the invention are also included in the present invention. Each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0102] The present invention can also be realized by executing the following process. That is, a software program that realizes the functions of the above-mentioned embodiments is supplied to a system or device via a network or various recording media, and a computer (or a CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the recording media on which the program is stored constitute the present invention.

Claims

1. an operation member that accepts an operation by a user; A display unit that displays a predetermined screen including an item and a live view image different from the item; A gaze detection means for detecting a gaze position of a user on the predetermined screen displayed on the display unit; a control means for controlling to execute a first process or a second process different from the first process, depending on whether or not the item is displayed at a gaze position detected by the gaze detection means on the predetermined screen when the user operates the operation member while the predetermined screen is being displayed, The control means when the item is not displayed at the gaze position detected by the gaze detection means on the predetermined screen, control is performed so as to execute the first process corresponding to the first item assigned to the operation member; An electronic device characterized by controlling, when the item is displayed at a gaze position detected by the gaze detection means on the specified screen, to execute the second process corresponding to a second item assigned to the item.

2. the operating member is a rotating member, 2 . The electronic device according to claim 1 , wherein the first process is a process of changing a setting value corresponding to the first item assigned to the operating member in accordance with an amount of rotation of the operating member.

3. The electronic device described in Claim 2, characterized in that the second processing is a processing of changing a setting value corresponding to the second item assigned to the item in accordance with the amount of rotation of the operating member.

4. the operation member receives a touch operation by a user; 2 . The electronic device according to claim 1 , wherein the first process is a process of changing a setting value corresponding to the first item assigned to the operating member in accordance with a movement amount of a touch operation on the operating member.

5. The electronic device described in Claim 4, characterized in that the second processing is a processing of changing a setting value corresponding to the second item assigned to the item depending on the amount of movement of the touch operation on the operating member.

6. An imaging means; The camera further includes a photographing mode setting means for setting the photographing mode to one of a plurality of photographing modes including a shutter speed priority mode and an aperture priority mode, 6. The electronic device according to claim 1, wherein the first item is a shutter speed in the shutter speed priority mode and an aperture value in the aperture priority mode.

7. A connection means for connecting to an external device via a network; an imaging means for capturing the live view image included on the screen; a recognition unit that executes an image recognition process on the live view image captured by the imaging unit and recognizes an external device that can be controlled by the electronic device, 7. The electronic device according to claim 1, wherein an item for controlling an external device recognized by the recognition means is assigned to the item, and the item is displayed on the screen together with the live view image.

8. a display step of displaying a predetermined screen including an item and a live view image different from the item; a gaze detection step of detecting a gaze position of a user on the predetermined screen displayed by the display step; a control step of controlling to execute a first process or a second process different from the first process, depending on whether or not the item is displayed at the gaze position detected on the predetermined screen by the gaze detection step, when the user operates an operating member while the predetermined screen is being displayed, The control step includes: When the item is not displayed at the gaze position detected on the predetermined screen by the gaze detection step, the first process corresponding to the first item assigned to the operation member is executed; A control method for an electronic device, characterized in that when the item is displayed at the gaze position detected by the gaze detection step on the specified screen, the control is performed to execute the second processing corresponding to the second item assigned to the item.

9. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 7.

Images