Electronic device, method for controlling electronic device, and program

Abstract

The electronic device includes: a detection means for detecting a line-of-sight position of a user viewing a display means for displaying a live view image captured by an imaging means; the operation component is used for receiving operation performed by a user; the mode switching component is used for switching between an AF mode and an MF mode; and a control means for, in response to a specific operation performed by a user on the operation means, executing (1) a first process in the AF mode in the live view image displayed on the display means, and (2) a second process in the AF mode in the live view image displayed on the display means. And (1) performing a first process for determining a position at which AF is to be performed based on a line-of-sight position on the live view image detected by the detection means in the MF mode, and (2) performing a second process for assisting manual focusing based on a line-of-sight position on the live view image detected by the detection means, in which the AF is to be performed on the basis of the line-of-sight position on the live view image detected by the detection means in the MF mode, in the second process, control is performed such that the second process is completed in response to completion of the specific operation.

Description

Technical Field

[0001] This invention relates to electronic devices, methods and procedures for controlling electronic devices. Background Technology

[0002] Electronic devices equipped with "eye-tracking" technology that allows operation based on the user's gaze are known. In particular, eye-tracking technology is effective when a user operates an electronic device (such as a digital camera or game console) and wishes to perform an intentional action quickly.

[0003] Electronic devices equipped with gaze input technology perform pre-determined processing at the gaze position corresponding to the detected gaze when they detect a user's gaze. Therefore, if the electronic device detects unintentional gaze movements, such as slight eye movements that occur even when the user attempts to fix their gaze on an object, the electronic device may perform an operation unintentional to the user.

[0004] Patent Document 1 describes a technique for displaying an item at a gaze position detected when a user performs a specific operation in order to prevent the execution of processing corresponding to unintentional gaze movement of the user, and for invalidating subsequently detected user gaze.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2022-68749 Summary of the Invention

[0008] The problem the invention aims to solve

[0009] However, the electronic device described in Patent Document 1 is configured such that gaze-input related processing performed in response to a specific operation by the user displays items only at a position corresponding to the gaze position. Therefore, when multiple processes are performed based on gaze input, corresponding operating components for each process are required, which may reduce the flexibility of other operations.

[0010] Therefore, the present invention provides an electronic device, a control method and program for the electronic device capable of performing multiple processes related to gaze input in response to a specific operation performed by a user.

[0011] Solution for solving the problem

[0012] To achieve the above objectives, the electronic device according to the present invention includes: a detection unit for detecting the gaze position of a user viewing a display unit for displaying a live view image captured by a camera unit; an operation unit for receiving an operation performed by the user; a mode switching unit for switching between an AF mode and a MF mode; and a control unit for, in response to a specific operation performed by the user on the operation unit, in the live view image displayed on the display unit, (1) performing a first process in the AF mode, the first process being used to determine the position to perform AF based on the gaze position on the live view image detected by the detection unit, and (2) performing a second process in the MF mode, the second process being used to assist manual focusing based on the gaze position on the live view image detected by the detection unit, wherein, in the second process, control is performed such that the second process is completed in response to the completion of the specific operation.

[0013] The effects of the invention

[0014] The electronic device, control method, and program according to the present invention enable the execution of multiple processes related to gaze input in response to a specific operation performed by a user. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the specification, serve to explain the principles of the invention.

[0016] Figure 1A This is an external view of a digital camera 100, which is an example of a camera device according to the first embodiment, and a perspective view of the front of the digital camera 100.

[0017] Figure 1B This is an external view of a digital camera 100, which is an example of a camera device according to the first embodiment, and a perspective view of the back of the digital camera 100.

[0018] Figure 2 This is a block diagram illustrating the configuration of a digital camera 100, which is an example of an imaging device according to the first embodiment.

[0019] Figure 3A This is a flowchart illustrating the processing of a digital camera 100, which is an example of a camera device according to the first embodiment, in camera mode.

[0020] Figure 3B This is a flowchart illustrating the processing of a digital camera 100, which is an example of a camera device according to the first embodiment, in camera mode.

[0021] Figure 3CThis is a flowchart illustrating the processing of a digital camera 100, which is an example of a camera device according to the first embodiment, in camera mode.

[0022] Figure 4A An example of a menu screen showing the switching of various settings of a digital camera 100, which is an example of a camera device according to the first embodiment, is shown.

[0023] Figure 4B An example of a menu screen showing the switching of various settings of a digital camera 100, which is an example of a camera device according to the first embodiment, is shown.

[0024] Figure 5A An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0025] Figure 5B An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0026] Figure 5C An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0027] Figure 5D An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0028] Figure 5E An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0029] Figure 5F An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0030] Figure 5G An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0031] Figure 5H An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0032] Figure 5IAn example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0033] Figure 5J An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0034] Figure 5K An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0035] Figure 5L An example of displaying an LV image on the screen in the video mode of a digital camera 100, which is an example of a video recording device according to the first embodiment, is shown.

[0036] Figure 6 This is a schematic diagram showing the display layer of the GUI component of a digital camera 100, which is an example of a camera device according to the first embodiment.

[0037] Figure 7 An example of a focus state being displayed on the display unit 28 of a digital camera 100, which is an example of an imaging device according to a second embodiment.

[0038] Figure 8A This is an external view of a head-mounted device (HMD) 810 and a game controller 820 used as an operating component, according to the third embodiment.

[0039] Figure 8B This is an external view of a head-mounted device including a head-mounted display (HMD) 810 and a game controller 820 used as an operating component, according to a third embodiment.

[0040] Figure 8C This is an external view of a head-mounted device including a head-mounted display (HMD) 810 and a game controller 820 used as an operating component, according to a third embodiment.

[0041] Figure 9A An example of a game screen displayed on an HMD 810 according to a third embodiment is shown.

[0042] Figure 9B An example of a game screen displayed on an HMD 810 according to a third embodiment is shown.

[0043] Figure 9C An example of a game screen displayed on an HMD 810 according to a third embodiment is shown.

[0044] Figure 9D An example of a game screen displayed on an HMD 810 according to a third embodiment is shown.

[0045] Figure 9E An example of a game screen displayed on an HMD 810 according to a third embodiment is shown.

[0046] Figure 9F An example of a game screen displayed on an HMD 810 according to a third embodiment is shown.

[0047] Figure 10A An example of a video editing screen of a personal computer (PC) according to a fourth embodiment is shown.

[0048] Figure 10B An example of a video editing screen of a personal computer (PC) according to a fourth embodiment is shown. Detailed Implementation

[0049] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Repeated descriptions of components having substantially the same functional configuration will be omitted in this specification and the drawings.

[0050] [First Embodiment]

[0051] <Camera Equipment Configuration>

[0052] Figure 1A and Figure 1B This is an external view of a digital camera 100, which is an example of a camera device according to the first embodiment. Figure 1A This is a stereoscopic view of the front of the digital camera 100. Figure 1B This is a 3D view of the back of the digital camera 100.

[0053] Display unit 28 is located on the back of digital camera 100 and displays images and various information items. Touch panel 70a allows touch operation to be detected on the display surface (touch operation surface, touch operation member) of display unit 28. Viewfinder external display unit 43 is located on top of digital camera 100 and displays various settings of digital camera 100, including shutter speed and aperture. Shutter button 61 is an operation member for providing imaging indication (video indication). Mode switch 60 is an operation member for switching between various modes. Terminal cover 40 is a cover for protecting the connector (not shown) used to connect digital camera 100 to external devices.

[0054] The main electronic dial 71 is a rotary operating member configured to change settings such as shutter speed and aperture by rotation, for example. The power switch 72 is an operating member used to switch the power of the digital camera 100 between ON and OFF. The secondary electronic dial 73 is a rotary operating member configured to move the selection box (cursor) or feed an image by rotation, for example. The four-way key 74 is configured to press the up, down, left, and right portions, thereby allowing processing corresponding to the pushed-pressed portions of the four-way key 74. The setting button 75 is a push button primarily used to determine the selected item.

[0055] The multi-controller (hereinafter referred to as "MC") 65 is capable of accepting instructions in eight directions and push-button operations at the center. The video button 76 is used to indicate whether to start or stop shooting (recording) motion images. The auto exposure (AE) lock button 77 is a push button that can be pressed to fix the exposure state while the camera is in standby mode.

[0056] Zoom button 78 is an operation button used to switch between ON and OFF magnification mode in the Live View display (LV display) on the standby screen in video mode. The Live View image (LV image) can be magnified or reduced by activating magnification mode and then operating the main electronic dial 71. In playback mode, zoom button 78 is used to magnify the playback image or increase the magnification. Playback button 79 is an operation button used to switch between video mode and playback mode. By pressing playback button 79 during video mode, the mode can be switched to playback mode, and the last image recorded on the recording medium 200 (described later) can be displayed on the display unit 28.

[0057] Menu button 81 is a push button used to indicate the display of a menu screen. When menu button 81 is pressed, a menu screen allowing various settings is displayed on display unit 28. Users can intuitively perform various settings using the menu screen displayed on display unit 28, four-way key 74, setting button 75, or MC 65.

[0058] The autofocus (AF)-ON button 82 is an operating component included in the operation unit 70. The AF-ON button 82 is a push button. AF (autofocus) processing can be started by pressing the AF-ON button 82. AF processing can be started primarily by pressing the shutter button 61; however, AF processing can also be started by pressing the AF-ON button 82. The AF-ON button 82 is positioned so that the user can easily operate it using the thumb of their right hand while holding the grip 90, even when the user is looking through the viewfinder (with their eye against the eyepiece 16).

[0059] The communication terminal 10 is used to communicate with the lens unit 150 (described later), to which the digital camera 100 is detachably attached. The eyepiece 16 is the eyepiece portion of an eyepiece viewfinder (endoscopic viewfinder) that allows the user to visually recognize the image displayed on the internal electronic viewfinder (EVF) 29 (described later) via the eyepiece 16. The eye detection unit 57 is an eye detection sensor that detects whether the user's (photographer's) eye is in contact with the eyepiece 16. The cover 202 is a cover for a slot to accommodate the recording medium 200 (described later). The grip 90 is a holder shaped to be easily held with the right hand when the user holds the digital camera 100. The shutter button 61 and the main electronic dial 71 are arranged in a position that can be operated together with the right index finger while holding the digital camera 100 using the grip 90 held by the little, ring, and middle fingers of the right hand. In the same state, the sub-electronic dial 73 and the AF-ON button 82 are arranged in a position that can be operated together with the right thumb.

[0060] <Description of the diagram>

[0061] Figure 2 This is a block diagram illustrating the configuration of a digital camera 100, which is an example of an imaging device according to a first embodiment. The lens unit 150 includes replaceable lenses.

[0062] Lens 103 typically comprises multiple lenses; however, for simplicity, in Figure 2 Only one lens is shown. Communication terminal 6 allows communication between the digital camera 100 and the lens unit 150. Communication terminal 10 allows communication between the digital camera 100 and the lens unit 150. The lens unit 150 communicates with the system control unit 50 via communication terminals 6 and 10. The lens unit 150 controls the aperture 1 using the internal lens system control circuit 4 via the aperture drive circuit 2. The lens unit 150 focuses by shifting the lens 103 via the lens system control circuit 4 using the AF drive circuit 3.

[0063] Shutter 101 is a focal plane shutter configured to freely control the exposure time of the imaging unit 22 under the control of the system control unit 50.

[0064] The camera unit 22 is an image sensor composed of a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) device, which converts optical images into electrical signals. The camera unit 22 may include a camera plane phase difference sensor that outputs defocus information to the system control unit 50.

[0065] The image processing unit 24 performs predetermined processing (e.g., pixel interpolation, resizing (such as reduction), and color conversion) on data from the analog-to-digital (A / D) converter 23 or the memory control unit 15. The image processing unit 24 performs predetermined calculations using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the calculation results obtained by the image processing unit 24. Therefore, it performs through-lens (TTL) type AF processing, automatic exposure (AE) processing, and flash pre-emission (EF) processing, etc. The image processing unit 24 also performs predetermined calculations using the captured image data and performs TTL type automatic white balance (AWB) processing based on the obtained calculation results.

[0066] The memory control unit 15 controls the data transmission / reception between the A / D converter 23, the image processing unit 24, and the memory 32. Output data from the A / D converter 23 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15 without going through the image processing unit 24. The memory 32 stores image data captured by the camera unit 22 and converted into digital data by the A / D converter 23, as well as image data to be displayed on the display unit 28 or EVF 29. The memory 32 has sufficient storage capacity to store a predetermined number of still images and a predetermined period of video or audio.

[0067] The memory 32 also serves as a memory for image display (video memory). Display image data written to the memory 32 is displayed by the display unit 28 or EVF 29 via the memory control unit 15. The display unit 28 and EVF 29 each display data on a display device such as a liquid crystal display (LCD) or an organic electroluminescent (EL) display based on signals from the memory control unit 15. LV display can be performed by sequentially transmitting data that has been converted from analog to digital by the A / D converter 23 and stored in the memory 32 to the display unit 28 or EVF 29.

[0068] The gaze detection unit 160 (receiving unit) detects the gaze of the user's eye toward the EVF 29 from the eyepiece 16. The gaze detection unit 160 includes a dichroic mirror 162, an imaging lens 163, a gaze detection sensor 164, a gaze detection circuit 165, and an infrared light-emitting diode 166.

[0069] An infrared LED 166 is a light-emitting device used to detect the user's gaze position in the viewfinder image and emits infrared light towards the user's eyeball (eye) 161. The infrared light emitted from the infrared LED 166 is reflected by the eyeball (eye) 161, and the reflected infrared light reaches a dichroic mirror 162. The dichroic mirror 162 reflects only infrared light and allows visible light to pass through. The infrared reflected light, whose optical path has been altered, is imaged on the imaging surface of the gaze detection sensor 164 via an imaging lens 163. The imaging lens 163 is an optical component constituting the gaze detection optical system. The gaze detection sensor 164 is an imaging device such as a CCD-type image sensor.

[0070] The gaze detection sensor 164 photoelectrically converts the incident infrared reflected light into an electrical signal and outputs the signal to the gaze detection circuit 165. The gaze detection circuit 165 detects the user's gaze position based on the output signal from the gaze detection sensor 164 and the movement of the user's eyeball (eye) 161, and outputs the detected information to the system control unit 50 and the gaze confirmation unit 170.

[0071] Based on the detection information received from the gaze detection circuit 165, when the user's gaze is fixed on a certain area for a period exceeding a predetermined threshold, the gaze confirmation unit 170 determines that the user is gazing at that area. Therefore, this area can be considered the location of the user's gaze (gaze area). An example of "gaze fixed on a certain area" is that during the predetermined period, the average position of the gaze movement is within that area, and its variation is less than a predetermined value. The predetermined threshold can be freely changed by the system control unit 50. The gaze confirmation unit 170 does not need to be set up as a separate block; the system control unit 50 can perform the same function as the gaze confirmation unit 170 based on the detection information received from the gaze detection circuit 165.

[0072] In this embodiment, the gaze detection unit 160 uses a method called the corneal reflection method to detect the gaze. The corneal reflection method is a method for detecting the direction and position of a gaze based on the positional relationship between the reflected light emitted from the infrared LED 166 and the pupil of the eyeball (eye) 161 (particularly the cornea) and the gaze. The method used to detect the gaze (its direction and position) is not limited and can be any method other than those described above. For example, a method called the scleral reflection method can be used, which utilizes the difference in reflectivity between the iris and the sclera.

[0073] The viewfinder external display unit 43 displays various settings, including shutter speed and aperture, via the viewfinder external display unit drive circuit 44.

[0074] The non-volatile memory 56 is an electrically erasable / recordable memory (e.g., flash read-only memory (ROM)). The non-volatile memory 56 stores constants and programs for the operating system control unit 50. As used herein, "program" refers to a program used to execute the various flowcharts described later in this embodiment.

[0075] The system control unit 50 is a control unit consisting of at least one processor or circuit, and controls the entire digital camera 100. The system control unit 50 performs the processing of this embodiment, described later, by executing programs stored in non-volatile memory 56. The system memory 52 is, for example, random access memory (RAM). The system control unit 50 deploys constants, variables, and programs for the operating system control unit 50, read from the non-volatile memory 56, into the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32 and the display unit 28, etc.

[0076] System timer 53 is a timer configured to measure the time used for various controls and the time of the internal clock.

[0077] The power control unit 80 includes a battery detection circuit, a DC-DC converter, and a switching circuit for switching between blocks to be energized, and detects whether a battery is installed, the battery type, and the remaining battery power. Based on its detection results and instructions from the system control unit 50, the power control unit 80 controls the DC-DC converter to supply the required voltage to the components including the recording medium 200 during the required periods. Examples of power supply units 30 include primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as nickel-cadmium (NiCd) batteries, nickel-metal hydride (NiMH) batteries, and lithium (Li) batteries, and an AC adapter.

[0078] Recording medium I / F 18 is an interface to recording medium 200 (such as a memory card or hard disk). An example of recording medium 200 is a memory card, such as a semiconductor memory or a disk, used to record captured images.

[0079] Communication unit 54 transmits and receives video and audio signals to and from an external device connected via a wireless or wired communication connection. Communication unit 54 can also be connected to a wireless local area network (LAN) and the Internet. Communication unit 54 can also be connected via... The communication unit 54 can communicate with external devices via Bluetooth or Bluetooth Low Energy. It can transmit images captured by the camera unit 22 (including LV images) and images recorded on the recording medium 200, and can receive image data and various other information from external devices.

[0080] The posture detection unit 55 detects the posture of the digital camera 100 relative to the direction of gravity. Based on the posture detected by the posture detection unit 55, it can be determined whether the image captured by the imaging unit 22 was taken while the digital camera 100 is held laterally or longitudinally. The system control unit 50 can add posture information corresponding to the posture detected by the posture detection unit 55 to the image file of the image captured by the imaging unit 22, and record the image with a rotational posture. Examples of posture detection units 55 include accelerometers and gyroscopes. By using an accelerometer or gyroscope as the posture detection unit 55, movement of the digital camera 100 (pan, pitch, rise, or remaining stationary, etc.) can be detected.

[0081] The eye detection unit 57 is an eye detection sensor used to detect the approach (eye-to-eye) and separation (away from eye) of the eye (object) 161 relative to the eyepiece 16 of the viewfinder (hereinafter referred to as the "viewfinder"). The system control unit 50 switches between display (display state) and non-display (non-display state) of the display unit 28 and the EVF 29 based on the state detected by the eye detection unit 57. More specifically, when the system is at least in camera standby mode and the display destination is automatically switched, during the non-eye-to-eye state, the display destination is set to the display unit 28, and the EVF 29 is disabled. During the eye-to-eye state, the display destination is set to the EVF 29, and the display unit 28 is disabled. An example of the eye detection unit 57 is an infrared proximity sensor capable of detecting the approach of an object to the eyepiece 16 of the viewfinder housing the EVF 29. When the object approaches, infrared light emitted from the light emitter (not shown) of the eye detection unit 57 is reflected by the object and received by the light receiver (not shown) of the infrared proximity sensor. The distance from the object to the eyepiece 16 (eye proximity distance) can also be determined based on the amount of infrared light received. In this way, the eye detection unit 57 performs eye proximity detection to detect the proximity distance from the object to the eyepiece 16. When an object is detected to have approached the eyepiece 16 from a non-eye-contact state (non-proximity state) within a predetermined distance, the eye detection unit 57 determines that an eye proximity has been established. When the detected approaching object moves away from the eye-contact state (proximity state) beyond a predetermined distance, the eye detection unit 57 determines that an eye separation has occurred. The threshold for detecting eye contact and the threshold for detecting eye separation can be different, for example, by setting a hysteresis. Once eye contact is detected, the system is considered to remain in the eye-contact state until eye separation is detected. Once eye separation is detected, the system is considered to remain in the non-eye-contact state until eye contact is detected. The infrared proximity sensor is merely one example; the eye detection unit 57 can be any other sensor capable of detecting the proximity of an eye or object that can be considered as eye contact.

[0082] The system control unit 50 can detect the following line of sight at EVF 29 by controlling the line of sight detection unit 160.

[0083] • The line of sight that was not previously directed at EVF 29 is now directed at EVF 29. In other words, line of sight input has begun.

[0084] • The line of sight is being input to EVF 29.

[0085] • A certain location of EVF 29 is being observed.

[0086] • The gaze state at EVF 29 has been removed. In other words, the gaze input has ended.

[0087] • No visual input was given to the EVF 29 (EVF 29 status was not viewed).

[0088] Such operations and statuses, along with the position (direction) of the line of sight at EVF 29, are sent to system control unit 50 via the internal bus. System control unit 50 determines how to perform line-of-sight input based on the sent information.

[0089] The operation unit 70 is a receiving unit that receives operations (user operations) from the user and is used to input various operation instructions to the system control unit 50. For example... Figure 2 As shown, the operation unit 70 includes a mode switch 60, a shutter button 61, a power switch 72, and a touch panel 70a. The operation unit 70 also includes a main electronic dial 71, a secondary electronic dial 73, a four-way key 74, a settings button 75, a video button 76, an AE lock button 77, a zoom button 78, a playback button 79, a menu button 81, and an MC 65 as other operation components 70b.

[0090] The mode switch 60 is an operational component used to switch between video recording modes. Examples of video recording modes include fully automatic video recording mode (Scene Intelligent Auto) and flexible AE video recording mode (Fv mode). Other examples of video recording modes include manual exposure mode (M mode), aperture priority AE mode (Av mode), shutter priority AE mode (Tv mode), and program AE video recording mode (P mode). Other examples include special scene modes (SCN mode) and custom modes for automatically recording video in the video recording mode selected by the user according to the subject or scene.

[0091] The mode switch 60 allows the user to switch directly between these modes. Alternatively, once the mode switch 60 switches to the camera mode list screen, different operating mechanisms can be used to selectively switch to one of the multiple display modes.

[0092] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. During operation on the shutter button 61 (i.e., by half-pressing) (an indication to prepare for recording), the first shutter switch 62 is turned on to generate a first shutter switch signal SW1. In response to the first shutter switch signal SW1, the system control unit 50 begins recording preparation processing such as AF processing, AE processing, AWB processing, and EF processing. When operation on the shutter button 61 is completed (i.e., by fully pressing) (an indication to start recording), the second shutter switch 64 is turned on to generate a second shutter switch signal SW2. In response to the second shutter switch signal SW2, the system control unit 50 begins a series of recording operations from reading signals from the imaging unit 22 to writing the captured image as an image file to the recording medium 200.

[0093] The touch panel 70a and the display unit 28 can be configured as a single unit. For example, the touch panel 70a is configured to have a light transmittance that does not interfere with the display of the display unit 28, and is mounted on the upper layer of the display surface of the display unit 28. The input coordinates on the touch panel 70a are associated with the display coordinates on the display surface of the display unit 28. This allows for the provision of a graphical user interface (GUI) that gives the user a direct view of the screen displayed on the display unit 28. The system control unit 50 can detect the following operations or states on the touch panel 70a.

[0094] • The operation of a finger or pen that has not yet touched the touch panel 70a making new contact with the touch panel 70a is the start of a touch (hereinafter referred to as "touch-down").

[0095] • The state in which the touch panel 70a is touched by a finger or pen (hereinafter referred to as "touch-on").

[0096] • An operation in which a finger or pen is moving while touching the touch panel 70a (hereinafter referred to as "touch-move").

[0097] • The operation of separating (releasing) a finger or pen that has touched the touch panel 70a from the touch panel 70a is the end of the touch (hereinafter referred to as "touch-up").

[0098] • The state of the touch panel 70a when it is not touched (hereinafter referred to as touch-off).

[0099] When a touch is detected, touch persistence is also detected. After a touch, touch persistence is typically detected continuously unless touch cessation is detected. Touch persistence is also detected even when touch movement is detected. Touch movement is not detected even when touch persistence is detected, unless the touch location is moved. The system enters a no-touch state after all touched fingers or stylus touches have ceased.

[0100] These operations and statuses, along with the position coordinates of the finger or pen touching the touch panel 70a, are sent to the system control unit 50 via an internal bus. The system control unit 50 determines what kind of operation (touch operation) has been performed on the touch panel 70a based on the sent information.

[0101] For touch movement, the direction of movement of a finger or pen moving on the touch panel 70a can be determined based on changes in the position coordinates of each vertical and horizontal component. When touch movement beyond a predetermined distance is detected, it is determined that a swipe operation has been performed.

[0102] A finger movement that involves quickly moving a finger a certain distance while it remains in contact with the touch panel 70a and then lifting it is called a swipe. In other words, a swipe is an operation in which the finger quickly glides across the touch panel 70a during a swipe motion. When a touch movement operation of more than a predetermined distance at a predetermined speed is detected and then the touch stops, it can be determined that a swipe has occurred (it can be determined that a swipe has occurred after a swipe operation).

[0103] A touch operation where multiple locations (e.g., two points) are touched (multi-touch) to bring the touch locations closer together is called a "pinch," and a touch operation where the touch locations are separated is called a "pinch separate." Pinch and pinch are collectively referred to as a pinch operation (or simply "pinch").

[0104] The touch panel 70a can be any type, such as a resistive film method, a capacitive method, a surface acoustic wave method, an infrared light method, an electromagnetic induction method, an image recognition method, or an optical sensor method. Examples of touch detection methods include methods that detect touch based on contact with the touch panel and methods that detect touch based on the proximity of a finger or pen to the touch panel; either of these can be used.

[0105] The digital camera 100 may be equipped with a voice input unit (not shown) that sends voice signals obtained from a built-in microphone or a voice input device connected via a voice input terminal to the system control unit 50. In this case, the system control unit 50 selects the input voice signal as needed, performs analog-to-digital conversion, applies level adjustment processing and specific frequency reduction processing, etc., and generates a voice signal.

[0106] In this embodiment, the user can set the method for specifying the position of the position indicator (e.g., AF frame) when performing a touch movement operation in the eye-catching state to an absolute position specification method or a relative position specification method.

[0107] The absolute position specification method associates the input coordinates on the touch panel 70a with the display coordinates on the display surface of the EVF 29. In the case of the absolute position specification method, when a touch operation is performed on the touch panel 70a, even without touch movement, the AF frame is displayed at the position associated with the touch position (the position of the input coordinates) (the AF frame moves from its position before the touch). The position displayed using the absolute position specification method is based on the touch position and is independent of the position before the touch. When a touch movement occurs after the touch, the position of the AF frame also moves based on the touch position after the touch movement.

[0108] The relative position specification method is a method in which the input coordinates on the touch panel 70a are not associated with the display coordinates on the display surface of the EVF 29. In the case of the relative position specification method, when a touch occurs on the touch panel 70a without any touch movement, the position of the AF frame does not change from its position before the touch. When a touch movement subsequently occurs, the position of the AF frame changes from its current position (the position set before the touch) by a distance corresponding to the amount of movement in the direction of the touch movement, regardless of the touch location.

[0109] Several AF methods, including "single-point AF" and "full-area AF," can be configured as an AF area (the method for setting the AF frame). Furthermore, whether to perform subject detection settings (tracking) is configurable.

[0110] "Single-point AF" is a method where the user designates a single location as the AF position using a single-point AF box. "Full-area AF" is a method where the AF position is automatically set based on automatic selection conditions when the user does not specify a tracking object. This AF area setting can be combined to reflect tracking settings. When tracking is set to "ON" and a human face is detected from the LV image, the system enters a mode that prioritizes tracking the detected face as the AF object. When multiple human faces are detected, one face is selected and set as the AF object based on priority conditions (such as large face size, face position close to the digital camera 100 (i.e., near the camera), face position near the center of the image, or face of a pre-registered person). When no human face is detected, a subject other than a face is selected and set as the AF object based on priority conditions (such as subject close to the digital camera 100 (i.e., near the camera), subject with high contrast, subject with high priority (such as animal or vehicle), or subject is a moving object). When the user specifies a tracking object, the tracking object is set as the AF object. In other words, the automatic selection criteria ensure that the score obtained by weighting using at least one of the factor criteria exemplified below is equal to or greater than a predetermined threshold or is the highest among all candidates.

[0111] • The face refers to the detected human face.

[0112] • Large facial size.

[0113] • The face is positioned close to the digital camera 100 (i.e., on the near side).

[0114] • The face is positioned near the center of the image.

[0115] • The face is the face of the person who is pre-registered.

[0116] • The subject is close to the digital camera 100 (i.e., on the near side).

[0117] • The subject has high contrast.

[0118] • Subjects with high priority, such as animals or vehicles.

[0119] • The subject is a moving object.

[0120] <Flowchart of processing in camera mode>

[0121] Figures 3A to 3CThis is a flowchart illustrating the processing in the recording mode of a digital camera 100, an example of a recording device according to the first embodiment. This processing is implemented by the system control unit 50 executing a program stored in the non-volatile memory 56 and loaded into the system memory 52. ​​This processing continues until the recording mode ends or the power to the recording device is turned off. In other words, this processing is repeated while the recording mode continues. When the digital camera 100 is started in recording mode, flags or control variables are initialized, and this processing begins. Figures 5A to 5L An example is shown of displaying an LV image on a screen in the recording mode of a digital camera 100, which is an example of a recording device according to the first embodiment.

[0122] In S301, the system control unit 50 initializes flags and control variables, etc.

[0123] In S302, such as Figure 5A As shown, the system control unit 50 instructs the display unit 28 to display LV image 501 in response to a signal detected by the camera unit 22.

[0124] In S303, the system control unit 50 determines whether a switching operation for determining whether to perform line-of-sight input-based processing has been performed on the operation unit 70. If the operation has been performed, the processing proceeds to S304; otherwise, the processing proceeds to S305.

[0125] Figure 4A and Figure 4B This section shows an example of a menu screen displaying various settings of a digital camera 100, which is an example of a camera device according to the first embodiment, being switched. Figure 4A The AF area setting 401 allows you to select either a single point or the entire AF area. Tracking 402 determines whether to track the subject. Focus mode 403 allows you to select between the digital camera 100's automatic focus adjustment mode (AF mode) or the user's manual focus adjustment mode (MF mode). Focus guide display 404 determines whether to display the focus guide described later.

[0126] In the first embodiment, the AF mode or MF mode is selected via a menu screen displayed on the main body of the digital camera 100; alternatively, the AF mode or MF mode can be selected using a focus mode switch provided on the lens unit 150. The focus mode switch is a switch that allows manual switching between MF mode and AF mode.

[0127] When choosing Figure 4A When the AF advanced setting is 405, the screen switches to... Figure 4B Advanced AF settings in the game. Figure 4BThe gaze input 406 is used to determine whether to perform gaze-based input processing. The indicator display 407 is used to determine whether to display the gaze indicator at a position based on the gaze location. The gaze confirmation 408 is used to determine whether to perform a gaze confirmation operation via the first shutter button 62.

[0128] In S304, the system control unit 50 switches between active 406a and inactive 406b of the line-of-sight input 406 according to the operation.

[0129] In S305, the system control unit 50 determines whether a switching operation has been performed on the operation unit 70 to determine whether to display the gaze indicator at the gaze position based on the detected gaze. If the operation has been performed, the process proceeds to S306; otherwise, the process proceeds to S307.

[0130] In S306, the system control unit 50 switches between enabling and disabling the gaze indicator display 407 based on the operation.

[0131] In S307, the system control unit 50 determines whether a switching operation has been performed on the operation unit 70 to determine whether a line-of-sight confirmation operation via the first shutter button (SW1) 62 should be performed. If the operation has been performed, the process proceeds to S308; otherwise, the process proceeds to S309.

[0132] In S308, the system control unit 50 switches between valid and invalid view confirmation 408 in SW1 according to the operation.

[0133] In S309, the system control unit 50 determines whether a switching operation to set the AF area to a single point or the entire area has been performed on the operation unit 70. If the operation has been performed, the process proceeds to S310; otherwise, the process proceeds to S311.

[0134] In S310, the system control unit 50 switches between single-point and full-area settings in AF area setting 401 according to the operation.

[0135] In S311, the system control unit 50 determines whether a switching operation for determining whether to track the subject has been performed on the operation unit 70. If the operation has been performed, the process proceeds to S312; otherwise, the process proceeds to S313.

[0136] In S312, the system control unit 50 switches between yes and no in tracking 402.

[0137] In S313, the system control unit 50 determines whether a switching operation to set the focus mode to AF mode or MF mode has been performed on the operation unit 70. If the operation has been performed, the process proceeds to S314; otherwise, the process proceeds to S315.

[0138] In S314, the system control unit 50 switches between AF and MF in the focusing mode 403 according to the operation.

[0139] In step S315, the system control unit 50 determines whether a switching operation for determining whether to display the focus guidance has been performed on the operation unit 70. If the operation has been performed, the process proceeds to step S316; otherwise, the process proceeds to step S317.

[0140] In S316, the system control unit 50 toggles between "yes" and "no" on the focus guidance display 404 based on the operation. The focus guidance is displayed to assist the user in manually adjusting the focus in MF mode, and the focus guidance indicates the adjustment direction and amount from the current position of the focusing object to the focus position.

[0141] Figure 5K This is an example of a focus guide display on an LV image. A subject detection frame 505 is superimposed on the LV image 501 at the location of the detected subject. A focus guide 511 is displayed next to the subject detection frame 505. The direction and amount of adjustment from the current position to the focus position are indicated by changing the display format of the focus guide 511. Whether the detected subject is in focus is indicated by changing the color of the focus guide 511.

[0142] In S317, the system control unit 50 determines whether a switching operation for other settings has already been performed on the operation unit 70. If the operation has been performed, the process proceeds to S318; otherwise, the process proceeds to S319.

[0143] In S318, the system control unit 50 switches other settings based on operation. "Other settings" include, for example, settings for function allocation that are activated when MC 65 is pressed.

[0144] In S319, the system control unit 50 overlays various setting values ​​related to processing in the camera mode onto the LV image 501 as camera information icons 502. Figure 5A This shows an example of the display of camera information icon 502 on LV image 501.

[0145] In S320, the system control unit 50 determines whether a specific lens is installed. A specific lens is, for example, a lens with a smaller AF range than usual. For such a lens, the user needs to be informed via a GUI or similar means of the range of AF available within the displayed LV image. If a specific lens is installed, the process proceeds to S321; otherwise, the process proceeds to S322.

[0146] In S321, such as Figure 5B As shown, the system control unit 50 overlays the camera information icon 502 onto the displayed LV image 501 and displays the AF available area 503. At this time, as in... Figure 6 As shown in the schematic diagram of the display layer of the GUI components of the digital camera 100, the AF available area 503 is displayed on a lower layer 604 than the layer 603 which displays the camera information icon 502.

[0147] In S322, the system control unit 50 determines whether the AF region is set to "full region". If the AF region is set to "full region", the process proceeds to S324; otherwise, the process proceeds to S323.

[0148] In S323, such as Figure 5B As shown, the system control unit 50 overlays a fixed AF frame 504, with dimensions corresponding to the current settings of the AF region, onto the LV image 501. At this time, as... Figure 6 As shown, the fixed AF frame 504 is displayed on a layer 601 that is higher than the layer 603 that displays the camera information icon 502 and the layer 602 that displays the gaze indicator described later.

[0149] In S324, the system control unit 50 determines whether to perform tracking. If tracking is to be performed, the process proceeds to S325; otherwise, the process proceeds to S327.

[0150] In S325, the system control unit 50 determines whether a subject has been detected in the LV image. If a subject is detected, the process proceeds to S326; otherwise, the process proceeds to S327.

[0151] In S326, such as Figure 5C As shown, the system control unit 50 overlays the subject detection frame 505 onto the detected subject in the LV image 501. Figure 6 As shown, the subject detection frame 505 is displayed on layer 601, similar to the fixed AF frame 504.

[0152] In S327, the system control unit 50 determines whether to perform line-of-sight input-based processing. If the line-of-sight input is valid, the processing proceeds to S328; otherwise, the processing proceeds to S331.

[0153] In S328, the system control unit 50 determines whether a gaze input has been detected. If a gaze input is detected, the process proceeds to S329; otherwise, it proceeds to S331.

[0154] In S329, the system control unit 50 determines whether the gaze indicator display is valid. If valid, the process proceeds to S330; otherwise, it proceeds to S331.

[0155] In S330, such as Figure 5D As shown, the system control unit 50 displays a gaze indicator 506 at the gaze position based on the detected gaze. At this time, the gaze indicator 506 is superimposed on the LV image 501. Figure 6 As shown, the gaze indicator 506 is displayed on layer 602, which is higher than layer 603, which displays camera information icon 502, and layer 604, which displays AF available area 503, and lower than layer 601, which displays various AF frames. Figure 5E This is a display example where the gaze indicator 506 and the subject detection box 505 are displayed in close proximity.

[0156] In S331, the system control unit 50 determines whether an operation has been performed on the zoom button 78 in the operation unit 70. If an operation has been performed, the process proceeds to S332; otherwise, the process proceeds to S339. When the zoom button 78 is pressed, partial magnification or demagnetization of the LV image is performed. If the LV image is not displayed in magnified mode, magnified display of the LV image is performed in response to the operation on the zoom button 78. If the LV image is displayed in magnified mode, magnified display of the LV image is demagnetized, and normal LV display is returned in response to the operation on the zoom button 78. In this embodiment, the system alternates between LV magnification display and demagnetization of LV magnification display in response to the pressing of the zoom button 78. However, it is also possible to sequentially switch between LV image magnification display (×5), LV magnified image display (×10), and demagnetization of LV magnification display in response to continuous pressing of the zoom button 78, thereby sequentially switching multiple magnification levels and demagnetization.

[0157] In S332, the system control unit 50 determines whether the LV image is displayed in magnified mode. If the LV image is displayed in magnified mode, the process proceeds to S333; otherwise, the process proceeds to S334.

[0158] In S333, the system control unit 50 demagnetizes the LV image in response to a user instruction.

[0159] In S334, the system control unit 50 determines whether a user's gaze is detected at the display unit 28 by switching between valid 406a and invalid 406b of the gaze input 406 in S303. If the user's gaze is detected, the process proceeds to S337; otherwise, the process proceeds to S335.

[0160] In S335, the system control unit 50 determines whether the fixed AF frame 504 is displayed on the LV image. If the fixed AF frame 504 is displayed, the process proceeds to S339; otherwise, the process proceeds to S336.

[0161] In step S336, the system control unit 50 determines whether the subject detection box 505 is displayed on the LV image. If the subject detection box 505 is displayed, the process proceeds to step S339; otherwise, the process proceeds to step S338.

[0162] In S337, the system control unit 50 displays the LV image in a magnified manner at the user's line-of-sight position. Figure 5L This shows an example of an LV image being magnified at the viewpoint and displayed in full screen, and is based on... Figure 5K The position of the gaze indicator 506 is used to magnify the LV image. The magnification range 512, the range indicating the entire image 513, and the magnification factor 514 are superimposed on the magnified LV image 515. During magnified display, the gaze indicator 506 is not displayed if the magnification position is moved or the subject is specified via an operating member. However, even during magnified display, the magnification position can be moved, the subject specified, and other operations can be performed based on the gaze. In the magnified LV image 515, the target focus position can be specified with high precision. This allows the user to manually focus while accurately identifying the subject to be focused.

[0163] A user's gaze can be used as a means to deactivate the state of displaying the LV image 515 in a magnified manner. For example, when a user gazes at a predetermined area at the end of the LV image 515, it can be determined that the user wishes to view an LV image outside the range of the LV image 515, and the magnified display can be deactivated. A specific GUI object can be overlaid on the LV image 515, and when a gaze is detected toward the object, the magnified display can be deactivated.

[0164] In S338, the system control unit 50 magnifies the LV image based on the center position of the screen.

[0165] In S339, the system control unit 50 magnifies the LV image based on the position of the fixed AF frame 504 or the subject detection frame 505. When the focus guide 511 is displayed, the subject detection frame 505 is also displayed. Therefore, the LV image is magnified based on the position of the subject detection frame 505. Even when the focus guide 511 is not displayed, if the subject detection frame 505 is displayed, the LV image is still magnified based on the position of the subject detection frame 505.

[0166] In S340, the system control unit 50 determines whether an operation to confirm the line-of-sight position has been performed on the operation unit 70. If a line-of-sight confirmation operation has been performed, the process proceeds to S341; otherwise, the process proceeds to S356. The line-of-sight confirmation operation in S340 is performed by pressing the AF-ON button 82. The line-of-sight confirmation function can be assigned to different buttons other than the AF-ON button. For example, the line-of-sight confirmation function can be assigned to the first shutter button 62, etc., and the line-of-sight position can be confirmed by half-pressing the first shutter button 62. If the system control unit 50 determines that... Figure 4B If the SW1 line of sight confirmation 408 on the advanced AF settings screen is switched to active (S308), then the line of sight confirmation function will be assigned to the first shutter button 62.

[0167] In S341, the system control unit 50 determines whether a user's gaze at the display unit 28 has been detected based on the switching between the valid 406a and invalid 406b of the gaze input 406 in S303. If a gaze is detected, the process proceeds to S342; otherwise, the process proceeds to S356.

[0168] In S342, the system control unit 50 determines whether the focusing mode is AF mode based on the focusing mode 403 switched by the user in S313. If the focusing mode is AF mode, the process proceeds to S343; if the focusing mode is MF mode, the process proceeds to S347.

[0169] In S343, the system control unit 50 determines whether to perform tracking based on the tracking 402 switched by the user in S311. If tracking is to be performed, the process proceeds to S345; otherwise, the process proceeds to S344.

[0170] In S344, the system control unit 50 determines whether to set an AF region across the entire region based on the AF region 401 switched by the user in S309. If an AF region is set across the entire region, the process proceeds to S345; otherwise, the process proceeds to S346.

[0171] In S345, the system control unit 50 tracks the subject located at a line-of-sight position based on the user's line of sight. Figure 5FThis shows an example of an LV image tracking a subject. Tracking frame 507 is superimposed on the LV image 501 at the gaze position based on the detected gaze. Tracking frame 507 is displayed as a fixed AF frame 504. Figure 6 On layer 601 as shown.

[0172] In S346, the system control unit 50 displays a fixed AF frame 504 at a line-of-sight position based on the user's line of sight.

[0173] In S347, the system control unit 50 determines whether the focus guide 511 is displayed on the LV image based on the focus guide display 404 switched by the user in S315. If the focus guide 511 is displayed, the process proceeds to S348; otherwise, the process proceeds to S353.

[0174] In S348, the system control unit 50 determines whether to perform tracking based on the tracking 402 switched by the user in S311. If tracking is to be performed, the process proceeds to S349; otherwise, the process proceeds to S352.

[0175] In S349, the system control unit 50 determines whether the subject detection frame 505 is displayed on the LV image. If the focus guide 511 is displayed on the LV image, the subject detection frame 505 is also displayed. If the subject detection frame 505 is displayed, the process proceeds to S350; otherwise, the process proceeds to S352.

[0176] In S350, the system control unit 50 determines whether the user's gaze position is next to the subject detection frame 505. If the gaze position is next to it, the process proceeds to S351; otherwise, the process proceeds to S356. The definition of "next to" for proceeding to S351 can vary depending on camera settings or conditions.

[0177] In S351, the system control unit 50 displays a tracking frame 507 at the user's line-of-sight position and tracks the subject.

[0178] In S352, the system control unit 50 displays a fixed frame at the line-of-sight position based on the user's line of sight.

[0179] In S353, the system control unit 50 magnifies the LV image at the user's line-of-sight position.

[0180] In S354, the system control unit 50 determines whether the gaze confirmation operation has been completed. If the gaze confirmation operation has been completed, the process proceeds to S355; otherwise, the process proceeds to S354. In response to the pressing of the zoom button 78, the LV image at the gaze position in S331 is magnified and displayed. In this case, the system control unit 50 controls the display so that the magnification remains even after the zoom button has been pressed, unless instructed by the user. Conversely, in response to the gaze confirmation operation, the LV image at the gaze position in S353 is magnified and displayed. In this case, if the system control unit 50 determines in S354 that the gaze confirmation operation has been completed, the system control unit 50 controls the display in S355 to demagnetize the LV image. In this way, when the LV image is magnified in response to the gaze confirmation operation, the system control unit 50 controls the display to temporarily magnify the LV image. Therefore, when the LV image is magnified in response to the gaze confirmation operation rather than the zoom button, the user can quickly switch to focusing operation.

[0181] In S356, the system control unit 50 determines whether another AF frame movement operation has been performed on the operation unit 70. If an operation has been performed, the process proceeds to S357; otherwise, the process proceeds to S358. Examples of AF frame movement operations include pressing MC 65 in eight directions and pressing the center to return the AF frame to the center of the screen.

[0182] In S357, the system control unit 50 moves the AF frame in response to the operation.

[0183] In S358, the system control unit 50 determines whether a camera operation has been performed on the operation unit 70. If the operation has been performed, the process proceeds to S359; otherwise, the process proceeds to S360.

[0184] In S359, the system control unit 50 performs camera processing.

[0185] In S360, the system control unit 50 determines whether other operations have already been performed on the operation unit 70. If other operations have been performed, the process proceeds to S361; otherwise, the process proceeds to S362. Examples of other operations include operations for changing various parameters (shutter speed and f-value, etc.).

[0186] In S361, the system control unit 50 performs other processing.

[0187] In S362, the system control unit 50 determines whether an exit operation has been performed on the operation unit 70. If so, the process is complete; otherwise, the process proceeds to S303.

[0188] Similar to shutter button 61, AF-ON button 82 may include a first switch and a second switch. A switch signal can be generated to activate the first function in response to a so-called half-press operation during the pressing of each button, and to activate the second function in response to a so-called full-press operation. The first and second functions may include, for example, confirmation of eye position, AE lock, LV image magnification, instruction for video recording, and instruction for AF execution. Components can also be provided to allow the user to freely assign these functions. For example, the first function can be assigned to operation for magnifying the LV image at a predetermined magnification, and the second function can be assigned to operation for magnifying the LV image at a magnification higher than the predetermined magnification. This allows for quick switching between multiple magnifications. By assigning the LV image magnification to the first function and the confirmation of eye position to the second function, it becomes possible to magnify the LV image and specify a more accurate position based on eye input. Focus mode settings can be included in the video information display 502.

[0189] In this way, the system control unit 50 controls the operation to perform different operations in response to the same line-of-sight confirmation operation by the user: in AF mode, the line-of-sight position is selected as the processing execution position, and in MF mode, a magnified display is provided at the line-of-sight position.

[0190] Immediately after displaying GUI components (such as the fixed AF box 504) at the position of the gaze indicator 506 based on the gaze confirmation operation, the gaze indicator 506 can be hidden for a predetermined period of time. This makes it easier to visually identify GUI components such as the fixed AF box 504 immediately after the confirmation operation.

[0191] The AF operation settings allow switching between single AF and servo AF. In single AF, the aforementioned display layer controls can be performed, while in servo AF, such controls are not required. This is because, when servo AF is set, the shooting scene involves a moving subject, and the positions of the gaze indicator 506 and the tracking frame 507 are constantly shifting; therefore, they are expected to overlap very little.

[0192] [Second Embodiment]

[0193] In the first embodiment, when a gaze confirmation operation is performed, the user's gaze is detected, the focus mode is MF mode, and no focus guidance is displayed, the system control unit 50 performs a magnified display of the LV image at the gaze position. In the second embodiment, in this case, the system control unit 50 performs a peaking display by emphasizing the focused portion of the subject within a predetermined range based on the gaze position.

[0194] Figure 7This is a diagram illustrating an example of the focus state displayed on the display unit 28 of a digital camera 100, which is an example of an imaging device according to the second embodiment. Among the child 701, father 702, and mother 703, the child 701 is in focus. The edges of the child 701 are highlighted, while the edges of the unfocused father 702 and mother 703 are weakened, thus emphasizing the edges of the child 701. Therefore, even when multiple subjects are photographed, the focused subject is displayed, making it easier for the user to manually focus on the desired subject even in MF mode.

[0195] In the first embodiment, an example of the invention being applicable to a digital camera has been described. However, the invention can be applied to situations requiring rapid and accurate selection of a subject or designated object.

[0196] [Third Embodiment]

[0197] Figures 8A to 8C This is an external view of a head-mounted device including a head-mounted display (HMD) and a game controller 820 used as an operating component, according to a third embodiment. The third embodiment assumes a user wearing the head-mounted device is playing an action game. When virtual game characters or the like are displayed on a virtual reality (VR) image on the HMD, the user can quickly select an attack target based on their gaze and confirm the action, or accurately determine the attack target by zooming in on the image.

[0198] Figure 8A and Figure 8B This is an external view of the head-mounted device 800. The head-mounted device 800 is mainly composed of an HMD 810. The HMD 810 is a VR device (standalone VR-HMD) that can be used independently without being connected to a personal computer (PC) or the like. The HMD 810 incorporates a system control unit, a gaze detection unit, and other components required to implement this embodiment.

[0199] Figure 8C This is an external view of the game controller 820. The game controller 820 receives instructions for displaying the game screen 900 (described later) on the HMD 810 and performs related operations. The game controller 820 includes a grip 821, multiple buttons 822 as pressable operating members including touchable portions, and a setting button 75. Similar to the grip 90 of the digital camera 100, the grip 821 is constructed and made of a material that allows the user to easily hold the game controller 820. The multiple buttons 822 can be assigned various functions, such as providing instructions for moving the indicator displayed on the game screen 900 (described later).

[0200] The multi-button 822 is an operating component capable of receiving both touch and press operations. The touch operating component mounted on the multi-button 822 can be a touch detection mechanism or an infrared sensor as described above. The infrared sensor is arranged in the multi-button 822 and applies infrared light to the top of the multi-button 822 (the part that contacts the user's finger). When the user touches the top of the multi-button 822, the infrared light is reflected by the user's finger. The movement of the user's finger can be detected by detecting this reflected light. Such an optical operating component is called an optical tracking indicator (OTP). Similar to the touch panel 70a, the OTP can detect movement operations as the movement of the user's finger (operating component) relative to the OTP. The multi-button 822 is not limited to an OTP and can also be a direction indicator component such as an MC 65, a four-way button 74, or a joystick. The HMD 810 and the game controller 820 can be connected via wired or wireless communication such as Bluetooth.

[0201] Figures 9A to 9F This shows an example of a game screen displayed on the HMD 810. Here, Figures 9A to 9C This demonstrates an example where the aiming assist mode (aiming assist mode) is invalid when the user has not yet selected an attack target, and Figures 9D to 9F This shows an example where the pattern is valid.

[0202] Figure 9A This example shows a game screen displaying multiple attack targets 901 to 904 virtually approaching a user wearing a headset 800 and playing an action game. Aiming assist mode is disabled. At this time, a gaze magnification box 910 is displayed at the location of the user's gaze detected by the gaze detection unit (not shown) based on the HMD 810. The gaze magnification box 910 moves from... Figure 9A Move to the position shown Figure 9B The location shown. Subsequently, when the user performs a visual confirmation operation (specifically, pressing the multi-button 822 or the setting button 75), as... Figure 9C As shown, the area within the magnification frame 910 is magnified. This allows the user to accurately pinpoint the target of the attack on the magnified display.

[0203] Figure 9D and Figure 9E This is an example of a screenshot from an action game when aim assist mode is active. Instead of displaying the magnification box 910, a gaze indicator 912 is shown, which the user can use to select the target. At this time, as... Figure 9ESimilar to the line-of-sight indicator 912, when a position slightly offset from the target is selected, the target is determined based on an automatic selection criterion of choosing the target closest to the line-of-sight indicator 912. As a result, if the target 901 meets the requirement of being the closest target, then... Figure 9D As shown, the position of the line-of-sight indicator 912 is automatically corrected. In this state, the user can give attack instructions.

[0204] Figure 9F This is a display example of a screen that makes it easier to select an attack target when aiming assist mode is active. In this case, instead of displaying a magnifying glass 910, a selection box 914 and a line-of-sight indicator 912 are displayed. The line-of-sight indicator 912 is displayed to follow the line of sight. The area outside the selection box 914 is displayed as a grayed-out area 913, indicating that the line-of-sight indicator 912 cannot be moved out of the selection box 914. This display prevents the user from selecting an unwanted attack target 904, thus making it easier to select the desired attack target.

[0205] In this way, the HMD 810's system control unit operates to perform different actions in response to the same line-of-sight confirmation action performed by the user: when aiming assist mode is disabled, a magnified display is provided at the line-of-sight location; and when aiming assist mode is enabled, the desired target is selected.

[0206] [Fourth Embodiment]

[0207] Figure 10A and Figure 10B An example of a video editing screen of a personal computer (PC) according to a fourth embodiment is shown. The PC includes a central processing unit (CPU), random access memory (RAM), read-only memory (ROM), storage device, network interface (I / F), gaze detection unit, and display unit, etc. Figure 10A The image shows a screen used to edit a video so that it can be viewed in a zoomed-in manner starting from a certain frame. Figure 10B This shows the screen used for editing the video to trim it. Therefore, the modes used for video editing include zoom-in editing mode (…). Figure 10A ) and editing mode ( Figure 10B Here, a frame refers to a single still image that makes up a video. For example, a one-minute video can be made by playing thirty still images (i.e., thirty frames) in sequence.

[0208] In the PC's zoom-in editing mode, the PC's CPU controls the display, causing the zoom-in editing screen 1001, including the video 1002 before zooming in and the preview video 1003 after zooming in, to be displayed on the display unit. The user plays back the video 1002 before zooming in and determines the frame to be zoomed in. When the user's gaze is detected in the frame to be zoomed in, the PC's CPU controls the display, causing the video to be zoomed in at the gaze position based on the user's gaze. At this time, the PC's CPU controls the display, causing the video to be zoomed in at the gaze position in response to the user's gaze confirmation operation. An example of a gaze confirmation operation is a mouse click. The zoomed-in frame can be viewed in the preview video 1003. Figure 10A In this process, the PC's CPU controls the display, causing a gaze indicator 1004 to be displayed at the user's gaze position, thereby allowing the user to confirm the location to be zoomed in on.

[0209] In the PC's editing mode, the PC's CPU controls the display, causing the editing screen 1005, which includes the video 1006 before editing and the preview video 1007 after editing, to be displayed on the display unit. The user plays back the video 1006 before editing and determines the first and last frames used to construct the desired video. Specifically, Figure 10B An example of determining the first frame is shown. The PC's CPU controls the display, causing cursor 1008 to be moved to a viewing position based on the user's gaze and displayed. This allows the first frame constituting the desired video to be determined. At this point, the PC's CPU controls the display, causing cursor 1008 to be moved to the viewing position in response to the user's gaze confirmation operation. An example of a gaze confirmation operation is a mouse click. In the preview video 1007 after editing, the first frame constituting the desired video can be confirmed. Figure 10B In this process, the PC's CPU controls the display, causing the gaze indicator 1004 to be displayed at the user's gaze position, thereby enabling the user to confirm on the screen the position where the user intends to move the cursor 1008.

[0210] In this way, the PC's CPU control operation enables different processing to be performed in response to the user's gaze confirmation operation: in zoom-in editing mode, a zoomed-in display is provided at the gaze position; and in clip editing mode, the first and last frames that constitute the desired video are selected.

[0211] refer to Figures 5A to 5L , Figures 9A to 9F as well as Figure 10A and Figure 10BThe described display form of the gaze indicator is merely illustrative. The display form can be not only a combination of outer and inner circles, but also only the outer circle or only the inner circle. Components can be configured to switch between such display forms. The order of layers can change depending on the display form. For example, in the case of using only the outer circle for the gaze indicator display form, other GUI components are not hidden by the inner circle, so gaze indicator 506, for example, can be displayed on a lower layer.

[0212] [Other Embodiments]

[0213] The various control operations described above, performed by the system control unit 50 of the camera device 100 or the CPU of the PC, can be executed by a single hardware component, or the control of the entire device can be performed by multiple hardware components (e.g., multiple processors or circuits) that share the processing.

[0214] While the present invention has been described in detail based on preferred embodiments, it is not limited to the specific embodiments described above. It should be understood that various modifications and variations can be made without departing from the spirit of the invention, and such modifications are also included within the scope of the invention. Furthermore, the above embodiments are merely examples of the invention, and features of the various embodiments can be appropriately combined.

[0215] The present invention can also be implemented by performing the following process. That is, software (programs) for implementing the functions of the above embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU or MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0216] This invention is not limited to the embodiments described above, and various modifications and alterations can be made without departing from the spirit and scope of the invention. Therefore, the appended claims are attached to publicly disclose the scope of the invention.

[0217] This application claims the benefit of Japanese Patent Application 2023-051690, filed on March 28, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An electronic device comprising: A detection component for detecting the position of a user's gaze as they view a display unit that shows a live view image captured by a camera unit; The operating component is used to receive operations performed by the user; A mode switching component for switching between AF mode and MF mode; as well as A control unit, configured to respond to a specific operation performed by the user on the operating unit, in the live view image displayed on the display unit, (1) In the AF mode, a first process is performed, the first process being used to determine the location to perform AF based on the gaze position on the real-time viewfinder image detected by the detection component, and (2) A second process is performed in the MF mode, the second process being used to assist manual focus adjustment based on the gaze position on the real-time viewfinder image detected by the detection component. In the second process, control is performed to complete the second process in response to the completion of the specific operation.

2. The electronic device according to claim 1, wherein, The second process for assisting manual focusing is a process for displaying the live view image in a magnified manner based on the line-of-sight position.

3. The electronic device according to claim 1, wherein, The second process used to assist manual focusing is a process for overlaying information related to the focus state onto the live view image.

4. The electronic device according to claim 3, wherein, The second processing used to assist manual focusing is a peak display processing for highlighting the focused portion in the live view image.

5. The electronic device according to claim 1, wherein, In the MF mode, when a guide for assisting manual focus is displayed in the live view image, the control unit controls itself so that the second processing is not performed even when the user performs the specific operation on the operation unit. Conversely, when the guide for assisting manual focus is not displayed in the live view image, the control unit controls itself so that the second processing is performed when the user performs the specific operation on the operation unit.

6. The electronic device according to claim 5, wherein, In the MF mode, when a guide for assisting manual focus is displayed in the live view image, when the user performs the specific operation on the operation component, a frame is displayed on the subject on which the guide is to be displayed or the subject on which the guide is to be displayed is tracked based on the line of sight detected by the detection component.

7. The electronic device according to claim 1, wherein, The specific operation of the operating component is an operation used to determine the location where the first process and the second process are to be performed based on the line-of-sight position.

8. The electronic device according to claim 1, wherein, The specific operation of the operating component is pressing the button that provides an instruction to perform AF or half-pressing the button that provides an instruction to take a picture.

9. The electronic device according to claim 1, in, The control unit controls the display of the live view image in partial magnification in response to the pressing of a zoom button (different from the operating unit), and maintains the magnified display even after the zoom button is pressed. In the second process performed in response to the specific operation of the operating component, the control component controls the real-time view image to be temporarily displayed in a magnified manner based on the line-of-sight position detected by the detection component.

10. The electronic device according to claim 1, in, In the live view image, the control unit displays a gaze indicator to follow the gaze position detected by the detection unit, and During the second processing performed by the control unit, the gaze indicator is not displayed.

11. The electronic device according to claim 1, wherein, The electronic device is a camera device, which includes the camera component and a recording component for recording the real-time view image captured by the camera component onto a recording medium.

12. A method for controlling an electronic device, the method comprising: The detection step is used to detect the position of the user's gaze when viewing the display unit used to display the live view image captured by the camera unit; Operation steps, used to receive operations performed by the user; The mode switching procedure is used to switch between AF mode and MF mode; as well as Control steps, used in response to a specific operation performed by the user on the operating component, in the live view image displayed on the display component, (1) Execute a first process in the AF mode, the first process being used to determine the location to perform AF based on the gaze position on the real-time viewfinder image detected by the detection component. (2) A second process is performed in the MF mode, the second process being used to assist manual focus adjustment based on the gaze position on the real-time viewfinder image detected by the detection component. In the second process, control is performed to complete the second process in response to the completion of the specific operation.

13. A program for causing a computer to perform a control method for an electronic device according to claim 12.

14. A computer-readable storage medium storing a program for causing a computer to perform the control method of the electronic device according to claim 12.

Images