Controlling device using wearable device

By integrating the gaze tracking sensor and camera in the head-mounted device, using gaze input and image data to control external electronic devices, the problem of head-mounted devices' wireless pairing difficulties and command conflicts is solved, and efficient inter-device collaborative operation is achieved.

CN118897620APending Publication Date: 2024-11-05APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410533164.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-04-24
Filing Date
2024-04-30
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

When controlling external electronic devices, existing head-mounted devices are prone to difficulty in controlling wireless pairing or command conflicts, resulting in inconvenience in operation.

Method used

By integrating gaze tracking sensors and cameras in a head-mounted device, using gaze inputs and image data to determine the position and user interface elements on the display, transmit relevant information to external electronic devices, achieve precise control of external devices, and suppress attachment devices. input instructions to avoid conflicts.

Benefits of technology

It realizes seamless control and operation of external electronic devices, improves user experience, avoids command conflicts, and enhances the collaborative working ability between devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897620A_ABST
    Figure CN118897620A_ABST
Patent Text Reader

Abstract

A head-mounted device may be used to control one or more external electronic devices. The gaze input and the camera image may be used to determine a gaze point relative to a display of the external electronic device. The external electronic device may receive information about a gaze input of the user from the head-mounted device, and may highlight a user interface element of a plurality of user interface elements targeted by the gaze input. The head-mounted device may receive input, such as keystroke information, from an accessory device and relay the input to an external electronic device being viewed. The head-mounted device may receive a display configuration request, determine layout information for a display in a physical environment of the head-mounted device, and transmit the layout information to an external device associated with the display.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 644,218 filed on April 24, 2024, U.S. Provisional Patent Application No. 63 / 550,949 filed on February 7, 2024, and U.S. Provisional Patent Application No. 63 / 500,499 filed on May 5, 2023, which are hereby incorporated by reference in their entirety. Technical Field

[0002] The present disclosure relates generally to electronic devices, and more particularly to head mounted devices. Background Art

[0003] Some electronic devices, such as head mounted devices, can be wirelessly paired with additional electronic devices, which can be difficult to control if you are not careful. Summary of the invention

[0004] An electronic device may include one or more sensors; one or more processors; and a memory storing instructions configured to be executed by the one or more processors, the instructions being used to obtain gaze input via a first subset of the one or more sensors, obtain an image including a display via a second subset of the one or more sensors, determine a location on the display corresponding to the gaze input using at least the gaze input and the image, and transmit information associated with the location on the display to an external electronic device.

[0005] The electronic device may include one or more sensors; a communication circuit; one or more processors; and a memory storing instructions configured to be executed by the one or more processors, the instructions being used to receive user input information from an accessory device using the communication circuit, obtain sensor data via one or more sensors, and transmit the user input information to an external electronic device using the communication circuit based on determining that the external electronic device is being viewed.

[0006] An electronic device may include one or more sensors; a communication circuit configured to communicate with an external electronic device having one or more associated displays; one or more processors; and a memory storing instructions configured to be executed by the one or more processors, the instructions being used to receive a request from the external electronic device, obtain an image of the one or more associated displays via a first subset of the one or more sensors, and based on the received request, use the image to determine layout information for the one or more associated displays, and transmit the layout information to the external electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a schematic diagram of an illustrative electronic device according to some embodiments.

[0008] Figure 2 is a view of an illustrative system including a head-mounted device and a display with content controlled by gaze input according to some embodiments.

[0009] Figure 3A is a view of an illustrative display presenting optical symbols for optically pairing with a head-mounted device according to some embodiments.

[0010] Figure 3B is a view of an illustrative display with a first user interface element targeted by gaze input according to some embodiments.

[0011] Figure 3C is a view of an illustrative display with a second user interface element targeted by gaze input according to some embodiments.

[0012] Figure 4 is a flowchart of an illustrative method for operating a head-mounted device according to some embodiments, wherein the head-mounted device uses gaze input sensed at the head-mounted device to control an external electronic device.

[0013] Figure 5A is a view of an illustrative system with a head-mounted device that relays input from an accessory device to a first external electronic device according to some embodiments.

[0014] Figure 5B is a view of an illustrative system with a head-mounted device that relays input from an accessory device to a second external electronic device according to some embodiments.

[0015] Figure 6 is a flowchart of an illustrative method for operating a head-mounted device that relays input from an accessory device to an external electronic device being viewed, according to some embodiments.

[0016] Figure 7 is a view of an illustrative system having a head-mounted device that obtains layout information for a display and transmits the layout information to an external electronic device associated with the display according to some embodiments.

[0017] Figure 8 is a flowchart of an exemplary method of operating a head-mounted device according to some embodiments, wherein the head-mounted device obtains layout information for a display and transmits the layout information to an external electronic device associated with the display.

[0018] Fig. 9 is a view of an illustrative display with shortcuts triggered by a gaze gesture according to some embodiments. DETAILED DESCRIPTION

[0019] exist Figure 1 A schematic diagram of an exemplary electronic device is shown in FIG. Figure 1 As shown, the electronic device 10 (sometimes referred to as a head-mounted device 10, a system 10, a head-mounted display 10, etc.) may have a control circuit 14. In addition to being a head-mounted device, the electronic device 10 may be other types of electronic devices, such as a cellular phone, a laptop computer, a speaker, a computer monitor, an electronic watch, a tablet computer, etc.

[0020] The control circuit 14 may be configured to perform operations in the head mounted device 10 using hardware (e.g., dedicated hardware or circuitry), firmware, and / or software. Software code and other data for performing operations in the head mounted device 10 are stored on a non-transitory computer readable storage medium (e.g., a tangible computer readable storage medium) in the control circuit 14. Software code may sometimes be referred to as software, data, program instructions, instructions, or code. Non-transitory computer readable storage media (sometimes commonly referred to as memory) may include non-volatile memory such as non-volatile random access memory (NVRAM), one or more hard disk drives (e.g., disk drives or solid-state drives), one or more removable flash drives, or other removable media, etc. The software stored on the non-transitory computer readable storage medium may be executed on a processing circuit of the control circuit 14. The processing circuit may include a dedicated integrated circuit having a processing circuit, one or more microprocessors, a digital signal processor, a graphics processing unit, a central processing unit (CPU), or other processing circuits.

[0021] The head mounted device 10 may include input-output circuitry 16. The input-output circuitry 16 may be used to allow a user to provide user input to the head mounted device 10. The input-output circuitry 16 may also be used to gather information about the environment in which the head mounted device 10 is operating. Output components in the circuitry 16 may allow the head mounted device 10 to provide output to the user.

[0022] like Figure 1As shown, the input-output circuit 16 may include a display, such as a display 18. The display 18 may be used to display images for the user of the head mounted device 10. The display 18 may be a transparent or semi-transparent display so that the user can observe the physical object through the display while overlaying the computer-generated content on top of the physical object by presenting the computer-generated image on the display. The transparent or semi-transparent display may be formed by a transparent or semi-transparent pixel array (e.g., a transparent organic light emitting diode display panel), or may be formed by a display device that provides an image to the user through a transparent structure such as a beam splitter, a holographic coupler, or other optical coupler (e.g., a display device such as a liquid crystal on a silicon display). Alternatively, the display 18 may be an opaque display that blocks light from the physical object when the user operates the head mounted device 10. In this type of arrangement, a perspective camera may be used to display the physical object to the user. The perspective camera may capture an image of the physical environment, and the physical environment image may be displayed on the display for viewing by the user. Additional computer-generated content (e.g., text, game content, other visual content, etc.) can optionally be overlaid on the image of the physical environment to provide the user with an extended reality environment. When the display 18 is opaque, the display can also optionally display the entire computer-generated content (e.g., without displaying the image of the physical environment).

[0023] Display 18 may include one or more optical systems (e.g., lenses) (sometimes referred to as optical components) that allow an observer to view images on display 18. A single display 18 may produce images for both eyes, or a pair of displays 18 may be used to display images. In a configuration with multiple displays (e.g., a left-eye display and a right-eye display), the focal length and position of the lenses may be selected so that any gaps that exist between the displays will not be visible to the user (e.g., so that the images of the left and right displays overlap or merge seamlessly). A display module (sometimes referred to as a display component) that generates different images for the left and right eyes of a user may be referred to as a stereoscopic display. A stereoscopic display may be able to present two-dimensional content (e.g., a user notification with text) and three-dimensional content (e.g., a simulation of a physical object such as a cube).

[0024] Display 18 may include an organic light emitting diode display or other display based on an array of light emitting diodes, a liquid crystal display, a liquid crystal on silicon display, a projector or a display based on projecting a light beam directly or indirectly onto a surface through specialized optical devices (e.g., a digital micromirror device), an electrophoretic display, a plasma display, an electrowetting display, or any other desired display.

[0025] Input-output circuitry 16 may include various other input-output devices. For example, input-output circuitry 16 may include one or more speakers 20 configured to play audio, and one or more microphones 30 configured to capture audio data from a user and / or from the physical environment surrounding the user.

[0026] Input-output circuitry 16 may include one or more cameras 22. Cameras 22 may include one or more outward-facing cameras (as an example, facing the physical environment around the user when the electronic device is mounted on the user's head). Cameras 22 may capture visible light images, infrared images, or any other desired type of image. If desired, the cameras may be stereo cameras. Outward-facing cameras may capture pass-through video for device 10. Cameras 22 may also include inward-facing cameras (e.g., for gaze detection).

[0027] like Figure 1 As shown, the input-output circuit 16 may include a position and motion sensor 24 (e.g., a compass, a gyroscope, an accelerometer, and / or other device for monitoring the position, orientation, and movement of the electronic device 10, a satellite navigation system circuit such as a global positioning system circuit for monitoring the user's position, etc.). For example, the control circuit 14 may use the sensor 24 to monitor the current direction of the user's head relative to the surrounding environment (e.g., the user's head posture). The camera in the camera 22 may also be considered as part of the position and motion sensor 24. The camera can be used for facial tracking (e.g., by capturing images of the user's chin, mouth, etc. when the device is worn on the user's head), body tracking (e.g., by capturing images of the user's torso, arms, hands, legs, etc. when the device is worn on the user's head) and / or for positioning (e.g., using visual odometry, visual inertial odometry, or other simultaneous localization and mapping (SLAM) techniques).

[0028] The input-output circuit 16 may include a gaze tracking sensor 26 (sometimes referred to as a gaze tracker 26, a gaze tracking system 26, a gaze detection sensor 26, etc.). The gaze tracking sensor 26 may include a camera and / or other gaze tracking sensor components (e.g., a light source that emits a light beam so that a reflection of the light beam from the user's eye can be detected) to monitor the user's eye. The gaze tracker 26 may face the user's eye and may track the user's gaze. The camera in the gaze tracking system may determine the position of the user's eye (e.g., the center of the user's pupil), may determine the orientation direction of the user's eye (the user's gaze direction), may determine the user's pupil size (e.g., so that light modulation and / or other optical parameters, and / or sequential amounts used to spatially adjust one or more of these parameters, and / or areas in which one or more of these optical parameters are adjusted, these areas are adjusted based on the pupil size), can be used to monitor the current focus of the lens in the user's eye (e.g., whether the user is focusing on the near field or the far field, which can be used to assess whether the user is daydreaming or thinking strategically or tactically), and / or other gaze information. The cameras in the gaze tracking system may sometimes be referred to as inward-facing cameras, gaze detection cameras, eye tracking cameras, gaze tracking cameras, or eye monitoring cameras. Other types of image sensors (e.g., infrared and / or visible light emitting diodes and photodetectors, etc.) may also be used to monitor the user's gaze if desired. The use of a gaze detection camera in the gaze tracker 26 is merely exemplary.

[0029] The input-output circuitry 16 may include one or more depth sensors 28. Each depth sensor may be a pixelated depth sensor (e.g., configured to measure multiple depths across a physical environment) or a point sensor (configured to measure a single depth in a physical environment). Each depth sensor (whether a pixelated depth sensor or a point sensor) may measure depth using phase detection (e.g., phase detection autofocus pixels) or light detection and ranging (LIDAR). Camera images (e.g., from one of the cameras 22) may also be used for monocular and / or stereo depth estimation. Any combination of depth sensors may be used to determine the depth of physical objects in a physical environment.

[0030] The input-output circuit 16 may include a button 32. The button may include a mechanical switch that detects a user press during operation of the head mounted device. Alternatively, the button 32 may be a virtual button that uses touch sensing to detect a user press.

[0031] If desired, the input-output circuit 16 may also include other sensors and input-output components (e.g., ambient light sensors, force sensors, temperature sensors, touch sensors, capacitive proximity sensors, light-based proximity sensors, other proximity sensors, strain gauges, gas sensors, pressure sensors, humidity sensors, magnetic sensors, audio components, tactile output devices such as actuators and / or vibration motors, light-emitting diodes, other light sources, etc.).

[0032] The head mounted device 10 may also include communication circuitry 56 to allow the head mounted device to communicate with external equipment (e.g., a tethered computer, a portable device, one or more external servers, or other electrical equipment). The communication circuitry 56 may be used for wired and wireless communication with external equipment.

[0033] The communication circuit 56 may include a radio frequency transceiver circuit formed by one or more integrated circuits, power amplifier circuits, low noise input amplifiers, passive radio frequency (RF) components, one or more antennas, transmission lines, and other circuits for processing RF wireless signals. Wireless signals may also be sent using light (e.g., using infrared communications).

[0034] For example, the RF transceiver circuitry in the wireless communication circuitry 56 may process wireless local area network (WLAN) communication frequency bands, such as 2.4 GHz and 5 GHz. (IEEE802.11) frequency band; Wireless Personal Area Network (WPAN) communication frequency band, such as 2.4 GHz Communication bands; cellular telephone communication bands, such as cellular low band (LB) (e.g., 600 MHz to 960 MHz), cellular low-mid band (LMB) (e.g., 1400 MHz to 1550 MHz), cellular mid band (MB) (e.g., 1700 MHz to 2200 MHz), cellular high band (HB) (e.g., 2300 MHz to 2700 MHz), cellular ultra-high band (UHB) (e.g., 3300 MHz to 5000 MHz), or other cellular communication bands between about 600 MHz and about 5000 MHz (e.g., 3G bands, 4G LTE bands, 5G new radio frequency range 1 (FR1) bands below 10 GHz, etc.); near field communication (NFC) bands (e.g., 13.56 MHz); satellite navigation bands (e.g., L1 global positioning system (GPS) band of 1575 MHz, L5 GPS band, Global Navigation Satellite System (GLONASS) band, BeiDou Navigation Satellite System (BDS) band, etc.); ultra-wideband (UWB) communication band supported by IEEE802.15.4 protocol and / or other UWB communication protocols (for example, a first UWB communication band of 6.5 GHz and / or a second UWB communication band of 8.0 GHz); and / or any other required communication band.

[0035] The radio frequency transceiver circuitry may include millimeter / centimeter wave transceiver circuitry that supports communications at frequencies between about 10 GHz and 300 GHz. For example, the millimeter wave / centimeter wave transceiver circuitry may support communications in an extremely high frequency (EHF) or millimeter wave communications band between about 30 GHz and 300 GHz and / or in a centimeter wave communications band between about 10 GHz and 30 GHz (sometimes referred to as a super high frequency (SHF) band). For example, the millimeter wave / centimeter wave transceiver circuitry may support communications in the following communications bands: the IEEE K communications band between about 18 GHz and 27 GHz, the K band between about 26.5 GHz and 40 GHz, and the IEEE K band between about 26.5 GHz and 40 GHz. a Communications band, K between approximately 12 GHz and 18 GHz u The mmWave / cmWave transceiver circuitry may support IEEE 802.11ad communications at 60 GHz (e.g., WiGig or 60 GHz Wi-Fi bands of approximately 57 GHz to 61 GHz) and / or 5th generation mobile networks or 5th generation wireless systems (5G) New Radio (NR) frequency range 2 (FR2) communications bands between approximately 24 GHz and 90 GHz, if desired.

[0036] The antenna in the wireless communication circuit 56 may include an antenna having a resonant element formed by a loop antenna structure, a patch antenna structure, an inverted-F antenna structure, a slot antenna structure, a planar inverted-F antenna structure, a spiral antenna structure, a dipole antenna structure, a monopole antenna structure, a combination of these designs, etc. Different types of antennas may be used for different frequency bands and frequency band combinations. For example, one type of antenna may be used when forming a local wireless link, and another type of antenna may be used when forming a remote wireless link antenna.

[0037] The head mounted device 10 can be paired with one or more additional electronic devices. In other words, a wireless link can be established between the electronic device 10 and the additional electronic device to allow fast and efficient communication between the device 10 and the additional electronic device.

[0038] Figure 2 A system 8 of electronic devices including a head mounted device 10 is shown. Figure 2As shown, system 8 also includes electronic devices 40, 42, and 48. Electronic devices 10, 40, 42, and 48 may be associated with the same user (e.g., logged into the cloud service using the same user ID), may exchange wireless communications, etc. In general, each of electronic devices 10, 40, 42, and 48 may be any desired type of electronic device (e.g., a cellular phone, a laptop computer, a speaker, a computer monitor, an electronic watch, a tablet computer, a head-mounted device, a remote control, a television, etc.). As an example, electronic device 42 may be a television or other device that includes a display. Electronic device 40 may be a source device that supplies images to electronic device 42. There may be a wired connection (e.g., a wireless connection) between devices 40 and 42. Figure 2 ) or wirelessly connected. Electronic device 40 may have a small console form factor without a display. Electronic device 40 may sometimes be referred to as a set-top box. Electronic device 48 may be a remote control configured to transmit signals to electronic device 40, as shown by transmission 50. Head mounted device 10 may both transmit signals to electronic device 40 and receive signals from electronic device 40, as shown by wireless link 52.

[0039] Each of the electronic devices 40, 42, and 48 may include any desired input-output components (e.g., similar to the combination of Figure 1 Input-output circuit as described above). Figure 2 As shown, electronic device 42 may include a display 44 coupled to housing 46. Display 44 may include an organic light emitting diode display or other display based on an array of light emitting diodes, a liquid crystal display, a liquid crystal on silicon display, a projector or a display based on projecting a light beam onto a surface directly or indirectly through specialized optics (e.g., a digital micromirror device), an electrophoretic display, a plasma display, an electrowetting display, or any other desired display.

[0040] Each of the electronic devices 40, 42 and 48 may optionally include communication circuitry (similar to Figure 1 Communication circuit 56 in the system 8) to exchange wired and / or wireless communications with other devices in the system 8.

[0041] During operation of the system 8, the remote control 48 may sometimes be used to control the electronic devices 40 and / or 42. The head mounted device 10 may also be used to control the electronic devices 40 and / or 42. For example, a user may provide user input to the head mounted device 10 indicating an intention to use gaze input to control the electronic devices 40 and / or 42. Subsequently, an optical marker associated with the electronic device 42 (e.g., displayed on the display 44 or visible on the housing 46) may be used by the head mounted device 10 to position the electronic device 42 relative to the head mounted device 10. The gaze input from the user on the head mounted device 10 may then be used to aim and / or select a user interface element on the display 44. For example, the user may gaze at a user interface element on the display 44. Ray tracing may be used to determine the user's gaze point on the display 44 (e.g., using gaze information from the gaze detection sensor 26). Information about the gaze point on the display 44 is then transmitted by the head mounted device 10 to the electronic devices 40 and / or 42 to control the electronic devices 40 and / or 42.

[0042] To avoid conflicting commands when controlling content presented on display 44, input from remote control 48 may be suppressed when head mounted device 10 is used to control electronic devices 40 and / or 42.

[0043] FIG. 3A to FIG. 3C is a view of an illustrative display showing how a head mounted device may be used to control content on the display. Figure 3A The electronic device 42 is shown after the user has provided user input associated with an intent to interact with the electronic device 42. As examples, the user input associated with the intent to interact with the electronic device 42 may include gaze input detected by the gaze detection sensor 26 (e.g., a gaze gesture, such as looking toward a predetermined corner of the display 18 on the head mounted device 10), touch input to the touch sensor (such as a swipe or press), the press of a button (such as button 32), a hand gesture detected by the camera 22, a head gesture detected by the position and motion sensor 24, a voice command detected by the microphone 30, and the like.

[0044] After receiving user input associated with an intent to interact with electronic device 42, head mounted device 10 may transmit information associated with the user input to electronic device 40 (e.g., in an arrangement where electronic device 40 controls content presented on display 44 of electronic device 42) or directly to electronic device 42 (e.g., in an arrangement where electronic device 40 is omitted from the system and electronic device 42 is a standalone device).

[0045] Transmitting information associated with the user input to the electronic device 40 and / or the electronic device 42 may cause the display 44 of the electronic device 42 to display the optical symbology 60. The optical symbology may then be used by the head mounted device 10 to determine the position of the electronic device 42 relative to the head mounted device 10. The optical symbology 60 may be displayed simultaneously with one or more user interface elements, such as the user interface elements 54-1, 54-2, and 54-3. The optical symbology 60 may be an icon associated with establishing gaze control of the electronic device 42 using the gaze detection sensor 26 on the head mounted device 10. Alternatively or in addition, the optical symbology may include one or more glyphs.

[0046] In general, the appearance of the optical symbol 60 can be selected to be obvious to the observer or not obvious to the observer. When the optical symbol is obvious to the observer, the optical symbol is intended to clearly indicate to the user that gaze control of the electronic device 42 is being established using the gaze detection sensor 26 on the head mounted device 10. In order to make the optical symbol unnoticeable to the observer, the optical symbol can be integrated into the user interface presented on the display 44 (or other content being displayed on the display 44). As an example, an icon or one or more glyphs that are part of a user interface element presented on the display 44 can be used as the optical symbol 60. Another option for an optical symbol that is unnoticeable to the observer is to present the optical symbol using invisible (e.g., infrared) light, which can be detected by the head mounted device 10 (but will not be seen by the user's eyes).

[0047] One or more cameras 22 in the head mounted device 10 may capture images of the electronic device 42. The head mounted device 10 may be aware of the size and shape of the optical symbology 60. Therefore, when the display 44 presents the optical symbology 60, the head mounted device 10 may recognize the optical symbology in the image from the camera 22 and use the image from the camera 22 to accurately determine the position of the display 44 relative to the head mounted device 10.

[0048] Figure 3A An example is shown in which an optical symbol 60 is displayed on the display 44. This example is illustrative only. If desired, the optical symbol 60 may alternatively be visible on a non-display portion of the electronic device 42 (such as the housing 46) (as shown by symbol 60').

[0049] Once the head mounted device has determined the position of the display 44 relative to the head mounted device 10, the gaze input from the gaze tracking sensor 26 can be used to determine the user's gaze point on the display 44. Figure 3BAs shown, ray tracing can be used to determine that gaze point 38 overlaps user interface element 54-1. This information can be transmitted from head mounted device 10 to electronic device 40 (e.g., in an arrangement where electronic device 40 controls content presented on display 44 of electronic device 42) or directly to electronic device 42 (e.g., in an arrangement where electronic device 40 is omitted from the system and electronic device 42 is a stand-alone device). Head mounted device 10 can optionally receive information about the size and layout of user interface elements on display 44 from electronic devices 40 and / or 42.

[0050] The transmitted information may include coordinate information (e.g., two-dimensional coordinates in distance units corresponding to a specific location on display 44, two-dimensional coordinates defined relative to the length and width of the display, two-dimensional coordinates in pixels, etc.). Alternatively, head mounted device 10 may use the size and layout of user interface elements 54 (received from electronic devices 40 and / or 42) to determine which user interface element 54 overlaps the gaze point. In this case, the information transmitted from head mounted device 10 to electronic devices 40 and / or 42 may include the selected user interface element (rather than specific coordinate information).

[0051] like Figure 3B As shown in , when electronic device 40 and / or 42 receives information from the head mounted device indicating that gaze point 38 overlaps with user interface element 54-1 (aims at the user interface element), visual indicator 58 identifying user interface element 54-1 may be presented on display 44. Visual indicator 58 may be an outline that highlights user interface element 54-1 as a target user interface element among user interface elements 54-1, 54-2, and 54-3. Visual indicator 58 may be a complete outline surrounding the user interface element (e.g., Figure 3B Alternatively or in addition, the color of the selected user interface element can be changed (e.g., the user interface element can be highlighted), a preview video associated with the user interface element can be played, and / or the size of the selected user interface element can be increased.

[0052] like Figure 3C As shown, if gaze point 38 changes to instead overlap a different user interface element, such as user interface element 54 - 2 , this information is transmitted to electronic devices 40 and / or 42 and visual indicator 58 is moved to instead highlight target user interface element 54 - 2 .

[0053] The user may provide additional user input to confirm an action associated with the target user interface element (e.g., selecting or clicking the target user interface element). For example, the user interface element identified by the gaze input may be selected in response to the gaze point overlapping the user interface element for longer than a given dwell time and / or in response to additional user input (e.g., a hand gesture, a head gesture, a touch input, a button press, a voice command, etc.).

[0054] Figure 4 1 is a flowchart of an illustrative method for operating a head mounted device that uses gaze input to the head mounted device to control an external electronic device. First, at box 102, the head mounted device may receive user input associated with an intention to interact with an external electronic device. The user input received at box 102 may include gaze input detected by the gaze detection sensor 26. For example, the user may perform a gaze gesture, such as looking at a predetermined corner (sometimes referred to as a hot corner) of the display 18 on the head mounted device 10. At box 102, the user may provide a touch input to the touch sensor, such as a swipe or press. At box 102, the user may press a button such as button 32. The user may perform a hand gesture detected by the camera 22 at box 102. At box 102, the user may perform a head gesture detected by the position and motion sensor 24. At box 102, the user may provide a voice command detected by the microphone 30.

[0055] At block 104, based on obtaining the user input at block 102, the head mounted device 10 may transmit an instruction to the external electronic device to suppress the user input from its own input device and / or the accessory electronic device. For example, the head mounted device 10 may transmit an instruction to the external electronic device to suppress the user input from the remote control (e.g., Figure 2 This prevents a situation where both the head mounted device 10 and the remote control 48 provide conflicting user inputs to the external electronic device at the same time. In another example, the head mounted device 10 may transmit instructions to suppress inputs to the external electronic device provided via the touch screen of the external electronic device to prevent simultaneous conflicting user inputs to the external electronic device. Also at box 104, the head mounted device 10 may transmit information identifying the user input (e.g., identifying the user's intent to interact with the external electronic device) to the external electronic device. This may cause the external electronic device to display an optical symbol that is later detected by the head mounted device 10 at box 110.

[0056] At block 104, the example of the electronic device 40 and / or 42 suppressing input from the remote control is merely illustrative. In general, the electronic device 40 and / or 42 may suppress input at any of its input devices in response to receiving information from the head mounted device 10 at block 104.

[0057] At box 106, the head mounted device may obtain gaze input (e.g., using gaze tracking sensor 26). Alternatively or in addition, a head direction vector may be obtained at box 106 (e.g., using position and motion sensor 24). At box 108, the head mounted device may obtain an image including a display (e.g., using camera 22). Note that the sensor data obtained at boxes 106 and 108 may be obtained only in response to user input detected at box 102. In other words, the gaze detection sensor 26 and / or the position and motion sensor 24 may be turned on (or have the sampling rate increased) at box 106, and the camera 22 may be turned on (or have the sampling rate increased) at box 108.

[0058] The display may be a display that is part of an external electronic device (e.g., in an arrangement where electronic device 40 is omitted and electronic device 42 is a separate electronic device that communicates directly with head mounted device 10). Alternatively, the display may be an additional electronic device that has a wired connection to electronic device 40 (e.g., Figure 2 A part of the electronic device 42).

[0059] At block 110, the head mounted device 10 may use the image captured at block 108 to identify an optical symbology associated with the external electronic device. The optical symbology may be displayed in a known shape, size, and / or position on the display 44 of the electronic device 42. Alternatively, the optical symbology may be fixed to the housing 46 of the electronic device 42 in a known shape, size, and / or position.

[0060] Thus, at box 112, the head mounted device 10 may use the size, orientation, and / or position of the optical symbol within the captured image to determine the position of the display 44 relative to the head mounted device 10. The detected optical symbol may be obvious to the user (e.g., an icon intended to indicate to the user that positioning of the electronic device 42 is in progress) or may not be obvious to the user (e.g., hidden in a user interface on the display 44 so that the positioning process is not highlighted). When the optical symbol is obvious to the user, the optical symbol may be displayed only in the optical pairing mode. In other words, once the position of the electronic device is determined at box 112, the optical symbol may no longer be presented on the display 44. When the optical symbol is not obvious to the user, the optical symbol may be displayed in the optical pairing mode and after the optical pairing is completed. In other words, the optical symbol may continue to be presented on the display 44 even after the position of the electronic device is determined at box 112.

[0061] After determining the position of the external electronic device at box 112, one or more of the camera 22, the position and motion sensor 24, and the depth sensor 28 (for example, using simultaneous positioning and mapping technology) can be used to continuously track the position of the external electronic device. Alternatively, box 110 and box 112 can be repeatedly performed to continuously track the position of the external electronic device. In general, box 110 and box 112 can be performed at any desired frequency to ensure that the real-time position of the display 44 relative to the head-mounted device 10 is known. Even if SLAM technology is used, box 110 and box 112 can still be performed intermittently. If the position of the display 44 relative to the head-mounted device 10 is changing rapidly, box 110 and box 112 can be performed more frequently, and if the position of the display 44 relative to the head-mounted device 10 has not changed (or has hardly changed), box 110 and box 112 can be performed less frequently.

[0062] At block 114, the head mounted device 10 may determine a location on the display corresponding to the gaze input using at least the input from block 106 (e.g., gaze input and / or head direction vector) and the image from block 108. Ray tracing may be performed at block 114 to identify the gaze point 38 on the display 44.

[0063] At box 116, the head mounted device 10 may transmit information associated with the location of the gaze point on the display to an external electronic device, such as electronic device 40 and / or 42. The information transmitted at box 116 may include coordinate information (e.g., two-dimensional coordinates identifying a location on display 44) or may include a target user interface element. For example, the head mounted device 10 may receive information about the size / layout of user interface elements on display 44, and thus may directly determine which user interface element is being targeted. When the head mounted device 10 receives information about the size / layout of user interface elements on display 44, the head mounted device may optionally generate a transparent virtual object associated with the user interface element to utilize the head mounted device's ability to detect alignment of the gaze point with the virtual object.

[0064] Blocks 114 and 116 may be performed repeatedly such that gaze input is continuously used to provide user input to the display 44 .

[0065] exist Figure 4 At any time during the method, the user can provide a user input associated with the intention of no longer interacting with the external electronic device. In other words, the user can turn off the gaze control at any desired time.

[0066] Consider an example where electronic device 42 is a television with a wired connection to electronic device 40. Electronic device 40 is a source device that supplies images to television 42 using the wired connection. Electronic device 48 is a remote control that provides input to source device 40. At a first time, a user may use remote control 48 to provide input to source device 40 and control content on television 42.

[0067] At block 102, the head mounted device 10 receives user input associated with an intent to interact with a source device 40 and / or a television 42. The user input may be a gaze gesture, such as a user gazing at a corner of a display 18 of the head mounted device 10. At block 104, based on receiving the user input at block 102, the head mounted device transmits an instruction to suppress the user input from the remote control 48 to the source device 40. This allows the head mounted device 10 to provide input to the source device 40 and control content on the television 42 without input from the remote control 48 that results in conflicting instructions. Also at block 104, the head mounted device 10 may transmit information about the user input from block 102 to the source device 40. In response, the source device 40 may instruct the television 42 to display the optical symbology 60.

[0068] At box 106, the head mounted device 10 may use the gaze tracking sensor 26 to obtain gaze input. At box 108, the head mounted device 10 may use the camera 22 to obtain an image of the display 44 including the television 42. At box 110, the head mounted device 10 may use the image from box 108 to identify the optical symbology 60 displayed on the display 44. The head mounted device 10 may store information about the position, size, and / or shape of the optical symbology and / or may receive information identifying the position, size, and / or shape of the optical symbology from the source device 40. At box 112, the head mounted device 10 may use the position, size, and / or shape of the optical symbology detected within the image from box 108 to determine the position of the television 42 relative to the head mounted device 10.

[0069] At block 114, the head mounted device 10 may determine a location on the display (e.g., gaze point 38) that corresponds to the gaze input from the user using at least the gaze input from block 106 and the image from block 108. For example, ray tracing may be performed using the gaze input and the position of the television 42 determined at block 112 to determine the gaze point on the display 44.

[0070] At block 116, head mounted device 10 may transmit the target user interface element to source device 40. The target user interface element may optionally be determined using information received from source device 40. Alternatively, at block 116, head mounted device 10 may transmit coordinate information about the gaze point on display 44 to source device 40.

[0071] In response to receiving the information sent at block 116, source device 40 may control display 44 of television 42 to display visual indicator 58 identifying the user interface element being targeted. The user may provide additional user input, such as a button press, to select the target user interface element.

[0072] Consider an example in which the electronic device 42 is a tablet computer and the electronic device 40 and the remote control 48 are omitted from the system 8. At box 102, the head mounted device 10 receives a user input associated with an intention to interact with the tablet computer 42. The user input may be a gaze gesture, such as a user gazing at a corner of the display 18 of the head mounted device 10. At box 104, based on receiving the user input at box 102, the head mounted device transmits instructions to suppress at least some additional user inputs (e.g., from a touch sensor in the display 44 when the display 44 is a touch-sensitive display). This allows the head mounted device 10 to provide input to the tablet computer 42 and control content on the tablet computer 42 without other user inputs that result in conflicting instructions. Also at box 104, the head mounted device 10 may transmit information about the user input from box 102 to the tablet computer 42. In response, the tablet computer 42 may instruct the display 44 to display the optical symbol 60.

[0073] At box 106, the head mounted device 10 may use the gaze tracking sensor 26 to obtain gaze input. At box 108, the head mounted device 10 may use the camera 22 to obtain an image of the display 44 including the tablet computer 42. At box 110, the head mounted device 10 may use the image from box 108 to identify the optical symbology 60 displayed on the display 44. The head mounted device 10 may store information about the position, size, and / or shape of the optical symbology and / or may receive information identifying the position, size, and / or shape of the optical symbology from the tablet computer 42. At box 112, the head mounted device 10 may use the position, size, and / or shape of the optical symbology detected within the image from box 108 to determine the position of the tablet computer 42 relative to the head mounted device 10.

[0074] At block 114, the head mounted device 10 may determine a location on the display (e.g., gaze point 38) that corresponds to the gaze and / or head direction input from the user using at least the input from block 106 and the image from block 108. For example, ray tracing may be performed using the gaze input and the position of the tablet computer 42 determined at block 112 to determine the gaze point on the display 44.

[0075] At block 116, the head mounted device 10 may transmit the target user interface element to the tablet computer 42. The target user interface element may optionally be determined using information received from the tablet computer 42. Alternatively, at block 116, the head mounted device 10 may transmit coordinate information about the gaze point on the display 44 to the tablet computer 42.

[0076] In response to receiving the information sent at block 116 , tablet computer 42 may control display 44 to display visual indicator 58 identifying the user interface element being targeted.

[0077] The user may provide additional user input, such as a button press, to select the target user interface element. Tablet computer 42 may then perform the function associated with the target user interface element.

[0078] Figure 4 The example in which the head mounted device 10 transmits information associated with the location of the gaze point on the display to the external electronic device during the operation of block 116 is merely illustrative. Alternatively or in addition, the head mounted device 10 may transmit additional information to the external electronic device during the operation of block 116. For example, hand gesture information or other information based on data from one or more sensors in the head mounted device 10 may be transmitted to the external electronic device.

[0079] As a first example, the head mounted device 10 may transmit information associated with the location of a gaze point on a display of the external electronic device to an external electronic device. The head mounted device may then receive information from the external electronic device about content on the display that overlaps with the user's gaze point. For example, the external electronic device may report to the head mounted device that the user of the head mounted device is viewing a web browser or window for a particular application. The head mounted device may then transmit additional information, such as hand gesture information, to the external electronic device for controlling the content on the display. The hand gesture may be specific to the type of content being viewed.

[0080] Consider an example where a user is viewing a web browser. After receiving information from an external electronic device identifying that the user is viewing a web browser, the external electronic device may detect a hand gesture associated with control of the web browser (e.g., a hand gesture associated with scrolling) and report the hand gesture to the external electronic device. The external electronic device then adjusts the web browser (e.g., scrolls) based on the hand gesture. If a hand gesture associated with scrolling the web browser is detected while the user is not viewing the web browser, the hand gesture information may not be transmitted to the external electronic device.

[0081] Consider an example in which a user is viewing a window for a particular application, such as a computer-aided design application. After receiving information from an external electronic device identifying that the user is viewing a computer-aided design application, the external electronic device may detect a hand gesture associated with control of the application (e.g., a hand gesture associated with rotating a three-dimensional object) and report the hand gesture to the external electronic device. The external electronic device then adjusts the application based on the hand gesture (e.g., rotates the three-dimensional object being presented by the application). If a hand gesture associated with rotating a three-dimensional object in a computer-aided design application is detected while the user is not viewing the computer-aided design application, the hand gesture information may not be transmitted to the external electronic device.

[0082] The example of the external electronic device reporting information about the type of content on the display that overlaps with the user's gaze point is merely illustrative. Alternatively or in addition, the head mounted device 10 may identify the type of content that overlaps with the user's gaze point and transmit information associated with the location of the gaze point on the display and / or additional information such as hand gesture information to the external electronic device.

[0083] If desired, the user may perform a gesture or otherwise provide instructions associated with the mouse cursor of the external electronic device. For example, the user may perform a gesture or provide instructions to reposition the mouse cursor at the location of the user's gaze point on the display. This can enable the user to easily find the mouse cursor (by aligning the mouse cursor to a location that the user has viewed). The gesture can be a hand gesture, a head gesture, or other desired type of gesture. Alternatively or in addition, the user may provide a verbal instruction or a button press that aligns the mouse cursor on the display of the external electronic device with the user's gaze point. In addition to the location of the gaze point, the head mounted device 10 may also send instructions to align the mouse cursor with the user's gaze point.

[0084] As another example, a user can use their gaze point to drag and drop content on the display of an external electronic device. The user can perform a gesture or provide instructions that cause the content on the external electronic device to be pinned to the location of the user's gaze point when the user changes their gaze point, allowing the user to control the location of the content with their gaze point. Additional instructions provided to the head-mounted device can cause the content to become non-pinned and remain at its current location. If desired, this type of gaze-based drag-and-drop process can be used to move content between multiple displays (e.g., displays of different external electronic devices) (e.g., content can be moved from a first display on a first external electronic device to a second display on a second external electronic device).

[0085] Another option for using a head mounted device to control one or more other electronic devices is to use the head mounted device to direct input from an accessory device to the electronic device being actively viewed. Figure 5A and Figure 5B A system of this type is shown in .

[0086] like Figure 5A As shown, in addition to the head mounted device 10, the system 8 may also include electronic devices 62, 66, and 70. The electronic devices 10, 62, 66, and 70 may be associated with the same user (e.g., logged into the cloud service using the same user ID), may exchange wireless communications, etc. In general, each of the electronic devices 10, 62, 66, and 70 may be any desired type of electronic device (e.g., a cellular phone, a laptop computer, a speaker, a computer monitor, an electronic watch, a tablet computer, a head mounted device, a wireless keyboard, a wireless mouse, etc.).

[0087] Each of the electronic devices 62, 66, and 70 may include any desired input-output components (e.g., similar to the combination of Figure 1 Each of the electronic devices 62, 66 and 70 may optionally include communication circuitry (similar to the input-output circuitry described in the Figure 1 Communication circuit 56 in the system 8) to exchange wired and / or wireless communications with other devices in the system 8.

[0088] System 8 shows an example with a head mounted device 10, a first electronic device 62 with a corresponding display 64, a second electronic device 66 with a corresponding display 68, and a keyboard 70 with keys 72. Head mounted device 10 can be paired with each of electronic devices 62, 66, and 70.

[0089] In some cases, the user of system 8 may wish to use keyboard 70 to provide input to electronic device 62 instead of electronic device 66. In other cases, the user of system 8 may wish to use keyboard 70 to provide input to electronic device 66 instead of electronic device 62. The user may optionally wirelessly pair and unpair the wireless keyboard each time the user wants to switch whether electronic device 62 or electronic device 66 receives keyboard input. However, this process may be more time-consuming than desired.

[0090] To allow the user to seamlessly switch whether the wireless keyboard 70 provides input to the electronic device 62 or the electronic device 66, the head mounted device 10 may use information from the gaze tracking sensor 26. The gaze point 38 may be determined using the gaze tracking sensor (optionally in combination with images from the camera 22 and / or depth information from the depth sensor 28).

[0091] The head mounted device 10 is paired with the wireless keyboard 70 and receives a wireless transmission 74 representing a user input to the keyboard 70. When the gaze point 38 is aimed at the electronic device 62 (eg, Figure 5A ), the head mounted device 10 relays the user input on the keyboard 70 to the electronic device 62, such as Figure 5A When the gaze point 38 is aimed at the electronic device 66 (such as Figure 5B ), the head mounted device 10 relays the user input on the keyboard 70 to the electronic device 66, such as Figure 5B As represented by transmission 78 in .

[0092] like Figure 5A As shown, relaying keyboard input to electronic device 62 may cause new text input 80 to appear on display 64. Figure 5B As shown, relaying keyboard input to the electronic device 66 may cause new text input 80 to appear on the display 68 .

[0093] exist Figure 5A and Figure 5B In the example of , gaze input collected using head mounted device 10 is used to direct input to accessory device 70 to an appropriate external electronic device. This example is illustrative only.

[0094] Alternatively or in addition, data collected by position and motion sensor 24 can be used to direct input to accessory device 70 to the appropriate external electronic device. For example, head gesture information can be used to determine which external electronic device to relay the accessory input to.

[0095] Alternatively or in addition, data collected by microphone 30 can be used to direct input to the accessory device 70 to the appropriate external electronic device. For example, voice commands detected by microphone 30 can be used to determine which external electronic device to relay the accessory input to.

[0096] These examples are merely illustrative. In general, input from any desired input component (e.g., buttons, touch sensors, cameras, position and motion sensors, gaze tracking sensors, depth sensors, etc.) can be used by the head mounted device 10 to determine which external electronic device to relay the accessory input to.

[0097] Figure 5A and Figure 5BThe example of an accessory as a wireless keyboard is merely illustrative. In general, input to any desired accessory (e.g., a mouse, touchpad, keyboard, etc.) can be directed to the appropriate external electronic device using input to the head mounted device 10. In general, the head mounted device 10 can transmit any desired output to the appropriate external electronic device based on information from one or more input components (e.g., output transmitted to the identified external electronic device need not originate from the accessory device).

[0098] Figure 6 1 is a flow chart of an exemplary method for operating a head mounted device. First, at block 122, the head mounted device may receive user input information from an accessory device (e.g., using communication circuitry 56). The user input information may include information from keys in a keyboard (e.g., text that has been typed into the keyboard, sometimes referred to as keystroke information), mouse position and / or click information from a mouse or touch pad, and the like.

[0099] Next, at block 124, the head mounted device may obtain sensor data. The sensor data may include gaze input from the door tracking sensor 26, images from the camera 22, head position information from the position and motion sensor 24, depth information from the depth sensor 28, voice command information from the microphone 30, and / or button press information from the button 32. The sensor data obtained at block 124 may be used to determine whether an external electronic device (e.g., among a plurality of visible external electronic devices) is being viewed by a user of the head mounted device 10.

[0100] Note that the sensor data obtained at block 124 may be obtained only in response to receiving user input information at block 122. In other words, the gaze detection sensor 26 may be turned on (or have its sampling rate increased) at block 124, and / or the camera 22 may be turned on (or have its sampling rate increased) at block 124.

[0101] At block 126, the head mounted device 10 may transmit user input information to the external electronic device based on determining that the external electronic device is being viewed. The user input information may be the user input information received from the accessory device at block 122. The user input information may be transmitted using the communication circuit 56 (e.g., using Bluetooth communication). The external electronic device that receives the user input information may use the user input information to update content presented by the external electronic device (e.g., display text based on keyboard presses included in the user input information).

[0102] Note that determining that the external electronic device is being viewed may include determining that a display associated with the external electronic device is being viewed. The display may be integrated into the external electronic device (e.g., a laptop computer having a display in a laptop computer housing) or an external display coupled to the external electronic device (e.g., an external monitor coupled to a desktop computer or laptop computer).

[0103] While determining that a display associated with an external electronic device is being viewed (and / or prior to the operation of block 126), the type of the display and / or external electronic device may also be determined. The type of the display and / or external electronic device may be determined (using one or more sensors in the head mounted device) by determining the size of the display and / or external electronic device, by determining the shape of the display and / or external electronic device, and / or by analyzing other visual features of the display and / or external electronic device (e.g., bezel size, button placement, speaker placement, etc.).

[0104] At block 128, the head mounted device 10 may optionally determine a location on the display of the external electronic device that corresponds to the sensor data (e.g., a gaze point). The head mounted device may transmit the gaze point to the external electronic device. This information may be used, for example, to select a window from a plurality of windows operating on the external electronic device to receive user input information.

[0105] Sensor data may be continuously obtained at box 124 so that the head mounted device 10 can continuously assess which (if any) external electronic device is being viewed.

[0106] At box 130, the head mounted device 10 may transmit user input information to the second external electronic device based on determining that the second external electronic device is being viewed. The user input information may be the user input information received from the accessory device at box 122. The user input information may be transmitted using the communication circuit 56 (e.g., using Bluetooth communication). The second external electronic device that receives the user input information may use the user input information to update the content presented by the external electronic device (e.g., display text based on keyboard presses included in the user input information).

[0107] Also at box 130, the head mounted device 10 may forgo transmitting the user input information to the external electronic device (as in box 126) based on determining that the second external electronic device is being viewed.

[0108] Consider an example where a head mounted device is wirelessly paired with a laptop computer 62, a tablet computer 66, and a wireless keyboard 70. At block 122, the head mounted device may receive user input information (e.g., keystroke information) from the wireless keyboard 70. At block 124, the head mounted device may obtain sensor data, such as gaze detection data and camera data, to determine whether the laptop computer 62 or the tablet computer 66 is being viewed.

[0109] At block 126, based on determining that the laptop computer 62 is being viewed, the head mounted device 10 may wirelessly transmit the keystroke information to the laptop computer 62. The laptop computer 62 may then take a corresponding action (e.g., display text corresponding to the received keystroke information).

[0110] At block 128, the head mounted device 10 may determine a location on the display of the laptop computer that corresponds to the sensor data. The location may be wirelessly transmitted to the laptop computer 62. The location may be used by the laptop computer 62 to, for example, select a window from a plurality of windows to receive keystroke information.

[0111] At block 130, based on determining that the tablet computer 66 is being viewed, the head mounted device 10 may forgo wirelessly transmitting the keystroke information to the laptop computer 62 and may wirelessly transmit the keystroke information to the tablet computer 66. The tablet computer 66 may then take a corresponding action (e.g., display text corresponding to the received keystroke information).

[0112] Figure 6 The techniques for determining to route input to an accessory external electronic device in the embodiment of the present invention can be applied to any type of user input (i.e., not just accessory input). For example, at block 126, a user's voice command, text dictation, head gesture, hand gesture, or any other type of input detected using the head mounted device 10 can be transmitted to the external electronic device being viewed.

[0113] Another option for using a head mounted device to control one or more other electronic devices is to use images from a camera 22 in the head mounted device 10 to select a layout for multiple connected displays. Figure 7 A system of this type is shown in .

[0114] like Figure 7As shown, system 8 may include displays 82A, 82B, and 82C. These displays may all be associated with (and controlled by) an external electronic device (sometimes referred to as a central hub / controller). The electronic device 10 and the external electronic devices associated with displays 82A, 82B, and 82C may be associated with the same user (e.g., logged into a cloud service using the same user ID), wireless communications may be exchanged, etc. The external electronic device may be, for example, a laptop computer including one of the displays, a desktop computer connected to three displays, etc. The external electronic device may have a display configuration (sometimes referred to as a display arrangement) that sets how content is divided between displays. Generally speaking, it may be desirable that the display configuration used by the external electronic device matches the layout of the displays in the physical environment (because this will provide an intuitive user experience for the user).

[0115] Consider an example in which a user drags a window on display 82A to the right. Based on the layout of the displays in the physical environment, when dragging to the right from display 82A, the window should appear on display 82B (rather than, for example, on display 82C). Consider another example in which a user drags a window on the right edge of display 82A downward. Based on the layout of the displays in the physical environment, when dragging downward from display 82A, the window should appear on display 82C (rather than, for example, on display 82B).

[0116] The controllers of displays 82A, 82B, and 82C may have a setup option where a user can set the display arrangement to match the layout of displays 82A, 82B, and 82C in their physical environment. As an option, a user may manually drag and drop boxes representing displays to set the display arrangement used by an external electronic device (e.g., a laptop, desktop or other computer, etc.) using the displays.

[0117] Alternatively, the head mounted device may capture images of the display using the camera 22, and then use this information to automatically set the display arrangement used by the external electronic device. When a new display is connected to the external electronic device or when the user manually selects an option to update the display arrangement, the external electronic device may send a request to the paired head mounted device 10. The head mounted device 10 may use one or more images captured by the camera 22 and / or other sensor data (e.g., depth information) to determine the layout information of the display. The head mounted device 10 then transmits (e.g., wirelessly using Bluetooth communication) the layout information to the external electronic device (e.g., Figure 7 86 in the transmission).

[0118] The external electronic device can then update the display arrangement based on the layout information received from the head mounted device 10. Figure 7An arrangement is shown in which display 82A is used to display frames 84 representing displays in a layout determined by head mounted device 10. As shown, the layout of the frames in the display arrangement matches the layout of the displays in the physical environment.

[0119] Figure 8 An illustrative method for operating a head mounted device that provides display layout information to an external electronic device is shown. First, at block 132, the head mounted device 10 may receive a request from an external electronic device (e.g., a computer) having one or more associated displays. The request may be a display configuration request. The request may be received wirelessly using the communication circuit 56.

[0120] In response to receiving the request, the head mounted device may obtain an image of one or more associated displays at block 134. The image may be obtained by camera 22. If desired, additional sensor data from one or more sensors may be collected at block 134.

[0121] Note that the sensor data obtained at block 134 may be obtained only in response to receiving a request at block 132. In other words, at block 134, the camera 22 may be turned on (or have the sampling rate increased).

[0122] At block 136, the head mounted device 10 may use the image from block 134 to determine layout information for one or more associated displays. The layout information may be determined using semantic segmentation or other desired techniques. The layout information may include the size of each display, the orientation of each display, and the position of the displays relative to each other. The layout information may include display type information for one or more displays.

[0123] If desired, the external electronic device can display one or more glyphs or symbols on each associated display to ensure that the head-mounted device can correctly distinguish which displays are associated with the external electronic device and / or which display is which display in the layout.

[0124] Finally, at block 138, the head mounted device may transmit the layout information to the external electronic device. The layout information may be transmitted wirelessly using the communication circuit 56. The external electronic device may use the received layout information to update its display arrangement for presenting content.

[0125] Fig. 9 is a diagram of an illustrative display showing how to use a head mounted device to control content on the display. Fig. 9As shown, the electronic device 42 may include a display 44 coupled to a housing 46. During operation of the head mounted device 10, the electronic device 42 having the display 44 may be detected (e.g., using the camera 22). One way to use the head mounted device 10 to control the electronic device 42 is to use the gaze gesture detected by the head mounted device 10 to trigger a shortcut in the electronic device 42. The head mounted device 10 may track the position of the user's gaze relative to the electronic device 42 (as shown in at least FIG. 3 to FIG. 4 ). Figure 4 shown and discussed in ).

[0126] consider Fig. 9 42 . An example in which a user initially has a gaze point at position 202. During operation of the head mounted device 10 and / or the electronic device 42, the user may move their gaze point from position 202 to a new position 204-1 in the upper left corner of the electronic device 42. Viewing the upper left corner of the electronic device 42 may trigger a first shortcut in the electronic device 42. In another example, during operation of the head mounted device 10 and / or the electronic device 42, the user may move their gaze point from position 202 to a new position 204-2 in the upper right corner of the electronic device 42. Viewing the upper right corner of the electronic device 42 may trigger a different second shortcut in the electronic device 42. In another example, during operation of the head mounted device 10 and / or the electronic device 42, the user may move their gaze point from position 202 to a new position 204-3 in the lower left corner of the electronic device 42. Viewing the lower left corner of the electronic device 42 may trigger a different third shortcut in the electronic device 42. In another example, during operation of the head mounted device 10 and / or the electronic device 42, the user may move their gaze point from the position 202 to a new position 204-4 in the lower right corner of the electronic device 42. Viewing the lower right corner of the electronic device 42 may trigger a fourth, different shortcut in the electronic device 42. The user's gaze point may need to stay on one of the positions 204-1, 204-2, 204-3, and 204-4 for longer than a threshold dwell time in order to trigger the shortcut associated with that position.

[0127] The shortcuts triggered for the electronic device 42 using the gaze gesture may be customizable and may be configured using the head mounted device 10 and / or the electronic device 42. The shortcuts may be application specific or general. For example, the shortcut associated with the gaze position 204-1 may be a general shortcut for triggering a digital voice assistant. The shortcuts associated with the gaze positions 204-2, 204-3, and 204-4 may be application specific shortcuts depending on the application running on the electronic device 42. For example, when the electronic device 42 is running a video streaming application, the viewing position 204-2 may trigger the closed captions to be toggled between on and off, the viewing position 204-3 may trigger the video to rewind or skip backward, and the viewing position 204-4 may trigger the video to fast forward or skip forward. At a later time, when the electronic device 42 is running a photo editing application, the viewing position 204-2 may trigger the display of a specific toolbar, the viewing position 204-3 may trigger the save file, and the viewing position 204-4 may trigger the opening of the cropping function.

[0128] If necessary, view the shortcut location (for example, Fig. 9 4 (one of the corners identified in the ). may result in an output from the head mounted device 10 indicating that a shortcut is about to be triggered. For example, the display 18 in the head mounted device 10 may present a visual indicator (e.g., highlighting a corner of the electronic device 42 being looked at, presenting an icon at a corner of the electronic device 42 being looked at, etc.), thereby identifying that a shortcut is about to be triggered. If the user continues to look at the location for an additional dwell time, the corresponding shortcut will be triggered.

[0129] When the head-mounted device 10 detects a gaze gesture associated with triggering a shortcut on the electronic device 42, the head-mounted device 10 may transmit information identifying the requested shortcut and / or the location of the gaze gesture to the electronic device 42 (e.g., using Bluetooth communication) (so that the electronic device 42 can identify the requested shortcut).

[0130] Figure 4 , Figure 6 and Figure 8 The order of the blocks in the example is merely exemplary, and if desired, the blocks may be executed in a different order. Figure 4 , Figure 6 and Figure 8 Omit one or more boxes from the .

[0131] As described above, one aspect of the present technology is to collect and use information such as sensor information. The present disclosure contemplates that, in some cases, data including personal information that uniquely identifies or can be used to contact or locate a specific person may be collected. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, username, password, biometric information, or any other identifying information or personal information.

[0132] The present disclosure recognizes that the use of such personal information in the disclosed technology can be used to benefit the user. For example, the personal information data can be used to deliver targeted content that the user is more interested in. Therefore, the use of such personal information data enables the user to have control over the delivered content. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health status, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0133] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to safeguard and protect access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0134] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, the present technology may be configured to allow users to choose to "opt in" or "opt out" of participating in the collection of personal information data at any time during or after registration for a service. For another example, a user may choose not to provide a particular type of user data. For another example, a user may choose to limit the length of time that user-specific data is retained. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be informed that their personal information data will be accessed when downloading an application ("app"), and then be reminded again just before the personal information data is accessed by the app.

[0135] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or characteristics of stored data (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0136] Thus, while the present disclosure broadly covers the use of information that may include personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data.

[0137] According to one embodiment, an electronic device is provided, comprising one or more sensors; one or more processors; and a memory storing instructions configured to be executed by the one or more processors, the instructions being used to obtain gaze input via a first subset of the one or more sensors, obtain an image via a second subset of the one or more sensors, the image comprising a display, determine a position on the display corresponding to the gaze input using at least the gaze input and the image, and transmit information associated with the position on the display to an external electronic device.

[0138] According to another embodiment, the first subset of one or more sensors includes gaze detection sensors and the second subset of one or more sensors includes cameras.

[0139] According to another embodiment, the instructions include instructions for obtaining, via a third subset of one or more sensors, user input associated with an intent to interact with an external electronic device, obtaining gaze input includes obtaining gaze input in response to obtaining user input associated with an intent to interact with an external electronic device, and the user input includes a second gaze input, a touch input, a button press, a hand gesture, or a head gesture.

[0140] According to another embodiment, the instructions include instructions for the following operations: based on obtaining user input associated with an intention to interact with the external electronic device, transmitting to the external electronic device an instruction to suppress user input from the accessory electronic device; and based on obtaining user input associated with the intention to interact with the external electronic device, transmitting to the external electronic device information associated with the user input.

[0141] According to another embodiment, the instructions include instructions for using an image to identify an optical symbol associated with an external electronic device, identifying the optical symbol includes identifying the optical symbol on a display of the external electronic device, and the optical symbol is displayed on the display of the external electronic device in a pairing mode.

[0142] According to another embodiment, the instructions include instructions for: determining the position of the external electronic device relative to the electronic device using optical symbols; and after determining the position of the external electronic device relative to the electronic device using the optical symbols, tracking the position of the external electronic device relative to the electronic device using at least a second subset of one or more sensors.

[0143] According to another embodiment, the information associated with the location on the display includes coordinate information.

[0144] According to another embodiment, the instructions include instructions for receiving user interface information from an external electronic device, the user interface information identifying one or more user interface elements displayed by the external electronic device, and the information associated with the location on the display includes a selected user interface element from the one or more user interface elements.

[0145] According to another embodiment, the instructions include instructions for transmitting the hand gesture information to an external electronic device based on the content type information.

[0146] According to one embodiment, a method of operating an electronic device including one or more sensors is provided, the method including obtaining gaze input via a first subset of the one or more sensors; obtaining an image via a second subset of the one or more sensors, the image including a display; determining a position on the display corresponding to the gaze input using at least the gaze input and the image; and transmitting information associated with the position on the display to an external electronic device.

[0147] According to another embodiment, the first subset of one or more sensors includes gaze detection sensors and the second subset of one or more sensors includes cameras.

[0148] According to another embodiment, the method includes obtaining, via a third subset of one or more sensors, user input associated with an intent to interact with an external electronic device, obtaining gaze input includes obtaining gaze input in response to obtaining user input associated with an intent to interact with an external electronic device, and the user input includes a second gaze input, a touch input, a button press, a hand gesture, or a head gesture.

[0149] According to another embodiment, the method includes transmitting an instruction to suppress user input from an accessory electronic device to an external electronic device based on obtaining user input associated with an intention to interact with the external electronic device; and transmitting information associated with the user input to the external electronic device based on obtaining user input associated with an intention to interact with the external electronic device.

[0150] According to another embodiment, the method includes using the image to identify an optical symbol associated with the external electronic device, identifying the optical symbol includes identifying the optical symbol on a display of the external electronic device, and the optical symbol is displayed on the display of the external electronic device in a pairing mode.

[0151] According to another embodiment, the method includes using optical symbols to determine the position of the external electronic device relative to the electronic device; and after using the optical symbols to determine the position of the external electronic device relative to the electronic device, using at least a second subset of one or more sensors to track the position of the external electronic device relative to the electronic device.

[0152] According to another embodiment, the information associated with the location on the display includes coordinate information.

[0153] According to another embodiment, the method includes receiving user interface information from an external electronic device, the user interface information identifying one or more user interface elements displayed by the external electronic device, and the information associated with the position on the display includes a selected user interface element from the one or more user interface elements.

[0154] According to another embodiment, the method includes transmitting the hand gesture information to an external electronic device based on the content type information.

[0155] According to one embodiment, a non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of an electronic device including one or more sensors, the one or more programs including instructions for the following operations: obtaining gaze input via a first subset of the one or more sensors; obtaining an image via a second subset of the one or more sensors, the image including a display; determining a location on the display corresponding to the gaze input using at least the gaze input and the image; and transmitting information associated with the location on the display to an external electronic device.

[0156] According to another embodiment, the first subset of one or more sensors includes gaze detection sensors and the second subset of one or more sensors includes cameras.

[0157] According to another embodiment, the instructions include instructions for obtaining, via a third subset of one or more sensors, user input associated with an intent to interact with an external electronic device, obtaining gaze input includes obtaining gaze input in response to obtaining user input associated with an intent to interact with an external electronic device, and the user input includes a second gaze input, a touch input, a button press, a hand gesture, or a head gesture.

[0158] According to another embodiment, the instructions include instructions for the following operations: based on obtaining user input associated with an intention to interact with the external electronic device, transmitting to the external electronic device an instruction to suppress user input from the accessory electronic device; and based on obtaining user input associated with the intention to interact with the external electronic device, transmitting to the external electronic device information associated with the user input.

[0159] According to another embodiment, the instructions include instructions for using an image to identify an optical symbol associated with an external electronic device, identifying the optical symbol includes identifying the optical symbol on a display of the external electronic device, and the optical symbol is displayed on the display of the external electronic device in a pairing mode.

[0160] According to another embodiment, the instructions include instructions for: determining the position of the external electronic device relative to the electronic device using optical symbols; and after determining the position of the external electronic device relative to the electronic device using the optical symbols, tracking the position of the external electronic device relative to the electronic device using at least a second subset of one or more sensors.

[0161] According to another embodiment, the information associated with the location on the display includes coordinate information.

[0162] According to another embodiment, the instructions include instructions for receiving user interface information from an external electronic device, the user interface information identifying one or more user interface elements displayed by the external electronic device, and the information associated with the location on the display includes a selected user interface element from the one or more user interface elements.

[0163] According to another embodiment, the instructions include instructions for transmitting the hand gesture information to an external electronic device based on the content type information.

[0164] The foregoing is merely illustrative and various modifications may be made to the embodiments described. The foregoing embodiments may be implemented independently or in any combination.

Claims

1. An electronic device, comprising: one or more sensors; one or more processors; and a memory storing instructions configured to be executed by the one or more processors, the instructions being used to: obtaining gaze input via a first subset of the one or more sensors; obtaining an image via a second subset of the one or more sensors, wherein the image includes a display; determining a location on the display corresponding to the gaze input using at least the gaze input and the image; as well as Information associated with the location on the display is transmitted to an external electronic device. 2 . The electronic device of claim 1 , wherein the first subset of the one or more sensors comprises a gaze detection sensor, and wherein the second subset of the one or more sensors comprises a camera.

3. The electronic device according to claim 1, wherein the instructions further comprise instructions for: Obtaining, via a third subset of the one or more sensors, user input associated with an intent to interact with the external electronic device, wherein obtaining the gaze input comprises obtaining the gaze input in response to obtaining the user input associated with the intent to interact with the external electronic device, and wherein the user input comprises a second gaze input, a touch input, a button press, a hand gesture, or a head gesture.

4. The electronic device according to claim 3, wherein the instructions further comprise instructions for performing the following operations: transmitting, in response to obtaining the user input associated with the intent to interact with the external electronic device, an instruction to suppress user input from an accessory electronic device to the external electronic device; and In accordance with obtaining the user input associated with the intent to interact with the external electronic device, information associated with the user input is transmitted to the external electronic device.

5. The electronic device according to claim 1, wherein the instructions further comprise instructions for: An optical symbology associated with the external electronic device is identified using the image, wherein identifying the optical symbology includes identifying the optical symbology on the display of the external electronic device, and wherein the optical symbology is displayed on the display of the external electronic device in a pairing mode.

6. The electronic device according to claim 5, wherein the instructions further comprise instructions for performing the following operations: determining a position of the external electronic device relative to the electronic device using the optical symbology; and After determining the position of the external electronic device relative to the electronic device using the optical symbology, tracking the position of the external electronic device relative to the electronic device using at least the second subset of the one or more sensors.

7. The electronic device of claim 1, wherein the information associated with the location on the display comprises coordinate information.

8. The electronic device according to claim 1, wherein the instructions further comprise instructions for: User interface information is received from the external electronic device, wherein the user interface information identifies one or more user interface elements displayed by the external electronic device, and wherein the information associated with the position on the display includes a selected user interface element from the one or more user interface elements.

9. The electronic device according to claim 1, wherein the instructions further comprise instructions for: The hand gesture information is transmitted to the external electronic device based on the content type information.

10. A method of operating an electronic device comprising one or more sensors, wherein the method comprises: obtaining gaze input via a first subset of the one or more sensors; obtaining an image via a second subset of the one or more sensors, wherein the image includes a display; determining a location on the display corresponding to the gaze input using at least the gaze input and the image; as well as Information associated with the location on the display is transmitted to an external electronic device. 11 . The method of claim 10 , wherein the first subset of the one or more sensors comprises a gaze detection sensor and the second subset of the one or more sensors comprises a camera.

12. The method according to claim 10, further comprising: Obtaining, via a third subset of the one or more sensors, user input associated with an intent to interact with the external electronic device, wherein obtaining the gaze input comprises obtaining the gaze input in response to obtaining the user input associated with the intent to interact with the external electronic device, and wherein the user input comprises a second gaze input, a touch input, a button press, a hand gesture, or a head gesture.

13. The method according to claim 12, further comprising: transmitting, in response to obtaining the user input associated with the intent to interact with the external electronic device, to the external electronic device an instruction to suppress user input from an accessory electronic device; as well as In accordance with obtaining the user input associated with the intent to interact with the external electronic device, information associated with the user input is transmitted to the external electronic device.

14. The method according to claim 10, further comprising: An optical symbology associated with the external electronic device is identified using the image, wherein identifying the optical symbology includes identifying the optical symbology on the display of the external electronic device, and wherein the optical symbology is displayed on the display of the external electronic device in a pairing mode.

15. The method according to claim 14, further comprising: determining a position of the external electronic device relative to the electronic device using the optical symbology; as well as After determining the position of the external electronic device relative to the electronic device using the optical symbology, tracking the position of the external electronic device relative to the electronic device using at least the second subset of the one or more sensors.

16. The method of claim 10, wherein the information associated with the location on the display comprises coordinate information.

17. The method according to claim 10, further comprising: User interface information is received from the external electronic device, wherein the user interface information identifies one or more user interface elements displayed by the external electronic device, and wherein the information associated with the position on the display includes a selected user interface element from the one or more user interface elements.

18. The method according to claim 10, further comprising: The hand gesture information is transmitted to the external electronic device based on the content type information.

19. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device including one or more sensors, wherein the one or more programs include instructions for: obtaining gaze input via a first subset of the one or more sensors; obtaining an image via a second subset of the one or more sensors, wherein the image includes a display; determining a location on the display corresponding to the gaze input using at least the gaze input and the image; as well as Information associated with the location on the display is transmitted to an external electronic device.

20. The non-transitory computer-readable storage medium of claim 19, wherein the first subset of the one or more sensors comprises a gaze detection sensor and the second subset of the one or more sensors comprises a camera.