SYSTEMS AND METHODS FOR DYNAMIC INPUT BEHAVIOR

Abstract

Some examples of the disclosure relate to systems and methods for dynamic input behavior. In some examples, an electronic device can automatically adjust an input detected by an intelligent input device to fulfill the user's intent, even if the input is already assigned to another action. In some examples, scroll input can be dampened based on gaze direction. For example, the electronic device can reduce the scrolling speed of a surface element when it detects that the user's gaze is searching for a particular element. In some examples, scroll input can perform different actions depending on the speed of the scroll input. In some examples, the electronic device can display context-sensitive hints about actions that can be performed when gaze is detected toward the hints.The displays and corresponding processes may change depending on the context.

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] This application claims priority over U.S. Preliminary Patent Application No. 63 / 700,172, filed on September 27, 2024, and U.S. Patent Application No. 19 / 326,464, filed on September 11, 2025, the entire disclosure of which is incorporated herein by reference for all purposes. AREA OF REVELATION

[0002] This refers generally to systems and methods for operating an electronic device and in particular to context-controlled input behavior in an electronic device. BACKGROUND OF THE REVELATION

[0003] Some computer graphics environments provide two-dimensional and / or three-dimensional environments in which at least some objects displayed to a user are virtual and generated by a computer. In some examples, a head-worn device is adapted to perform operations based on context-controlled user input. SUMMARY OF THE REVELATION

[0004] Some examples of the disclosure relate to systems and methods for dynamic input behavior. In some examples, the system of the present disclosure may include an electronic device (e.g., a head-worn device) with an intelligent input device. In some examples, an input detected by the intelligent input device may perform different actions based on the determined intention of the input. For example, the electronic device may automatically adjust an input detected by the intelligent input device (e.g., a gesture) to fulfill the user's intention, even if the input is already assigned to another action. In some examples, a scrolling input (e.g., a swipe gesture) may be dampened based on the direction of gaze.For example, the electronic device can reduce the scrolling speed of a surface element when it detects that the user's gaze is searching for a specific element. In some examples, a scroll input can perform different actions depending on the scrolling speed. For instance, a scroll input that scrolls a surface element at a slower speed can stop displaying the element when performed at a faster speed. In some examples, the electronic device can display context-sensitive hints about actions that can be performed when the user's gaze is detected. The displays and corresponding actions can change depending on the context.

[0005] The complete descriptions of these examples are included in the drawings and the detailed description, and it is understood that this summary does not in any way limit the scope of the revelation. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] For a better understanding of the various examples described herein, please refer to the detailed description below, along with the accompanying drawings. The same reference numerals throughout the drawings often refer to corresponding parts. Fig. Figure 1 illustrates an electronic device that represents an augmented reality environment, according to some examples from the revelation. Fig. Sections 2A-2B illustrate block diagrams of exemplary architectures for electronic devices according to some examples of the disclosure. Fig. 3A-3C illustrate an example of an electronic device with an intelligent input device according to examples in the disclosure. Fig. Figures 4A-4B illustrate an example of an electronic device with attention-based scroll stabilization according to examples in the disclosure. Fig. Figures 5A-5C illustrate an example of an electronic device with speed-based swipe detection according to examples in the disclosure. Fig. Figures 6A-6D illustrate an example of an electronic device with context-controlled active display areas according to examples in the disclosure. Fig. Figures 7A-7D illustrate an example of an electronic device with context-controlled active display areas according to examples in the disclosure. Fig. Figures 8A-8D illustrate an example of an electronic device with context-controlled active display areas according to examples in the disclosure. Fig. Figures 9A-9D illustrate an example of an electronic device with context-controlled active display areas according to examples in the disclosure. Fig. Figures 10A-10D illustrate an example of an electronic device with context-controlled active display areas according to examples in the disclosure. Fig. 11 illustrates an exemplary flowchart of a process 1100 according to an example from the revelation Fig. 12 illustrates an exemplary flowchart of a process 1200 according to an example from the revelation Fig. 13 illustrates an exemplary flowchart of a process 1300 according to an example from the revelation Fig. 14 illustrates an exemplary flowchart of a process 1400 according to an example from the revelation DETAILED DESCRIPTION

[0007] Some examples of the disclosure relate to systems and methods for dynamic input behavior. In some examples, the system of the present disclosure may include an electronic device (e.g., a head-worn device) with an intelligent input device. In some examples, an input detected by the intelligent input device may perform different actions based on the determined intention of the input. For example, the electronic device may automatically adjust an input detected by the intelligent input device (e.g., a gesture) to fulfill the user's intention, even if the input is already assigned to another action. In some examples, a scrolling input (e.g., a swipe gesture) may be dampened based on the direction of gaze.For example, the electronic device can reduce the scrolling speed of a surface element when it detects that the user's gaze is searching for a specific element. In some examples, a scroll input can perform different actions depending on the scrolling speed. For instance, a scroll input that scrolls a surface element at a slower speed can stop displaying the element when performed at a faster speed. In some examples, the electronic device can display context-sensitive hints about actions that can be performed when the user's gaze is detected. The displays and corresponding actions can change depending on the context.

[0008] Fig. Figure 1 illustrates an electronic device 101 that presents a three-dimensional environment (e.g., an augmented reality (XR) environment or a computer-generated reality (CGR) environment, optionally including representations of physical and / or virtual objects) according to some examples of the disclosure. In some examples, as in Fig. As shown in Figure 1, the electronic device 101 is a head-mounted display or other head-mounted device configured to be worn on the head of a user of the electronic device 101. Examples of the electronic device 101 are given below with reference to the architecture of the block diagram from Figure 1. Fig. 2A described. As in Fig. As shown in Figure 1, the electronic device 101 and the table 106 are located in a physical environment. The physical environment can include physical features such as a physical surface (e.g., floor, walls) or a physical object (e.g., table, lamp, etc.). In some embodiments, the electronic device 101 can be configured to detect and / or capture images of the physical environment, including the table 106 (illustrated in the field of view of the electronic device 101).

[0009] In some examples, such as in Fig. As shown in Figure 1, the electronic device 101 includes one or more internal image sensors 114a oriented towards the user's face (e.g., eye-tracking cameras, which are described below in relation to Fig. (described in sections 2A-2B). In some examples, internal image sensors 114a are used for eye tracking (e.g., to detect the user's gaze). Internal image sensors 114a are optionally arranged in the left and right areas of the display 120 to enable eye tracking of the user's left and right eyes. In some examples, the electronic device 101 also includes external image sensors 114b and 114c, which are directed outwards from the user, to detect and / or capture the physical environment of the electronic device 101 and / or movements of the user's hands or other body parts.

[0010] In some examples, the Display 120 has a user-visible field of view. In some examples, the user-visible field of view is the same as the field of view of the external image sensors 114b and 114c. Because the Display 120 is optionally part of a head-mounted device, its field of view is optionally the same as, or similar to, the user's eye field of view. In some examples, the user-visible field of view differs from the field of view of the external image sensors 114b and 114c (e.g., narrower than the field of view of the external image sensors 114b and 114c). In other examples, the field of view of the Display 120 may be smaller than the user's eye field of view. A user's viewing angle determines what content is visible in the field of view. A viewing angle generally specifies a location and direction relative to the three-dimensional environment.When a user's viewing angle changes, the field of view of the three-dimensional environment also changes accordingly. In some examples, the electronic device 101 can be an optically transparent device, where the display 120 is a transparent or translucent display through which parts of the physical environment can be viewed directly. In some examples, the display 120 can be contained within a transparent lens and overlap all or part of the transparent lens. In other examples, the electronic device can be a video passthrough device, where the display 120 is an opaque display configured to show images of the physical environment captured by the external image sensors 114b and 114c. While in . Fig. 1. Where a single display is shown, it is understood that display 120 optionally includes more than one display. For example, display 120 optionally includes a stereo display pair (e.g., left and right display fields for the user's left and right eyes, respectively) with displayed outputs that are merged (e.g., by the user's brain) to represent the information in Fig. To generate the view of the content shown in point 1. In some examples, as below with reference to Fig. As explained in more detail in 2A - 2B, display 120 includes or corresponds to a transparent or translucent surface (e.g. a lens) that is not equipped with display functions (and therefore, for example, is unable to generate and display the virtual object 104) and alternatively represents a direct view of the physical environment in the user's field of vision (e.g. the user's eye field of vision).

[0011] In some examples, the electronic device 101 is configured to display a virtual object 104 in the three-dimensional environment (e.g., in response to a trigger). The virtual object 104 is displayed by a Fig. Figure 1 illustrates a cube that is not present in the physical environment but is displayed in the three-dimensional environment, which is positioned on the top of the table 106 (e.g., a real table or a representation thereof). Optionally, the virtual object 104 is displayed on the surface of the table 106 in the three-dimensional environment via the display 120 of the electronic device 101 in response to the detection of the flat surface of the table 106 in the physical environment 100.

[0012] It is understood that virtual object 104 is a representative virtual object, and one or more different virtual objects (e.g., of different dimensions, such as two-dimensional or other three-dimensional virtual objects) can be enclosed and rendered in a three-dimensional environment. For example, the virtual object can represent an application or a user interface displayed in the three-dimensional environment. In some examples, the virtual object can represent content corresponding to the application and / or displayed via the user interface in the three-dimensional environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g.,Air gestures, such as pinch air gestures, tap air gestures and / or touch air gestures), so that a user can virtually touch, tap, move, rotate or otherwise interact with the virtual object 104.

[0013] As discussed herein, one or more air pinch gestures performed by a user (e.g., with hand 103 in Fig. 1) is detected by one or more input devices of the electronic device 101 and interpreted as one or more user inputs directed at content displayed by the electronic device 101. Additionally or alternatively, in some examples, the one or more user inputs interpreted by the electronic device 101 as directed at content displayed by the electronic device 101 (e.g., the virtual object 104) are detected via one or more hardware input devices (e.g., controllers, touchpads, proximity sensors, keys, sliders, buttons, etc.) and not via the one or more input devices configured to detect air gestures, such as the one or more air pinch gestures performed by the user. Such a description is intended to be illustrative rather than limiting.The user can optionally provide user input using various air gestures and / or other input methods.

[0014] In some examples, the electronic device 101 can be configured to communicate with a second electronic device, such as a companion device. For example, the electronic device 101, as in Fig. Figure 1 illustrates, optionally in conjunction with the electronic device 160. In some examples, the electronic device 160 corresponds to a mobile electronic device, such as a smartphone, tablet computer, smartwatch, laptop computer, or other electronic device. In other examples, the electronic device 160 corresponds to a non-mobile electronic device that is generally stationary and cannot be easily moved within the physical environment (e.g., desktop computer, server, etc.). Additional examples of the electronic device 160 are given below with reference to the architecture of the block diagram from Fig. 2B described. In some examples, electronic device 101 and electronic device 160 are assigned to the same user. For example, in Fig. 1. The electronic device 101 may be positioned on a user's head (e.g., attached to it), and the electronic device 160 may be positioned near the electronic device 101, such as in the user's hand 103 (e.g., the hand 103 holds the electronic device 160), in the user's pocket or bag, or on a surface near the user. The electronic device 101 and the electronic device 160 are optionally linked to the same user account (e.g., the user is logged into the user account on both the electronic device 101 and the electronic device 160). Further details regarding communication between the electronic device 101 and the electronic device 160 are described below with reference to the Fig. 2A-2B provided.

[0015] In some examples, displaying an object in a three-dimensional environment can enable or trigger interaction with one or more user interface objects within that environment. For instance, initiating the display of the object in the three-dimensional environment might involve interaction with one or more virtual options / affordances displayed within that environment. In some examples, when the display of an object in the three-dimensional environment is initiated, a user's gaze can be tracked as input to identify one or more virtual options / affordances available for selection. The gaze might, for example, be used to identify one or more virtual options / offers to be selected using a further selection input.In some examples, a virtual option or offer can be selected using hand tracking input, which is captured via an input device communicating with the electronic device. In some examples, objects displayed in the three-dimensional environment can be moved and / or reoriented in the three-dimensional environment in accordance with the motion input captured by the device.

[0016] The following explanation describes an electronic device that communicates with one or more displays and one or more input devices. It is understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. Furthermore, as described above, it is understood that the described electronic device, the display, and the touch-sensitive surface are optionally distributed among two or more devices.Therefore, as used in this disclosure, information displayed on or by the electronic device is optionally used to describe information output by the electronic device for display on a separate display device (touch-sensitive or not). Similarly, as used in this disclosure, input received by the electronic device (e.g., touch input received on a touch-sensitive surface of the electronic device or touch input received on the surface of an input pen) is optionally used to describe input received on a separate input device from which the electronic device receives input information.

[0017] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disc authoring software development application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video camera application, an internet browsing application, a digital music playback application, a television channel browsing application, and / or a digital video playback application.

[0018] Fig. Figures 2A-2B illustrate block diagrams of exemplary architectures for electronic devices according to some examples of the disclosure. In some examples, the electronic device 201 and / or the electronic device 260 include one or more electronic devices. For example, the electronic device 201 may be a portable device, an auxiliary device communicating with another device, a head-mounted display, a head-worn loudspeaker, etc. In some examples, the electronic device 201 corresponds to the one described above with reference to Fig. 1 electronic device 101 described above. In some examples, the electronic device 260 corresponds to the one described above with reference to Fig. 1 described electronic device 160.

[0019] As in Fig. As illustrated in Figure 2, the electronic device 201 optionally includes one or more sensors, such as one or more hand tracking sensors 202, one or more location sensors 204A, one or more image sensors 206A (optionally the internal image sensors 114a and / or external image sensors 114b and 114c in Fig. 1 accordingly), one or more touch-sensitive surfaces 209A, one or more motion and / or orientation sensors 210A, one or more eye-tracking sensors 212, one or more microphones 213A or other audio sensors, one or more body-tracking sensors (e.g., torso and / or head-tracking sensors), etc. The electronic device 201 optionally includes one or more output devices, such as one or more display-generating components 214A, which may optionally be connected to the display 120. Fig. 1, one or more loudspeakers 216A, one or more haptic output devices (not shown), etc. The electronic device 201 optionally includes one or more processors 218A, one or more memories 220A, and / or a communication switching logic 222A. One or more communication buses 208A are optionally used for communication between the aforementioned components of the electronic device 201.

[0020] Furthermore, the electronic device 260 optionally includes the same or similar components as the electronic device 201. As in Fig. As shown in Figure 2B, the electronic device 260 optionally includes, for example, one or more location sensors 204B, one or more image sensors 206B, one or more touch-sensitive surfaces 209B, one or more orientation sensors 210B, one or more microphones 213B, one or more display generation components 214B, one or more loudspeakers 216B, one or more processors 218B, one or more memories 220B, and / or the communication switching logic 222B. One or more communication buses 208B are optionally used for communication between the aforementioned components of the electronic device 260.

[0021] Electronic devices 201 and 260 are optionally configured to communicate between the two electronic devices via a wired or wireless connection (e.g., via the communication switching logic 222A, 222B). For example, electronic device 260 can, as shown in Fig. 2A, as specified, to function as an accompanying device to the electronic device 201. In some examples, the electronic device 260, for instance, processes sensor inputs from the electronic devices 201 and 260 and / or generates content for display using the display generation components 214A of the electronic device 201.

[0022] The 222A and 222B communication switching logic optionally includes switching logic for communicating with electronic devices, networks such as the Internet, intranets, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs). The 222A and 222B communication switching logic optionally includes switching logic for communication via near-field communication (NFC) and / or short-range communication, such as Bluetooth®, etc. In some examples, the 222A and 222B communication switching logic includes or supports Wi-Fi (e.g., an 802.11 protocol), Ethernet, ultra-wideband (UWB), radio frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), or any other communication protocol or combination thereof.

[0023] The one or more processors 218A, 218B include one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, the one or more processors 218A, 218B include one or more microprocessors, one or more central processing units, one or more application-specific integrated circuits, one or more field-programmable gate arrays, one or more programmable logic devices, or a combination of such devices. In some examples, the memories 220A and / or 220B are a non-transient, computer-readable storage medium (e.g.,Flash memory, random-access memory, or other volatile or non-volatile working memory or storage) that stores computer-readable instructions configured to be executed by one or more 218A, 218B processors to perform the techniques, processes, and / or procedures described below. In some examples, the 220A and / or 220B memory may include more than one non-transient, computer-readable storage medium. A non-transient, computer-readable storage medium may be any medium (except, for example, a signal) that can tangiblely contain or store computer-executable instructions for use by, or in connection with, the instruction execution system, instruction execution device, or instruction execution device. In some examples, the storage medium is a transient, computer-readable storage medium.In some examples, the storage medium is a non-transient, computer-readable storage medium. Non-transient, computer-readable storage mediums can include, but are not limited to, magnetic, optical, and / or semiconductor storage. Examples of such storage include magnetic disks, optical disks based on Compact Disc (CD), Digital Versatile Disc (DVD), or Blu-ray technologies, and persistent semiconductor storage such as flash memory, solid-state drives, and the like.

[0024] In some examples, one or more display generating components 214A, 214B include a single display (e.g., a liquid crystal display (LCD), organic light-emitting diode (OLED), or other types of displays). In some examples, the one or more display generating components 214A, 214B include multiple displays. In some examples, the one or more display generating components 214A, 214B may include a touch-enabled display (e.g., a touchscreen), a projector, a holographic projector, a retina projector, a transparent or translucent display, etc. In some examples, the electronic device includes no display generating components or multiple display generating components 214A or 214B.For example, some electronic devices include transparent or translucent lenses or other surfaces not configured to display or present virtual content, instead of one or more display generating components 214A or 214B. It is understood, however, that in such cases, the electronic device 201 and / or the electronic device 260 may optionally be combined with one or more of the other components described in the following. Fig. 2A and Fig. The electronic devices 201 and / or 260 are equipped with components illustrated in Figure 2B and described herein, such as the one or more hand tracking sensors 202, the one or more eye tracking sensors 212, the one or more image sensors 206A, and / or the one or more motion and / or orientation sensors 210A. Alternatively, in some examples, the one or more display generating components 214A or 214B are provided separately from the electronic devices 201 and / or 260. For example, the one or more display generating components 214A, 214B are connected to the electronic device 201 (and / or the electronic device 260) but are not integrated into the electronic device 201 and / or the electronic device 260 (e.g., within an enclosure of the electronic devices 201, 260).In some examples, the electronic devices 201 and 260 include one or more touch-sensitive surfaces 209A and 209B, respectively, for receiving user input, such as taps, swipes, or other gestures (e.g., hand- or finger-based gestures). In some examples, the one or more display-generating components 214A, 214B, and the one or more touch-sensitive surfaces 209A, 209B form one or more touch-sensitive displays (e.g., a touchscreen that is integrated into each of the electronic devices 201 and 260 or is located outside of each of the electronic devices 201 and 260 and communicates with each of the electronic devices 201 and 260).

[0025] The electronic devices 201 and 260 optionally include one or more image sensors 206A and 206B, respectively. The one or more image sensors 206A and 206B optionally include one or more visible light image sensors, such as CCD (charge-coupled device) sensors and / or CMOS (complementary metal-oxide semiconductor) sensors, which can be operated to obtain images of physical objects from the real environment. The one or more image sensors 206A and 206B also optionally include one or more infrared (IR) sensors, such as a passive or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real environment.The one or more image sensors 206A, 206B optionally include one or more cameras configured to detect movement of physical objects in the real-world environment. The one or more image sensors 206A, 206B optionally also include one or more depth sensors configured to detect the distance of physical objects from the electronic device 201, 260. In some examples, information from one or more depth sensors can enable the device to identify objects in the real-world environment and distinguish them from other objects in the real-world environment. In some examples, one or more depth sensors can enable the device to determine the texture and / or topography of objects in the real-world environment.In some examples, the one or more image sensors 206A or 206B are contained in an electronic device other than the electronic devices 201 and / or 260. For example, the one or more image sensors 206A, 206B are connected to the electronic device 201, 260, but are not integrated into the electronic device 201, 260 (e.g., within a housing of the electronic device 201, 260). In particular, in some examples, the one or more cameras of the one or more image sensors 206A, 206B are integrated into and / or coupled to one or more devices separate from the electronic devices 201 and / or 260 (but are, for example, connected to the electronic devices 201 and / or 260), such as one or more input and / or output devices (e.g.,one or more loudspeakers and / or one or more microphones (such as earphones or headphones) enclosing one or more image sensors 206A, 206B. In some examples, the electronic device 201 or the electronic device 260 corresponds to a head-worn loudspeaker (e.g., headphones or earphones). In such cases, the electronic device 201 or the electronic device 260 is connected to a subset of the others in the . Fig. 2A and Fig. 2B illustrated and described herein are equipped with the components. In some of these examples, the electronic device 201 or the electronic device 260 is equipped with one or more image sensors 206A, 206B, one or more motion and / or orientation sensors 210A, 210B and / or loudspeakers 216A, 216B.

[0026] In some examples, the electronic device 201, 260 uses CCD sensors, event cameras, and depth sensors in combination to detect the physical environment around the electronic device 201, 260. In some examples, the one or more image sensors 206A, 206B include a first image sensor and a second image sensor. The first and second image sensors work together and are optionally configured to capture different information about physical objects in the real environment. In some examples, the first image sensor is a visible light image sensor, and the second image sensor is a depth sensor. In some examples, the electronic device 201, 260 uses the one or more image sensors 206A, 206B to detect the position and orientation of the electronic device 201, 260 and / or the one or more display generating components 214A, 214B in the real environment.For example, the electronic device 201, 260 uses the one or more image sensors 206A, 206B to track the position and orientation of the one or more display generating components 214A, 214B relative to one or more solid objects in the real environment.

[0027] In some examples, the electronic devices 201 and 260 include one or more microphones 213A and 213B, respectively, or other audio sensors. The electronic device 201, 260 optionally uses the one or more microphones 213A, 213B to detect sound from the user and / or from the user's real-world environment. In some examples, the one or more microphones 213A, 213B include an array of microphones (e.g., a plurality of microphones) that optionally work together to identify ambient noise or to localize the sound source within the real-world environment.

[0028] The electronic devices 201 and 260 include one or more location sensors 204A and 204B, respectively, for detecting the location of the electronic device 201 and / or the one or more display generating components 214A, or the location of the electronic device 260 and / or the one or more display generating components 214B. The one or more location sensors 204A, 204B may include a GPS (Global Positioning System) receiver that receives data from one or more satellites and enables the electronic devices 201, 260 to determine the absolute position of the device in the physical world.

[0029] The electronic devices 201 and 260 include one or more orientation sensors 210A and 210B, respectively, for detecting the orientation and / or movement of the electronic device 201 and / or the one or more display-generating components 214A, or the orientation and / or movement of the electronic device 260 and / or the one or more display-generating components 214B. For example, the electronic device 201, 260 uses the one or more orientation sensors 210A, 210B to track changes in the position and / or orientation of the electronic device 201, 260 and / or the one or more display-generating components 214A, 214B, as with respect to physical objects in the real environment. The one or more orientation sensors 210A, 210B optionally include one or more gyroscopes and / or one or more accelerometers.

[0030] In some examples, the electronic device 201 includes one or more hand tracking sensors 202 and / or one or more eye tracking sensors 212. It is understood that, although referred to as hand tracking or eye tracking sensors, the electronic device 201 may additionally or alternatively optionally include one or more other body tracking sensors, such as one or more leg, one or more torso, and / or one or more head tracking sensors. The one or more hand tracking sensors 202 are configured to track the position and / or location of one or more sections of the user's hands and / or movements of one or more sections of the user's hands in relation to the three-dimensional environment relative to the one or more display generating components 214A and / or relative to another defined coordinate system.The one or more eye-tracking sensors 212 are configured to track the position and movement of a user's gaze (e.g., a user's attention, including eyes, face, or generally the head) with respect to the real or three-dimensional environment and / or relative to the one or more display-generating components 214A. In some examples, the one or more hand-tracking sensors 202 and / or the one or more eye-tracking sensors 212 are implemented together with the one or more display-generating components 214A. In other examples, the one or more hand-tracking sensors 202 and / or the one or more eye-tracking sensors 212 are implemented separately from the one or more display-generating components 214A.In some examples, the electronic device 201 alternatively does not include the one or more hand tracking sensors 202 and / or the one or more eye tracking sensors 212. In some of these examples, the one or more display generating components 214A can be used by the electronic device 260 to provide a three-dimensional environment, and the electronic device 260 can use inputs and other data received via the one or more sensors (e.g.,The electronic device 201 receives inputs and data acquired by the one or more location sensors 204A, the one or more image sensors 206A, the one or more touch-sensitive surfaces 209A, the one or more motion and / or orientation sensors 210A, and / or the one or more microphones 213A or other audio sensors, as inputs and data to be processed by the one or more processors 218B of the electronic device 260. Additionally or alternatively, the electronic device 260 optionally includes no other inputs. Fig. The components shown in Figure 2B, such as the one or more location sensors 204B, the one or more image sensors 206B, the one or more touch-sensitive surfaces 209B, etc., are included. In some of these examples, the one or more display generating components 214A can be used by the electronic device 260 to provide a three-dimensional environment, and the electronic device 260 can use as input inputs and other data acquired via the one or more motion and / or orientation sensors 210A (and / or the one or more microphones 213A) of the electronic device 201.

[0031] In some examples, the one or more hand tracking sensors 202 (and / or other body tracking sensors such as leg, torso, and / or head tracking sensors) can use the one or more image sensors 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that detect three-dimensional information from the real world, including one or more body parts (e.g., hands, legs, or torso of a human user). In some examples, the hands can be resolved with sufficient resolution to distinguish fingers and their respective positions. In some examples, the one or more image sensors 206A are positioned relative to the user to define a field of view for the one or more image sensors 206A and an interaction space in which finger / hand position, orientation, and / or movement detected by the image sensors are used as inputs (e.g.,(to distinguish between the user's hand at rest and the hands of other people in the real environment). Tracking fingers / hands for input (e.g., gestures, touch, tapping, etc.) can be advantageous because the user does not need to touch, hold, or wear any kind of beacon, sensor, or other device.

[0032] In some examples, the one or more eye-tracking sensors 212 include at least one eye-tracking camera (e.g., infrared cameras (IR cameras)) and / or light sources (e.g., IR light sources, such as LEDs) that emit light toward a user's eyes. The eye-tracking cameras may be directed toward a user's eyes to receive reflected IR light from the light sources directly or indirectly from the eyes. In some examples, both eyes are tracked separately by respective eye-tracking cameras and light sources, and a gaze / view can be determined from tracking both eyes. In some examples, one eye (e.g., a dominant eye) is tracked by one or more respective eye-tracking cameras / light sources.

[0033] The electronic devices 201 and 260 are not related to the components and configuration of Fig. 2A - 2B are limited, but may include fewer, different, or additional components in several configurations. In some examples, the electronic device 201 and / or the electronic device 260 may each be implemented among several electronic devices (e.g., as a system). In some of these examples, each (or several) of the electronic devices may include one or more of the components discussed above, such as various sensors, one or more display generating components, one or more loudspeakers, one or more processors, one or more memories, and / or communication switching logic. A person or persons who use the electronic device 201 and / or the electronic device 260 shall optionally be referred to herein as the user(s) of the device.

[0034] Attention now turns to context-controlled interactions for an electronic device, including with one or more virtual objects displayed in a three-dimensional environment presented on an electronic device (e.g., according to electronic device 201).

[0035] Fig. Figures 3A-3C illustrate an example of an electronic device 300 with an intelligent input device according to examples from the disclosure. In some examples, the electronic device 300 is essentially similar to the electronic devices 101 and 201 described above. Therefore, the electronic device 300 can be associated with one or more displays and one or more input devices. For example, the electronic device 300 can be a head-mounted device (e.g., a head-mounted display) worn by a user of the electronic device 300. In some examples, the electronic device 300 includes a display-generating component 312 (e.g., the display-generating component 214 described above in relation to the electronic device 201).The one or more input devices may include physical user interface devices, such as a touch-sensitive surface 316 (e.g., the touch-sensitive surface described above with respect to electronic device 201), a physical keyboard, a mouse, a joystick, a hand tracking device (e.g., the hand tracking sensors 202 described above with respect to electronic device 201), an eye tracking device (e.g., the eye tracking sensors 212 described above with respect to electronic device 201), a pen, and other input devices. In some examples, the one or more input devices may include one or more sensors for detecting eye movements (e.g.,the eye tracking sensors 212 described above in relation to the electronic device 201), which can be used to determine attention or gaze position and / or gaze movement, which can optionally be used to detect gaze-only inputs based on gaze movement and / or dwell.

[0036] In some examples, such as in the Fig. As illustrated in Figures 3A-3C, the one or more input devices can include a touch-sensitive surface 316. The touch-sensitive surface 316 is configured to detect user contact (e.g., the user's fingers and / or hands) and / or touch by a pointing device such as a stylus. The touch-sensitive surface 316 can detect user inputs such as taps, swipes, and other gestures. In some examples, the touch-sensitive surface 316 is located on a surface of the electronic device 300. In other examples, the touch-sensitive surface is located on a different device that is associated with the electronic device 300.

[0037] In some examples, the electronic device 300 can assign an operation to a specific input device (such as the touch-sensitive surface 316) at the input device, which can be performed in response to the electronic device 300 receiving that input type at the input device. Accordingly, the electronic device 300 can perform the operation assigned to that input type in response to the detection of an input of that type at the input device. For example, the electronic device 300 can assign one operation to a tap input (e.g., an input in which the user of the electronic device places a finger on the touch-sensitive surface 316 and then removes it), another operation to a double-tap input, and another operation to a swipe input.In some examples, the input assignments, or equivalently, the mapping of input types to operations, can be based on the application and, in particular, the interface of an application displayed by the electronic device. For example, in the context of a music application that plays music, a tap input can be assigned to execute a pause / play operation, while a swipe input (where the user moves their finger across the touch-sensitive surface 316) can be assigned to increase and / or decrease the volume (depending on the direction of the swipe input).

[0038] In some examples, the context in which the input type is applied may not justify executing the operation, even if an input type is assigned to a particular operation. Therefore, in one or more examples, the electronic device 300 can override the input assignments or mappings based on the current context of the electronic device 300. The electronic device 300 can thus automatically adapt certain inputs (e.g., gestures) detected at an input device (e.g., a smart input device) to fulfill the user's intent, even if the detected inputs are already assigned to other operations.

[0039] Fig. 3A illustrates an exemplary input-to-operation mapping according to examples from the revelation. In the example of Fig. 3A displays the electronic device 300, which presents a first user interface in an environment 310. The first user interface is a media player interface 314. In Fig. 3A A first operation is assigned or mapped to a first input type while the electronic device 300 displays the media player interface 314 (which is displayed as part of the electronic device running a media application for playing media). For example, while the media player interface 314 is displayed and media playback is in progress, a 'pause' operation can be assigned and / or mapped to a tap input (e.g., the first input type) detected on the touch-sensitive surface 316 (e.g., a first input device). While the media application is running, the electronic device assigns the pause operation to a detected tap on the touch-sensitive surface 316. In some examples, a second operation can be assigned to a second input type.For example, while the media player interface 314 is displayed and media playback is in progress, an operation “Clear User Interface” (e.g. “Clear user interface”) can be assigned and / or mapped to a double-tap input (e.g. of the second input type) that is detected on the touch-sensitive surface 316 (e.g. of a first input device).

[0040] As in Fig. As shown in Figure 3A, while displaying the media player interface 314 (e.g., the first user interface), the electronic device 300 detects a tap input 302 (e.g., the first input type) on the touch-sensitive surface 316 (e.g., of the first input device). The electronic device 300 can determine the intent of the tap input 302 based on a current context. In some examples, a context may include an environment 310 displayed on the screen, such as an environment described in relation to the electronic device 101. In some examples, the context may include the user's location within a three-dimensional environment and / or the virtual objects displayed in the three-dimensional environment by the one or more screens 312. In some examples, the context may include an event or occurrence within the environment 310.In some examples, the context may include one or more applications that the electronic device 300 presents on the one or more displays 312. In some examples, the context may include one or more user interfaces that the electronic device 300 presents on the one or more displays 312. In some examples, the context may include the physical environment of the electronic device 300, as detected by the various sensors of the electronic device 300, as described above in relation to the electronic device 201 (e.g.,one or more hand tracking sensors 202, one or more location sensors 204, one or more image sensors 206, one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors). The context may include, for example, the electronic device 300 detecting, via the one or more image sensors 206, that the user is indoors and not outdoors, or that the user is watching television or reading a book. The context may also include, for example, the electronic device detecting, via the one or more body tracking sensors, that the user is standing or sitting.

[0041] In some examples, the electronic device 300 can detect an intent for the tap input 302 (e.g., the first input type) based on the current context, which includes the media player interface 314 (e.g., the first user interface) and the user's gaze direction 318. For example, when the electronic device 300 detects the tap input 302, playback is in progress and the user's gaze 318 (e.g., as detected by one or more eye-tracking sensors 212) is not directed at the media player interface 314. As previously described, a "pause" operation has been assigned and / or associated with the tap input (e.g., the first input type) on the touch-sensitive surface 316. Based on the current context, the electronic device 300 determines that the intent of the tap input is to request the "pause" operation.Accordingly, the electronic device can execute the "pause" operation (e.g., stop playback) in response to the recognition of the touch input 302. The response of the electronic device 300 to the recognition of the touch input 302 thus reflects the mapping of the "pause" operation to the touch input 302 on the touch-sensitive surface 316.

[0042] Fig. Figure 3B illustrates an input-to-operation assignment override operation according to examples from the disclosure. In the example of Fig. 3B detects the electronic device 300, while it is displaying the media player interface 314 (e.g., the first user interface), a double-tap input 304 (e.g., the second input type) on the touch-sensitive surface 316 (e.g., of the first input device). The electronic device 300 can determine the intent of the double-tap input 304 based on the current context, which includes the media player interface 314 (e.g., the first user interface) and the user's viewing direction 318. For example, when the electronic device 300 detects the double-tap input 304, playback on the media player interface 314 begins. Unlike in Fig. 3A, where the user's gaze 318 was directed towards the environment 310, the electronic device 300 detects in Fig. 3B, that the user's gaze 318 is directed towards the media player interface 314. While, as previously described, the double-tap input 304 (e.g., the second input type) on the touch-sensitive surface 316 has been assigned and / or mapped to an operation "Clear user interface", the electronic device 300 determines, based on the current context (e.g., playback in progress and gaze 318 directed towards the media player interface 314), that the intention of the double-tap input 304 (e.g., the second input type) is a request to execute the "Pause" operation (e.g., the first operation). Accordingly, in response to the detection of the double-tap input 304, the electronic device can execute the "Pause" operation (e.g., stop playback). The electronic device 300 has thus performed the second operation that was assigned to the double-tap input 304 (e.g.,The operation "Clear user interface") was overridden based on the determination that the intent of the double-tap input 304 (based on the current media player interface context and gaze direction 318) was a request to perform the first operation (a "pause" operation) instead. In one or more examples, the computer system determined from the context in which the double-tap input was received that it was more likely the user intended to perform a pause operation (and may have accidentally double-tapped when they only meant to tap once) and therefore overridden the operation assigned to a double-tap in order to perform the operation normally assigned to a single tap (e.g., a pause operation).

[0043] In some examples, the intention based on the current context has a confidence level, and determining the intention includes determining a confidence level associated with the intention. The confidence level of the intention based on the current context can be influenced by various factors that contribute to determining the current context, such as the initial interface (e.g., media player interface 314) and the user's gaze 318. For example, ambiguity in the user's gaze direction 318 can affect (e.g., decrease) the confidence level in the intention, as explained in more detail below. Therefore, in some examples, the electronic device 300 optionally performs the initial operation on the electronic device 300 in accordance with a finding that the confidence level of the determined intention is above a confidence threshold.

[0044] Fig. Figure 3C illustrates an example context which, when recognized by the electronic device, causes the device to override the assigned operation according to one or more examples. In the example of Fig. 3C determines that the electronic device 300 determines that the intent of the first input with the second input type (e.g., the double-tap input 304) is uncertain and / or ambiguous and therefore does not constitute a clear request to perform the first operation (e.g., the "pause" operation) or the second operation (e.g., "clear user interface") on the electronic device. As previously described, the electronic device 300 can determine the intent of the double-tap input 304 based on the current context, which includes the media player interface 314 (e.g., the first user interface) and the user's viewing direction 318. For example, if the electronic device 300 detects the double-tap input 304, playback is running on the media player interface 314. Unlike in Fig. 3B, where the user's gaze 318 was directed towards the media player interface 314, is in Fig. 3C, however, the position of the gaze 318 is uncertain, so the electronic device 300 cannot determine the intention of the double-tap input 304 with sufficient certainty. For example, the user's gaze 318 may not be directed at the media player interface 314 long enough to indicate an intention. In some examples, the user's gaze 318 may be directed at an edge of the media player interface 314, so the electronic device 300 cannot determine whether the user is looking at the media player interface 314 or the surrounding environment 310.

[0045] As in Fig. As shown in Figure 3C, the electronic device 300, in accordance with a finding that the intention for the double-tap input 304 is not a request to perform the "pause" operation, can display a second user interface 322 that includes a first selectable option 324 (e.g., "Pause?") for performing the first operation (e.g., pause playback) and a second selectable option 326 (e.g., "Clear user interface?") for performing the second operation (e.g., clear user interface), which is different from the first operation. In some examples, the second selectable option 326 corresponds to the assigned operation for a double tap, while the first selectable option 324 corresponds to a possible override operation based on a low-confidence-level intention.In some examples, the electronic device 300 displays the second user interface 322 in accordance with a determination that the confidence level associated with the intent does not exceed a confidence threshold (e.g., is not above it). The second user interface 322 provides the user with the opportunity to clarify the intent of an ambiguous input detected by the electronic device 300 on the touch-sensitive surface 316. Thus, the electronic device 300 can detect a second input directed at the first selectable option 324 (e.g., "Pause?") or the second selectable option 326 (e.g., "Clear user interface?"). In response, the electronic device 300 can perform the first operation on the electronic device (e.g.,(Stop playback). And in accordance with a finding that the second input is directed to the second selectable option 326 (e.g., "Clear user interface?"), the electronic device can perform the second operation on the electronic device (e.g., clear user interface). In one or more examples, the selectable options 324 and 326 may be accompanied by an audio notification (e.g., a tone is played) indicating that the electronic device 300 has little confidence in which operation should be performed in response to a particular input based on the context in which the input was made.

[0046] In some examples, after determining that the confidence level associated with the intent does not exceed a confidence threshold, the electronic device 300 optionally refrains from displaying the second user interface 322 and instead performs the second operation (e.g., the operation assigned to the second input type). For example, after determining that the confidence level associated with the determined intent of the double-tap input 304 (e.g., a request to perform a "pause" operation on the electronic device 300) does not exceed a confidence threshold, the electronic device 300 optionally performs the "clear user interface" operation, which is the operation assigned or associated with the double-tap input 304.The electronic device 300 therefore optionally dispenses with the input assignment or mapping for an input type and the display of a second user interface 322 to clarify the intent of the input and instead performs the operation assigned to the input type of the input if the confidence level of the intent is below the confidence threshold.

[0047] By automatically adapting an input detected by an intelligent input device to fulfill the user's intent, even if the input is already assigned to another action as described above, ambiguities and misinterpretations of user inputs are reduced, thus minimizing erroneous inputs. This improves the reliability and efficiency of the user's interaction with the electronic device and saves computing resources that would otherwise be used to correct erroneous user input.

[0048] In one or more examples, inputs such as a swipe gesture can be used to perform a scrolling operation on the electronic device 300. In one or more examples, and as described in more detail below, the electronic device 300 can detect, based on the user's eye movements, whether the user is searching for something specific while scrolling through a user interface (as opposed to scrolling and navigating without a specific intention). In some examples, and as described in more detail below, the system can dampen the scrolling speed to make it easier for the user to search while the user interface scrolls based on the user's eye movements.

[0049] Fig. Figures 4A-4B illustrate an example of an electronic device 400 that features attention-based scroll stabilization as described in the examples disclosed. In some examples, the electronic device 400 is essentially similar to the electronic devices 101, 201, and 300 described above. Therefore, the electronic device 400 can be associated with one or more displays and one or more input devices. For example, the electronic device 400 can be a head-mounted device (e.g., a head-mounted display) worn by a user of the electronic device 400. In some examples, the electronic device 400 includes a display-generating component 412 (e.g., the display-generating component 214 described above in relation to the electronic device 201).In some examples, the electronic device 400 can display a three-dimensional environment 410 on the display-generating component (or display) 412. In some examples, the three-dimensional environment 410 is visible to the user of the electronic device 400 through the display-generating component 412 (e.g., optionally through a transparent and / or translucent display). For example, the three-dimensional environment 410 is visible to the user of the electronic device 400 while the user is wearing the electronic device 400. In some examples, the display-generating component is configured to display one or more virtual objects (e.g., virtual content contained in a virtual window or user interface) in the three-dimensional environment 410. In some examples, the one or more virtual objects are displayed within a virtual environment (e.g., superimposed).In some examples, one or more virtual objects are displayed within a representation of a user's physical environment (e.g., superimposed). In some examples, one or more virtual objects include one or more user interface elements, such as the movie list 414. In some examples, one or more user interface elements are scrollable, such as the scrollable movie list 414 displayed by the electronic device 400 in the three-dimensional environment 410.

[0050] Fig. Figure 4A illustrates an exemplary scroll operation based on the speed of a scroll input according to examples from the disclosure. In some examples, the electronic device 400 can detect a scroll input 402 via a first input device 416 of one or more input devices. The scroll input 402 can correspond to a request to scroll one or more user interface elements, for example, the scrollable movie list 414. It is understood that the one or more user interface elements in Fig. Although 4A is represented as a scrollable movie list 414, one or more of the interface elements can be any scrollable interface element (e.g., any interface element that can scroll in response to user input (e.g., scroll input 402)). Examples of scrollable interface elements include interfaces for content elements (e.g., interface elements that include a variety of representations of content elements such as videos, photos, documents, files), notification interfaces, document interfaces, text, and other examples.

[0051] In some examples, the first input device 416 can be a physical user interface device (e.g., a touch-sensitive surface described above with respect to the electronic device 201), such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device (e.g., the hand tracking sensors 202 described above with respect to the electronic device 201), an eye tracking device (e.g., the eye tracking sensors 212 described above with respect to the electronic device 201), a stylus, etc. In the illustrated example, the first input device 416 is a touch-sensitive surface arranged on a surface of the electronic device 400. In some examples, the scroll input 402 can be a gesture input such as swiping (e.g., with a finger or a stylus).In some examples, swiping a prompt to scroll the scrollable movie list 414 (e.g., one or more surface elements) in a direction that corresponds to (e.g., coincides with) a direction of swiping. In some examples, the scroll input 402 (e.g., a swipe gesture) can have an initial input speed 424, as shown in the input speed bar 422. The input speed 424 represents a speed of a gesture (e.g., a swipe gesture) that is recognized by the electronic device 400 as a scroll input 402.

[0052] In some examples, the one or more input devices may include one or more eye-tracking sensors (e.g., the eye-tracking sensors 212 described above in relation to the electronic device 201), which can be used to determine attention or gaze position and / or gaze movement, optionally enabling the detection of gaze-only inputs based on gaze movement and / or dwell time. Gaze and / or attention information can be combined with hand-tracking information to determine interactions between the user and one or more user interfaces based on direct and / or indirect inputs such as air gestures or inputs using one or more hardware input devices such as one or more buttons.

[0053] In some examples, the electronic device 400 can scroll the one or more user interface elements, such as the scrollable movie list 414, in response to the detection of the scroll input 402 (e.g., the swipe motion). In particular, the electronic device 400 can scroll the one or more user interface elements at a first speed 428 in accordance with a finding that the user's attention is not directed toward the one or more user interface elements. In some examples, attention is based on gaze 418, which indicates the location in the three-dimensional environment 410 at which the electronic device 400 detects the user's gaze as directed (e.g., via one or more sensors of the one or more input devices). Fig. 4A, attention (e.g., based on gaze 418) is represented as if it were directed to a different location in the three-dimensional environment 410 than the scrollable movie list 414. Accordingly, if the user swipes or performs a swipe gesture on the input device 416 (e.g., if the electronic device 400 recognizes the swipe gesture as scroll input 402) while the scrollable movie list 414 is displayed in the three-dimensional environment 410, the electronic device 400 scrolls the movie list 414. In some examples, the electronic device 400 scrolls the one or more user interface elements at a first speed 428, as shown in the speed bar 426.In some examples, the first speed 428 is a speed above a speed at which a user can recognize or read the elements of the film list 414, because the device determines that the user is not specifically looking at the film list 414.

[0054] In some examples, as described below, the electronic device 400 reduces the scrolling speed when the user's attention shifts to the one or more user interface elements, while the electronic device 400 detects a scroll input 402 (e.g., while the user is scrolling the one or more user interface elements), even while maintaining the scroll input speed to make it easier for the user to see the scrolling one or more user interface elements.

[0055] Accordingly, in some examples, the electronic device 400 can, in response to the detection of scroll input and in accordance with a finding that the user's attention is directed towards one or more user interface elements, scroll one or more user interface elements at a second speed that is slower than the first. Fig. Figure 4B illustrates the electronic device 400, which, in response to the scroll input 402 and in accordance with a finding that the user's attention is directed toward one or more surface elements, scrolls one or more user interface elements (e.g., the scrollable movie list 414) at a second speed 432. As illustrated by the input speed bar 422, the input speed 424 of the scroll input 402 is in Fig. 4B the same as the input speed 424 of the scroll input 402 in Fig. 4A. Unlike in Fig. 4A will be in Fig. 4B the user's attention (e.g., based on gaze 418) is directed to one or more surface elements (e.g., the user's gaze 418 is directed to the scrollable movie list 414). Accordingly, in response to the detection of the scroll input 402 and in accordance with the finding that the user's attention (e.g., based on gaze 418) is directed to the scrollable movie list 414, the electronic device 400 scrolls the scrollable movie list 414 at a second speed 432, which differs from the first speed 428. In some examples, the electronic device 400 can scroll the scrollable movie list 414 at the second speed 432, even though the input speed 424 of the scrolling remains the same as if the user's attention (e.g., based on gaze 418) were not directed to the scrollable movie list 414. In some examples, as in Fig. As illustrated in Figure 4B, the second speed 432 is lower or slower than the first input speed 424 (e.g., scrolling is slowed down), effectively dampening the scroll input 402 and / or the effect of the swipe gesture and stabilizing scrolling when the gaze 418 is directed toward the one or more surface elements (e.g., attention-based scroll stabilization). Dampening and / or stabilizing the scroll input can make it easier for the user to see the scrolling one or more user interface elements when the user's attention shifts to the one or more user interface elements while the electronic device 400 detects a scroll input 402 (e.g., while the user is scrolling the one or more user interface elements).

[0056] In some examples, the electronic device 400 reduces the scrolling speed based on the level of attention directed toward one or more user interface elements. In one or more examples, a level of attention reflects the degree to which the user is focused on the one or more user interface elements, as detected by the electronic device 400. The level of attention includes, for example, eye movement while the gaze 418 is directed toward the one or more user interfaces. For example, greater eye movement may indicate that the user is less focused on the one or more user interfaces (e.g., a lower level of attention), whereas lesser eye movement may indicate that the user is more focused on the one or more user interface elements (e.g., a higher level of attention).a higher level of attention). In some examples, dwell time can be a measure of the level of attention, such that a longer dwell time can indicate a higher level of attention and a shorter dwell time a lower level of attention. Thus, in some examples, in accordance with a finding that a level of attention is a first level, the electronic device 400 can scroll the one or more user interface elements at a first speed. In some examples, in accordance with a finding that the level of attention is a second level, which is higher than the first, the electronic device can scroll the one or more user interface elements at a third speed, which is slower than the first.

[0057] Dampening scroll input based on gaze direction, such as reducing the scroll speed of an interface element based on the detection that the user's gaze is searching for a specific element as described above, reduces unnecessary movements in the user interface and thus improves the energy efficiency and readability of the user interface, which increases the efficiency of the user's interaction with the electronic device and minimizes the probability of erroneous user input, thereby saving computing resources that would otherwise have to be spent correcting erroneous user input.

[0058] In some examples, scroll input can be used to perform different actions based on the speed (e.g., velocity) of the scroll input. As described in more detail below, a user scrolling to navigate a user interface can cause the electronic device to move the user interface out of their field of vision by increasing the scrolling speed.

[0059] Fig. Figures 5A-5C illustrate an example of an electronic device 500 with speed-based wipe detection according to examples in the disclosure. As in Fig. As illustrated in Figure 5A, the electronic device 500 is, in some examples, essentially similar to the electronic devices 101, 201, 300, and 400 described above. Thus, the electronic device 500 can be associated with one or more displays 512 (e.g., the display generating component 214 described above in relation to the electronic device 201) and one or more input devices 516. For example, the electronic device 500 can be a head-mounted device (e.g., a head-mounted display) worn by a user of the electronic device 500. In some examples, the electronic device 500 can represent a three-dimensional environment 510 on the display generating component (or display) 512.

[0060] In some examples, the electronic device 500 can detect a scroll input 502 via a first input device of one or more input devices 516. The scroll input 502 can correspond to a request to scroll one or more user interface elements, for example, the scrollable movie list 514. It is understood that the one or more user interface elements in Fig. 5A is represented as a scrollable movie list 514, but the one or more interface elements can be any scrollable interface element (e.g., any interface element that can scroll in response to user input). Examples of scrollable interface elements include interfaces for content elements (e.g., interfaces that include a variety of representations of content elements such as videos, photos, documents, and files), notification interfaces, document interfaces, and other examples.

[0061] In some examples, such as in Fig. As illustrated in Figure 5A, a first input device of the one or more input devices 516 can be a physical user interface device, such as a touch-sensitive surface (e.g., the touch-sensitive surface described above with respect to the electronic device 201), a physical keyboard, a mouse, a joystick, a hand tracking device (e.g., the hand tracking sensors 202 described above with respect to the electronic device 201), an eye tracking device (e.g., the eye tracking sensors 212 described above with respect to the electronic device 201), a pen, etc. In the illustrated example, the first input device of the one or more input devices 516 is a touch-sensitive surface arranged on a surface of the electronic device 500. In some examples, the scroll input 502 can be a gesture input such as a swipe (e.g., a scrolling motion).B. with a finger or a stylus). In some examples, swiping in response to a prompt to scroll the scrollable movie list 514 (e.g., one or more surface elements) in a direction that corresponds to (e.g., coincides with) a direction of swiping. In some examples, the scroll input (e.g., swipe gesture) may have an initial input speed, which is displayed as the scroll input speed 524 in the input speed bar 522. The scroll input speed 524 represents a speed of a gesture (e.g., a swipe gesture) that is recognized by the electronic device 500 as scroll input.

[0062] In some examples, the electronic device 500 can scroll one or more user interface elements, such as the scrollable movie list 514, in response to the detection of the scroll input 502 (e.g., the swipe motion). In particular, the electronic device 500 can scroll one or more user interface elements in accordance with a finding that the speed of the scroll input 502 is below an input speed threshold 526. As in Fig. As shown in Figure 5A, the speed of the scroll input detected by the electronic device 500 is displayed as the scroll input speed 524 in the scroll input speed bar 522. Furthermore, the scroll input speed 524 is below or lower than the input speed threshold 526. Accordingly, in response to the detection that the scroll input 502 has a scroll input speed 524 below the input speed threshold 526, the electronic device 500 scrolls one or more user interfaces (e.g., the scrollable movie list 514). In some examples, as described with respect to the electronic device 400 and in the Fig. As illustrated in Figures 4A-4B, the electronic device 500 can scroll the one or more user interfaces at a first speed (e.g., the first speed 428) and / or at a second speed (e.g., the second speed 432), depending on where the electronic device 500 detects that the user's attention is directed (e.g., based on gaze 418), and / or at a third speed, depending on how much attention is directed to the one or more user interface elements.

[0063] In some examples, the electronic device 500 can, in accordance with a finding that the scroll input speed 502 (e.g., swiping) is at or above the input speed threshold 526, terminate the display of one or more user interface elements, as in the example of Fig. 5B illustrates this. Fig. Figure 5B illustrates, in one or more examples, how the electronic device recognizes a scroll input 502. As shown in Fig. 5A is the scroll input 502, a swipe gesture. Unlike the scroll input 502 from Fig. 5A displays the scroll input 502. Fig. 5B, however, shows a speed of 528, which is above the input speed threshold of 526 (e.g., the user swiped faster on the first input device of one or more input devices 516 than in Fig. 5A and at a speed exceeding the input speed threshold 526). Accordingly, in response, the electronic device 500 terminates the display of one or more user interfaces (e.g., the scrollable movie list 514). In some examples, the electronic device 500 terminates the display of one or more user interface elements (e.g., the scrollable movie list 514) by displaying an animation of the one or more user interface elements moving out of the three-dimensional environment 510. In some examples, the electronic device 500 displays the animation of the one or more user interface elements moving in a direction corresponding to a scroll input direction. Fig. 5B, one or more user interface elements (e.g., the scrollable movie list 514) are displayed as if they were moving out of the user's line of sight and / or the three-dimensional environment 510. In Fig. 5C, one or more user interface elements were removed from the user's line of sight and / or the three-dimensional environment 510. In particular, the electronic device 500 stopped displaying one or more user interface elements (e.g., the scrollable movie list 514) in the three-dimensional environment 510 in response to the detection of a scroll input 502 whose scroll input speed 528 exceeded the input speed threshold 526.

[0064] In some examples, the scroll input 502 (e.g., a swipe gesture) whose speed exceeds the input speed threshold 526 and thus causes the electronic device 500 to stop displaying one or more user interface elements may have a direction that matches the direction of a scroll input 502 (e.g., a swipe gesture) whose speed is below the input speed threshold 526 and also causes the electronic device 500 to scroll the one or more user interface elements. Accordingly, a user can cause the electronic device 500 to stop displaying a user interface element that they have scrolled with a scroll input 502 (e.g., a swipe gesture) by performing the same gesture (e.g., in the same direction) quickly enough to exceed the input speed threshold 526.

[0065] By performing various actions based on the speed of a scroll input, as described above, the number of inputs required to operate the electronic device is reduced, thus improving user interface navigation, increasing the efficiency of the user's interaction with the electronic device and conserving the electronic device's computing resources.

[0066] In one or more examples, in addition to the touch inputs described above, the user can also make inputs by looking at the electronic device. For example, as described in detail below, the user can direct their gaze to a specific part of the display to initiate an operation that is executed based on the context in which the gaze input occurs.

[0067] Fig. Figures 6A-6D illustrate an example of an electronic device 600 with context-controlled active display areas according to examples from the disclosure. In some examples, the electronic device 600 is essentially similar to the electronic devices 101, 201, 300, 400, and 500 described above. Therefore, the electronic device 600 can be associated with one or more displays 612 (e.g., the display generating component 214 described above in relation to electronic device 201) and one or more input devices. For example, the electronic device 600 can be a head-mounted device (e.g., a head-mounted display) worn by a user of the electronic device 600. In some examples, the one or more displays 612 include one or more areas 614. Fig. 6A comprises the one or more displays 612 and four areas 614 (614a, 614b, 614c, and 614d). However, it is understood that the one or more displays 612 of an electronic device 600 can have any number of areas 614 (e.g., 1, 2, 3, 4, 5, 8, 10, etc., areas 614). In some examples, an area of ​​the one or more areas 614 can correspond to a corner of the one or more displays 612. Such an area can be called a "corner area." In some examples, an area of ​​the one or more areas 614 can correspond to a border area and be called a "border area." However, it is understood that an area can refer to any location on the one or more displays 612 of the electronic device 600 (e.g., a corner area, a border area, a central area, or any other area).

[0068] In some examples, the electronic device 600 also includes one or more visual indicators 622 (e.g., 622a, 622b, 622c, and 622d), each visual indicator 622 being associated with a region 614 of the one or more regions 614. As described with respect to the electronic device 201, a visual indicator 622 is an output device, and one or more communication buses 208 are optionally used for communication between the one or more visual indicators 622 and other components of the electronic device 600. In some examples, a visual indicator 622 is a light-emitting diode (“LED”). In some examples, as in the Fig. As illustrated in Figures 6A-6D, an area 614 is located next to the visual indicator 622 (e.g., LED) with which it is linked. In some examples, the electronic device 600 can change properties (e.g., brightness, color) of a visual indicator 622 (e.g., LED) based on the detection of attention directed toward the corresponding area 614, as described below.

[0069] In some examples, the electronic device 600 can represent a first context on one or more displays 612. In some examples, a context on the electronic device 600 can include an environment displayed on the screen, such as a three-dimensional environment as described with respect to the electronic devices 400 and 500 and in the Fig. 4A - 4B and Fig. Figures 5A-5C show (e.g., the three-dimensional environments 410 and 510). In some examples, the context may include the user's location within the three-dimensional environment and / or the virtual objects displayed in the three-dimensional environment. In some examples, the context may include an event or occurrence within the three-dimensional environment. In some examples, the context may include one or more applications that the electronic device 600 presents on the one or more displays 612. In some examples, the context may include one or more user interfaces that the electronic device 600 presents on the one or more displays 612.In some examples, the context may include the physical environment of the electronic device 600, as detected by the various sensors of the electronic device 600, as described above in relation to the electronic device 201 (e.g., one or more hand tracking sensors 202, one or more location sensors 204, one or more image sensors 206, one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body tracking sensors (e.g., torso and / or head tracking sensors)). For example, the context may include the electronic device 600 detecting, via the one or more image sensors 206, that the user is indoors and not outdoors, or that the user is watching television or reading a book.The context may include, for example, the electronic device detecting whether the user is standing or sitting via one or more body tracking sensors.

[0070] In some examples, the electronic device 600, in accordance with a determination that a first context is present on the electronic device 600, can display an indicator 616 in one or more areas 614, corresponding to an operation to be performed on the electronic device 600. As described below, the corresponding operation can be performed when the electronic device 600 detects that attention is directed toward area 614. As described in Fig. As shown in Figure 6A, the electronic device 600 displays a media player interface 632 in the three-dimensional environment, which represents a context (e.g., the first context) present on the electronic device 600. In accordance with the first context, which includes the media player interface 632, the electronic device 600 displays a first display 616a-1 (e.g., "Home") in the first area (e.g., area 614a), corresponding to a first operation to be performed on the electronic device (e.g., displaying the home screen). In some examples, such as those illustrated, the electronic device 600 can display first displays 616 (e.g., "Home") in several areas 614 (e.g., 614a-614d), in accordance with the first context, which includes the media player interface 632.Displays 616a-1, 616b-1, 616c-1, and 616d-1) correspond to the first operations to be performed on the electronic device 600. In some examples, the electronic device optionally does not display a display 616 in area 614, even though that area 614 corresponds to an operation to be performed on the electronic device. Each display 616n-1 is associated with an area 614n of one or more areas 614. The displays 616 thus serve to inform the user which operation will be performed when they focus their attention on a particular area 614. In some examples, such as those illustrated, a display 616 may be a label that names the operation corresponding to the area (e.g., "Home," "Search," etc.). In some examples, the display 616 may be a symbol that represents and / or corresponds to the operation associated with area 614.

[0071] In accordance with the finding that a first context (e.g., the media player interface 632) is present on the electronic device 600 and that the user's attention is directed toward a first area 614 of the plurality of areas 614, the electronic device 600 can perform a first operation on the electronic device. In some examples, the attention is based on the gaze 618, which indicates the location of the one or more displays 612 toward which the electronic device 600 recognizes the user's gaze as directed (e.g., via one or more sensors of the one or more input devices). Fig. In 6A, view 618 is shown in such a way that it is directed towards the media player interface 632 and away from areas 614a-614d. Fig. In 6B, view 618 is depicted as if it were directed closer to area 614a, without actually being directed at the area itself. Fig. In Figure 6C, gaze 618 is depicted as directed toward area 614a, and the electronic device 600 recognizes that attention (e.g., based on gaze 618) is directed toward area 614a. Area 614a includes the indicator 616a (e.g., "Home"), which indicates that the operation corresponding to area 614a is a request to display the home interface or home screen of the electronic device 600. Accordingly, the electronic device 600, as shown in Figure 6C, performs the following action: Fig. 6D shows that, in response to the detection that the user's attention (e.g., based on gaze 618) is directed towards area 614a (e.g., the first area), the "Home" operation (e.g., the first operation) is executed on the electronic device, thus displaying the Home screen 634.

[0072] Furthermore, the home screen displays 634 in Fig. 6D represents a second context present on the electronic device 600, distinct from the first context of the media player interface 632. Therefore, in some examples, in accordance with the finding that the second context, distinct from the first, is present on the electronic device 600, the electronic device 600 can display a second display 616a-2 (e.g., "Photos") in the first area (e.g., area 614a), which is distinct from the first display 616a-1 (e.g., "Home," as shown in Figure 6D). Fig. 6A - 6C) differs and corresponds to a second operation (e.g., opening a photo app). In some examples, the electronic device 600 can display second displays 616 (e.g., 616a-2, 616b-2, 616c-2 and 616d-2, respectively, “Photos”, “Settings”, “Language” or “Voice Assistant” and “Apps”) in several areas 614 (e.g., 614a - 614d), which differ from the first displays and correspond to second operations to be performed on the electronic device, which differ from the first operations.

[0073] Fig. Figures 7A-7D illustrate an example of an electronic device 600 with context-controlled active display areas according to examples in the disclosure. Fig. 7A A second context (e.g., a context that includes the home screen 634) is present on the electronic device 600, which is the same context as in Fig. 6D is shown. In accordance with the finding that the second context (e.g., the home screen 634) is present on the electronic device 600 and the user's attention is directed to the first area 614a of the multitude of areas 614, the electronic device 600 can perform a second operation on the electronic device (e.g., open the photo app) that differs from the first operation (e.g., display the home interface). Fig. 7A detects that the gaze is directed 618 towards the home screen 634 and away from one of the areas 614a-614d. In Fig. In 7B, view 618 is depicted as being closer to area 614a, without actually being directed at the area itself. Fig. 7C shows that gaze 618 is directed at area 614a, and the electronic device 600 recognizes that attention (e.g., based on gaze 618) is directed at area 614a. Area 614a includes display 616a-2 (e.g., "Photos"), which is distinct from displays 616a-1 ("Home") of the first context (e.g., the media player interface 632) and indicates that the operation corresponding to area 614a in the second context is opening the Photos app. Accordingly, the electronic device 600, as shown in Fig. 7D shows that, in response to the detection that the user's attention (e.g., based on gaze 618) is directed towards area 614a (e.g., the first area), the operation "Photos" (e.g., the second operation) is executed on the electronic device, thus opening the photo app.

[0074] Fig. Figures 8A-8D illustrate an example of an electronic device 600 with context-controlled active display areas according to examples in the disclosure. Fig. 8A is the first context present on the electronic device 600. This is the same context as in Fig. 6A - 6C (e.g., a context that includes the media player interface 632) and that displays the same displays 616a-1 - 616d-1 (e.g., "Home", "Search", "Back", "Next") in one or more areas 614a - 614d of the one or more displays 612. In accordance with a finding that the first context (e.g., the media player interface 632) is present on the electronic device 600 and the user's attention is directed to a second area 614b of the plurality of areas 614, the electronic device 600 can perform a third operation on the electronic device (e.g., display the settings) that is different from the first operation (e.g., the "Home" operation). In Fig. 8A detects that the view is directed 618 towards the media player interface 632 and away from one of the areas 614a-614d. In Fig. In 8B, view 618 is depicted as being directed closer to area 614b, without being directed at the area itself. Fig. In Figure 8C, gaze 618 is depicted as directed toward area 614b, and the electronic device 600 recognizes that attention (e.g., based on gaze 618) is directed toward area 614b. Area 614b includes display 616b-1 (e.g., "Search"), which is distinct from display 616a-1 ("Home") and indicates that the operation corresponding to area 614b is a request to display the search interface of the media player interface 632. Accordingly, the electronic device 600, as shown in Figure 8C, executes the following command: Fig. 8D shows that, in response to the detection that the user's attention (e.g., based on gaze 618) is directed towards area 614b (e.g., the second area), the "Search" operation (e.g., the third operation) is executed on the electronic device, thus displaying the search surface 636.

[0075] Fig. Figures 9A-9D illustrate an example of an electronic device 600 with context-controlled active display areas according to examples in the disclosure. Fig. 9A is the second context (e.g., a context that includes the home screen 634) present on the electronic device 600. This is the same context as in the Fig. 7A - 7C (e.g., a context that includes the home screen 634) and displays the same displays 616a-2 - 616d-2 (e.g., "Photos", "Settings", "Language" or "Voice Assistant", "Apps") in one or more areas 614a - 614d of the one or more displays 612. In accordance with a finding that the second context (e.g., the home screen 634) is present on the electronic device 600 and the user's attention is directed to the second area 614b of the plurality of areas 614, the electronic device 600 can perform a fourth operation on the electronic device (e.g., the display settings interface) that is different from the second operation (e.g., opening the Photos app). In Fig. In 9A, the view 618 is depicted as being directed towards the home screen 634 and away from one of the areas 614a-614d. Fig. In 9B, view 618 is depicted as being directed closer to area 614b, without being directed at the area itself. Fig. 9C shows that gaze 618 is directed towards area 614b, and the electronic device 600 recognizes that attention (e.g., based on gaze 618) is directed towards area 614b. Area 614b includes display 616b-2 (e.g., "Settings"), which is distinct from display 616a-2 (e.g., "Search") of the first context (e.g., the media player interface 632) and indicates that the operation corresponding to area 614b is to display the settings interface. Accordingly, the electronic device 600, as shown in Fig. 9D shows that, in response to the detection that the user's attention (e.g., based on gaze 618) is directed towards area 614b (e.g., the second area), the operation "Settings" (e.g., the fourth operation) is executed on the electronic device, thus displaying the setting interface 638.

[0076] In some examples, upon detecting that a user's gaze is near an area 614, the electronic device 600 can modify the visual properties (e.g., brightness and / or color) of a visual indicator 622 associated with that area to provide feedback to the user. For example, the electronic device 600 can vary the brightness of a visual indicator 622 associated with an area 614 based on the distance of the user's gaze 618 from the area. In some examples, when the user's gaze 618 is detected within an area 614, the electronic device 600 can change the color of the corresponding visual indicator 622 based on the operation associated with the area.

[0077] Fig. Figures 10A-10D illustrate an example of an electronic device 600 with context-controlled active display areas according to examples in the disclosure. Fig. 10A is the first context present on the electronic device 600. This is the same context as in Fig. 6A - 6C (e.g., a context that includes the media player interface 632) and that displays the same displays 616a-1 - 616d-1 (e.g., "Home", "Search", "Previous", "Next") in one or more areas 614a - 614d of the one or more displays 612. As previously described, the electronic device 600 further includes one or more visual indicators 622 (e.g., 622a, 622b, 622c, and 622d), each visual indicator 622 being associated with an area 614 of the one or more areas 614. In some examples, the electronic device can vary the brightness of a visual indicator 622 based on a distance of the user's gaze 618 from the area 614 corresponding to the visual indicator. Accordingly, the electronic device 600 can, as in Fig. As shown in Figure 10A, in accordance with a finding that the user's gaze 618 has a first distance d1 (as shown in the distance indicator 642) from the first area 614a, the first visual indicator 622 (e.g., LED 622a) is set to the first brightness b1. As shown in Fig. As shown in Figure 10B, the electronic device 600, in accordance with the finding that the user's gaze 618 has a second distance d2 from the first area 614a, which is smaller than the first distance d1, can adjust the first visual indicator to a second brightness b2, which is greater than the first brightness b1. The electronic device 600 can thus illuminate the visual indicator (e.g., LED 622) with a greater brightness when the user's gaze 618 approaches the area 614 corresponding to the visual indicator (e.g., LED 622).

[0078] Furthermore, the electronic device can reduce the brightness of an LED 622n when the gaze 618 moves away from the area 614n connected to the LED 622n. For example, the electronic device 600, as in Fig. As shown in Figure 10D, in accordance with the finding that the user's gaze 618 is at a distance d4 from the first region 614a that is greater than the distance d1, the first visual indicator (e.g., LED 622a) is set to a brightness b4 that is lower than the first brightness b1. The electronic device 600 can thus illuminate the LED 622n at a lower brightness when the user's gaze 618 moves away from the area 614n corresponding to the LED 622n.

[0079] In some examples, the electronic device, based on the determination that the user's attention is directed to the first area (e.g., area 614a), changes the brightness 644 of the first visual indicator (e.g., LED 622a) from a first brightness to a second brightness that is greater than the first brightness. As in Fig. As shown in 10A, the user's attention (e.g., based on gaze 618) is not directed towards an area 614 (e.g., area 614a). Accordingly, the brightness 644 of LED 622a is b1. Fig. 10C, where the user's attention (e.g., based on gaze 618) is directed toward area 614a, the brightness of LED 622a is displayed at b3, which is higher than b1. The electronic device 600 thus increases the brightness of LED 622a when attention (e.g., based on gaze 618) is directed toward area 614a associated with LED 622a. Similarly, upon detection that attention (e.g., based on gaze 618) is directed toward one of areas 614a-614d, the electronic device 600 can increase the brightness of the visual indicator 622 (e.g., LED 622a-622d) associated with the area toward which attention is directed.

[0080] Furthermore, in some examples, the color of a visual indicator (e.g., LED 622a) changes when the electronic device 600 detects that attention (e.g., based on gaze 618) is directed toward the corresponding area (e.g., area 614a). Therefore, based on the detection that the first context is present on the electronic device 600 and that the user's attention is directed toward the first area among the multitude of areas, the electronic device 600 changes the color of the first visual indicator from a first color to a second color.

[0081] For example, in Fig. 10A and Fig. As shown in Figure 10B, the electronic device 600 sets the color of the LED 622a associated with the area 614a to green (e.g., the first color) when the electronic device represents a first context (e.g., a context that includes the media player interface 632) and the user's gaze 618 is not directed at the area 614a. Fig. 10C, where the user's gaze 618 is directed towards area 614a, the electronic device 600 sets the color of the LED 622a connected to area 614a to yellow (e.g. the second color).

[0082] In some examples, the color of a visual indicator 622 can change with the corresponding operation (which, as previously described, can change depending on the context). Therefore, based on the determination that the second context is present on the electronic device 600 and the user's attention is directed to the first area of ​​the plurality of areas, the electronic device 600 can change the color of the first visual indicator to a third color that differs from the second color. For example, the electronic device 600 can change the color of the first visual indicator (e.g., LED 622a) to blue (a third color) when it is determined that the second context (e.g., the home screen 634, as in Fig. 7A - 7C shown) is present on the electronic device 600 and the user's attention (e.g. based on the view 618) is directed to the first area 614a of the multitude of areas.

[0083] By displaying context-related action prompts and performing context-related actions based on the captured gaze in an area of ​​the display, as described above, the number of inputs and / or input types required to operate the electronic device is reduced, which improves the navigation and flexibility of the user interface, which in turn increases the efficiency of the user's interaction with the electronic device and conserves the computing resources of the electronic device.

[0084] It is understood that although the different functions described above are described separately with regard to different electronic devices, in some examples some and / or all of the described functions can be implemented together in the same electronic device.

[0085] It is understood that the examples shown and described here are merely illustrative and that additional and / or alternative elements can be provided within the three-dimensional environment to automatically adapt input detected by an intelligent input device to fulfill the user's intent, even if the input is already assigned to another action; to dampen scrolling input based on gaze direction; to execute different actions based on the speed of scrolling input; and / or to display context-sensitive hints about actions that can be performed when gaze is detected by the hints. It is understood that the appearance, shape, form, and size of the various user interface elements and objects shown and described herein are exemplary and that alternative appearances, shapes, and / or sizes may be provided.For example, the virtual objects representing application user interfaces (e.g., the media player interface 314) may be provided in shapes other than those shown, such as rectangular, circular, triangular, etc. In some examples, the various selectable offerings (e.g., first and second selectable options 324 and 326 and / or movie lists 414 and 514) described herein may be selected verbally via verbal user commands (e.g., the verbal command "select option" or "select virtual object"). Additionally or alternatively, in some examples, the various options, user interface elements, controls, etc., described herein may be selected and / or manipulated by user input received via one or more input devices communicating with the electronic device(s).Selection input can be received, for example, via physical input devices such as a mouse, trackpad, keyboard, etc., which communicate with the electronic device(s), or via a physical button that is integrated into the electronic device(s).

[0086] Fig. Figure 11 illustrates an exemplary flowchart of a method 1100 according to an example of the disclosure. In some examples, the method 1100 begins with an electronic device that communicates with one or more displays and one or more input devices. In some examples, the electronic device is a head-mounted display that communicates with the electronic device 101. Fig. 1 and / or the device 201 from Fig. 2A is similar to or corresponds to. As in Fig. As shown in Figure 11, in some examples, when displaying a first user interface in which a first operation is assigned to a first input type, the electronic device detects a first input via a first input device of one or more input devices (1102), wherein the first input corresponds to a second input type that is different from the first input type. For example, when displaying a media player interface 314 in which a tap input is assigned a "pause" operation (e.g., to interrupt playback), the electronic device (e.g., the electronic device 300) can detect a double-tap input via the touch-sensitive surface 316, as shown in Figure 11. Fig. 3B shown.

[0087] In some examples, (1104) the electronic device determines an intention based on a current context, the current context including the initial user interface and a user's gaze direction of the electronic device. The current context may, for example, include the media player interface 314 and the gaze direction 318, as in the Fig. 3A and Fig. 3B shown.

[0088] In some examples, the electronic device (1106), in accordance with the finding that the intention of the first input is a request to perform the first operation on the electronic device, performs the first operation on the electronic device in response to the recognition of the first input. As in Fig. As shown in Figure 3B, the electronic device 300 can, for example, determine that the intent behind the double-tap input is a request to perform the "pause" operation on the electronic device (e.g., instead of the "clear user interface" operation assigned to the double-tap input). In accordance with the determination that the intent of the double-tap input is a request to perform the "pause" operation, the electronic device 300, in response to the detection of the double-tap input, can perform the "pause" operation (e.g., pause playback) and thus override the input assignment or mapping of the double-tap input (e.g., "clear user interface") based on the context of the electronic device 300.

[0089] Fig. Figure 12 illustrates an exemplary flowchart of a method 1200 according to an example of the disclosure. In some examples, the method 1200 begins at an electronic device connected to one or more displays and one or more input devices. In some examples, the electronic device is a head-mounted display connected to the electronic device 101. Fig. 1 and / or the device 201 from Fig. 2A is similar to or corresponds to. As in Fig. As shown in Figure 12, (1202) in some examples, during the display of a three-dimensional environment with one or more user interface elements, the electronic device recognizes, via a first input device of the one or more input devices, a scroll input corresponding to a request to scroll the one or more user interface elements. As shown, for example, in the Fig. 4A and Fig. As shown in Figure 4B, the electronic device 400 recognizes, via one or more input devices (e.g. the touch-sensitive surface 416), a scroll input 402 (e.g. a swipe gesture) which corresponds to a request to scroll the movie list 414.

[0090] In some examples, in response to the detection of scroll input and in accordance with a finding that a user's attention to the electronic device is not focused on the one or more user interface elements, the electronic device may scroll the one or more user interface elements at a first speed (1204). As in Fig. As shown in Figure 4A, in some examples attention is based on gaze, for example, the gaze 418 of the user. In accordance with the finding that the gaze 418 of the user of the electronic device is not directed at the film list 414, the electronic device 400 scrolls the film list 414 at a first scrolling speed 428.

[0091] In some examples, the electronic device (1206) scrolls the one or more user interface elements at a second speed that is slower than the first speed, in accordance with an observation that the user's attention to the one or more user interface elements is focused on the electronic device. As in Fig. As shown in Figure 4B, the electronic device 400 scrolls the film list 414 in accordance with a finding that the gaze 418 of the user of the electronic device is not directed at the film list 414, with a second scrolling speed 432 which is slower than the first scrolling speed 428.

[0092] Fig. Figure 13 illustrates an exemplary flowchart of a method 1300 according to an example of the disclosure. In some examples, the method 1300 begins at an electronic device connected to one or more displays and one or more input devices. In some examples, the electronic device is a head-mounted display connected to the electronic device 101. Fig. 1 and / or the device 201 from Fig. 2A is similar to or corresponds to. As in Fig. As shown in Figure 13, (1302) in some examples, the electronic device recognizes, during the display of a three-dimensional environment with one or more user interface elements, a scroll input corresponding to a request to scroll the one or more user interface elements via a first input device of the one or more input devices. As shown, for example, in the Fig. As shown in Figures 5A-5C, the electronic device 500 recognizes, via one or more input devices (e.g., the touch-sensitive surface 516), a scroll input 502 (e.g., a swipe gesture) which corresponds to a request to scroll the movie list 514.

[0093] In some examples, (1304) the electronic device scrolls through one or more user interface elements in response to the detection of scroll input in accordance with a finding that the scroll input speed is below an input speed threshold. For example, in Fig. As shown in Figure 5A, the electronic device 500 scrolls through the film list 514 in accordance with a finding that the speed 524 of the scroll input 502 (e.g. a swipe gesture) is below an input speed threshold 526.

[0094] In some examples (1306), the electronic device, in response to the detection of scroll input, stops displaying one or more user interface elements after determining that the scroll input speed reaches or exceeds the input speed threshold. For example, in Fig. As shown in Figures 5B-5C, the electronic device 500 terminates the display of the movie list 414 when it is detected that the speed 528 of the scroll input 502 (e.g., a swipe gesture) is at or above the input speed threshold 526. Fig. 5B The electronic device 500 terminates the display of the film list 414 by scrolling the film list 414 out of one or more displays 512 (right). In Fig. 5C has finished displaying the film list 514 on the electronic device 500.

[0095] Fig. Figure 14 illustrates an exemplary flowchart of a method 1300 according to an example of the disclosure. In some examples, the method 1400 begins with an electronic device connected to one or more displays having a plurality of areas and one or more input devices. In some examples, the electronic device is a head-mounted display connected to the electronic device 101. Fig. 1 and / or the device 201 from Fig. 2A resembles or corresponds to. As in the Fig. As shown in 6A - 6D, an area 614 can correspond to the multitude of areas of a corner of one or more displays 612.

[0096] In some examples, the electronic device performs a first operation on the electronic device in accordance with a finding that a first context exists at the electronic device and that the attention of a user of the electronic device is directed to a first area of ​​the plurality of areas (1402). As in the Fig. As shown in 6A-6C, the first context in some examples may include a user interface such as a media player interface 632. In some examples, the user's attention is based on the user's gaze 618, which is in Fig. 6C is directed towards area 614a, where the display 616a-1 (e.g., "Home") is shown. In Fig. 6D the electronic device 600 performs the first operation (e.g. displaying the home screen 634) that is part of the area 614a (e.g. “Home”) Fig. 6C corresponds.

[0097] In some examples, the electronic device, after finding that a second context exists on the electronic device which is different from the first context, and the user's attention is directed to the first area of ​​the plurality of areas, performs a second operation on the electronic device (1404) which is different from the first operation. As in the Fig. As shown in Figures 7A-C, the second context in some examples can include the home screen 634, which is different from the media player interface 632. Accordingly, as shown by Figure 616a-2, the second operation (e.g., "Photos"), which corresponds to area 614a, is different from the first operation (e.g., "Home"). Fig. 7C is the user's view 618 directed towards area 614a and in Fig. 7D the electronic device 600 performs the second operation (e.g. opens the Photos app) which is from area 614a (e.g. “Photos”). Fig. 7C corresponds.

[0098] It is understood that processes or procedures 1100, 1200, 1300, and 1400 are examples and that more, fewer, or different operations can be performed in the same or a different sequence (e.g., within a process). Additionally, the operations described above in processes or procedures 1100, 1200, 1300, and 1400 are optionally implemented by one or more functional modules in an information processing device such as a general-purpose processor (e.g., with reference to...). Fig. 2A-2B described) or application-specific chips, and / or by other components of Fig. 2A-2B will be executed.

[0099] Therefore, in accordance with the foregoing, some examples of the disclosure relate to a method that includes: in an electronic device communicating with one or more displays and one or more input devices: while displaying a first user interface, wherein a first operation is assigned to a first input type, detecting a first input via a first input device of the one or more input devices, wherein the first input is of a second input type that is different from the first input type; determining an intent based on a current context, wherein the current context includes the first user interface and a viewing direction of a user of the electronic device; and in accordance with a finding,that the intention of the first input is a request to perform the first operation on the electronic device: performing the first operation on the electronic device in response to the recognition of the first input. In addition to or alternatively to one or more of the examples disclosed above, in some examples, in accordance with a finding that the intention of the first input is not a request to perform the first operation, the method may further include displaying a second user interface that includes a first selectable option for performing the first operation and a second selectable option for performing a second operation that is different from the first operation. In addition to or alternatively to one or more of the examples disclosed above, the method may further include, in some examples, recognizing a second input.which is directed to the first or second selectable option, and in response thereto, according to a finding that the second input is directed to the first selectable option, the execution of the first operation on the electronic device, and according to a finding that the second input is directed to the second selectable option, the execution of the second operation on the electronic device. In addition to or alternatively to one or more of the examples disclosed above, determining the intent in some examples may include determining a confidence level associated with the intent and, in accordance with the finding that the confidence level is above a confidence threshold,The method may include performing the first operation on the electronic device. In addition to or as an alternative to one or more of the examples disclosed above, in some examples, in accordance with a finding that the confidence level does not exceed the confidence threshold, the method may further include displaying a second user interface that includes a first selectable option for performing the first operation and a second selectable option for performing a second operation.

[0100] In accordance with the foregoing, some examples of the disclosure relate to a method that includes: in an electronic device communicating with one or more displays and one or more input devices: during the display of a three-dimensional environment with one or more user interface elements, the detection of a scroll input via a first input device of the one or more input devices, corresponding to a request to scroll the one or more user interface elements; and in response to the detection of the scroll input: in accordance with a finding that the attention of a user of the electronic device is not directed toward the one or more user interface elements, scrolling the one or more user interface elements at a first speed; and in accordance with a finding,that the user's attention of the electronic device is directed toward one or more user interface elements, scrolling of one or more user interface elements at a second speed that is slower than the first speed. In addition to or alternatively to one or more of the examples disclosed above, in some examples the user's attention may include the user's gaze, and wherein the finding that the user's attention is directed toward one or more user interface elements includes a finding that the user's gaze is directed toward one or more user interface elements. In addition to or alternatively to one or more of the examples disclosed above, the method may further include in some examples: in accordance with a finding that a degree of attention is a first degree,scrolling one or more user interface elements at the second speed; and in accordance with a finding that a level of attention is a second level that is higher than the first level, scrolling one or more user interface elements at a third speed that is slower than the first speed.

[0101] In accordance with the foregoing, some examples of the disclosure relate to a method that includes: in an electronic device communicating with one or more displays and one or more input devices: during the display of a three-dimensional environment with one or more user interface elements, the detection of a scroll input via a first input device of the one or more input devices, corresponding to a request to scroll the one or more user interface elements; and in response to the detection of the scroll input: scrolling the one or more user interface elements in accordance with a finding that the scroll input speed is below an input speed threshold; and in accordance with a finding that the scroll input speed reaches or exceeds the input speed threshold.Ending the display of one or more user interface elements. In addition to or alternatively to one or more of the examples disclosed above, scroll input in some examples may include a swipe gesture. In addition to or alternatively to one or more of the examples disclosed above, in some examples, scroll input at a speed below the input speed threshold may be a first scroll input, and scroll input at or above the input speed threshold may be a second scroll input, and one direction of the first scroll input may correspond to one direction of the second scroll input. In addition to or alternatively to one or more of the examples disclosed above, in some examples, ending the display of one or more user interface elements may involve displaying an animation of one or moreInclude user interface elements that move in one direction of scroll input.

[0102] In accordance with the foregoing, some examples of the disclosure relate to a method that includes: on an electronic device in communication with one or more displays having a plurality of areas and one or more input devices: in accordance with a finding that a first context exists on the electronic device and the attention of a user of the electronic device is directed to a first area of ​​the plurality of areas, performing a first operation on the electronic device; and in accordance with a finding that a second context exists on the electronic device which is different from the first context and the attention of the user of the electronic device is directed to the first area of ​​the plurality of areas, performing a second operation on the electronic device.which differs from the first operation. In addition to or alternatively to one or more of the examples disclosed above, the method may in some examples further include: in accordance with a finding that the first context is present on the electronic device and the user's attention is directed to a second area of ​​the plurality of areas, performing a third operation on the electronic device that differs from the first operation; and in accordance with a finding that the second context is present on the electronic device and the user's attention is directed to the second area of ​​the plurality of areas, performing a fourth operation on the electronic device.which differs from the second operation. In addition to or alternatively to one or more of the examples disclosed above, in some examples the method may further include, in accordance with a finding that the first context is present on the electronic device, displaying in the first area of ​​a first display corresponding to the first operation, and in accordance with a finding that the second context is present on the electronic device, displaying in the first area of ​​a second display that differs from the first display and corresponds to the second operation. In addition to or alternatively to one or more of the examples disclosed above, in some examples the electronic device may include a first visual indicator linked to the first area, and the method may further include, in accordance with the finding thatthat the user's attention is directed towards the first area, include changing the brightness of the first visual indicator from a first brightness to a second brightness that is greater than the first brightness. In addition to or alternatively to one or more of the examples disclosed above, in some examples the user's attention includes the user's gaze, and the method may further include: in accordance with a finding that the user's gaze is at a first distance from the first area, setting the brightness of the first visual indicator to the first brightness; and in accordance with a finding that the user's gaze is at a second distance from the first area that is less than the first distance,Adjusting the brightness of the first visual indicator to the second brightness level. In addition to or alternatively to one or more of the examples disclosed above, in some examples the first visual indicator has a color, and the method may further include: upon establishing that the first context is present on the electronic device and the user's attention is directed to the first area of ​​the plurality of areas, changing the color of the first visual indicator from a first color to a second color. In addition to or alternatively to one or more of the examples disclosed above, in some examples the method may further include: upon establishing that the second context is present on the electronic device and the user's attention is directed to the first area of ​​the plurality of areas, changing the color of the first visual indicator to a third color.which differs from the second color.

[0103] Some examples of the disclosure relate to an electronic device comprising: one or more processors; a memory; and one or more programs stored in the working memory and configured to be executed by the one or more processors, the one or more programs containing instructions for carrying out one of the above-mentioned procedures.

[0104] Some examples of the disclosure relate to a non-transitory, computer-readable storage medium that stores one or more programs, wherein the one or more programs comprise instructions which, when executed by one or more processors of an electronic device, cause the electronic device to perform one of the above-mentioned procedures.

[0105] Some examples of the disclosure relate to an electronic device comprising one or more processors, memory, and means for carrying out one of the foregoing methods.

[0106] Some examples of the disclosure relate to a non-transitory, computer-readable storage medium that stores one or more programs, wherein the one or more programs comprise instructions which, when executed by one or more processors of an electronic device, cause the electronic device to perform one of the above-mentioned procedures.

[0107] Some examples of the disclosure relate to an electronic device comprising one or more processors, memory, and means for carrying out one of the foregoing methods.

[0108] Some examples of the disclosure relate to an information processing device for use in an electronic device, the information processing device comprising means for carrying out one of the methods disclosed above.

[0109] The foregoing description was provided for the purpose of explanation with reference to specific examples. However, the preceding illustrative discussions are neither intended to be exhaustive nor to limit revelation to the exact forms revealed. In light of the foregoing teachings, many modifications and variations are possible. The examples were chosen and described to best explain the principles of revelation and their practical application, thus enabling other professionals to make the best possible use of revelation and the various examples described, with different modifications, as appropriate for their intended particular use. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 700.172

[0001] US 19 / 326,464

[0001]

Claims

[1] Procedure, encompassing: on an electronic device in conjunction with one or more displays and one or more input devices: during the display of a first user interface, wherein a first operation is assigned to a first input type, detection of a first input via a first input device of one or more input devices, wherein the first input corresponds to a second input type that differs from the first input type; Determining an intent based on a current context, wherein the current context includes the initial user interface and a user's viewing direction of the electronic device; and according to a finding that the intention of the first input is a request to perform the first operation on the electronic device: Performing the first operation on the electronic device in response to the detection of the first input. [2] Method according to claim 1, further comprising: Based on the finding that the intention of the first input is not to request the first operation, a second user interface is displayed, containing a first selectable option to perform the first operation and a second selectable option to perform a second operation that differs from the first operation. [3] Method according to any one of claims 1 to 2, further comprising: Capturing a second input directed at the first or second selectable option, and in response, according to a determination that the second input is directed at the first selectable option, performing the first operation on the electronic device, and Based on a finding that the second input is directed to the second selectable option, the second operation is performed on the electronic device. [4] Method according to any one of claims 1 to 3, wherein determining the intention includes determining a level of confidence associated with the intention, and: Based on a finding that the confidence level is above a confidence threshold, the first operation on the electronic device is performed. [5] The method of claim 4, further comprising: According to a determination that the confidence level does not exceed the confidence threshold, display a second user interface that includes a first selectable option to perform the first operation and a second selectable option to perform a second operation. [6] Procedure, comprehensive: on an electronic device in conjunction with one or more displays and one or more input devices: During the rendering of a three-dimensional environment that includes one or more user interface elements, detection via a first input device of the one or more input devices of a scroll input that corresponds to a request to scroll the one or more user interface elements and in response to the detection of scroll input: in accordance with a finding that the attention of a user of the electronic device is not focused on the one or more user interface elements, scrolling of the one or more user interface elements at a first speed and In accordance with a finding that the user's attention is focused on one or more user interface elements, scrolling of one or more user interface elements at a second speed that is slower than the first speed. [7] Method according to claim 6, wherein the user's attention includes a view of the user and wherein the finding that the user's attention is directed towards the one or more user interface elements includes a finding that the user's view is directed towards the one or more user interface elements. [8] The method of claim 6, wherein the method further comprises: In accordance with a finding that a level of attention is a first level, scrolling of one or more user interface elements at the second speed; and in accordance with a finding that an attention level is a second level that is higher than the first level, scrolling of one or more user interface elements at a third speed that is slower than the first speed. [9] Procedures, comprehensive: on an electronic device in conjunction with one or more displays and one or more input devices: During the rendering of a three-dimensional environment that includes one or more user interface elements, detection via a first input device of the one or more input devices of a scroll input that corresponds to a request to scroll the one or more user interface elements and in response to the detection of scroll input: in accordance with a finding that a scroll input speed is below an input speed threshold, scrolling of one or more user interface elements and In accordance with a finding that the scroll input speed reaches or exceeds the input speed threshold, the display of one or more user interface elements is terminated. [10] Method according to claim 9, wherein the scroll input includes a swipe gesture. [11] Method according to one of claims 9 to 10, wherein the scroll input at a speed below the input speed threshold is a first scroll input and the scroll input at a speed at or above the input speed threshold is a second scroll input and a direction of the first scroll input corresponds to a direction of the second scroll input. [12] Method according to any one of claims 9 to 11, wherein ending the display of one or more user interface elements includes displaying an animation of one or more user interface elements moving in a direction of scroll input. [13] Procedures, including: on an electronic device in conjunction with one or more displays with a multitude of areas and one or more input devices: in accordance with a finding that a first context exists on the electronic device and that a user's attention to the electronic device is directed to a first area of ​​the multitude of areas, performing a first operation on the electronic device and In accordance with the finding that a second context, different from the first context, exists on the electronic device and the user's attention is directed to the first area of ​​the multitude of areas, performing a second operation on the electronic device that differs from the first operation. [14] The method of claim 13, further comprising: in accordance with a finding that the first context is present on the electronic device and the user's attention is directed to a second area of ​​the multitude of areas, performing a third operation on the electronic device that differs from the first operation and In accordance with the finding that the second context is present on the electronic device and the user's attention is directed to the second area of ​​the multitude of areas, performing a fourth operation on the electronic device that differs from the second operation. [15] Method according to any one of claims 13 to 14, further comprising: in accordance with a finding that the first context is present on the electronic device, displaying a first display in the first area corresponding to the first operation, and In accordance with the finding that the second context is present on the electronic device, displaying a second display in the first area, which differs from the first display and corresponds to the second operation. [16] Method according to any one of claims 13 to 15, wherein the electronic device includes a first visual indicator linked to the first region, and the method further comprises: In accordance with the finding that the user's attention is directed towards the first area, changing the brightness of the first visual indicator from a first brightness to a second brightness that is greater than the first brightness. [17] Method according to claim 16, wherein the user's attention includes a glance by the user and wherein the method further comprises: In accordance with a finding that the user's gaze is located at a first distance from the first area, adjusting the brightness of the first visual indicator to the first brightness; and In accordance with a finding that the user's gaze is located at a second distance from the first area, which is smaller than the first distance, adjust the brightness of the first visual indicator to the second brightness. [18] Method according to any one of claims 16 to 17, wherein the first visual indicator has a color, and wherein the method further comprises: In accordance with a finding that the first context is present on the electronic device and the user's attention is directed to the first area of ​​the multitude of areas, changing the color of the first visual indicator from a first color to a second color. [19] The method of claim 18, further comprising: In accordance with a finding that the second context is present on the electronic device and the user's attention is directed to the first area of ​​the multitude of areas, changing the color of the first visual indicator to a third color that differs from the second color. [20] Electronic device comprising: one or more processors; a storage facility; and one or more programs that are stored in memory and configured to be executed by the one or more processors, wherein the one or more programs include instructions for carrying out a method according to any one of claims 1 to 19. [21] Non-transitory, computer-readable storage medium that stores one or more programs comprising instructions which, when executed by one or more processors of an electronic device, cause the electronic device to perform a method according to any one of claims 1 to 19.

Images