Gaze-based user interaction

Abstract

The present disclosure relates to gaze-based user interactions. In an exemplary process for interacting with a user interface object using eye gaze, an affordance associated with a first object is displayed. A gaze direction or a gaze depth is determined. When the gaze direction or the gaze depth is determined to correspond to a gaze on the affordance, a first input representing a user instruction to take an action on the affordance is received, and the affordance is selected in response to receiving the first input.

Description

[0001] This application is a divisional application of the Chinese patent application with international application number PCT / US2018 / 053428, international application date September 28, 2018, national application number 201880051530.5, and invention name “Gaze-based user interaction”. Technical Field

[0002] The present disclosure generally relates to user interfaces for interacting with electronic devices, and more particularly, to interacting with electronic devices using eye gaze. Background Art

[0003] Conventional electronic devices use input mechanisms such as keyboards, buttons, joysticks, and touch screens to receive input from users. Some conventional devices also include a screen that displays content in response to the user's input. Such input mechanisms and displays provide an interface for users to interact with the electronic device. Summary of the invention

[0004] The present disclosure describes the technology for using eye gaze to interact with electronic devices. According to some embodiments, the user uses his or her eyes to interact with the user interface object displayed on the electronic device. In some exemplary embodiments, the technology provides a more natural and more effective interface by allowing the user to operate the device mainly using eye gaze and eye gestures (e.g., eye movement, blinking and staring). It is also described to use eye gaze to quickly specify an initial position (e.g., for selecting or placing an object), and then move the specified position without using the technology of eye gaze, because due to the uncertainty and instability of the user's eye gaze position, it may be difficult to accurately locate the specified position using eye gaze. The technology can be applied to conventional user interfaces on devices such as desktop computers, laptop computers, tablet computers and smart phones. The technology is also advantageous for computer-generated reality (including virtual reality and mixed reality) devices and applications, as described in more detail below.

[0005] According to some embodiments, an enable representation associated with a first displayed object is displayed, and a gaze direction or a gaze depth is determined. A determination is made as to whether the gaze direction or the gaze depth corresponds to a gaze on the enable representation. Upon determining that the gaze direction or the gaze depth corresponds to a gaze on the enable representation, a first input representing an instruction to take an action on the enable representation is received, and the enable representation is selected in response to receiving the first input.

[0006] According to some embodiments, a first enable representation and a second enable representation are displayed simultaneously, and a first gaze direction or a first gaze depth of one or more eyes is determined. It is determined whether the first gaze direction or the first gaze depth corresponds to a gaze on both the first enable representation and the second enable representation. In response to determining that the first gaze direction or the first gaze depth corresponds to a gaze on both the first enable representation and the second enable representation, the first enable representation and the second enable representation are enlarged.

[0007] According to some embodiments, an electronic device is adapted to display a field of view of a computer-generated three-dimensional reality environment, and the field of view is presented from an observation perspective. A first object is displayed simultaneously with a second object, wherein the first object is presented closer to an observation position than the second object. A gaze position is determined. Based on determining that the gaze position corresponds to a gaze on the first object, a display of the second object is visually altered. Based on determining that the gaze position corresponds to a gaze on the second object, a display of the first object is visually altered.

[0008] According to some embodiments, a first user input is received at a first time. In response to receiving the first user input, a selection point is specified at a first location corresponding to a gaze location at the first time. While maintaining the specification of the selection point, a second user input is received. In response to receiving the second user input, the selection point is moved to a second location different from the first location, wherein moving the selection point to the second location is not based on the gaze location. While the selection point is at the second location, a third user input is received. In response to receiving the third user input, the selection point is confirmed at the second location.

[0009] According to some embodiments, a first user input is received at a first time. In response to receiving the first user input, a first object among a plurality of objects corresponding to a gaze position at the first time is specified. While maintaining the specification of the first object, a second user input is received. In response to receiving the second user input, the specification of the first object is stopped and a second object among a plurality of objects is specified, wherein the specification of the second object is not based on the gaze position. While maintaining the specification of the second object, a third user input is received. In response to receiving the third user input, the second object is selected.

[0010] According to some embodiments, an object is selected. While maintaining the selection of the object, a first user input is received at a first time. In response to receiving the first user input, a placement point is specified at a first location based on a gaze location at the first time, wherein the first location corresponds to the gaze location at the first time. While maintaining the specification of the placement point, a second user input is received. In response to receiving the second user input, the placement point is moved to a second location different from the first location, wherein moving the placement point to the second location is not based on the gaze location. A third user input is received, and in response to receiving the third user input, the selected object is placed at the second location. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.

[0012] Figure 1A and Figure 1B Depicted are exemplary systems for use in various computer-generated reality techniques, including virtual reality and mixed reality.

[0013] Figure 1C to Figure 1E An embodiment of the system in the form of a mobile device is shown.

[0014] Figure 1F to Figure 1H An embodiment of the system in the form of a head mounted display (HMD) device is shown.

[0015] Fig. 1I An embodiment of the system in the form of a heads-up display (HUD) device is shown.

[0016] Figure 2 A user is shown viewing an object according to various embodiments.

[0017] Figure 3 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0018] Figure 4 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0019] Figure 5 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0020] Figure 6 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0021] Figure 7 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0022] Figure 8 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0023] Fig. 9 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0024] Fig.10User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0025] Fig.11 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0026] Fig.12 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0027] Fig.13 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0028] Fig.14 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0029] Fig.15 User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0030] Fig.16 A flowchart is shown of an exemplary process for interacting with an electronic device using eye gaze according to various embodiments.

[0031] Fig.17 A flowchart is shown of an exemplary process for interacting with an electronic device using eye gaze according to various embodiments.

[0032] Fig.18 A flowchart is shown of an exemplary process for interacting with an electronic device using eye gaze according to various embodiments.

[0033] FIG. 19A to FIG. 19Y User interfaces for interacting with electronic devices using eye gaze are shown according to various embodiments.

[0034] Fig. 20 A flowchart is shown of an exemplary process for interacting with an electronic device using eye gaze according to various embodiments.

[0035] Fig.21 A flowchart is shown of an exemplary process for interacting with an electronic device using eye gaze according to various embodiments.

[0036] Fig. 22 A flowchart is shown of an exemplary process for interacting with an electronic device using eye gaze according to various embodiments. DETAILED DESCRIPTION

[0037] The following description sets forth exemplary methods, parameters, etc. However, it should be appreciated that such description is not intended to limit the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0038] This disclosure describes various embodiments of electronic systems and techniques for using such systems in connection with various computer-generated reality technologies, including virtual reality and mixed reality that incorporate sensory input from the physical environment.

[0039] The physical environment (or real environment) refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical items (or physical objects or real objects), such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0040] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0041] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object, which creates a 3D or spatial audio environment that provides the perception of a point audio source in a 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0042] Examples of CGR include virtual reality and mixed reality.

[0043] A virtual reality (VR) environment (virtual environment) refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment, and / or through a simulation of a subset of the person's physical movements within the computer-generated environment.

[0044] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0045] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0046] Examples of mixed reality include augmented reality and augmented virtuality.

[0047] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0048] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0049] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more features of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the position of the sun in the physical environment.

[0050] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, silicon-based liquid crystal, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0051] Figure 1A and Figure 1B An exemplary system 100 for use with various computer-generated reality technologies, including virtual reality and mixed reality, is depicted.

[0052] In some embodiments, such as Figure 1A As shown, system 100 includes device 100a. Device 100a includes various components, such as processor 102, RF circuit 104, memory 106, image sensor 108, orientation sensor 110, microphone 112, position sensor 116, speaker 118, display 120, and touch-sensitive surface 122. These components are optionally in communication via a communication bus 150 of device 100a.

[0053] In some embodiments, elements of system 100 are implemented in a base station device (e.g., a computing device such as a remote server, a mobile device, or a laptop computer), and other elements of system 100 are implemented in a head mounted display (HMD) device designed to be worn by a user, wherein the HMD device communicates with the base station device. In some embodiments, device 100a is implemented in either the base station device or the HMD device.

[0054] like Figure 1B As shown, in some embodiments, system 100 includes two (or more) devices in communication, such as through a wired connection or a wireless connection. The first device 100b (e.g., a base station device) includes a processor 102, an RF circuit 104, and a memory 106. These components optionally communicate through a communication bus 150 of the device 100b. The second device 100c (e.g., a head-mounted device) includes various components, such as a processor 102, an RF circuit 104, a memory 106, an image sensor 108, an orientation sensor 110, a microphone 112, a position sensor 116, a speaker 118, a display 120, and a touch-sensitive surface 122. These components optionally communicate through a communication bus 150 of the device 100c.

[0055] In some embodiments, system 100 is a mobile device, such as in Figure 1C to Figure 1E In some embodiments, the system 100 is a head mounted display (HMD) device, such as in the embodiment described with respect to the device 100a in FIG. Figure 1F to Figure 1H In some embodiments, the system 100 is a wearable HUD device, such as in the embodiment described with respect to device 100a in FIG. Fig. 1I In the implementation scheme described in device 100a.

[0056] System 100 includes processor 102 and memory 106. Processor 102 includes one or more general purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some embodiments, memory 106 is one or more non-transitory computer-readable storage media (e.g., flash memory, random access memory) storing computer-readable instructions configured to be executed by processor 102 to perform the following techniques.

[0057] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with electronic devices, networks (such as the Internet, an intranet), and / or wireless networks (such as cellular networks and wireless local area networks (LANs)). RF circuitry 104 optionally includes circuitry for communicating using near field communications and / or short range communications (such as ) circuit for communication.

[0058] System 100 includes a display 120. In some embodiments, display 120 includes a first display (e.g., a left eye display panel) and a second display (e.g., a right eye display panel), each display being used to display images to a corresponding eye of a user. Corresponding images are displayed simultaneously on the first display and the second display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby producing a parallax effect that provides the user with a stereoscopic effect of an object on the display. In some embodiments, display 120 includes a single display. For each eye of the user, the corresponding image is simultaneously displayed on a first area and a second area of ​​a single display. Optionally, the corresponding images include representations of the same virtual object and / or the same physical object from different viewpoints, thereby producing a parallax effect that provides the user with a stereoscopic effect of an object on a single display.

[0059] In some embodiments, system 100 includes a touch-sensitive surface 122 for receiving user input, such as tap input and swipe input. In some embodiments, display 120 and touch-sensitive surface 122 form a touch-sensitive display.

[0060] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more visible light image sensors (such as a charge coupled device (CCD) sensor) and / or a complementary metal oxide semiconductor (CMOS) sensor operable to obtain an image of a physical object from a real environment. The image sensor also optionally includes one or more infrared (IR) sensors, such as a passive IR sensor or an active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter, such as an IR point emitter, for emitting infrared light into the real environment. Image sensor 108 also optionally includes one or more event cameras configured to capture the movement of physical objects in the real environment. Image sensor 108 also optionally includes one or more depth sensors configured to detect the distance of physical objects from system 100. In some embodiments, system 100 uses CCD sensors, event cameras, and depth sensors in combination to detect the physical environment around system 100. In some embodiments, image sensor 108 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are optionally configured to capture images of physical objects in the real environment from two different perspectives. In some embodiments, the system 100 uses the image sensor 108 to receive user input, such as gestures. In some embodiments, the system 100 uses the image sensor 108 to detect the position and orientation of the system 100 and / or the display 120 in the real environment. For example, the system 100 uses the image sensor 108 to track the position and orientation of the display 120 relative to one or more fixed objects in the real environment.

[0061] In some embodiments, the system 100 includes a microphone 112. The system 100 uses the microphone 112 to detect sounds from the user and / or the user's real environment. In some embodiments, the microphone 112 includes a microphone array (including multiple microphones), which are optionally operated in series, such as to identify ambient noise or locate a sound source in the space of the real environment.

[0062] System 100 includes an orientation sensor 120 for detecting orientation and / or movement of system 110 and / or display 100. For example, system 100 uses orientation sensor 110 to track changes in position and / or orientation of system 100 and / or display 120, such as with respect to physical objects in a real environment. Orientation sensor 110 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0063] Figure 1C to Figure 1E An embodiment of the system 100 is shown in the form of an apparatus 100a. Figures 1C to 1E , device 100a is a mobile device, such as a cellular phone. Figure 1C A device 100a that performs virtual reality technology is shown. The device 100a displays a virtual environment 160 on a display 120, the virtual environment including virtual objects such as a sun 160a, a bird 160b, and a beach 160c. The displayed virtual environment 160 and the virtual objects of the virtual environment 160 (e.g., 160a, 160b, 160c) are all computer-generated images. Note that Figure 1C The virtual reality environment shown in does not include representations of physical objects from the real environment 180, such as the person 180a and the tree 180b, even though these elements of the real environment 180 are within the field of view of the image sensor 108 of the device 100a.

[0064] Figure 1D A device 100a is shown that performs mixed reality technology, particularly augmented reality technology, using pass-through video. The device 100a is displaying a representation 170 of a real environment 180 with virtual objects on a display 120. The representation 170 of the real environment 180 includes a representation 170a of a person 180a and a representation 170b of a tree 180b. For example, the device uses an image sensor 108 to capture images of the real environment 180, which are pass-through for display on the display 120. The device 100a covers a hat 160d on the head of the representation 170a of the person 180a, which is a virtual object generated by the device 100a. The device 100a tracks the position and / or orientation of physical objects relative to the position and / or orientation of the device 100a so that the virtual objects can interact with the physical objects of the real environment from the augmented reality environment. In this embodiment, device 100a takes into account the movement of device 100a and person 180a to display hat 160d as being on the head of representation 170a of person 180a even though device 100a and person 180a move relative to each other.

[0065] Figure 1EA device 100a that performs mixed reality technology, particularly augmented virtual technology, is shown. The device 100a displays a virtual environment 160 with representations of physical objects on a display 120. The virtual environment 160 includes virtual objects (e.g., the sun 160a, the bird 160b) and a representation 170a of a person 180a. For example, the device 100a uses an image sensor 108 to capture an image of a person 180 in a real environment 180a. The device 100a places the representation 170a of the person 180a in the virtual environment 160 to be displayed on the display 120. The device 100a optionally tracks the position and / or orientation of the physical object relative to the position and / or orientation of the device 100a so that the virtual object can interact with the physical object from the real environment 180. In this embodiment, the device 100a takes into account the movement of the device 100a and the person 180a to display the hat 160d as being on the head of the representation 170a of the person 180a. Notably, in this embodiment, when performing mixed reality techniques, device 100a does not display a representation of tree 180b even though tree 180b is also within the field of view of the image sensor of device 100a.

[0066] Figure 1F to Figure 1H An embodiment of the system 100 is shown in the form of a device 100a. Figures 1F to 1H As shown, device 100a is an HMD device configured to be worn on a user's head, with each eye of the user viewing a respective display 120a and 120b. Figure 1F A device 100a that performs virtual reality technology is shown. The device 100a displays a virtual environment 160 on a display 120a and a display 120b, and the virtual environment includes virtual objects, such as a sun 160a, a bird 160b, and a beach 160c. The displayed virtual environment 160 and virtual objects (e.g., 160a, 160b, 160c) are computer-generated images. In this embodiment, the device 100a displays corresponding images on the display 120a and the display 120b at the same time. The corresponding images include the same virtual environment 160 and virtual objects (e.g., 160a, 160b, 160c) from different viewpoints, thereby producing a parallax effect that provides the user with a stereoscopic effect of the objects on the display. It should be noted that Figure 1F The virtual reality environment shown in does not include representations of physical objects from the real environment, such as person 180a and tree 180b, even though person 180a and tree 180b are within the field of view of the image sensor of device 100a when performing the virtual reality techniques.

[0067] Figure 1GA device 100a is shown that performs augmented reality technology using pass-through video. The device 100a is displaying a representation 170 of a real environment 180 with virtual objects on displays 120a and 120b. The representation 170 of the real environment 180 includes a representation 170a of a person 180a and a representation 170b of a tree 180b. For example, the device 100a uses an image sensor 108 to capture images of the real environment 120b, which are pass-through to be displayed on displays 180 and 120a. The device 100a overlays a computer-generated hat 160d (virtual object) on the head of the representation 170a of the person 180a to be displayed on each display 120a and 120b. The device 100a tracks the position and / or orientation of the physical object relative to the position and / or orientation of the device 100a so that the virtual object can interact with the physical object from the real environment 180. In this embodiment, device 100a takes into account the movement of device 100a and person 180a to display hat 160d as being on the head of representation 170a of person 180a.

[0068] Figure 1H A device 100a for performing mixed reality technology, particularly augmented virtual technology, using transparent video is shown. The device 100a displays a virtual environment 160 with representations of physical objects on displays 120a and 120b. The virtual environment 160 includes a representation 170a of a virtual object (e.g., the sun 160a, a bird 160b) and a person 180a. For example, the device 100a uses an image sensor 108 to capture an image of the person 180. The device 100a places the representation 170a of the person 180a in the virtual environment to be displayed on displays 120a and 120b. The device 100a optionally tracks the position and / or orientation of the physical object relative to the position and / or orientation of the device 100a so that the virtual object can interact with the physical object from the real environment 180. In this embodiment, the device 100a takes into account the movement of the device 100a and the person 180a to display the hat 160d as being on the head of the representation 170a of the person 180a. Notably, in this embodiment, when performing mixed reality techniques, device 100a does not display a representation of tree 180b even though tree 180b is also within the field of view of image sensor 108 of device 100a.

[0069] Fig. 1I An embodiment of the system 100 is shown in the form of an apparatus 100a. Fig. 1I In the embodiment, device 100a is a HUD device (eg, an eyeglass device) configured to be worn on a user's head, wherein each eye of the user views a corresponding head-up display 120c and 120d. Fig. 1IA device 100a is shown that performs augmented reality techniques using head-up displays 120c and 120d. The head-up displays 120c and 120d are (at least partially) transparent displays, thus allowing a user to view a real environment 180 in conjunction with the head-up displays 120c and 120d. The device 100a displays a virtual hat 160d (virtual object) on each head-up display 120c and 120d. The device 100a tracks the position and / or orientation of physical objects in the real environment relative to the position and / or orientation of the device 100a and relative to the position of the user's eyes so that the virtual objects can interact with physical objects from the real environment 180. In this embodiment, the device 100a takes into account the movement of the device 100a, the movement of the user's eyes relative to the device 100a, and the movement of the person 180a at the position on the displays 120c and 120d to display the hat 160d, so that the user appears to the hat 160d to be located on the head of the person 180a.

[0070] Reference now Figures 2 to 15 , describes exemplary techniques for interacting with an electronic device using eye gaze.

[0071] Figure 2 A top view of a user 200 is shown with his gaze focused on an object 210. The user's gaze is defined by the visual axis of each of the user's eyes. The direction of the visual axis defines the user's gaze direction, and the distance at which the axes converge defines the gaze depth. The gaze direction may also be referred to as a gaze vector or line of sight. Figure 2 , the gaze direction is in the direction of the object 210, and the gaze depth is the distance D relative to the user.

[0072] In some embodiments, the center of the user's cornea, the center of the user's pupil, and / or the center of rotation of the user's eyeball is determined to determine the location of the visual axis of the user's eye, and can therefore be used to determine the user's gaze direction and / or gaze depth. In some embodiments, the gaze depth is determined based on the convergence point of the visual axes of the user's eyes (or the location of the minimum distance between the visual axes of the user's eyes) or some other measurement of the focus of the user's eyes. Optionally, the gaze depth is used to estimate the distance at which the user's eyes are focused.

[0073] exist Figure 2 , rays 201A and 201B are cast along the visual axes of the left and right eyes of the user 200, respectively, and are optionally used in a so-called ray casting to determine the gaze direction and / or gaze depth of the user. Figure 2Also shown are cones 202A and 202B having angular ranges 203A and 203B, respectively. Cones 202A and 202B are also projected along the visual axes of the left eye and right eye of the user 200, respectively, and are optionally used to determine the user's gaze direction and / or gaze depth in the so-called cone projection. Due to factors such as eye movement, sensor movement, sampling frequency, sensor delay, sensor resolution, sensor misalignment, etc., the gaze direction and gaze depth cannot often be determined absolutely accurately or precisely. Therefore, in some embodiments, angular resolution or (estimated) angular error is associated with the gaze direction. In some embodiments, depth resolution is associated with gaze depth. Optionally, the angular range of the cone (e.g., the angular range 203A and 203B of cones 202A and 202B) represents the angular resolution of the user's gaze direction.

[0074] Figure 3 An electronic device 300 having a display 302 is depicted. The electronic device 300 displays a virtual environment 304 including a virtual object 306. In some embodiments, the environment 304 is a CGR environment (e.g., a VR or MR environment). In the illustrated embodiment, the object 306 is an affordance that the user 200 can interact with using gaze. In some embodiments, the affordance 306 is associated with a physical object (e.g., a device or other equipment that can be controlled via interaction with the affordance 306). Figure 3 Also shown is a view from above user 200, which shows the gaze direction of user 200. The visual axis of each of the user's eyes is extrapolated to the plane of the displayed representation of virtual environment 304, which corresponds to the plane of display 302 of device 300. Point 308 represents the gaze direction of user 200 on display 302.

[0075] like Figure 3 As shown, the gaze direction of user 200 corresponds to the direction of affordance 306. The term "affordance" refers to a graphical user interface object with which a user can interact. Examples of affordances include user interactive images (e.g., icons), buttons, and text (e.g., hyperlinks). Electronic device 300 is configured to determine the gaze direction of user 200. Device 300 captures data from a sensor directed toward the user and determines the gaze direction based on the data captured from the sensor. In some embodiments of presenting a three-dimensional representation of scene 300, such as below with respect to Figures 9 to 12 In the described embodiment, device 300 also (or alternatively) determines a gaze depth and whether the gaze depth corresponds to affordance 306. Optionally, determining whether the gaze depth corresponds to a depth of the affordance is based at least in part on a depth resolution of the gaze depth.

[0076] In the illustrated embodiment, the device 300 includes an image sensor 310 that points to the user 200 and captures image data of the user's 200 eyes. In some embodiments, the device 300 includes an event camera that detects event data from the user (e.g., the user's eyes) based on the detected light intensity changes over time and uses the event data to determine the gaze direction and / or gaze depth. Optionally, the device 300 uses both image data and event data (e.g., from an image sensor and an independent event camera or a sensor configured to capture image data and event data) to determine the gaze direction and / or gaze depth. Optionally, the device 300 uses ray casting and / or cone casting to determine the gaze direction and / or gaze depth.

[0077] Based on the gaze direction, device 300 determines that the gaze direction corresponds to enable representation 306 because the gaze direction is in the same direction as enable representation 306 (e.g., a ray or cone projected from the eye of user 200 at least partially intersects with enable representation 306 or is within an error margin of enable representation 306). Optionally, determining that the gaze direction corresponds to enable representation 306 is based at least in part on an angular resolution of the gaze direction. In some embodiments of presenting a three-dimensional representation of a scene, device 300 also (or alternatively) determines whether the gaze depth corresponds to the depth of enable representation 306. Optionally, determining whether the gaze depth corresponds to the depth of the enable representation is based at least in part on a depth resolution of the gaze depth. Optionally, enable representation 306 is also located at the gaze depth (or within a depth range based on the depth resolution of the gaze depth).

[0078] In some embodiments, the gaze direction and / or gaze depth are determined to continue to correspond to the gaze at the enable representation even when the gaze direction and / or gaze depth no longer overlaps with the enable representation (e.g., once the gaze direction and / or gaze depth are initially determined to correspond to the gaze at the enable representation, the gaze direction and / or gaze depth are considered to correspond to the gaze at the enable representation for at least a predetermined amount of time or for a predetermined amount of time after the user looks away from the enable representation).

[0079] Although the gaze direction is determined to correspond to the gaze at the affordance 306, the device 300 receives input (referred to as a "confirmation action") representing an instruction to take an action on the affordance corresponding to the first object. For example, upon determining that the user 200 is viewing affordance 306, a confirmation action is received.

[0080] In response to receiving the confirmation action, device 300 selects affordance 306. That is, affordance 306 is selected in response to a combination of the user viewing affordance 306 and providing a confirmation action. Confirmation actions are useful in preventing false positives (e.g., device 300 incorrectly determines that user 200 wishes to select or act on affordance 306). Non-limiting examples of confirmation actions include eye gestures, body gestures, voice inputs, controller inputs, or combinations thereof.

[0081] Examples of eye gestures include a single blink, multiple blinks, a predetermined number of blinks, a predetermined number of blinks within a predetermined amount of time, blinks of a predetermined duration (e.g., closing the eyes for one second), a blink pattern (e.g., one slow blink followed by two rapid blinks), blinking an eye, blinking a specific eye, a pattern of blinking an eye (e.g., left, right, left, each eye blinking for a specified duration), a predetermined eye movement (e.g., a quick look up), a "long" look or dwell (e.g., maintaining gaze in the direction of enable representation 306 (or in a direction corresponding to enable representation 306) for a predetermined amount of time continuously), or an eye movement that meets some other predetermined criteria.

[0082] Examples of gestures include placing a hand at a position corresponding to the position of affordance 306 (e.g., between the user and the display of affordance 306), waving, pointing motion (e.g., at affordance 306), or a gesture with a predefined motion pattern. In some embodiments, the gesture confirmation action depends on the position of the gesture (e.g., the gesture must be at a specific position). In some embodiments, the gesture confirmation action does not depend on the position of the gesture (e.g., the gesture is independent of position).

[0083] Examples of speech input include speech commands (e.g., "pick that up" or "turn on the light"). In some embodiments, the speech input explicitly identifies an object associated with affordance 306 (e.g., "select the box"). In some embodiments, the speech input does not explicitly identify an object associated with an affordance, but instead uses an otherwise ambiguous pronoun (e.g., "grab that") to refer to the object.

[0084] With respect to controller input, in some embodiments, the device 300 communicates with a controller that is configured to receive input via, for example, a button, a trigger, a joystick, a scroll wheel, a knob, a keyboard, or a touch-sensitive surface (e.g., a trackpad or a touch-sensitive display). In some embodiments, the controller and the device 300 are connected wirelessly or via a wired connection. Examples of controller input include the pressing of a button, the pulling of a trigger, the movement of a joystick, the rotation of a scroll wheel, the rotation of a knob, the pressing of a button on a keyboard, or a contact or gesture (e.g., a tap or a swipe) on a touch-sensitive surface.

[0085] In some embodiments, selecting enable representation 306 includes applying focus to enable representation 306. Optionally, device 300 provides an indication that enable representation 306 has been selected. In some embodiments, the indication includes an audio output (e.g., a beep), a visual indication (e.g., outlining or highlighting the selected enable representation), or a tactile output. Optionally, enable representation 306 remains selected for a predetermined amount of time (e.g., focus is maintained on enable representation 306 for a predetermined amount of time). Optionally, enable representation 306 remains selected until a deselection input is received. In some embodiments, the deselection input is the same input as the confirmation action. In some embodiments, the deselection input is an input different from the confirmation action. In some embodiments, the deselection input includes an eye gesture, a body gesture, a voice input, a controller input, or a combination or portion thereof, such as the exemplary inputs described above.

[0086] In some embodiments, enable representation 306 remains selected until an action associated with enable representation 306 (or an object associated with it) is performed. Figure 4 306. Upon selecting affordance 306, device 300 receives input (e.g., an eye gesture, a body gesture, a voice input, a controller input, or a combination or portion thereof, such as the exemplary inputs described above). In the illustrated example, the input includes user 200 changing the position of his eyes so that his gaze direction moves from position 308 to position 309 on display 302. Figure 4 Position 400 as shown. In response to receiving the input, device 300 performs an action associated with enable indication 306 based on the input. In some embodiments, in response to the input causing device 300 to select enable indication 306, an action associated with enable indication 306 is performed (e.g., selecting enable indication 306 includes performing the action associated with enable indication 306). Figure 4 In the example shown, device 300 moves affordance 306 according to a change in the gaze direction of user 200, and enables affordance 306 on display 302 from Figure 3 The indicator shown indicates that the position of 306 is translated upward and leftward to Figure 4 Position shown.

[0087] In addition to moving the affordance, exemplary actions include transforming the affordance or the representation of an object associated with the affordance (e.g., rotating, twisting, stretching, compressing, enlarging, and / or shrinking affordance 306) and changing the state of a device associated with the affordance (e.g., turning a light on or off). For example, in some embodiments, the affordance is a virtual dial associated with a thermostat. The user can select the virtual dial and then adjust the temperature of the thermostat. In some embodiments, some aspects of the position of the affordance (or an object associated with it) are automatically determined as the object is moved. For example, if a virtual picture frame that is initially lying flat on a horizontal surface is moved to a wall, the picture frame automatically rotates to a vertical orientation to lie flat against the wall.

[0088] Now turn to Figure 5 , which describes techniques related to parsing and selecting closely spaced objects. Figure 5 A virtual environment 500 is shown displayed on device 300. In some embodiments, environment 500 is a CGR environment (e.g., a VR or MR environment). Virtual environment 500 includes affordances 502 and affordances 504, each of which is associated with a corresponding box on top of a virtual table 506 and is displayed simultaneously on display 302. The dashed circle represents the gaze direction 508 of user 200 determined by device 300. The radius of the circle represents the angular uncertainty of gaze direction 508. Figure 5 As shown, gaze direction 508 overlaps with enable representation 502 and enable representation 504, indicating that user 200 is interested in one of the enable representations. Although gaze direction 508 is slightly more toward enable representation 502, the angular uncertainty of gaze direction 508 is greater than the angular spacing between enable representation 502 and enable representation 504, which prevents device 300 from determining with sufficiently high confidence that gaze direction 508 corresponds to a specific one of enable representation 502 and enable representation 504. In other words, device 300 cannot resolve with sufficient confidence which enable representation user 200 expects to select. Instead, device 200 determines that gaze direction 508 corresponds to both enable representation 502 and enable representation 504. In some embodiments of presenting a three-dimensional representation of a scene, the depth separation between the enable representations may be less than the angular resolution or depth resolution of the gaze position.

[0089] In response to determining that gaze direction 508 corresponds to both affordance 502 and affordance 504 , device 300 enlarges affordance 502 and affordance 504 . Figure 6 Affordance 502 and affordance 504 are shown after being magnified (e.g., expanded). Affordance 502 and affordance 504 appear as if they have been moved from the top of table 506 and positioned closer to user 200. Figure 6, enable representation 502 and enable representation 504 are enlarged by the same amount so that their relative sizes and positions remain the same (e.g., enable representation 502 continues to appear in front of enable representation 504). Enlarging enable representation 502 and enable representation 504 increases the angular range of enable representation 502 and enable representation 504 and increases the angular spacing between enable representation 502 and enable representation 504. Optionally, the amount of scaling is based on the size of the enable representation and / or the resolution of the gaze direction (e.g., enable representation 502 and enable representation 504 are enlarged so that enable representation 504 is a predetermined minimum size). In some embodiments, enable representation 502 and enable representation 504 are scaled so that device 300 can resolve (with a predetermined confidence level) which enable representation user 200 is attempting to focus on.

[0090] In some embodiments, enablement representation 502 and enablement representation 504 are enlarged based on determining that the gaze of user 200 meets predefined criteria (e.g., gaze direction 508 corresponds to enablement representation 502 and enablement representation 504 continuously for a predetermined amount of time or for a predetermined amount of time in a predefined time window (e.g., 3 seconds during a 4 second window)). In some embodiments, enablement representation 502 and enablement representation 504 are enlarged in response to device 300 receiving input (e.g., eye gestures, gestures, voice input, or controller input as described above) when gaze direction 508 corresponds to both enablement representation 502 and enablement representation 504. In this way, user 200 can have improved control over when the device zooms content. In addition, in this way, device 300 can reduce or limit the situations in which it invokes the zoom function to resolve gaze ambiguity for situations that require resolution, thereby reducing user stress and improving the user experience. Optionally, the affordance corresponding to the gaze direction is enlarged based on the input (e.g., a long button press and / or a heavy button press results in more enlargement than a short button press and / or a light button press). In some embodiments, the affordance is enlarged based on a voice command (e.g., "enlarge 40%"). This allows user 200 to have enhanced control over the zoom.

[0091] Enlarging affordances 502 and affordances 504 provides user 200 with an improved view of the affordances and allows user 200 to more easily and confidently select one of the affordances. Figure 6As shown in , after enable representation 502 and enable representation 504 are enlarged, user 200 decides that he wants to select enable representation 502 and moves his gaze to gaze direction 510 on enable representation 502. Notably, gaze direction 510 no longer overlaps with enable representation 504. Therefore, device 300 determines (e.g., with a relatively high degree of confidence) that gaze direction 510 corresponds to the direction of enable representation 502 (and does not correspond to the direction of enable representation 504). When gaze direction 510 is determined to correspond to the direction of enlarged enable representation 502, user 200 selects enable representation 502 using a confirmation action, such as one of the confirmation actions described above. Optionally, device 300 performs an action associated with enable representation 502 in response to and based on a confirmation input by user 200 for selecting to enlarge enable representation 502 and / or in response to further input when enabling representation 502 is selected. In response to confirmation input by user 200 for selecting to enlarge affordance 502, device 300 optionally reduces (eg, zooms out) affordance 502 and affordance 504 back to a previous state (eg, Figure 5 ). In some embodiments, affordance 502 remains selected after being reduced to a previous state.

[0092] In the above relative to Figures 5 and 6 In the described embodiment, device 300 only magnifies enable representation 502 and enable representation 504. In some embodiments, in addition to magnifying enable representation 502 and enable representation 504, device 300 also displays an enlarged view of at least a portion of the environment surrounding enable representation 502 and enable representation 504. Figure 7 An exemplary embodiment is shown in which device 300 determines a portion of the surrounding of virtual environment 500 and includes an affordance corresponding to gaze direction 508. The portion is designated by rectangle 700 and includes, for example, a portion of table 506 in addition to affordance 502 and affordance 504. Figure 8 As shown, in response to determining that gaze direction 508 corresponds to both affordance 502 and affordance 504, device 300 zooms in on the portion of virtual environment 500 designated by rectangle 700 that includes affordance 502 and affordance 504. Although a portion of virtual environment 500 is zoomed in along with affordance 502 and affordance 504, the portion of virtual environment 500 that is designated by rectangle 700 may still be zoomed in as described above with respect to Figures 5 and 6The selection is made and acts on the enablement representation. In addition, although the above-described embodiments relate to virtual environments, similar techniques may be applied to other CGR environments, including mixed reality environments. For example, in some embodiments, the device includes a transparent display that displays enablement representation 502 and enablement representation 504 superimposed on the user's real-time view of the physical environment. The device also includes a user sensor for capturing user eye data and a scene sensor for capturing an image of the physical environment in which enablement representation 502 and enablement representation 504 are displayed. In response to determining that the user's gaze direction corresponds to enablement representation 502 and enablement representation 504, the exemplary device captures data of at least the physical environment surrounding enablement representation 502 and enablement representation 504 and displays an enlarged representation (e.g., image) of the physical environment surrounding enablement representation 502 and enablement representation 504.

[0093] In the above-described embodiments, affordances 502 and affordances 504 are displayed as two-dimensional representations of a virtual environment. Figure 1F to Figure 1H The affordances are displayed in a three-dimensional (3D) representation of an environment on the virtual reality HMD 100a shown in FIG. Fig. 9 A 3D representation of a virtual environment 902 displayed on an HMD 900 is shown. In some embodiments, environment 902 is a CGR environment (e.g., a VR or MR environment). Virtual environment 902 includes an affordance 904 and an affordance 906. Affordance 904 has a first depth, and affordance 906 has a second depth greater than the first depth of affordance 904. Since virtual environment 902 is a 3D representation, device 900 determines a gaze position based on data captured from the user's eyes, and in the illustrated embodiment, the gaze position includes a gaze direction and a gaze depth. In some embodiments, determining a gaze position includes determining a gaze direction, but not necessarily a gaze depth. In some embodiments, determining a gaze position includes determining a gaze depth, but not necessarily a gaze direction.

[0094] exist Fig. 9 , the radius of the cylinder surrounding gaze location 908 represents the angular resolution of the gaze direction, and the length of the cylinder represents the depth resolution of the gaze depth (e.g., the uncertainty of the gaze depth). Based on the gaze direction, angular resolution, gaze depth, and depth resolution, device 900 determines whether the position of enable representation 904 and / or enable representation 906 corresponds to the gaze location. In some embodiments, device 900 determines whether the position of enable representation 904 and / or enable representation 906 corresponds to the gaze location based on the gaze direction (and optionally the angular resolution) regardless of the gaze depth, or based on the gaze depth (and optionally the depth resolution) regardless of the gaze direction.

[0095] In some embodiments, device 900 enhances display of the farther affordance (e.g., affordance 906) in response to determining that the gaze location corresponds to both affordance 904 and affordance 906. Fig.10 In the illustrated embodiment, enabling representation 906 is enhanced by making enabling representation 906 brighter relative to enabling representation 904 (e.g., by increasing the brightness of enabling representation 906, decreasing the brightness of enabling representation 904, or a combination of the two). In some embodiments, enhancing the enabling representation includes changing the visual appearance of the enabling representation itself (e.g., by making the enabling representation brighter or changing the color of the enabling representation). In some embodiments, enhancing the enabling representation includes weakening the visual appearance of other aspects of the environment (e.g., by making another enabling representation or the surrounding environment appear blurred). Similarly, in a 2D representation of a 3D environment, smaller objects or objects with larger depth values ​​in the 3D environment are optionally enhanced.

[0096] In some embodiments, in response to device 900 determining that gaze location 908 corresponds to both affordance 904 and affordance 906 (e.g., device 900 cannot resolve which affordance the user is looking at), device 900 magnifies affordance 904 and affordance 906. In some embodiments that provide a 3D representation, the affordance is magnified from the user's perspective by moving the affordance toward the user and displaying the affordance at a depth that appears closer to the user. Fig.11 Shows something like Figure 6 An embodiment of the illustrated embodiment in which enable representation 904 and enable representation 906 are enlarged (e.g., moved closer to the user) while maintaining their relative size and position. Fig.12 1 shows an embodiment in which affordance 904 and affordance 906 are enlarged and repositioned relative to each other so that affordance 904 and affordance 906 are displayed side by side at the same depth. It should be appreciated that similar techniques can also be applied to 2D representations of an environment. For example, in the above description with respect to Figure 5 When determining that gaze direction 508 corresponds to a gaze at both affordance 502 and affordance 504, affordance 502 and affordance 504 are optionally enlarged by different amounts relative to each other and / or repositioned relative to each other so that affordance 502 and affordance 504 are displayed side by side. Additionally, when affordance 904 and affordance 906 are enlarged, device 900 may also determine whether the updated gaze location corresponds to one of the enlarged affordances and reposition the affordances in a manner similar to the previous relative gaze location. Figure 3 to Figure 4 and Figures 6 to 8 The described techniques enable selection of representations and / or the performance of actions thereon.

[0097] Now turn to Fig.13, techniques for changing the display of an object based on its depth are described. Fig.13 Device 300 is shown again. Device 300 displays environment 1300 (eg, a CGR environment) while displaying object 1302 and object 1304. Fig.13 As shown, object 1302 is closer (e.g., has a smaller depth value) than object 1304. Fig.13 , object 1302 partially obscures the view of object 1304. Gaze location 1306 is located on object 1302. Gaze location 1306 optionally includes a gaze direction or a gaze depth, or both. According to any of the above techniques, device 300 optionally determines whether the gaze location corresponds to object 1302 and / or object 1304 based on the gaze direction or the gaze depth, or both.

[0098] Device 300 visually changes object 1302 and / or object 1304 based on whether gaze location 1306 corresponds to object 1302 or object 1304. Device 300 determines whether gaze location 1306 corresponds to object 1302 or object 1304 according to any of the above-mentioned techniques. In some embodiments, in response to determining that gaze location 1306 corresponds to object 1302, device 300 visually changes the display of object 1304; and in response to determining that gaze location 1306 corresponds to object 1304, device 300 visually changes the display of object 1302. For example, if the user focus determined by direction or depth or both is determined to be on one of the objects, the visual appearance of the other object is changed to emphasize the object on which the user has focus. Fig.14 , the device 300 determines that the gaze position 1306 corresponds to the object 1302, and in response, visually changes the object 1304 in a manner that emphasizes the object 1302 and / or de-emphasizes the object 1304. Examples of visually changing the object to de-emphasize the object include making the object appear blurry or unclear, reducing the resolution of the object, reducing the brightness of the object, reducing the contrast of the object, increasing the transparency of the object, and stopping displaying the object. In some embodiments, in response to receiving an input (e.g., an eye pose, a gesture, a voice input, or a controller input) and determining that the gaze position 1306 corresponds to the object 1302 or the object 1304, respectively, the device 300 visually changes the object 1302 or the object 1304. Optionally, the device 300 changes the object 1302 and the object 1304 according to determining that the direction of the two objects corresponds to the gaze direction (which indicates that one of the objects may block the other object, and it will be beneficial to distinguish the objects).

[0099] Optionally, the device 300 also visually changes the display of the object (e.g., object 1302) corresponding to the gaze position to enhance the appearance of the object. Examples of visually enhancing an object include making the object clearer, improving the resolution of the object, increasing the brightness of the object, increasing the contrast of the object, reducing the transparency of the object, highlighting the object, and making the object appear.

[0100] exist Fig.15 , the user has moved his gaze position from 1306 to position 1500 corresponding to object 1304. In response, device 300 visually changes object 1302 and returns object 1304 to Fig.13 The appearance originally shown in . Fig.15 In the illustrated embodiment, device 300 makes object 1302 semi-transparent so that the user can better see the object he is trying to focus on. Optionally, device 300 removes object 1302 to provide an unobstructed view of object 1304.

[0101] It should be recognized that the above Figures 2 to 15 The embodiments discussed are exemplary and are not intended to be limiting. For example, although the virtual environment is described Figures 2 to 12 , but the technology can be similarly applied to other CGR environments, including mixed reality environments.

[0102] Now turn to Fig.16 , which shows a flowchart of an exemplary process 1600 for interacting with an electronic device using eye gaze. Process 1600 can be performed using a user device (e.g., 100a, 300, or 900). The user device is, for example, a handheld mobile device, a head-mounted device, or a heads-up device. In some embodiments, process 1600 is performed using two or more electronic devices, such as a user device that is communicatively coupled to another device (such as a base device). In these embodiments, the operations of process 1600 are distributed between the user device and the other device in any manner. In addition, the display of the user device can be transparent or opaque. Process 1600 can be applied to CGR environments (including virtual reality environments and mixed reality environments), as well as enable representations corresponding to virtual objects or physical objects. Although in Fig.16 1600 are depicted in a particular order, but these blocks may be performed in other orders. In addition, one or more blocks of process 1600 may be partially performed, optionally performed, performed in combination with another (some) blocks, and / or additional blocks may be performed.

[0103] At box 1602, the device displays an enable representation associated with a first object (e.g., the displayed object).

[0104] At box 1604, the device determines a gaze direction or a gaze depth (e.g., of one or more eyes). In some embodiments, data is captured from a sensor pointed at the user, and the gaze direction or the gaze depth is determined based on the data captured from the sensor. In some embodiments, determining the gaze direction or the gaze depth includes determining the gaze direction. In some embodiments, determining the gaze direction or the gaze depth includes determining the gaze depth. Optionally, the gaze direction or the gaze depth is determined using ray casting or cone casting. Optionally, the angular range of the cone used for the cone casting is based on the angular resolution of the gaze direction.

[0105] At box 1606, the device determines whether the gaze direction or the gaze depth corresponds to a gaze at the enable representation. In some embodiments, determining that the gaze direction or the gaze depth corresponds to the depth of the enable representation includes determining that the gaze is directed toward the enable representation. In some embodiments, determining that the gaze is directed toward the enable representation is based at least in part on an angular resolution of the gaze direction. In some embodiments, determining that the gaze direction or the gaze depth corresponds to a gaze at the enable representation includes determining that the gaze depth corresponds to the depth of the enable representation. In some embodiments, determining that the gaze depth corresponds to the depth of the enable representation is based at least in part on a depth resolution of the gaze depth.

[0106] At box 1608, when the gaze direction or gaze depth is determined to correspond to a gaze at the enabling representation, the device receives a first input representing an instruction to take an action on the enabling representation corresponding to the first object. In some embodiments, the first input includes an eye gesture, a hand gesture, a voice input, and / or a controller input.

[0107] At box 1610, the device selects an affordance in response to receiving a first input. Optionally, when the affordance is selected, a second input is received, and an action associated with the selected affordance is performed in response to receiving the second input and according to the second input. In some embodiments, the second input includes an eye gesture, a hand gesture, a voice input, or an input on a controller.

[0108] Now turn to Fig.17, which shows a flowchart of an exemplary process 1700 for interacting with an electronic device using eye gaze. Process 1700 can be performed using a user device (e.g., 100a, 300, or 900). The user device is, for example, a handheld mobile device, a head-mounted device, or a heads-up device. In some embodiments, process 1700 is performed using two or more electronic devices, such as a user device that is communicatively coupled to another device (such as a base device). In these embodiments, the operations of process 1700 are distributed between the user device and the other device in any manner. In addition, the display of the user device can be transparent or opaque. Process 1700 can be applied to CGR environments (including virtual reality environments and mixed reality environments), as well as enable representations corresponding to virtual objects or physical objects. Although in Fig.17 1700, but these blocks may be executed in other orders. In addition, one or more blocks of process 1700 may be partially executed, optionally executed, executed in combination with another (some) blocks, and / or additional blocks may be executed.

[0109] At box 1702, the device displays a first enable representation and a second enable representation. Optionally, the first enable representation and the second enable representation are displayed simultaneously. In some embodiments, the first enable representation and the second enable representation are displayed together with a two-dimensional representation or a three-dimensional representation of an environment (e.g., a CGR environment) that includes the first enable representation and the second enable representation. Optionally, the first enable representation is displayed at a first depth in the three-dimensional representation of the environment, and the second enable representation is displayed at a second depth in the three-dimensional representation of the environment, where the first depth is different from the second depth.

[0110] At block 1704, the device determines a first gaze direction or a first gaze depth (e.g., of one or more eyes). In some embodiments, data is captured from a sensor directed toward the user, and the gaze direction or gaze depth is determined based on the data captured from the sensor. Optionally, the gaze direction or gaze depth is determined using ray casting or cone casting. In some embodiments, the angular range of the cone used for cone casting is based on the angular resolution of the gaze direction.

[0111] At box 1706, the device determines whether the first gaze direction or the first gaze depth corresponds to a gaze at both the first enable representation and the second enable representation. Optionally, in response to determining that the first gaze direction or the first gaze depth corresponds to both the first enable representation and the second enable representation, the display of the first enable representation is enhanced according to the first depth being greater than the second depth; and the display of the second enable representation is enhanced according to the second depth being greater than the first depth. In some embodiments, determining the gaze direction, and determining that the gaze direction or the gaze depth corresponds to both the first enable representation and the second enable representation includes determining that the gaze direction corresponds to both the first enable representation and the second enable representation. Optionally, determining that the gaze direction corresponds to both the first enable representation and the second enable representation is based at least in part on an angular resolution of the gaze direction. In some embodiments, determining the gaze direction or the gaze depth includes determining the gaze depth, and determining that the gaze direction or the gaze depth corresponds to both the first enable representation and the second enable representation includes determining that the gaze depth corresponds to both the first enable representation and the second enable representation. Optionally, determining that the gaze depth corresponds to both the first enable representation and the second enable representation is based at least in part on a depth resolution of the gaze depth.

[0112] At box 1708, in response to determining that the first gaze direction or the first gaze depth corresponds to the gaze at both the first enable representation and the second enable representation, the device enhances the first enable representation and the second enable representation. In some embodiments, the first enable representation and the second enable representation are enlarged based on determining that the user's gaze meets a predefined standard. In some embodiments, a third input is received, and in response to determining that the first gaze direction or the first gaze depth corresponds to both the first enable representation and the second enable representation and receiving the third input, the first enable representation and the second enable representation are enlarged. In some embodiments, the third input includes an eye gesture, a gesture, a voice input, or a controller input. In some embodiments, enlarging the first enable representation and the second enable representation includes displaying an enlarged view of at least a portion of the environment (e.g., a CGR environment) surrounding the first enable representation and the second enable representation. In some embodiments, the enlarged view of at least a portion of the environment surrounding the first enable representation and the second enable representation is a representation of a virtual environment. In some embodiments, the enlarged view of at least a portion of the environment surrounding the first enable representation and the second enable representation is a representation of a physical environment. In some embodiments, enlarging the first enable representation and the second enable representation includes displaying the first enable representation at a third depth in the three-dimensional representation of the environment and displaying the second enable representation at a fourth depth in the three-dimensional representation of the environment, wherein the third depth is the same as the fourth depth.

[0113] Optionally, after amplifying the first enable representation and the second enable representation, a second gaze direction or a second gaze depth is determined, and the second gaze direction or the second gaze depth is determined to correspond to the gaze at the first enable representation. When the second gaze direction or the second gaze depth is determined to correspond to the gaze at the first enable representation, a first input representing a user instruction to take an action on the first enable representation is received, and the first enable representation is selected in response to receiving the first input. Optionally, the first input includes an eye gesture, a hand gesture, a voice input, or a controller input.

[0114] In some embodiments, the first enable representation or the second enable representation is reduced in response to receiving the first input. Optionally, when the first enable representation is selected, a second input is received, and in response to receiving the second input, an action associated with the first enable representation according to the second input is performed. In some embodiments, the second input includes an eye gesture, a hand gesture, a voice input, or a controller input.

[0115] Now turn to Fig.18 , which shows a flowchart of an exemplary process 1800 for interacting with an electronic device using eye gaze. Process 1800 can be performed using a user device (e.g., 100a, 300, or 900). The user device is, for example, a handheld mobile device, a head-mounted device, or a heads-up device. In some embodiments, process 1800 is performed using two or more electronic devices, such as a user device that is communicatively coupled to another device (such as a base device). In these embodiments, the operations of process 1800 are distributed between the user device and the other device in any manner. In addition, the display of the user device can be transparent or opaque. Process 1800 can be applied to CGR environments (including virtual reality environments and mixed reality environments), as well as virtual objects, physical objects, and their representations. Although in Fig.18 1800 are depicted in a particular order, but these blocks may be performed in other orders. In addition, one or more blocks of process 1800 may be partially performed, optionally performed, performed in combination with another (some) blocks, and / or additional blocks may be performed.

[0116] The device is adapted to display a field of view of a computer-generated three-dimensional reality environment. The field of view is presented from an observation perspective. At block 1802, the device displays a first object and a second object. Optionally, the first object and the second object are displayed simultaneously. In some embodiments, the first object and the second object are displayed such that from an observation perspective, the first object appears (e.g., is presented) to be closer than the second object.

[0117] At block 1804, the device determines a gaze location (e.g., of one or more eyes). In some embodiments, data is captured from a sensor pointed at the user, and the gaze location is determined based on the data captured from the sensor. In some embodiments, the gaze location is determined using ray casting or cone casting. Optionally, the angular range of the cone used for cone casting is based on the angular resolution of the gaze direction.

[0118] At box 1806, the device determines whether the gaze position corresponds to a gaze at a first object or a second object. In some embodiments, the gaze direction is determined, and determining that the gaze position corresponds to a gaze at the first object or the second object includes determining that the gaze is directed to the first object or the second object. Optionally, determining whether the gaze is directed to the first object or the second object is based at least in part on the angular resolution of the gaze direction. In some embodiments, the gaze depth is determined, and determining that the gaze position corresponds to a gaze at the first object or the second object includes determining that the gaze depth corresponds to the depth of the first object or the second object (e.g., as presented in the field of view). Optionally, determining that the gaze depth corresponds to the depth of the first object or the second object is based at least in part on the depth resolution of the gaze depth.

[0119] At box 1808, the device visually changes the display of the second object according to the determination that the gaze position corresponds to the gaze at the first object. In some embodiments, the second object is changed in response to determining that the gaze position corresponds to the gaze at the first object and receiving input. The input optionally includes eye gestures, gestures, voice input, or controller input. Optionally, the device enhances the display (e.g., display resolution) of the first object. Optionally, after visually changing the second object, the device determines the second gaze position of the user, and according to determining that the second gaze position corresponds to the gaze at the second object, visually changes the display of the first object and displays the second object according to its initial appearance.

[0120] At box 1810, the device visually changes the display of the first object based on determining that the gaze position corresponds to the gaze at the second object. In some embodiments, the first object is changed in response to determining that the gaze position corresponds to the gaze at the second object and receiving input. The input optionally includes an eye gesture, a gesture, a voice input, or a controller input. Optionally, the device enhances the display of the second object (e.g., display resolution).

[0121] Executable instructions for performing the features of the above-described methods 1600, 1700, and / or 1800 are optionally included in a transient or non-transitory computer-readable storage medium (e.g., memory 106) or other computer program product configured to be executed by one or more processors (e.g., processor 102). In addition, some operations in method 1600 (e.g., block 1610) are optionally included in method 1700 and / or method 1800, some operations in method 1700 (e.g., block 1708) are optionally included in method 1600 and / or method 1800, and some operations in method 1800 (e.g., blocks 1806, 1808, and / or 1810) are optionally included in method 1600 and / or method 1700.

[0122] Steering FIG. 19A to FIG. 19Y , describes a technology that provides dual modality for selecting and / or placing objects (e.g., virtual objects, physical objects, and affordances corresponding to virtual objects and physical objects), for example, in a CGR environment. In a first mode (e.g., a "gaze engaged" mode), a location or object is initially specified based on the location of a user's gaze. After the initial specification, a second mode (e.g., a "gaze disengaged" mode) is used to move the specified location or specify a different object without gaze. The user can quickly move the gaze location, which makes it effective for quickly identifying general areas. However, as described above, there is uncertainty in the location of the user's gaze, which makes it difficult to use gaze to specify a precise location. Displaying a persistent visual indicator at an estimated gaze location may be ineffective for specifying a precise location because the indicator may distract the user and cause the user's gaze to follow the indicator rather than focus on the desired specified point. The dual modality technology allows the user to quickly make a rough initial specification and then make fine adjustments independent of gaze (e.g., based solely on manual input) to specify a specific point or object.

[0123] Fig.19A A user 200 is shown using a device 1900 to interact with a virtual environment 1902. In some embodiments, the environment 1902 is a CGR environment (e.g., a VR or MR environment). The device 1900 includes a virtual reality HMD 1900a and an input device 1900b. In some embodiments, the HMD 1900a is a virtual reality HMD of the device 100a (e.g., a virtual reality HMD in FIG. 190b). Figure 1F to Figure 1I ), and the input device 1900b communicates with the HMD 1900a (e.g., via the communication bus 150, such as Figure 1A to Figure 1B ). View 1902a shows a view of a virtual environment 1902 displayed to a user 200 on an HMD 1900a, and view 1902b shows a perspective view of the virtual environment 1902 including the user 200. Fig.19AAlso shown is an input device 1900b that includes a touch-sensitive surface 1904 (eg, Figure 1A to Figure 1B 1902) that allows the user 200 to provide input to interact with the virtual environment 1902. The device 1900 includes a sensor (e.g., an image sensor on the HMD 1900A) to determine the gaze 1906 (e.g., gaze direction and / or gaze depth) of the user 200 (as described above). In some embodiments, the device 1900 includes sensors configured to detect various types of user input, including (but not limited to) eye gestures, body gestures, and voice input. In some embodiments, the input device includes a controller configured to receive button input (e.g., up, down, left, right, enter, etc.).

[0124] Virtual environment 1902 includes a stack of photos 1908, including individual photos 1908a-1908e placed on table 1912. Gaze 1906 seen in view 1902b indicates that user 200 is viewing the stack of photos 1908. In some embodiments, the line representing gaze 1906 is not visible in virtual environment 1902, as shown, for example, in view 1902a.

[0125] like Fig.19A , device 1900 receives user input 1910a (e.g., a touch gesture on touch-sensitive surface 1904) while gaze 1906 is directed toward a stack of photos 1908. In some embodiments, user input 1910a includes eye gestures, body gestures, voice input, controller input, or a combination thereof in addition to or in lieu of a touch gesture on touch-sensitive surface 1904.

[0126] In some embodiments, the response to the user input 1910a depends on the characteristics of the user input 1910a. For example, based on determining that the user input 1910a is a first type of input (e.g., a tap on the touch-sensitive surface 1904), the stack of photos 1908 is selected, such as by the surrounding Fig.19B 1908. In some embodiments, device 1900 deselects the stack of photos 1908 in response to receiving further input (e.g., selecting an exit button).

[0127] Alternatively, based on determining that user input 1910a is a different type of input (e.g., a touch and hold on touch-sensitive surface 1904), photos 1908a-1908e are presented, such as Fig.19C , making it easier for user 200 to select a specific photo from stack 1908. Fig.19C, photos 1908a-1908e are moved from table 1912 and presented vertically and spread in the middle of the field of view of user 200. In response to receiving user input 1910a, photo 1908a in the left position is designated (e.g., initially selected). The designation of photo 1908a is indicated by a focus indicator 1914, which includes a thick border around photo 1908a. In some embodiments, focus indicator 1914 includes a pointer, cursor, point, ball, highlight, outline stroke, or artifact image that visually identifies the designated object. In some embodiments, device 1900 undesignates photo 1908a and returns photo 1908 to table 1912 in response to receiving further input (e.g., selecting an exit button or lifting of touch).

[0128] Fig.19B and Fig.19C The responses shown in FIG. 1 are based on gaze 1906, and more specifically, based on the gaze position of user 200 when making user input 1910a. Since the gaze position of user 200 is located on the stack of photos 1908, the stack of photos 1908 is selected ( Fig.19B ) or is specified and re-presented for further selection. When the response to the user input is based on gaze 1906, the device 1900 is in a gaze engagement mode, and the gaze 1906 of the user 200 is engaged with the user input. Fig.19A This is shown by the gaze 1906 shown by the solid line.

[0129] In some embodiments, the response to user input 1910a depends on whether gaze 1906 corresponds to more than one selectable object. In some embodiments, if there is ambiguity or uncertainty about the object to be selected, device 1900 does not confirm the selection. For example, device 1900 displays photos 1908a-1908e and specifies photo 1908a (e) based on determining that the location of gaze 1906 corresponds to multiple unresolvable selectable objects (e.g., a stack of photos 1908). Fig.19C ). In some such embodiments, based on determining that the location of gaze 1906 corresponds to only a single selectable object (e.g., as described below, Fig.19M 1918 ), device 1900 selects a single selectable object (e.g., rather than specifying an object or providing the ability to further refine the selection).

[0130] In the illustrated embodiment, in response to receiving user input 1910a, device 1900 also switches to a gaze-away mode, in which the response to the user input is not based on gaze 1906 of user 200, and gaze 1906 is away from further user input. Fig.19C This is shown by the gaze 1906 shown by the dotted line.

[0131] Steering Fig.19D When specifying a photo 1908a, device 1900 receives user input 1910b. Fig.19D In some embodiments, user input 1910b includes a swipe or drag gesture from left to right. In some embodiments, user input 1910b is a continuation of user input 1910a (e.g., user input 1910a includes contact maintained on touch-sensitive surface 1904, and user input 1910b includes movement of the contact). In some embodiments, user input 1910b includes pressing of a directional button or a verbal command (“move right”). In response to receiving user input 1910b, focus indicator 1914 is moved from photo 1908a according to user input 1910b specifying photo 1908b (e.g., in the direction thereof), such as Fig.19E as shown in .

[0132] Notably, since fixation 1906 is released, photo 1908b is designated in response to receiving user input 1910b, even though fixation 1906 was positioned on photo 1908a when user input 1910b was made. Focus indicator 1914 moves to a location (e.g., an object) that does not correspond to the location of fixation 1906. More generally, moving focus indicator 1914 to designate photo 1908b is not based on fixation 1906. In some embodiments, focus indicator 1914 is moved based solely on characteristics of user input 1910b (e.g., location, direction, speed, duration, etc.).

[0133] like Fig.19E As shown in FIG. 1 , gaze 1906 remains disengaged, and in response to receiving further user input 1910c while gaze 1906 is positioned on photo 1908a, focus indicator 1914 is moved from photo 1908b to designate photo 1908c, as shown in FIG. Fig.19F as shown in .

[0134] Steering Figure 19G , while designating photo 1908c, device 1900 receives user input 1910d (e.g., a click, double-click, or finger lift). In response to receiving user input 1910d, the currently designated object, i.e., photo 1908c, is selected. In response to receiving user input 1910d, focus indicator 1914 remains on photo 1908c, and other photos 1908a, 1908b, 1908d, and 1908e are returned to table 1912, as shown. Fig.19H In addition, in response to receiving user input 1910d, gaze 1906 of user 200 is reengaged.

[0135] Relative to FIG. 19A to FIG. 19HThe described techniques provide bimodal operation in which user 200 can effectively select a specific object (e.g., one of photos 1908) that would be difficult to distinguish using gaze alone (e.g., due to uncertainty in gaze position) when photos 1908 are stacked on table 1912. User 200 can use gaze 1906 to quickly specify a group of objects, and then use input independent of gaze 1906 to navigate the group of objects and select a specific object.

[0136] Steering Fig.19I , while maintaining the selection of photo 1908c, user 200 moves gaze 1906 to a position on wall 1916 in environment 1902 between photo 1908f and photo 1908g. In response to the movement of gaze 1906, photo 1908c is moved to a position corresponding to the gaze position. In some embodiments, photo 1908c is maintained at Fig.19I 1908c or is visually moved and / or modified (e.g., so as not to obstruct user 200's view of virtual environment 1902) until a placement location for photo 1908c is specified or selected, as described below.

[0137] In such Fig.19I While positioning photo 1908c as shown, device 1900 receives user input 1910e (e.g., a touch on touch-sensitive surface 1904). In response to receiving user input 1910e, a placement location for photo 1908c is specified based on the location of gaze 1906 at the time of user input 1910e. Fig.19J As shown in , in response to receiving user input 1910e, the selected photo 1908c is placed at the location of gaze 1906 and remains selected, and gaze 1906 is disengaged. In some embodiments, the placement location is indicated by a pointer, cursor, point, ball, highlight, outline stroke, or artifact image (e.g., of the object being placed).

[0138] In some embodiments, the response to the user input 1910e depends on the characteristics of the user input 1910e. In some embodiments, based on the user input 1910e including a first type of input (e.g., a touch on the touch-sensitive surface 1904), the device 1900 specifies a temporary placement location on the wall 1916 for the photo 1908c, the photo 1908c remains selected, and the gaze 1906 is disengaged, as described above; and based on the user input 1910e including a second type of input (e.g., a click on the touch-sensitive surface 1904), the photo 1908c is placed on the wall 1916, the photo 1908c is deselected, and the gaze 1906 is reengaged. Thus, by using different inputs, the user 200 can choose to specify a temporary placement location and maintain the selection of the photo 1908c to adjust the location through further input (as described below), or accept the gaze location as the placement location and deselect the photo 1908c.

[0139] return Fig.19J , while the photo 1908c remains selected and positioned at the initially designated location, the device 1900 receives user input 1910f comprising a downward swipe or drag gesture. In response to receiving the user input 1910f, the photo 1908c is moved downward according to the user input 1910f and independently of the location of the gaze 1906, as shown in FIG. Figure 19K According to the technique, user 200 can use gaze 1906 to quickly and roughly specify an initial placement location, and then make fine adjustments to the location that are independent of gaze. Once photo 1908c is in the desired location (e.g., aligned with photo 1908f and photo 1908g), user 200 provides input 1910g. In response to input 1910g, Figure 19K The position of the photo 1908c in FIG. 1 is selected as the final placement position, and the image 1906 is reattached. Figure 19L As shown in , in response to receiving user input 1910g, focus is removed from photo 1908c (photo 1908c is deselected), and photo 1908c remains in the selected placement location as user 200 moves gaze 1906 to a different location.

[0140] Now turn to Fig.19M ,like Figure 19L The virtual environment 1902 is configured as shown in , where a cup 1918 is added. Fig.19M , device 1900 receives user input 1910h (e.g., a touch on touch-sensitive surface 1904, a press of a button, or a body gesture) while gaze 1906 is positioned between cup 1918 and the stack of photos 1908. In response to receiving user input 1910h, a selection point represented by focus indicator 1920 is specified at a location corresponding to the location of gaze 1906, as shown in FIG. Fig.19N In some embodiments, focus indicator 1920 includes a pointer, cursor, dot, or sphere. In some embodiments, gaze 1906 is determined (e.g., measured or estimated) before user input (e.g., the most recently measured or estimated position) or after user input (e.g., in response).

[0141] In some embodiments, the response to user input 1910h is context-dependent. In some embodiments, the response is based on the content at the gaze location. For example, device 1900 may respond differently depending on whether an object, multiple unresolvable objects, a menu affordance, or no object was at the gaze location when user input 1910h was made. For example, if device 1900 determines with a predetermined amount of certainty that gaze 1906 corresponds to a stack of photos 1908, then as described in reference FIG. 19A to FIG. 19C Focus indicator 1914 is displayed as described rather than focus indicator 1920. In some embodiments, multiple objects include menu options associated with the objects (e.g., menu affordance 1924 described below).

[0142] In some embodiments, the response is based on whether the object is currently selected. For example, if the object is not currently selected, the device 1900 can operate in a selection mode and perform a selection action (e.g., selecting an object ( Fig.19B ), specify the selection point ( Fig.19N ), or display multiple objects for selection ( Fig.19C )); if an object is currently selected, the device 1900 may operate in a placement mode and perform a placement action (e.g., placing the object at the gaze location, or displaying an artifact image of the selected object at the specified placement location; for example, see the above FIG. 19H to FIG. 19J and the following Figures 19P to 19Q description).

[0143] return Fig.19N , displaying a focus indicator 1920 based on determining that there are multiple selectable objects corresponding to the location of gaze 1906 when user input 1910h is made. For example, if device 1900 cannot determine with sufficient certainty whether gaze 1906 corresponds to the stack of photos 1908 or cup 1918, focus indicator 1920 is displayed so that user 200 can indicate which object he wants to select. In the illustrated embodiment, user 200 wishes to select cup 1918 and provides user input 1910i, which includes a right-to-left swipe or drag gesture on touch-sensitive surface 1904 to move focus indicator 1920 to cup 1918. In response to receiving user input 1910i, the selection point is moved to a location corresponding to cup 1918, as shown. Fig.19OWhen the selection point is positioned on the cup 1918, the device 1900 receives a user input 1910j (e.g., a click). In response to receiving the user input 1910j, the selection point is confirmed and an object corresponding to the current position of the selection point is selected. Figure 19P As shown in , cup 1918 is selected, as indicated by focus indicator 1915 (eg, highlighted) around cup 1918, and gaze 1906 is reengaged.

[0144] like Figure 19P , the user 200 moves the gaze 1906 to the table 1922 while the cup 1918 remains selected. In the illustrated embodiment, the cup 1918 remains displayed in the same position (e.g., it does not move with the gaze 1906 even though the gaze 1906 is engaged).

[0145] In 1906 Figure 19P When positioning as shown, device 1900 receives user input 1910k. In response to receiving user input 1910k, the placement point is specified by a focus indicator 1920 located at a position corresponding to gaze 1906, and gaze 1906 is disengaged, as shown. Figure 19Q In some embodiments, the focus indicator 1920 includes a pointer, cursor, dot, ball, highlight, outline, or artifact image of a selected object (e.g., cup 1918).

[0146] In some embodiments, when cup 1918 is selected, the response to user input 1910k depends on whether there is more than one placement location that corresponds to the location of gaze 1906 when user input 1910k is made. Figure 19P In some embodiments, based on determining that the location of gaze 1906 corresponds to multiple possible selectable placement locations, a placement point is specified (e.g., if a variety of possible placement locations exist at or near the gaze location, the device 1900 does not confirm a placement location for the selected object). In some embodiments, based on determining that the location of gaze 1906 corresponds to only a single selectable placement location and in response to receiving user input 1910k, the device 1900 places the selected object at the gaze location, deselects the object, and then reengages gaze 1906.

[0147] Steering Figure 19Q , while specifying the placement point, the device 1900 receives user input 19101, which includes a diagonal swipe or drag gesture upward and to the right on the touch-sensitive surface 1904. In response to receiving the user input 19101, the placement point is moved according to the user input 19101, as shown by the focus indicator 1920 moving toward the center of the table 1922, as shown in FIG. Figure 19RIn response to receiving user input 1910m to confirm the placement point, the selected object (e.g., cup 1918) is placed at the confirmed placement point on the table 1922, as shown in FIG. Figure 19S Gaze 1906 is also reengaged in response to receiving user input 1910m.

[0148] Now turn to Figure 19T ,like Fig.19M The virtual environment 1902 is configured as shown, wherein a menu affordance 1924 is added, which is associated with objects (e.g., table 1912, photo 1908, and cup 1918) near the menu affordance 1924. Figure 19T , device 1900 receives user input 1910n (e.g., touch) while gaze 1906 is positioned on menu affordance 1924. In response to receiving user input 1910n, menu affordance 1924 is selected and gaze 1906 is disengaged from the user input, as shown in FIG. Figure 19U as shown in .

[0149] Selection of menu affordance 1924 causes display of menu options 1926a-1926d, which may be cycled through and selected via input independent of gaze 1906. Figure 19U As shown in , in response to selection of menu affordance 1924, menu option 1926a (select table) is initially designated using focus indicator 1928 (e.g., thick border).

[0150] like Figure 19V As shown in , device 1900 receives user input 1910o including a downward swipe or drag gesture. In response to receiving user input 1910o, focus indicator 1928 moves downward from menu option 1926a (select table) to menu option 1926b (select photo) according to user input 1910o, regardless of the location of gaze 1906, as shown in FIG. Figure 19W shown.

[0151] exist Figure 19W In FIG. 1 , device 1900 receives user input 1910p including an additional downward movement. In response to receiving user input 1910p, focus indicator 1928 moves downward from menu option 1926b (select photo) to menu option 1926c (select cup) according to user input 1910p, again without regard to the position of gaze 1906, as shown in FIG. Fig.19X shown.

[0152] exist Fig.19X In the embodiment, when specifying menu option 1926c, device 1900 receives user input 1910q. In response to receiving user input 1910q, an object corresponding to menu option 1926c is selected, such as Figure 19Y As shown. Figure 19Y In response to receiving user input 1910r, cup 1918 is selected and moved to the center of the field of view of user 200.

[0153] Furthermore, although the above reference FIG. 19A to FIG. 19Y The described embodiments relate to virtual environments, but similar techniques may be applied to other CGR environments, including mixed reality environments.

[0154] Now turn to Fig. 20 , which shows a flowchart of an exemplary process 2000 for interacting with an electronic device using eye gaze. Process 2000 can be performed using a user device (e.g., 100a, 300, 900, or 1900). The user device is, for example, a handheld mobile device, a head-mounted device, or a heads-up device. In some embodiments, process 2000 is performed using two or more electronic devices, such as a user device that is communicatively coupled to another device (such as a base device). In these embodiments, the operations of process 2000 are distributed between the user device and the other device in any manner. In addition, the display of the user device can be transparent or opaque. Process 2000 can be applied to CGR environments (including virtual reality environments and mixed reality environments) as well as virtual objects, physical objects, and representations corresponding to virtual objects and physical objects (affordance representations). Although in Fig. 20 2000 are depicted in a particular order, but these blocks may be performed in other orders. In addition, one or more blocks of process 2000 may be partially performed, optionally performed, performed in combination with another (some) blocks, and / or additional blocks may be performed.

[0155] At box 2002, the device receives a first user input (e.g., a contact on a touch-sensitive surface, a press of a button, or a body gesture) at a first time. At box 2004, in response to receiving the first user input, the device specifies a selection point at a first location based on a gaze location at the first time, where the first location corresponds to the gaze location at the first time. In some embodiments, the gaze location at the first time is determined (e.g., measured or estimated) before the first user input (e.g., the most recently measured or estimated location) or after the first user input (e.g., in response).

[0156] In some embodiments, a focus indicator is displayed at the gaze position. In some embodiments, the focus indicator includes a pointer, cursor, point, ball, highlight, outline stroke, or artifact image (e.g., of a specified or selected object). In some embodiments, the focus indicator specifies a selection point corresponding to an object at the gaze position. In some embodiments, the device disengages gaze from the user input in response to a first user input.

[0157] In some embodiments, the response to the first input is context-sensitive (e.g., the response is based on the content located at the gaze location (e.g., an object, multiple unresolvable objects, a menu affordance, or no object) or whether an object is currently selected). For example, if an object is not currently selected, the device operates in selection mode and performs a selection action (e.g., displaying multiple objects for selection), and if an object is currently selected, the device operates in placement mode and performs a placement action (e.g., displaying an artifact image of the selected object at a temporary placement location).

[0158] In some embodiments, a selection point is specified at the first position based on determining that the first position corresponds to a plurality of optional objects. In some embodiments, the plurality of objects are a group of closely spaced objects that cannot be resolved based on the user's gaze. In some embodiments, the plurality of objects are menu options associated with an object at the gaze position (e.g., a menu affordance). For example, if it is determined that there is ambiguity or uncertainty about the object to be selected, the device temporarily specifies a selection point instead of confirming the selection of the object. In some such embodiments, in response to receiving a first user input, the device selects a single optional object (e.g., rather than specifying a selection point at the first position) based on determining that the first position corresponds only to a single optional object.

[0159] In some embodiments, based on determining that the first user input is a first type of input (e.g., a touch on a touch-sensitive surface, a press of a button, or a body gesture), a selection point is specified at a first location. In some such embodiments, in response to receiving the first user input and based on determining that the first user input is a second type of input that is different from the first type of input (e.g., a click (as opposed to a touch) on a touch-sensitive surface, a press of a different button, or a different body gesture), the device confirms the selection point at the first location.

[0160] While maintaining the designation of the selection point, the device performs the operations of boxes 2006, 2008, 2010, and 2012. At box 2006, the device receives a second user input (e.g., movement of a contact on a touch-sensitive surface, or pressing of a directional button). At box 2008, in response to receiving the second user input, the device moves the selection point to a second position different from the first position, wherein moving the selection point to the second position is not based on the gaze position. For example, the device moves the focus indicator to a different object, selection point, or placement point. In some embodiments, the selection point is moved only based on the characteristics of the second input (e.g., position, direction, speed, duration, etc.). In some embodiments, the movement of the selection point is independent of (not based on) the gaze position. In some embodiments, the second position is different from the gaze position associated with the second user input.

[0161] In some embodiments, a first user input is received while the device is in a first mode (e.g., a gaze-engaged mode, in which a response to the user input is based on the user's gaze), and a selection point is specified at a first location in accordance with the device being in the first mode. In some such embodiments, in response to receiving the first user input, the device switches from the first mode to a second mode (e.g., a gaze-disengaged mode, in which a response to the user input is not based on the user's gaze, such that the gaze is disengaged from further input). In some such embodiments, a second input is received while the device is in a second mode, and the selection point is moved to a second location in accordance with the device being in the second mode.

[0162] At box 2010, the device receives a third user input (e.g., a click, double-click, or contact lifts off the touch-sensitive surface) while the selection point is in the second position. At box 2012, in response to receiving the third user input, the device confirms the selection point at the second position. In some embodiments, the device re-engages the input with the gaze upon confirmation (e.g., switches from a gaze-disengaged mode to a gaze-engaged mode). In some embodiments, the third input is received while the device is in the second mode (gaze-disengaged mode), and the selection point is confirmed at the second position based on the device being in the second mode.

[0163] In some embodiments, confirming the selection point selects an object corresponding to the location of the selection point (e.g., the second location). For example, in response to receiving the third user input, the device selects the object corresponding to the second location.

[0164] In some embodiments, confirming the selection point places the object at the location of the selection point. For example, before receiving the first user input, the device selects the object at a third location different from the second location, and in response to receiving the third user input, places the object at the second location. In some embodiments, before receiving the first user input, the device selects the object at a third location different from the second location, wherein the selection point is specified at the first location based on determining that the first location corresponds to multiple optional placement locations (e.g., if there is ambiguity or uncertainty about the location to be selected, the device does not confirm the placement location). In some such embodiments, in response to receiving the first user input and based on determining that the first location corresponds only to a single optional placement location, the device places the object at a single optional placement location (e.g., if there is only one option, the device bypasses temporarily specifying the placement location).

[0165] In some embodiments, the first user input is the same type of input as the second user input or the third user input. In some embodiments, a single tap or press of a button is used to specify a point or object, and another single tap or press of the same button is used to confirm the specified point or object. In some embodiments, the device decides which action to take based on the mode in which the device is operating (e.g., selection mode or placement mode).

[0166] Now turn to Fig.21 , which shows a flowchart of an exemplary process 2100 for interacting with an electronic device using eye gaze. Process 2100 can be performed using a user device (e.g., 100a, 300, 900, or 1900). The user device is, for example, a handheld mobile device, a head-mounted device, or a heads-up device. In some embodiments, process 2100 is performed using two or more electronic devices, such as a user device that is communicatively coupled to another device (such as a base device). In these embodiments, the operations of process 2100 are distributed between the user device and the other device in any manner. In addition, the display of the user device can be transparent or opaque. Process 2100 can be applied to CGR environments (including virtual reality environments and mixed reality environments) as well as virtual objects, physical objects, and representations corresponding to virtual objects and physical objects (affordance representations). Although in Fig.21 2100, but these blocks may be performed in other orders. In addition, one or more blocks of process 2100 may be partially performed, optionally performed, performed in combination with another (some) blocks, and / or additional blocks may be performed.

[0167] At box 2102, the device receives a first user input at a first time. At box 2104, in response to receiving the first user input, the device specifies a first object among multiple objects based on the gaze position (e.g., the positions of the multiple objects correspond to the gaze position at the first time). In some embodiments, a focus indicator specifies the first object. In some embodiments, multiple objects are highlighted or enlarged, or menu options corresponding to the menu affordances at the gaze position are displayed.

[0168] In some embodiments, the first object is specified based on determining that the gaze position at the first time corresponds to a gaze at more than one object (e.g., multiple objects). For example, if there is ambiguity or uncertainty about the object to be selected, the device does not confirm the selection. In some such embodiments, in response to receiving the first user input and based on determining that the gaze position at the first time corresponds to a gaze on only a single selectable object, the device selects the single selectable object.

[0169] In some embodiments, a first user input is received while the device is in a first mode (e.g., a gaze-engaged mode, in which a response to the user input is based on the user's gaze), and a first object is specified based on the electronic device being in the first mode. In some such embodiments, in response to receiving the first user input, the device switches from the first mode to a second mode (e.g., switches to a gaze-disengaged mode, in which a response to the user input is not based on the user's gaze, such that the gaze is disengaged from further input). In some such embodiments, a second input is received while the electronic device is in a second mode, and a second object is specified based on the electronic device being in the second mode.

[0170] In some embodiments, based on determining that the first user input is a first type of input (e.g., a touch on a touch-sensitive surface, a press of a button, or a body gesture), the first object is designated. In some such embodiments, in response to receiving the first user input and based on determining that the first user input is a second type of input different from the first type of input (e.g., a click (as opposed to a touch) on the touch-sensitive surface, a press of a different button, or a different body gesture), the device selects multiple objects.

[0171] While maintaining the designation of the first object, the device performs the operations of boxes 2106 and 2108. At box 2106, the device receives a second user input. At box 2108, in response to receiving the second user input, the device stops designating the first object and designates a second object among the plurality of objects (e.g., moves the focus indicator to a different object), wherein designating the second object is not based on the gaze position. In some embodiments, the second object is selected based only on the features of the second user input (e.g., position, direction, speed, duration, etc.). In some embodiments, the second user input is at a second time, and the position of the second object at the second time is different from the gaze position at the second time.

[0172] While maintaining the designation of the second object, the device performs the operations of blocks 2110 and 2112. At block 2110, the device receives a third user input. At block 2112, in response to receiving the third user input, the device selects the second object.

[0173] In some embodiments, after selecting the second object, the device receives a fourth user input at a second time. In response to receiving the fourth user input and based on determining that the fourth user input is an input of the first type, the device places the second object at the gaze position at the second time. In response to receiving the fourth user input and based on determining that the fourth user input is an input of a second type different from the first type of input, the device specifies a placement point corresponding to the gaze position at the second time. In some such embodiments, while maintaining the designation of the placement position, the device receives a fifth user input, and in response to receiving the fifth user input, places the second object at the current position of the placement point.

[0174] In some embodiments, the first user input is the same type of input as the second user input or the third user input. In some embodiments, a single tap or press of a button is used to specify the first object, and another single tap or press of the same button is used to select the second object.

[0175] Now turn to Fig. 22 , which shows a flowchart of an exemplary process 2200 for interacting with an electronic device using eye gaze. Process 2200 can be performed using a user device (e.g., 100a, 300, 900, or 1900). The user device is, for example, a handheld mobile device, a head-mounted device, or a heads-up device. In some embodiments, process 2200 is performed using two or more electronic devices, such as a user device that is communicatively coupled to another device (such as a base device). In these embodiments, the operations of process 2200 are distributed between the user device and the other device in any manner. In addition, the display of the user device can be transparent or opaque. Process 2200 can be applied to CGR environments (including virtual reality environments and mixed reality environments) as well as virtual objects, physical objects, and representations corresponding to virtual objects and physical objects (affordance representations). Although in Fig. 22 The blocks of process 2200 are depicted in a particular order, but these blocks may be performed in other orders. In addition, one or more blocks of process 2200 may be partially performed, optionally performed, performed in combination with another (some) blocks, and / or additional blocks may be performed.

[0176] At block 2202, the device selects an object (e.g., as described in process 2100). While maintaining the selection of the object, the device performs the operations of blocks 2204, 2206, 2208, 2210, 2212, and 2214. At block 2204, the device receives a first user input at a first time. At block 2206, in response to receiving the first user input, the device specifies a placement point at a first location based on a gaze location at the first time, wherein the first location corresponds to the gaze location at the first time.

[0177] In some embodiments, based on determining that the first user input is a first type of input (e.g., a touch on a touch-sensitive surface, a press of a button, or a body gesture), a placement point is specified at a first location. In some such embodiments, in response to receiving the first user input and based on determining that the first user input is a second type of input that is different from the first type of input (e.g., a click (as opposed to a touch) on the touch-sensitive surface, a press of a different button, or a different body gesture), the device places the selected object at the first location.

[0178] In some embodiments, based on determining that the first position corresponds to multiple selectable placement locations, a placement point is specified at the first position (e.g., the device does not confirm a placement location when there is ambiguity or uncertainty about the location to be selected for placement). In some such embodiments, in response to receiving the first user input and based on determining that the first position corresponds to only a single selectable placement location, the device places the selected object at the single selectable placement location.

[0179] While maintaining the designation of the placement point, the device performs the operations of blocks 2208, 2210, 2212, and 2214. At block 2208, the device receives a second user input. At block 2210, in response to receiving the second user input, the device moves the placement point to a second location that is different from the first location, wherein moving the placement point to the second location is not based on a gaze position (e.g., a gaze position when the second user input was made). In some embodiments, the second location is different from a gaze position associated with the second user input.

[0180] In some embodiments, a first user input is received while the electronic device is in a first mode (e.g., a gaze-engaged mode, in which a response to user input is based on a gaze position), and a placement point is specified at a first position in accordance with the electronic device being in the first mode. In some such embodiments, in response to receiving the first user input, the device switches from the first mode to a second mode (e.g., switches to a gaze-disengaged mode, in which a response to user input is not based on a user's gaze, such that gaze is disengaged from further input in response to the first input). In some such embodiments, a second user input is received while the electronic device is in a second mode, and the placement point is moved to a second position in accordance with the electronic device being in the second mode.

[0181] At box 2212, the device receives a third user input. At box 2214, in response to receiving the third user input, the device places the selected object at the second location and optionally deselects the object. In some embodiments, the first user input is the same type of input as the second user input or the third user input. In some embodiments, a single tap or press of a button is used to specify a placement point at a first location, and another single tap or press of the same button is used to place the selected object at a second location.

[0182] Executable instructions for performing the features of processes 2000, 2100, and / or 2200 described above are optionally included in a transitory or non-transitory computer-readable storage medium (e.g., memory 106) or other computer program product configured to be executed by one or more processors (e.g., processor 102). Some operations in process 2000 are optionally included in process 2100 and / or process 2200 (e.g., block 2004 and / or block 2008 are included in block 2104 and / or block 2108, respectively), some operations in process 2100 are optionally included in process 2000 and / or process 2200 (e.g., block 2202 includes block 2112), and some operations in process 2200 are optionally included in process 2000 and / or process 2100 (e.g., block 2112 includes block 2202). In addition, some operations in processes 2000, 2100 and / or 2200 (e.g., boxes 2004, 2008, 2104, 2108, 2206 and / or 2210) are optionally included in processes 1600, 1700 and / or 1800, and some operations in processes 1600, 1700 and / or 1800 (e.g., boxes 1604, 1606, 1704, 1706, 1804 and / or 1806) are optionally included in processes 2000, 2100 and / or 2200.

[0183] As described above, one aspect of the inventive technology involves using data about user gaze. In the inventive technology, gaze information can be used to benefit the user. For example, the user's gaze can be used to infer the user's attention to a particular portion of a computer-generated reality environment and allow the user to interact with specific objects in that portion of the field of view. However, some users may consider gaze information to be sensitive or personal in nature.

[0184] Entities that collect, use, transmit, store, or otherwise affect gaze information detected by CGR systems should comply with generally accepted privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Users' gaze information should be collected for the entity's lawful and reasonable uses and should not be shared or sold outside of those lawful uses. In addition, such collection / sharing should be conducted after receiving the user's informed consent. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such information data and ensure that other parties (if any) who have access to the gaze information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be tailored to the specific type of gaze information data being collected and / or accessed and appropriate to applicable laws and standards, including jurisdiction-specific considerations.

[0185] The present disclosure also contemplates implementation schemes in which users selectively block the use or access of gaze information. An entity implementing the present technology may decide whether to provide specific features while allowing users to "opt in" or "opt out" of participating in the use of gaze information at any time during registration for the service or thereafter. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of gaze information. For example, when downloading an application, users may be notified that their personal gaze data will be accessed. Users may also be transparently informed of why certain gaze information is being used to provide certain features. For example, in a virtual reality system that uses gaze information to determine where the user is looking, the user may be informed that their gaze information is being used to determine which field of view of the virtual environment the system should present, thereby enabling the user to make an informed decision about when to allow the use of gaze information.

[0186] Nonetheless, it is an object of the present disclosure that gaze information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risk can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., user names, device names, etc.), controlling the amount or specificity of the data stored (e.g., collecting mathematical coordinates of where a user is looking in a coordinate system, but avoiding collecting information about what is being viewed at the coordinates), controlling how the data is stored (e.g., locally), and / or other methods.

[0187] The foregoing descriptions of specific embodiments have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the scope of the claims to the precise forms disclosed, and it is to be understood that many modifications and variations are possible in light of the above teachings.

Claims

1. A method implemented on an electronic device in communication with one or more input devices, comprising: When the electronic device is in gaze engagement mode: receiving, via the one or more input devices, a first user input at a first time; determining a gaze location at the first time; In response to receiving the first user input, and based on determining that a plurality of possible selectable positions are at the gaze location at the first time: specifying a selection point at a first position of the plurality of possible selectable positions; and switching the electronic device from the gaze-engaged mode to a gaze-disengaged mode; and When the electronic device is in the gaze-off mode: while maintaining the designation of the selection point, receiving a second user input via the one or more input devices; in response to receiving the second user input, moving the selection point to a second position of the plurality of possible selectable positions, wherein the second position is different than the first position, and wherein moving the selection point to the second position is not based on the gaze position; receiving a third user input via the one or more input devices when the selection point is at the second position; as well as In response to receiving the third user input, the selection point is confirmed at the second location.

2. The method according to claim 1, further comprising: In response to receiving the third user input, an object corresponding to the second position is selected.

3. The method according to claim 1, further comprising: prior to receiving the first user input, selecting an object located at a third location different from the second location; as well as In response to receiving the third user input, the object is placed at the second location. The method of claim 1 , wherein the first user input is the same type of input as the second user input or the third user input. The method of claim 1 , wherein the second location is different than a gaze location associated with the second user input.

6. The method of claim 1 , wherein the selection point is specified at the first location based on determining that the first user input is a first type of input, and wherein the method further comprises: In response to receiving the first user input: Based on determining that the first user input is a second type of input different from the first type of input, the selection point is confirmed at the first location.

7. A computer-readable storage medium storing one or more programs, wherein the one or more programs are configured to be executed by one or more processors of an electronic device, wherein the one or more programs include instructions for: When the electronic device is in gaze engagement mode: Receiving a first user input at a first time; determining a gaze location at the first time; In response to receiving the first user input and based on determining a plurality of possible selectable positions at the gaze location at the first time: specifying a selection point at a first position of the plurality of possible selectable positions; and switching the electronic device from the gaze-engaged mode to a gaze-disengaged mode; and When the electronic device is in the gaze-off mode: receiving a second user input while maintaining the designation of the selection point; in response to receiving the second user input, moving the selection point to a second position of the plurality of possible selectable positions, wherein the second position is different than the first position, and wherein moving the selection point to the second position is not based on the gaze position; When the selection point is at the second position, receiving a third user input; and In response to receiving the third user input, the selection point is confirmed at the second location.

8. The computer-readable storage medium of claim 7, wherein the one or more programs further include instructions for: In response to receiving the third user input, an object corresponding to the second position is selected.

9. The computer-readable storage medium of claim 7, wherein the one or more programs further comprise instructions for: Prior to receiving the first user input, selecting an object at a third location different from the second location; and In response to receiving the third user input, the object is placed at the second location.

10. The computer-readable storage medium of claim 7, wherein the first user input is the same type of input as the second user input or the third user input.

11. The computer-readable storage medium of claim 7, wherein the second location is different than a gaze location associated with the second user input.

12. The computer-readable storage medium of claim 7, wherein the selection point is specified at the first location based on determining that the first user input is a first type of input, and wherein the one or more programs further include instructions for: In response to receiving the first user input: Based on determining that the first user input is a second type of input different from the first type of input, the selection point is confirmed at the first location.

13. An electronic device comprising: one or more processors; as well as A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions, the instructions being used to: When the electronic device is in gaze engagement mode: Receiving a first user input at a first time; determining a gaze location at the first time; In response to receiving the first user input, and based on determining that a plurality of possible selectable positions are at the gaze location at the first time: specifying a selection point at a first position of the plurality of possible selectable positions; and switching the electronic device from the gaze-engaged mode to a gaze-disengaged mode; and When the electronic device is in the gaze-off mode: receiving a second user input while maintaining the designation of the selection point; in response to receiving the second user input, moving the selection point to a second position of the plurality of possible selectable positions, wherein the second position is different than the first position, and wherein moving the selection point to the second position is not based on the gaze position; When the selection point is at the second position, receiving a third user input; as well as In response to receiving the third user input, the selection point is confirmed at the second location.

14. The electronic device of claim 13, wherein the one or more programs further include instructions for: In response to receiving the third user input, an object corresponding to the second position is selected.

15. The electronic device according to claim 13, wherein the one or more programs further include instructions for: Prior to receiving the first user input, selecting an object at a third location different from the second location; and In response to receiving the third user input, the object is placed at the second location. 16 . The electronic device of claim 13 , wherein the first user input is the same type of input as the second user input or the third user input.

17. The electronic device of claim 13, wherein the second location is different than a gaze location associated with the second user input.

Images