How to interact with objects in the environment

Abstract

A method for interacting with objects and user interface elements in a computer-generated environment provides an efficient and intuitive user experience. In some embodiments, a user can interact with an object directly or indirectly. In some embodiments, during indirect manipulation, manipulation of a virtual object is scaled. In some embodiments, during direct manipulation, manipulation of a virtual object is not scaled. In some embodiments, an object can be reconfigured from an indirect manipulation mode to a direct manipulation mode by moving the object to a distinct position in the three-dimensional environment in response to a distinct gesture.

Description

[Technical Field]

[0001] The present invention relates generally to methods for interacting with objects in a computer-generated environment. [Background technology]

[0002] A computer-generated environment is an environment in which at least some of the objects displayed for viewing by a user are generated using a computer, and the user can interact with the objects displayed in the computer-generated environment by moving the objects, rotating the objects, etc. Summary of the Invention

[0003] Some embodiments described in this disclosure are directed to methods for interacting with virtual objects in a computer-generated environment. Some embodiments described in this disclosure are directed to methods for performing direct and indirect manipulation of virtual objects. These interactions provide a more efficient and intuitive user experience. A full description of these embodiments is provided in the "Drawings" and "Detailed Description," and it should be understood that this "Summary" is not intended to limit the scope of the present disclosure in any way. [Brief explanation of the drawings]

[0004] For a better understanding of the various described embodiments, please refer to the following detailed description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0005] [Figure 1] 1 illustrates an electronic device displaying a computer-generated environment according to some embodiments of the present disclosure.

[0006] [Figure 2A] 1 illustrates a block diagram of an example architecture for one or more devices according to some embodiments of the present disclosure. [Figure 2B] 1 illustrates a block diagram of an example architecture for one or more devices according to some embodiments of the present disclosure.

[0007] [Figure 3] 1 illustrates a method for displaying a three-dimensional environment having one or more virtual objects according to some embodiments of the present disclosure.

[0008] [Figure 4A] 1 illustrates a method for indirectly manipulating a virtual object according to some embodiments of the present disclosure. [Figure 4B] 1 illustrates a method for indirectly manipulating a virtual object according to some embodiments of the present disclosure. [Figure 4C] 1 illustrates a method for indirectly manipulating a virtual object according to some embodiments of the present disclosure. [Figure 4D] 1 illustrates a method for indirectly manipulating a virtual object according to some embodiments of the present disclosure.

[0009] [Figure 5A] 1 illustrates a method for directly manipulating a virtual object according to some embodiments of the present disclosure. [Figure 5B] 1 illustrates a method for directly manipulating a virtual object according to some embodiments of the present disclosure. [Figure 5C] 1 illustrates a method for directly manipulating a virtual object according to some embodiments of the present disclosure. [Figure 5D] 1 illustrates a method for directly manipulating a virtual object according to some embodiments of the present disclosure.

[0010] [Figure 6A] 1 illustrates a method for moving a virtual object according to some embodiments of the present disclosure. [Figure 6B] 1 illustrates a method for moving a virtual object according to some embodiments of the present disclosure.

[0011] [Figure 7] FIG. 1 is a flow diagram illustrating a method for manipulating a virtual object according to an embodiment of the present disclosure.

[0012] [Figure 8] FIG. 10 is a flow diagram illustrating a method for moving a virtual object by an amount based on the virtual object's distance to a user, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] In the following description of the embodiments, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments that may be implemented. It is to be understood that other embodiments may optionally be utilized and structural changes may optionally be made without departing from the scope of the disclosed embodiments.

[0014] A person can interact with and / or sense a physical environment or world without the aid of an electronic device. The physical environment may include physical features, such as physical objects or surfaces. An example of a physical environment is a physical forest containing physical plants and animals. A person can directly sense and / or interact with a physical environment through various means, such as hearing, sight, taste, touch, and smell. In contrast, a person can use an electronic device to interact with and / or sense a wholly or partially simulated extended reality (XR) environment. For example, an XR environment may include mixed reality (MR) content, augmented reality (AR) content, virtual reality (VR) content, etc. An XR environment is often referred to herein as a computer-generated environment. In an XR system, some of a person's body movements or a representation thereof may be tracked, and properties of virtual objects simulated within the XR environment may be adjusted accordingly to behave according to at least one law of physics. For example, an XR system can detect movement of a user's head and adjust the graphical and auditory content presented to the user as if such views and sounds were changing in the physical environment. In another example, an XR system can detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment and adjust the graphical and auditory content presented to that user as if such views and sounds were changing in the physical environment. In some situations, an XR system can adjust a characteristic(s) of the graphical content in response to other inputs, such as expressions of body movement (e.g., voice commands).

[0015] Many different types of electronic devices can enable a user to interact with and / or sense an XR environment. A non-exclusive list of examples includes heads-up displays (HUDs), head-mounted devices, projection-based devices, windows or vehicle windshields with built-in display capabilities, displays formed as lenses placed over the user's eyes (e.g., contact lenses), headphones / earphones, input devices with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop / laptop computers. A head-mounted device may have one or more speaker(s) and an opaque display. Other head-mounted devices may be configured to accept an opaque external display (e.g., a smartphone). A head-mounted device may include one or more image sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted device may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium that directs light passing through it toward the user's eyes. The display may utilize various display technologies, such as uLED, OLED, LED, liquid crystal on silicon, laser scanning light source, digital light projection, or a combination thereof. The medium may be a light guide, an optical reflector, a holographic medium, an optical combiner, or a combination thereof. In some implementations, the transparent or translucent display can be selectively controlled to be opaque. A projection device may utilize retinal projection technology that projects an image onto the user's retina. A projection device may also project virtual objects into a physical environment (e.g., as a hologram or onto a physical surface).

[0016] FIG. 1 illustrates an electronic device 100 that can be configured to display a computer-generated environment, according to some embodiments of the present disclosure. In some embodiments, the electronic device 100 is a portable electronic device such as a tablet computer, a laptop computer, or a smartphone, among other possibilities. An exemplary architecture of the electronic device 100 is described in further detail with reference to FIGS. 2A-2B . FIG. 1 illustrates the electronic device 100 and a table 104A disposed within a physical environment 102. In some embodiments, the electronic device 100 is configured to capture and / or display an area of ​​the physical environment 102 that includes the table 104A (shown within the field of view of the electronic device 100). In some embodiments, the electronic device 100 is configured to display one or more virtual objects within the computer-generated environment that are not present within the physical environment 102 but are displayed within the computer-generated environment (e.g., placed on or otherwise secured to a top surface of a computer-generated representation 104B of the real-world table 104A). 1 , for example, an object 106 (e.g., a virtual object) not present in the physical environment is optionally displayed on the surface of table 104B in the computer-generated environment displayed via device 100 in response to detecting the plane of table 104A in physical environment 102. It should be understood that object 106 is a representative object, and that one or more different objects (e.g., of various dimensions, such as two-dimensional or three-dimensional objects) may be included and rendered in the two-dimensional or three-dimensional computer-generated environment. For example, a virtual object may include an application or user interface displayed in the computer-generated environment. In addition, it should be understood that a three-dimensional (3D) environment (or 3D object) described herein may be a representation of a 3D environment (or 3D object) displayed in a two-dimensional (2D) context (e.g., displayed on a 2D display screen).

[0017] 2A-2B show exemplary block diagrams of architecture for one or more devices according to some embodiments of the present disclosure. The blocks in FIG. 2A may represent information processing equipment for use in the device. In some embodiments, electronic device 200 is optionally a portable device such as a mobile phone, smartphone, tablet computer, notebook computer, or auxiliary device capable of communicating with other devices. 2A , device 200 includes various sensors (e.g., one or more hand-tracking sensor(s) 202, one or more location sensor(s) 204, one or more image sensor(s) 206, one or more touch-sensitive surface(s) 209, one or more movement and / or orientation sensor(s) 210, one or more eye-tracking sensor(s) 212, one or more microphone(s) 213 or other audio sensors), one or more display generating component(s) 214, one or more speaker(s) 216, one or more processor(s) 218, one or more memories 220, and / or communication circuitry 222. One or more communication buses 208 are optionally used for communication between the above-mentioned components of device 200.

[0018] Communications circuitry 222 optionally includes circuitry for communicating with electronic devices and networks such as the Internet, an intranet, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs), etc. Communications circuitry 222 optionally includes circuitry for communicating using short-range communications such as Near-Field Communication (NFC) and / or Bluetooth®.

[0019] The processor(s) 218 ​​optionally include one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors (DSPs). In some embodiments, the memory 220 is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage) that stores computer-readable instructions configured to be executed by the processor(s) 218 ​​to perform the techniques, processes, and / or methods described below. In some embodiments, the memory 220 includes two or more non-transitory computer-readable storage media. A non-transitory computer-readable storage medium may be any medium (other than a signal) that can tangibly store or carry computer-executable instructions used by or in connection with an instruction execution system, apparatus, or device. In some embodiments, the storage medium is a transient computer-readable storage medium. In some embodiments, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic, optical, and / or semiconductor storage devices, such as magnetic disks, CDs, DVDs, or optical discs based on Blu-ray technology, and persistent solid-state memory, such as flash and solid-state drives.

[0020] Display generating component(s) 214 optionally include a single display (e.g., a Liquid-Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or other type of display). In some embodiments, display generating component(s) 214 include multiple displays. In some embodiments, display generating component(s) 214 include a display having a touch-sensitive surface (e.g., a touchscreen), a projector, a holographic projector, a retinal projector, etc.

[0021] In some embodiments, device 200 includes touch-sensitive surface(s) 209 configured to receive user input (touch and / or proximity input), such as tap and swipe inputs or other gestures. In some embodiments, display generating component(s) 214 and touch-sensitive surface(s) 209 together form touch-sensitive display(s) (e.g., a touchscreen built into device 200 or a touchscreen external to device 200 that can communicate with device 200). It should be understood that device 200 optionally includes or receives input from one or more other physical user interface devices other than the touch-sensitive surface, such as a physical keyboard, a mouse, a stylus, and / or a joystick (or any other suitable input device).

[0022] The image sensor(s) 206 optionally include one or more visible light image sensors, such as a Charged Coupled Device (CCD) sensor, and / or a Complementary Metal-Oxide-Semiconductor (CMOS) sensor operable to acquire images of physical objects from the real-world environment. The image sensor(s) 206 optionally include one or more IR or NIR sensors, such as passive or active Infrared (IR) or Near Infrared (NIR) sensors, for detecting infrared or near-infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensor(s) 206 also optionally include one or more cameras configured to capture movement of physical objects in the real-world environment. Image sensor(s) 206 optionally include one or more depth sensors configured to detect the distance of a physical object from device 200. In some embodiments, information from the one or more depth sensors may enable the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some embodiments, the one or more depth sensors may enable the device to determine the texture and / or topography of an object in the real-world environment.

[0023] In some embodiments, device 200 detects the physical environment around device 200 using a combination of a CCD sensor, an event camera, and a depth sensor. In some embodiments, image sensor(s) 206 include a first image sensor and a second image sensor. The first image sensor and the second image sensor cooperate and are optionally configured to capture different information of physical objects in the real-world environment. In some embodiments, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some embodiments, device 200 uses image sensor(s) 206 to detect the position and orientation of device 200 and / or display generating component(s) 214 within the real-world environment. For example, device 200 uses image sensor(s) 206 to track the position and orientation of display generating component(s) 214 relative to one or more fixed objects in the real-world environment.

[0024] In some embodiments, device 200 optionally includes hand tracking sensor(s) 202 and / or eye tracking sensor(s) 212. Hand tracking sensor(s) 202 are configured to track the position / location of a user's hands and / or fingers and / or the movement of the user's hands and / or fingers relative to the computer-generated environment, relative to the display generating component(s) 214, and / or relative to another coordinate system. Eye tracking sensor(s) 212 are configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or the computer-generated environment and / or relative to the display generating component(s) 214. The user's gaze can include the direction the eyes are pointed and, optionally, their intersection with a particular point or region in space and / or their intersection with a particular object. In some embodiments, the hand tracking sensor(s) 202 and / or the eye tracking sensor(s) 212 are implemented integrally with (e.g., in the same device as) the display generating component(s) 214. In some embodiments, the hand tracking sensor(s) 202 and / or the eye tracking sensor(s) 212 are implemented separately from (e.g., in a different device than) the display generating component(s) 214.

[0025] In some embodiments, the hand tracking sensor(s) 202 use image sensor(s) 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that capture three-dimensional information from the real world, including one or more hands. In some examples, the hand can be resolved with sufficient resolution to distinguish fingers and their respective positions. In some embodiments, the one or more image sensor(s) 206 are positioned relative to the user such that the field of view of the image sensor(s) and an interaction space in which the position, orientation, and / or movement of the fingers / hands captured by the image sensors are used as input (e.g., to distinguish from the user's stationary hand or other hands of other people in the real-world environment) are used as input. Tracking fingers / hands for input (e.g., gestures) can be advantageous in that it provides a means of input that does not require the user to touch or hold an input device, and using an image sensor enables tracking without requiring the user to wear beacons, sensors, etc. on their hands / fingers.

[0026] In some embodiments, the eye tracking sensor(s) 212 include one or more eye tracking cameras (e.g., IR cameras) and / or illumination sources (e.g., IR light sources / LEDs) that emit light toward the user's eyes. The eye tracking cameras may be aimed at the user's eyes to receive reflected light from the light source directly or indirectly from the eyes. In some embodiments, both eyes are tracked separately by respective eye tracking cameras and illumination sources, and gaze can be determined from tracking of both eyes. In some embodiments, one eye (e.g., the dominant eye) is tracked by a separate eye tracking camera / illumination source(s).

[0027] Device 200 optionally includes microphone(s) 213 or other audio sensors. Device 200 uses microphone(s) 213 to detect sounds from the user and / or the user's real-world environment. In some embodiments, microphone(s) 213 optionally include an array of microphones working in concert (e.g., to identify ambient noise or locate a sound source within the space of the real-world environment). In some embodiments, audio and / or voice input, as authorized by a user of the electronic device, may be used to interact with a user interface or computer-generated environment, captured using one or more audio sensors (e.g., microphones).

[0028] Device 200 optionally includes location sensor(s) 204 configured to detect the location of device 200 and / or display generation component(s) 214. For example, location sensor(s) 204 optionally include a GPS receiver that receives data from one or more satellites, allowing device 200 to determine the absolute position of the device in the physical world.

[0029] Device 200 optionally includes motion and / or orientation sensor(s) 210 configured to detect orientation and / or movement of device 200 and / or display generating component(s) 214. For example, device 200 uses orientation sensor(s) 210 to track changes in position and / or orientation (e.g., relative to physical objects in a real-world environment) of device 200 and / or display generating component(s) 214. Orientation sensor(s) 210 optionally include one or more gyroscopes, one or more accelerometers, and / or one or more inertial measurement units (IMUs).

[0030] While the architecture of FIG. 2A is an example architecture, it should be understood that device 200 is not limited to the components and configuration of FIG. 2A. For example, a device may include fewer, additional, or other components, in the same or different configuration. In some embodiments, as shown in FIG. 2B, system 250 may be divided among multiple devices. For example, first device 260 optionally includes processor(s) 218A, one or more memories 220A, and communication circuitry 222A, which optionally communicate via communication bus(es) 208A. Second device 270 (e.g., corresponding to device 200) optionally includes various sensors (e.g., one or more hand-tracking sensor(s) 202, one or more location sensor(s) 204, one or more image sensor(s) 206, one or more touch-sensitive surface(s) 209, one or more movement and / or orientation sensor(s) 210, one or more eye-tracking sensor(s) 212, one or more microphone(s) 213 or other audio sensors, etc.), one or more display generating component(s) 214, one or more speaker(s) 216, one or more processor(s) 218B, one or more memories 220B, and / or communications circuitry 222B. One or more communications buses 208B are optionally used for communication between the above-mentioned components of device 270. Details of the components of devices 260 and 270 are similar to the corresponding components described above with respect to device 200 and will not be repeated here for the sake of brevity. First device 260 and second device 270 optionally communicate via a wired or wireless connection between the two devices (e.g., via communication circuitry 222A-222B).

[0031] Device 200 or system 250 generally supports applications displayable within a variety of computer-generated environments, such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disc authoring application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo / video management application, a digital camera application, a digital video camera application, a web browsing application, a digital music playback application, a television channel browsing application, and / or a digital video playback application.

[0032] The computer-generated environment may be displayed using an electronic device (e.g., electronic device 100, device 200, device 270), including using one or more display generation components. The computer-generated environment may optionally include various graphical user interfaces ("Graphical User Interfaces (GUIs") and / or user interface objects.

[0033] In some embodiments, the electronic device can detect or estimate real-world lighting characteristics. The estimation of lighting characteristics can provide some understanding of the lighting in an environment. For example, the estimation of lighting characteristics can provide an indication of which areas of the real-world environment are bright or dark. The estimation of lighting characteristics may provide an indication of the position of a light source (e.g., a parametric light source, a directional light source, a point light source, an area light source, etc.) and / or the orientation of the light source. In some embodiments, the lighting characteristics are estimated as a per-voxel incident light field indicating brightness, color, and / or direction. For example, the lighting characteristics can be parameterized as an Image-Based Lighting (IBL) environment map. It should be understood that other parameterizations of lighting characteristics are also possible. In some examples, the lighting characteristics are estimated per pixel using a triangle mesh with lighting characteristics defining lighting for each vertex or each face. In addition, it should be understood that the estimation of lighting characteristics is optionally derived from an intermediate representation (e.g., an environment map).

[0034] In some embodiments, sensors such as cameras (e.g., image sensor(s) 206) are used to capture images of the real-world environment. The images can be processed by processing circuitry (one or more of processor(s) 218) to locate and measure light sources. In some embodiments, light can be determined from reflections and / or shadows cast by light sources in the environment. In some embodiments, (e.g., supervised) deep learning or other artificial intelligence or machine learning is used to estimate lighting characteristics based on input image(s).

[0035] As described herein, computer-generated environments including various graphics user interfaces (“GUIs”) may be displayed using an electronic device such as electronic device 100 or device 200 that includes one or more display generation components. The computer-generated environment may include one or more virtual objects. In some embodiments, the one or more virtual objects may interact with or be manipulated within the three-dimensional environment. For example, a user may move or rotate a virtual object. As described in further detail below, interaction with a virtual object may be direct or indirect, and the device may automatically interpret user input as either a direct or indirect manipulation based on context such as the position of the user's hand and / or the position of the virtual object to be manipulated.

[0036] FIG. 3 illustrates a method for displaying a three-dimensional environment 300 having one or more virtual objects according to some embodiments of the present disclosure. In FIG. 3A, an electronic device (e.g., device 100 or 200 described above) displays the three-dimensional environment 300. In some embodiments, the three-dimensional environment 300 includes one or more real-world objects (e.g., representations of objects in the device's surrounding physical environment) and / or one or more virtual objects (e.g., representations of objects generated and displayed by the device that are not necessarily based on real-world objects in the device's surrounding physical environment). For example, in FIG. 3A, the table 302 and picture frame 304 may be representations of real-world objects in the device's surrounding physical environment. In some embodiments, the table 302 and picture frame 304 are displayed by a display generation component by capturing one or more images of the table 302 and picture frame 304 (e.g., in the device's surrounding physical environment) and displaying representations (e.g., photorealistic representations, simplified representations, caricatures, etc.) of the table and picture frame, respectively, within the three-dimensional environment 300. In some embodiments, table 302 and picture frame 304 are passively provided by the device via a transparent or semi-transparent display so as not to obscure the user's view of table 302 and picture frame 304. In Figure 3A, cube 306 is a virtual object that is displayed above table 302 in three-dimensional environment 300 but is not present in the physical environment around the device. In some embodiments, a virtual device, such as cube 306 displayed as being placed on top of table 302 in Figure 3, can interact with representations of real-world objects both when the representations are actively displayed by the device and when the representations are passively displayed by the device.

[0037] In some embodiments, table 302 and picture frame 304 are representations of real-world objects in the device's surrounding environment and therefore cannot be manipulated by a user via the device. For example, because table 302 exists in the device's surrounding physical environment, to move or otherwise manipulate table 302, a user can move or manipulate table 302 within three-dimensional environment 300 by physically moving or manipulating table 302 within the device's surrounding physical environment. In contrast, because cube 306 is a virtual object, cube 306 can be manipulated by a device user via the device (e.g., without requiring the user to manipulate objects in the physical world around the device), as described in further detail below.

[0038] 4A-4D illustrate a method for indirectly manipulating a virtual object according to some embodiments of the present disclosure. In FIG. 4A, a device (e.g., device 100 or device 200) displays, via a display generation component, a three-dimensional environment 400 (e.g., similar to three-dimensional environment 300) including a cube 406 on a table 402. In some embodiments, cube 406 is a virtual object similar to cube 306 described above with respect to FIG. 3. It should be understood that while FIG. 4A shows cube 406 twice, the second cube 406, shown near the bottom of the figure (e.g., near hand 410), is not shown in three-dimensional environment 400 and is shown in FIG. 4A for the purpose of illustrating the distance of hand 410 from cube 406 (e.g., on table 402) when performing gesture A, as described in further detail below. In other words, the three-dimensional environment 400 does not include two copies of the cube 406 (e.g., the second cube 406 near the hand 410 is a replica of the cube 406 on the table 402, shown for illustrative purposes only; this replica is not shown in Figures 4B-4D).

[0039] In FIG. 4A , hand 410 is a user's hand of a device, and the device can track the position of hand 410 and / or detect gestures made by hand 410 (e.g., via one or more hand tracking sensors). In some embodiments, a representation of hand 410 is displayed within three-dimensional environment 400; for example, when hand 410 is held in front of the device, the device can capture an image of hand 410 and display a representation of hand 410 (or passively provide visibility of hand 410) at a corresponding location within the three-dimensional environment. In other embodiments, hand 410 may be a real-world object in the physical environment that is passively provided by the device through a transparent or semi-transparent display so as not to obscure the user's view of the hand. As used herein, reference to a physical object, such as a hand, can refer to either a representation of that physical object presented on a display or the physical object itself as passively provided by a transparent or semi-transparent display. Thus, as the user moves hand 410, the representation of hand 410 moves accordingly within three-dimensional environment 400.

[0040] In some embodiments, a user can use hands 410 to interact with virtual objects in three-dimensional environment 400 as if the user were interacting with real-world objects in the physical environment around the device. In some embodiments, a user's interaction with a virtual object can refer to either a direct manipulation interaction or an indirect manipulation interaction. In some embodiments, a direct manipulation interaction includes an interaction in which a user uses one or both hands to directly manipulate a virtual object while intersecting with (or within a threshold distance of) the virtual object. In some embodiments, an indirect manipulation interaction includes an interaction in which a user uses one or both hands to manipulate a virtual object without either one or both hands intersecting with (or within a threshold distance of) the virtual object.

[0041] Returning to FIG. 4A, the device may detect when the line of sight 408 is pointing to a virtual object. (e.g. cube 406 ) 4A , the hand 410 is detected (e.g., via one or more hand tracking sensors) performing a first gesture (e.g., "Gesture A") corresponding to a selection input while the hand 410 is pointed at the cube 406. In some embodiments, the gaze 408 may be detected via one or more eye tracking sensors to determine the location or object the user's eyes are looking at or pointing toward. In FIG. 4A , the hand 410 is farther away from the cube 406 than a threshold distance 412 when the hand 410 performs the first gesture.

[0042] In some embodiments, the distance between the hand 410 and the cube 406 is determined based on the distance between the location of the hand 410 in the physical world and a corresponding location of the cube 406 on the table 402 in the physical world. For example, the cube 406 is displayed at a location in the three-dimensional environment 400 that has a corresponding location in the physical world, and the distance between the corresponding location of the cube 406 in the physical world and the location of the user's hand 410 in the physical world is used to determine whether the hand 410 is farther away from the cube 406 than a threshold distance 412. In some embodiments, the distance can be determined based on the distance between the location of the hand 410 in the three-dimensional environment and the cube 406 in the three-dimensional environment 400. For example, a representation of the hand 410 is displayed at a distinct location in the three-dimensional environment 400, and the distance between the distinct position of the hand 410 in the three-dimensional environment 400 and the position of the cube 406 in the three-dimensional environment 400 is used to determine whether the hand 410 is farther away from the cube 406 than a threshold distance 412. For example, if the hand 410 is held one foot in front of the user (e.g., not extended toward the cube 406) and the cube 406 is six feet away from the user, the hand 410 is determined to be five feet away from the hand 410. In some embodiments, the threshold distance 412 may be one inch, three inches, six inches, one foot, three feet, etc.

[0043] In some embodiments, the first gesture corresponding to the selection input may be a pinch gesture (e.g., a pinch between the thumb and index finger of hand 410) using two or more fingers or one or both hands of the user. In some embodiments, the first gesture corresponding to the selection input may be a pointing gesture or a tap gesture using the fingers of hand 410 (e.g., the index finger of hand 410). In some embodiments, the gesture predetermined to correspond to the selection input may be any gesture.

[0044] In some embodiments, pursuant to a determination that a selection gesture (e.g., a pinch gesture, “Gesture A”) was performed by hand 410 while hand 410 was more than a threshold distance 412 from cube 406 (e.g., optionally more than a threshold distance 412 from any virtual object), the device is configured in an indirect manipulation mode in which user input is directed to the virtual object the user's gaze was directed at when the input was received. For example, in FIG. 4A , gaze 408 was directed at cube 406 (e.g., looking at cube 406, focusing on cube 406, etc.) when hand 410 was performing the selection input. Thus, the selection input was performed on cube 406 (e.g., cube 406 was selected as the target for manipulation). In some embodiments, cube 406 remains selected while hand 410 maintains the selection gesture. While cube 406 remains selected, manipulation gestures with hand 410 cause manipulation actions to be performed on cube 406 (eg, even though line of sight 408 is optionally moved away from cube 406).

[0045] 4B illustrates a method for moving a virtual object within a three-dimensional environment 400. In FIG. 4B, the device detects that the hand 410 is moving to the right (e.g., along the "x" axis) by a discrete amount 414 while maintaining a selection gesture. In some embodiments, moving the hand 410 to the right by a discrete amount 414 corresponds to an angular movement of the hand 410 by a discrete angle 416. For example, to move the hand 410 by the discrete amount 414, the user rotates the user's discrete arm by a discrete angle 416. In some embodiments, the discrete angle 416 is the angle formed between a first line extending outward from the device's position to the previous position of the hand and a second line extending outward from the device's position to the hand's new position.

[0046] 4B , in response to detecting that hand 410 is moving rightward by discrete amount 414 while maintaining a selection gesture, cube 406 is similarly moved rightward (e.g., along the “x” axis) within three dimensional environment 400 by a second discrete amount 418. In some embodiments, second discrete amount 418 is different from discrete amount 414. In some embodiments, second discrete amount 418 is discrete amount 414 scaled by a scaling factor. In some embodiments, the scaling factor is based on the distance of cube 406 from the user (e.g., the distance of cube 406 from a “camera” of three dimensional environment 400, the distance of cube 406 from a location in three dimensional environment 400 associated with the user, and / or the location from which the user is viewing three dimensional environment 400). In some embodiments, second discrete amount 418 is calculated such that the angular change made by cube 406 is the same as the angular change made by hand 410. For example, second discrete angle 420 (e.g., the angle formed between a first line extending outward from the device's position to the previous position of cube 406 and a second line extending outward from the device's position to the new position of cube 406) is equal to discrete angle 416. Thus, in some embodiments, the scaling factor used for second discrete quantity 414 is calculated based on the distance of cube 406 from the user and the distance of hand 410 from the user (e.g., the ratio of the two distances).

[0047] In some embodiments, as described in more detail below, the movement of the cube 406 can move in any direction based on the movement of the hand 410 (e.g., the cube 406 exhibits six degrees of freedom). In some embodiments, the movement of the cube 406 can be fixed in one dimension based on the movement of the hand 410. For example, if the initial movement of the hand 410 is along the x-direction (e.g., the horizontal component of the movement of the hand 410 is larger than other components of the movement of the hand 410, for the first 0.1 seconds, 0.3 seconds, 0.5 seconds, 1 second, or for the first 1 cm, 3 cm, 10 cm, or for the entire movement, etc.), the movement of the cube 406 is fixed to only horizontal movement (e.g., the cube 406 moves only horizontally based on the horizontal component of the movement of the cube 406, and does not move vertically and / or change depth even if the hand 410 includes vertical and / or depth movement components and / or moves vertically and / or changes depth) until the selection input is terminated.

[0048] FIG. 4C illustrates a method for rotating a virtual object within the three-dimensional environment 400. In FIG. 4C, the device detects that the hand 410 is rotating by a discrete amount 422 while maintaining a selection gesture. In some embodiments, the rotation of the hand 410 is along a yaw orientation (e.g., clockwise, such that the fingers rotate to the right relative to the wrist and the wrist rotates to the left relative to the fingers). In some embodiments, the rotation of the hand 410 is along a roll orientation (e.g., such that the fingers and wrist maintain their respective positions relative to each other, but the hand 410 reveals portions of the hand 410 that previously faced in other directions (e.g., portions that were previously obscured and / or facing outward relative to the device)). In some embodiments, a rotation of the hand 410 (e.g., along any orientation) that does not include lateral movement (e.g., horizontal movement, vertical movement, or change in depth) or includes lateral movement of less than a threshold amount (e.g., less than 1 inch, less than 3 inches, less than 6 inches, less than 1 foot, etc.) is interpreted as a request to rotate the cube 406.

[0049] 4C , in response to detecting that the hand 410 is rotating by the discrete amount 422 while maintaining the selection gesture, the cube 406 is rotated by a second discrete amount 424 in accordance with the rotation of the hand 410. In some embodiments, the cube 406 rotates in the same direction as the rotation of the hand 410. For example, if the hand 410 is rotated in a yaw direction, the cube 406 rotates in a yaw direction, and if the hand 410 is rotated in a roll direction, the cube 406 rotates in a roll direction. In some embodiments, the second discrete amount 424 by which the cube 406 is rotated is the same as the discrete amount 422 by which the hand 410 is rotated. For example, if the hand 410 makes a 90-degree rotation, the cube 406 is rotated 90 degrees in the same direction.

[0050] In some embodiments, the second discrete amount 424 by which the cube 406 is rotated is different (e.g., the rotation is attenuated or amplified) from the discrete amount 422 of rotation by the hand 410. For example, if the cube 406 can only rotate 180 degrees (e.g., if a property of the cube 406 is, for example, that the cube 406 cannot be turned upside down), the rotation of the cube 406 may be scaled by half (e.g., a 90 degree rotation of the hand 410 rotates the cube 406 45 degrees). In another example, if cube 406 can be rotated by 180 degrees, cube 406 rotates 180 degrees in response to a 180-degree rotation of hand 410, but cube 406 does not rotate in response to further rotation by hand 410 (e.g., rotations beyond 180 degrees), or cube 406 exhibits a rubberbanding effect or resistance to further rotation by hand 410 (e.g., cube 406 temporarily rotates beyond its maximum amount while hand 410 continues to rotate, but returns to its maximum rotation value once the rotation and / or input ends).

[0051] 4D illustrates a method for moving a virtual object toward or away from a user within a three-dimensional environment 400. In FIG. 4D, the device detects that the hand 410 is moving toward the user by a discrete amount 426 (e.g., pulling the hand 410 back from an extended position toward the user's body and / or toward the device) while maintaining a selection gesture. Thus, the distance between the hand 410 and the device is reduced (e.g., movement in the z direction).

[0052] 4D , in response to detecting movement of hand 410 moving a discrete amount 426 toward the user and / or device while maintaining a selection gesture, cube 406 is moved a second discrete amount 428 toward the user (e.g., closer to the “camera” of three-dimensional environment 400). In some embodiments, the amount cube 406 moves (e.g., second discrete amount 428) is the same as the amount of movement by hand 410 (e.g., discrete amount 426), optionally along the same direction as hand 410. In some embodiments, the amount cube 406 moves (e.g., second discrete amount 428) is different from the amount of movement by hand 410 (e.g., discrete amount 426), optionally along the same direction as hand 410. In some embodiments, the amount cube 406 moves is based on the distance of cube 406 from the user and / or the distance of hand 410 from the user. For example, if the cube 406 is farther away from the user, the cube 406 will move a greater amount than if the cube 406 were closer to the user in response to the same amount of movement by the hand 410. For example, if the hand 410 moves 6 inches toward the user (e.g., toward the device, toward the device's camera), the cube 406 may move closer 2 feet if the cube 406 is farther away from the user, but may move closer 6 inches if the cube 406 is closer to the user.

[0053] In some embodiments, the amount of movement by cube 406 is scaled based on the ratio of the distance cube 406 is from the user and / or device to the distance hand 410 is from the user and / or device when the selection input (e.g., a pinch gesture) is first received. For example, if hand 410 is 2 feet from the user (e.g., 2 feet from the user's eye, 2 feet from the device, and 2 feet from the device's camera) and cube 406 is 10 feet from the user (e.g., 10 feet from the user's eye, 10 feet from the device, and 10 feet from the device's camera) when the selection input is received, then the scaling factor is 5 (e.g., the distance of cube 406 divided by the distance of hand 410). Thus, a 1 inch movement of hand 410 along the z-axis (e.g., toward or away from the user) causes cube 406 to move 5 inches along the same direction (e.g., toward or away from the user). Thus, cube 406 moves closer to the user as the user brings hand 410 closer to the user, such that as hand 410 moves closer to the user, cube 406 also moves closer to the user. In this manner, the user can use hand 410 to bring cube 406 from its initial position to the user without requiring the user to perform input multiple times. In some embodiments, cube 406 is brought to the user's location. In some embodiments, cube 406 is brought to the location of hand 410 such that cube 406 is in contact with hand 410 or within a threshold distance (e.g., 1 inch, 3 inches, 6 inches, etc.) of hand 410. In some embodiments, once cube 406 is brought to the location of hand 410, the user can use hand 410 to perform direct manipulation of cube 406, as described in further detail below with reference to FIGS. 5A-5D and 6A-6B.

[0054] In some embodiments, instead of scaling movement based on distance (e.g., of cube 406 and / or hand 410) from the user, movement is based on distance (e.g., of cube 406 and / or hand 410) from a location that is a predetermined distance in front of the user (e.g., a predetermined reference location that is, optionally, the user's location or a location in front of the user). For example, the reference location may be the user's location, the user's face location, the device's location (e.g., as described above), or 3 inches in front of the user (or the user's face, or the device), 6 inches in front of the user (or the user's face, or the device), 1 foot in front, 3 feet in front, etc. Thus, by using a reference location that is not exactly the user's location, the user can bring cube 406 to the user and / or hand 410 from a distant location by moving hand 410 to a reference location that is slightly in front of the user (e.g., without requiring the user to move hand 410 to the user's location, which may be an awkward gesture).

[0055] In some embodiments, the scaling of the cube 406's movement is applied to both movement toward and movement away from the user. In some embodiments, the scaling is applied only to movement toward the user (e.g., along the z-axis), while movement away from the user is scaled differently (e.g., scaled one-to-one with the movement of the hand 410). In some embodiments, the scaling is applied to movement along a particular direction based on the context and / or type of element being manipulated. For example, if a user is moving a virtual object in a direction not intended by the designer of the three-dimensional environment, the movement of the virtual object may be attenuated (e.g., scaled less), whereas if the user is moving the virtual object in a direction intended by the designer, the movement of the virtual object may be amplified (e.g., scaled more). Thus, the scaling factor may differ based on the movement direction to provide the user with feedback about whether a particular movement direction is consistent or intended.

[0056] It should be understood that the above-described movement of a virtual object is not limited to only one type of manipulation at a time or to only movement along one axis at a time. For example, a user can move a virtual object (e.g., cube 406) in both the x and y directions (e.g., as in FIG. 4B) and the z direction (e.g., to change depth as in FIG. 4D) while simultaneously rotating the virtual object (e.g., as shown in FIG. 4C). Thus, the device can determine the different movement and / or rotation components of the hand 410 and perform the appropriate manipulation on the virtual object. For example, if the hand 410 is moving leftward and simultaneously moving toward the user (e.g., while maintaining selection of cube 406), the device can simultaneously move cube 406 leftward as described above with respect to FIG. 4B and move cube 406 toward the user as described above with respect to FIG. 4D. Similarly, if the hand 410 is moving leftward and simultaneously rotating, the device can simultaneously move cube 406 leftward as described above with respect to FIG. 4B and simultaneously rotate cube 406 as described above with respect to FIG. 4C.

[0057] Thus, as described above, when performing an indirection manipulation, the direction, magnitude, and / or speed of the manipulation may depend on the direction, magnitude, and / or speed of the user's hand movement. For example, when performing a translation manipulation, if the user's hand moves to the right, the virtual object being manipulated moves to the right; if the user's hand moves to the left, the virtual object moves to the left; if the user's hand moves forward (e.g., away from the user), the virtual object moves forward (e.g., away from the user), etc. Similarly, if the hand moves fast, the virtual object optionally moves fast; if the hand moves slow, the virtual object optionally moves slower. Also, as described above, the amount of movement depends on the amount of hand movement (e.g., optionally scaled based on the distance from the user, as described above). In some embodiments, when performing a rotation manipulation, the direction, magnitude, and / or speed of the rotation depends on the direction, magnitude, and / or speed of the rotation of the user's hand, similar to that described above for the translation manipulation.

[0058] 5A-5D illustrate a method of directly manipulating a virtual object according to some embodiments of the present disclosure. In FIG. 5A, a device, via a display generating component, displays a three-dimensional environment 500 (e.g., similar to three-dimensional environments 300 and 400) including a cube 506 on a table 502. In some embodiments, cube 506 is a virtual object similar to cubes 306 and 406 described above with respect to FIGS. 3 and 4A-4D. As described above with respect to FIG. 4A, FIG. 5A illustrates cube 506 twice; however, it should be understood that the second cube 506, shown near the bottom of the figure (e.g., near hand 510), is not shown in three-dimensional environment 500 and is shown in FIG. 5A for the purpose of illustrating the distance of hand 510 from cube 506 (e.g., on table 502) when performing gesture A, as described in further detail below. In other words, the three-dimensional environment 500 does not include two copies of the cube 506 (e.g., the second cube 506 near the hand 510 is a replica of the cube 506 on the table 502 and is shown for illustrative purposes; the replica is not shown in Figures 5B-5D).

[0059] As described above, direct manipulation is an interaction with a virtual object in which a user uses one or more hands to intersect with the virtual object when manipulating the virtual object. For example, grabbing a virtual object similar to grabbing a physical object and moving the hands that are gripping the virtual object is an example of moving a virtual object via direct manipulation. In some embodiments, whether a user is performing a direct manipulation action or an indirect manipulation action on a virtual object depends on whether the user's hands are within a threshold distance from the virtual object being manipulated. For example, if the user's hands are in contact with the virtual object (e.g., at least a portion of the user's hands are in a location in physical space that makes it appear as if a portion of the hand is in contact with or intersecting with the virtual object in the three-dimensional environment), the user is directly interacting with the virtual object. In some embodiments, the device can interpret the user's interaction as direct manipulation if the user's hands are within a threshold distance 512 (e.g., within 1 inch, within 6 inches, within 1 foot, within 3 feet, etc.) of the virtual object being manipulated. In some embodiments, user input is directed toward the virtual object when the hands 510 are within the threshold distance 512 of the virtual object. For example, if the hand 510 is within the threshold distance 512 of one virtual object, the user's input is directed to that virtual object (optionally regardless of whether the user's gaze is directed toward that virtual object). If the hand 510 is within the threshold distance 512 of two virtual objects, the user's input may be directed to the closer virtual object, or the virtual object closer to the portion of the hand 510 performing the input (e.g., closer to the pinch location if the selection input is a pinch gesture), or the virtual object toward which the user's gaze is directed. If the hand 510 is not within the threshold distance 512 of either virtual object, the device may determine whether the user is performing indirect manipulation of the virtual object (e.g., when the user's gaze is directed toward a particular virtual object), as described above with respect to FIGS. 4A-4D .

[0060] 5A , the device detects that the hand 510 is performing a gesture corresponding to a selection input (e.g., a "gesture A," a pinch gesture, a tap gesture, a poke gesture, etc.) when the hand 510 is within a threshold distance 512 of the cube 506. In some embodiments, in response to the hand 510 performing the selection input when the hand 510 is within the threshold distance 512 of the cube 506 (and optionally, the hand 510 is not within the threshold distance 512 of any other virtual object), the cube 506 is selected as a target for input such that further user input (e.g., an object manipulation input, etc.) is performed on the cube 506. In FIG. 5A , the cube 506 is selected as a target for input even though the user's line of sight 514 is directed toward the table 502 when the selection input is performed. Thus, in some embodiments, a user can interact with a virtual object through direct manipulation of the virtual object without the user needing to see the virtual object.

[0061] 5B , in response to selecting cube 506 as a target for input, in some embodiments, cube 506 is automatically rotated by a discrete amount 516 such that cube 506 is aligned with one or more axes and / or one or more surfaces of the object. For example, the orientation of cube 506 is snapped to the nearest axis such that at least one boundary of cube 506 is aligned with the x-axis (e.g., perfectly horizontal), y-axis (e.g., perfectly vertical), or z-axis (e.g., perfectly flat). In some embodiments, cube 506 is automatically snapped to an upward orientation (e.g., aligned with gravity and / or other objects in the environment). In some embodiments, in response to selecting cube 506 as a target for input, cube 506 snaps to the same orientation as hand 510. For example, if hand 510 is oriented diagonally at a 30-degree angle (e.g., as shown in FIG. 5B ), cube 506 can be snapped to a rotated orientation of 30 degrees. In some embodiments, cube 506 does not change orientation in response to being selected for input, but rather maintains the orientation it had when the selection input was received (e.g., as shown in FIG. 5A ). In some embodiments, cube 506 automatically snaps into orientation on the surface of table 502 (e.g., so that the bottom of cube 506 is flush with the top surface of table 502).

[0062] 5C illustrates a method for moving a virtual object within the three-dimensional environment 500. In FIG. 5C, the device detects that the hand 510 is moving to the right (e.g., along the “x” axis) by a discrete amount 518 while maintaining a selection gesture (e.g., while maintaining a pinch gesture, a pointing gesture, a tap gesture, etc.). In response to detecting the hand 510 moving rightward, the device optionally moves the cube 506 to the right by a second discrete amount 520. In some embodiments, the cube 506 moves the same amount as the hand 510, such that the relative distance and / or relative position between the cube 506 and the hand 510 is maintained. For example, if the cube 506 was 3 inches in front of the hand 510 when the selection input was received, then in response to (and optionally while receiving) the user input, the cube 506 moves with the movement of the hand 510 and remains 3 inches in front of the hand 510. In some embodiments, the movement of the cube 506 along the x and y directions is scaled one-to-one with the movement of the hand 510. Thus, in some embodiments, the movement of the cube 506 simulates the hand 510 physically holding and moving the cube 506, where the cube 506 moves in the same direction, by the same amount, and at the same speed as the hand 510 (e.g., during indirect manipulation, the cube 506 optionally moves more or less than the movement of the hand 510, as described above with reference to FIG. 4B ). In some embodiments, the movement of the cube 506 during direct manipulation is not fixed to a discrete movement orientation, but can move in any direction (e.g., six degrees of freedom) based on the movement of the hand (e.g., during execution of some embodiments of indirect manipulation, the movement of the virtual object is fixed to one movement orientation, such as the x, y, or z axis, and hand movement in other directions is filtered or ignored, or otherwise does not move the virtual object in those other directions).

[0063] 5D illustrates a method for moving a virtual object toward or away from a user within three-dimensional environment 500. In FIG. 5D , the device detects that hand 510 is moving forward (e.g., away from the user and / or device along the z-direction) by a discrete amount 522 while maintaining a selection gesture (e.g., while maintaining a pinch gesture, a pointing gesture, a tap gesture, etc.). In response to detecting that hand 510 is moving further away, the device optionally moves cube 506 further away by a second discrete amount 524. In some embodiments, cube 506 moves the same amount as hand 510 such that the relative distance and / or relative position between cube 506 and hand 510 is maintained. Thus, changes in the distance of cube 506 from the user and / or device (e.g., away from and towards the user) are optionally scaled one-to-one with movements of hand 510 (e.g., during indirect manipulation, movements towards and / or away from the user are optionally not scaled one-to-one with movements of hand 510).

[0064] In some embodiments, while performing direct manipulation of cube 506, rotating hand 510 while maintaining a selection gesture causes cube 506 to rotate as well (optionally exhibiting the same or similar behavior as described above with respect to FIG. 4C).

[0065] Thus, as described above, when a user is performing direct manipulation of a virtual object, the movement of the virtual object is optionally scaled one-to-one with the movement of the hand performing the selection input, but when performing indirect manipulation of a virtual object, the movement of the virtual object is not necessarily scaled one-to-one with the movement of the hand performing the selection input. In some embodiments, the rotation input is scaled the same amount regardless of whether the manipulation is direct or indirect. In some embodiments, whether a user is performing direct or indirect manipulation input is based on whether the user's hand is within a threshold distance of the virtual object when the selection input (e.g., a selection gesture) is received.

[0066] Thus, as described above, when performing a direct manipulation, the direction, magnitude, and / or speed of the manipulation may depend on the direction, magnitude, and / or speed of the user's hand movement. For example, when performing a translation operation, if the user's hand moves to the right, the virtual object being manipulated moves to the right; if the user's hand moves to the left, the virtual object moves to the left; if the user's hand moves forward (e.g., away from the user), the virtual object moves forward (e.g., away from the user), etc. Similarly, if the hand moves fast, the virtual object optionally moves fast; if the hand moves slow, the virtual object optionally moves slower. Also, as described above, the amount of movement is scaled one-to-one with the amount of hand movement (e.g., as opposed to being scaled by distance as described above in FIGS. 4A-4D ). In some embodiments, when performing a rotation operation, the direction, magnitude, and / or speed of the rotation depend on the direction, magnitude, and / or speed of the rotation of the user's hand, similar to what is described above for translation operations.

[0067] 6A-6B illustrate a method of moving a virtual object according to some embodiments of the present disclosure. In FIG. 6A, a device, via a display generating component, displays a three-dimensional environment 600 (e.g., similar to three-dimensional environments 300, 400, and 500) including a cube 606 on a table 602. In some embodiments, cube 606 is a virtual object similar to cubes 306, 406, and 506 described above with respect to FIGS. 3, 4A-4D, and 5A-5D. As described above with respect to FIGS. 4A and 5A, FIG. 6A illustrates cube 606 twice; however, it should be understood that the second cube 606, shown near the bottom of the figure (e.g., near hand 610), is not shown in three-dimensional environment 600 and is shown in FIG. 6A for the purpose of illustrating the distance of hand 610 from cube 606 (e.g., on table 602) when performing gesture B, as described in further detail below. In other words, the three-dimensional environment 600 does not include two copies of the cube 606 (e.g., the second cube 606 near the hand 610 is a duplicate of the cube 606 on the table 602 and is shown for illustrative purposes; the duplicate is not shown in FIG. 6B).

[0068] 6A , the device detects that the hand 610 has performed a discrete gesture (e.g., “Gesture B”) between cubes that are more than a threshold distance 612 away from 606. In some embodiments, the discrete gesture includes a pinch gesture (e.g., between the thumb and index finger of the hand, or between any two or more fingers of one or both hands, as described above with respect to “Gesture A”). In some embodiments, the discrete gesture includes a pinch gesture followed by a predetermined movement and / or rotation of the hand 610 while maintaining the pinch gesture (e.g., Gesture B includes Gesture A followed by a discrete movement by the hand 610). For example, a tag gesture by the hand 610 (e.g., an upward rotation of the hand 610 such that the fingers and / or pinch location are moved closer to and / or rotated toward the user, optionally while maintaining the wrist position). In some embodiments, the discrete gesture includes a pinch gesture followed by movement of the hand 610 to bring the cube 606 from a remote location to the location of the hand 610 by bringing the hand 610 to the user's location or to a predetermined reference location in front of the user (e.g., as described above with reference to FIG. 4D ). In some embodiments, the discrete gesture corresponds to a request to move the cube 606 to a location for direct manipulation (e.g., a location associated with the hand 610). In some embodiments, because the discrete gesture is indirect manipulation input (e.g., the hand 610 is more than a threshold distance 612 from the cube 606), the device uses the line of sight 614 to determine that the user's input is directed toward the cube 606. It should be understood that the discrete gesture may be any gesture (e.g., including, but not limited to, selecting a selectable option to snap the cube 606 to the location of the hand 610) predetermined to correspond to a request to move the cube 606 to a location for direct manipulation.

[0069] In some embodiments, in response to detecting a discrete gesture (e.g., gesture B) by hand 610 while line of sight 615 is directed toward cube 606, the device moves cube 606 to a location associated with hand 610, as shown in FIG. 6B . In some embodiments, the discrete gesture includes a pinch gesture, and cube 606 is moved to the pinch location (e.g., a portion of cube 606 is positioned at the pinch location, and hand 610 appears to be pinching a portion of cube 606) or to a location within a predetermined distance (e.g., 1 inch, 3 inches, 6 inches, etc.) of the pinch. Thus, after moving cube 606 to the pinch location, the user can perform direct manipulation on cube 606 by maintaining the pinch gesture (e.g., maintaining a selection input) and performing direct manipulation gestures (e.g., moving sideways, moving forward or backward, rotating, etc.) similar to those described above with respect to FIGS. 5A-5D . In some embodiments, moving cube 606 to a pinch location allows the user to use direct manipulation input to manipulate objects at locations within three-dimensional environment 600 that may otherwise be far away and out of the user's reach.

[0070] While the above figures and descriptions describe movement along a particular direction or rotation along a particular direction, it should be understood that this is merely exemplary, and that the virtual object can exhibit the same or similar behavior for movement or rotation along any direction. For example, the virtual object can be moved leftward and exhibit a response to user input similar to the example above relating to moving the virtual object rightward. Similarly, the virtual object can be rotated counterclockwise and exhibit a response to user input similar to the example above relating to rotating the virtual object clockwise.

[0071] While the above figures and descriptions describe manipulation of virtual objects, it will be appreciated that the above methods may be applied to any type of user interface element or control element. For example, buttons, sliders, dials, knobs, etc. may be moved or rotated according to the direct or indirect manipulation methods described above.

[0072] 7 is a flow diagram illustrating a method 700 for manipulating a virtual object according to an embodiment of the present disclosure. Method 700 is optionally performed in an electronic device, such as device 100, device 200, when displaying selectable options on a plane, as described above with reference to FIGS. 3A-3C, 4A-4B, 5A-5B, and 6A-6B. Some operations in method 700 are optionally combined (e.g., with each other or with operations in method 800), and / or the order of some operations is optionally changed. As described below, method 700 provides a method for manipulating a virtual object (e.g., as described above with respect to FIGS. 3-6B) according to an embodiment of the present disclosure.

[0073] In some embodiments, an electronic device (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), a computer such as device 100 and / or device 200) that is capable of communicating with display-generating components (e.g., a display built into the electronic device (optionally, such as a touchscreen display) and / or an external display such as a monitor, projector, television, etc.) and one or more input devices (e.g., a touchscreen, a (e.g., external) mouse, a (optionally, built-in or external) trackpad, a (optionally, built-in or external) touchpad, a (e.g., external) remote control device, another mobile device (e.g., separate from the electronic device), a (e.g., external) handheld device, a (e.g., external) controller, a camera (e.g., a visible light camera), a depth sensor and / or a motion sensor (e.g., a hand tracking sensor, a hand motion sensor), etc.) includes cube 306 in FIG. 3 via the display-generating components. 3D A computer-generated environment including a first user interface element, such as environment 300, is presented (702).

[0074] In some embodiments, while presenting the computer-generated environment, the electronic device receives (704) multiple user inputs (e.g., a sequence) including selection inputs and manipulation inputs, such as hand 410 performing a gesture (e.g., gesture A) corresponding to the selection input in FIG. 4A and hand 410 moving while maintaining the gestures in FIGS. 4B-4D.

[0075] In some embodiments, pursuant to determining that a hand representation of a user of the electronic device is within a threshold distance from the first user interface element when a selection input is received, such as hand 510 within threshold distance 512 from cube 506 in FIG. 5A , the electronic device manipulates (706) the first user interface element in accordance with the manipulation input, such as moving cube 506 in accordance with movement of hand 510 in FIGS. 5C-5D . In some embodiments, manipulating the first user interface element includes a translation action, a rotation action, a resize action, or any other suitable manipulation action. In some embodiments, the threshold distance is 1 inch, 3 inches, 6 inches, 1 foot, 3 feet, etc.

[0076] In some embodiments, pursuant to a determination (708) that the hand representation of the user of the electronic device is not within a threshold distance from the first user interface element when the selection input is received, e.g., hand 410 is farther than threshold distance 412 from cube 406 in FIG. 4A , the gaze of the user of the electronic device is directed toward the first user interface element, e.g., when hand 410 is performing a selection input (e.g., “gesture A”) in FIG. 4A , cube 406 is manipulated in accordance with the movement of hand 410 in FIGS. 4B-4D . In accordance with a determination that the gaze of a user of the electronic device is not directed at the first user interface element, the electronic device manipulates (710) the first user interface element in accordance with the manipulation input; in accordance with a determination that the gaze of a user of the electronic device is not directed at the first user interface element, such that if the gaze 408 was not directed at the cube 406 when the hand 410 was performing the selection input, the cube 406 is optionally not manipulated in accordance with the movement of the hand 410 (712). In some embodiments, if the gaze was directed at another object when the selection input was received, the other object is manipulated in accordance with the movement of the hand 410. In some embodiments, a non-virtual object is not manipulable (e.g., the user input is optionally discarded or ignored, and / or a notice is displayed to the user indicating that the object is not manipulable), such that the non-virtual object is not manipulated in accordance with the movement of the hand 410 if the gaze is directed at an object that is not a virtual object (e.g., a representation or depiction of a real-world object).

[0077] In some embodiments, in accordance with a determination that a representation of a hand of a user of the electronic device is within a threshold distance from the second user interface element when the selection input is received, the electronic device manipulates the second user interface element in accordance with the manipulation input. For example, if the user's hand is within a threshold distance of either virtual object, the individual virtual object closest to the hand and / or closest to a pinch point of the hand is selected as the target of the input (e.g., such that subsequent movement of the hand causes manipulation of the individual virtual object).

[0078] In some embodiments, in accordance with a determination that a hand representation of a user of the electronic device is not within a threshold distance from a second user interface element when a selection input is received, the electronic device operates the second user interface element in accordance with the operation input in accordance with a determination that a gaze of the user of the electronic device is directed toward the second user interface element, and in accordance with a determination that the gaze of the user of the electronic device is not directed toward the second user interface element, the electronic device refrains from operating the second user interface element in accordance with the operation input. For example, if the user's hand is not within a threshold distance of any virtual object, the object toward which the user's gaze is directed is the object selected as a target for input in response to detecting the selection input. In some embodiments, if the gaze is directed toward a first virtual object, the first virtual object is selected as a target for operation, whereas if the gaze is directed toward a second virtual object, the second virtual object is selected as a target for operation. As described herein, determining whether a user's gaze is directed toward a particular object or location is based on one or more eye tracking sensors. In some embodiments, if a user's gaze is directed at a particular location in the physical world that maps to (e.g., corresponds to) a particular location in the three-dimensional environment, the user's gaze is considered to be directed at the corresponding location in the three-dimensional environment (e.g., if a virtual object is at that corresponding location in the three-dimensional environment, the user's gaze is interpreted as being directed at that virtual object).

[0079] In some embodiments, the manipulation input includes a movement of the user's hand, such as the horizontal movement of hand 410 in Figure 4B and the movement toward the user in Figure 4D. In some embodiments, in accordance with a determination that a representation of a hand of a user of the electronic device is within a threshold distance from the first user interface element when the selection input is received, manipulating the first user interface element in accordance with the manipulation input includes moving the first user interface element by an amount equal to the amount of the user's hand movement, e.g., so that cube 506 moves rightward by the same amount as the rightward movement of hand 510 in Figure 5C. In some embodiments, in accordance with a determination that a representation of a hand of a user of the electronic device is not within a threshold distance from the first user interface element when the selection input is received, manipulating the first user interface element in accordance with the manipulation input includes moving the first user interface element by an amount unequal to the amount of the user's hand movement, so that cube 406 moves rightward by a greater amount than the rightward movement of hand 410 in Figure 4B.

[0080] In some embodiments, in response to receiving the selection input and before manipulating the first user interface element in accordance with the manipulation input, the electronic device changes the orientation of the first user interface element based on the orientation of the user's hand, e.g., such that in FIG. 5B cube 516 optionally snaps into a particular orientation based on the orientation of hand 510. In some embodiments, cube 516 snaps into its "up" orientation. In some embodiments, cube 516 snaps into the nearest axis. In some embodiments, cube 516 snaps into the same orientation as hand 510 (e.g., if hand 510 is held at an angle, cube 516 snaps into the same diagonal angle).

[0081] In some embodiments, the manipulation input includes a rotation of the user's hand, and manipulating the first user interface element in accordance with the manipulation input includes rotating the first user interface element, such as rotating cube 406 in accordance with the rotation of hand 410 in FIG. 4C . In some embodiments, the virtual object is rotated in the same direction and by the same amount as the rotation of the hand. For example, if the hand rotates in a yaw direction, the virtual object rotates in a yaw direction; if the hand rotates in a pitch direction, the virtual object rotates in a pitch direction, etc. Similarly, if the hand rotates 30 degrees, the virtual object optionally rotates 30 degrees. In some embodiments, the user can perform both a rotation operation and a translation operation simultaneously by both rotating and translating the user's hand while maintaining a selection input.

[0082] In some embodiments, the first user interface element includes a control element, such as a button, a slider, a dial, or any other suitable control element. In some embodiments, in response to manipulating the first user interface element according to the operational input, the electronic device performs an operation associated with the control element. For example, a user may manipulate the control element in a manner similar to that described above with respect to virtual objects, and manipulating the control element optionally performs one or more functions associated with the control element. For example, sliding a volume slider may change the volume accordingly, etc.

[0083] In some embodiments, when a selection input is received, in accordance with a determination that the user's hand representation is not within a threshold distance from the first user interface element, upon detecting a predefined gesture (e.g., "gesture B") corresponding to a request to move cube 606 to a position for direct manipulation (e.g., a remote request to directly manipulate) in FIG. 6A , in accordance with a determination that the plurality of user inputs includes a predefined gesture by the user's hand, such that cube 606 in FIG. 6B is moved toward the user, optionally to or near the location of the pinch by hand 610, the first user interface element is moved to a location in the computer-generated environment associated with the user's hand representation. Thus, by performing a particular gesture, the user can move (e.g., fly) an object to (or within a threshold distance of) the hand location such that the user can perform a direct manipulation operation on the object. In this manner, the user can directly manipulate an object without relying on indirect manipulation actions and without the user having to walk toward the object. In some embodiments, after completing the manipulation operation, such as after detecting the end of the selection input (e.g., the end of a pinch gesture, the end of gesture B, and / or the detection of another gesture corresponding to a request to return the virtual object to its original position), cube 606 is returned to its original position before the user input (optionally maintaining any manipulation performed while held by the user, such as rotation). In some embodiments, after completing the manipulation operation, such as after detecting the end of the selection input, cube 606 remains in the location it was in when the selection input was ended (e.g., cube 606 does not return to its original position, but remains in the position where the user placed it).

[0084] 8 is a flow diagram illustrating a method 800 for moving a virtual object by an amount based on the virtual object's distance to a user, according to some embodiments of the present disclosure. Method 800 is optionally performed in an electronic device, such as device 100, device 200, when displaying selectable options on a surface, as described above with reference to FIGS. 3A-3C, 4A-4B, 5A-5B, and 6A-6B. Some operations in method 800 are optionally combined (e.g., with each other or with operations in method 700), and / or the order of some operations is optionally changed. As described below, method 800 provides a method for moving a virtual object by an amount based on the virtual object's distance to a user (e.g., as described above with reference to FIGS. 3-6B), according to some embodiments of the present disclosure.

[0085] In some embodiments, an electronic device (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), a computer such as device 100 and / or device 200) that is capable of communicating with display-generating components (e.g., a display built into the electronic device (optionally, such as a touchscreen display) and / or an external display such as a monitor, projector, television, etc.) and one or more input devices (e.g., a touchscreen, a (e.g., external) mouse, a (optionally, built-in or external) trackpad, a (optionally, built-in or external) touchpad, a (e.g., external) remote control device, another mobile device (e.g., separate from the electronic device), a (e.g., external) handheld device, a (e.g., external) controller, a camera (e.g., a visible light camera), a depth sensor and / or a motion sensor (e.g., a hand tracking sensor, a hand motion sensor), etc.) includes cube 306 in FIG. 3 via the display-generating components. 3D A computer-generated environment including a first user interface element, such as environment 300, is presented (802).

[0086] In some embodiments, while presenting the computer-generated environment, the electronic device receives user input including a movement component directed toward a first user interface element (804), such as a rightward movement of hand 410 in FIG. 4B . In some embodiments, in accordance with a determination that the electronic device is in a first manipulation mode, the electronic device moves the first user interface element by a first amount in accordance with the movement component, such as moving cube 506 by amount 520 while in the direct manipulation mode in FIG. 5C . In some embodiments, in accordance with a determination that the electronic device is in a second manipulation mode different from the first manipulation mode, the electronic device moves the first user interface element by a second amount greater than the first amount in accordance with the movement component, such as moving cube 406 by amount 418 while in the indirect manipulation mode in FIG. 4B .

[0087] In some embodiments, the first operation mode is a direct operation mode in which a representation of a user's hand on the electronic device is within a threshold distance of the first user interface element when user input is received, such as when hand 510 is within threshold distance 512 of cube 406 in FIG. 5A , and the second operation mode is an indirect operation mode in which a representation of the user's hand is not within a threshold distance of the first user interface element when user input is received, such as when hand 410 is farther away than threshold distance 412 of cube 506 in FIG. 4A .

[0088] In some embodiments, the first amount is the same as the movement of the user input movement component, as shown in FIG. 5C, and the second amount is a different amount than the movement of the user input movement component, as shown in FIG. 4B.

[0089] In some embodiments, the second amount is the amount of movement of a movement component of the user input scaled by a scaling factor, such as the movement of cube 406 in FIG. 4B scaled by a scaling factor based on the distance of cube 406 from the user and / or the distance of hand 410 from the user.

[0090] In some embodiments, a first scaling factor is set as the scaling factor in accordance with a determination that the movement of the moving component is in a first direction relative to a user of the electronic device, and a second scaling factor different from the first scaling factor is set as the scaling factor in accordance with a determination that the movement of the moving component is in a second direction relative to the user that is different from the first direction. For example, if the object is moving away from the user, the scaling factor is optionally not based on the distance of the object from the user and / or the distance of the hand from the user (e.g., optionally, the scaling factor is 1), whereas if the object is moving towards the user, the scaling factor is optionally based on the distance of the object from the user and / or the distance of the hand from the user (e.g., optionally, the scaling factor is greater than 1), as shown in FIG.

[0091] In some embodiments, the second scaling factor is based at least on a distance of the first user interface element from a predetermined reference location in the computer-generated environment, such as that described in FIG. 4B (e.g., a location in the three-dimensional environment corresponding to the location of the head of a user of the electronic device, the location of the user of the electronic device, the location of the electronic device, any of the above, 1 inch in front, 3 inches in front, 6 inches in front, 1 foot in front, 3 feet in front), and a distance of the representation of the user's hand from the predetermined reference location (e.g., a distance from a location in the three-dimensional environment corresponding to the user's hand to a location of the head of a user of the electronic device, the location of the user of the electronic device, the location of the electronic device, any of the above, 1 inch in front, 3 inches in front, 6 inches in front, 1 foot in front, 3 feet in front).

[0092] In some embodiments, the movement component of the user input includes a lateral movement component parallel to the user of the electronic device (e.g., horizontal and / or vertical movement while maintaining the same distance from the user), as shown in FIG. Individual quantity 414 Rightward movement In action The angle of the second amount of movement relative to the user of the electronic device is the same as the angle of movement of the lateral movement component of the user input relative to the user of the electronic device, such that the hand 410 is moved rightward by an amount such that angle of change 420 of the movement of the hand 410 due to the user input is the same as angle of change 416. Thus, in some embodiments, the scaling factor for lateral movement is proportional to the ratio of the distance of the object from the user to the distance of the hand from the user.

[0093] The foregoing has been described with reference to specific embodiments for purposes of explanation. However, the exemplary discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments were chosen and described in order to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suited to the particular uses contemplated.

Claims

1. In an electronic device in communication with a display, presenting via the display a three-dimensional environment including a first user interface element; receiving a plurality of user inputs, including selection inputs and manipulation inputs, while presenting the three-dimensional environment; manipulating the first user interface element in accordance with the manipulation input in accordance with a determination that a hand of a user of the electronic device is within a threshold distance from the first user interface element when the selection input is received; in response to determining that the hand of the user of the electronic device is not within the threshold distance from the first user interface element when the selection input is received; In accordance with a determination that a gaze of the user of the electronic device is directed toward the first user interface element, operating the first user interface element in accordance with the operation input; and refraining from operating the first user interface element in accordance with the operation input in accordance with a determination that the line of sight of the user of the electronic device is not directed toward the first user interface element.

2. manipulating the second user interface element in accordance with the manipulation input in accordance with a determination that the hand of the user of the electronic device is within the threshold distance from a second user interface element when the selection input is received; and in response to determining that the hand of the user of the electronic device is not within the threshold distance from the second user interface element when the selection input is received; manipulating the second user interface element in accordance with the manipulation input in accordance with a determination that the line of sight of the user of the electronic device is directed toward the second user interface element; in accordance with determining that the line of sight of the user of the electronic device is not directed toward the second user interface element, refraining from operating the second user interface element in accordance with the operation input; The method of claim 1 further comprising:

3. the operation input includes movement of the user's hand, and manipulating the first user interface element in accordance with the manipulation input in accordance with the determination that the hand of the user of the electronic device is within the threshold distance from the first user interface element when the selection input is received includes moving the first user interface element by an amount approximately equal to an amount of the movement of the user's hand within the three-dimensional environment; 3. The method of claim 1, wherein, in accordance with the determination that the hand of the user of the electronic device is not within the threshold distance from the first user interface element when the selection input is received, manipulating the first user interface element in accordance with the manipulation input comprises moving the first user interface element an amount unequal to the amount of the movement of the hand of the user.

4. 4. The method of claim 1, further comprising, in response to receiving the selection input, changing an orientation of the first user interface element based on an orientation of the user's hand before manipulating the first user interface element in accordance with the manipulation input.

5. 5. The method of claim 1, wherein the operational input comprises a rotation of the user's hand, and wherein manipulating the first user interface element in accordance with the operational input comprises rotating the first user interface element.

6. the first user interface element comprises a control element, and the method further comprises: The method of claim 1 , further comprising: performing an action associated with the control element in response to manipulating the first user interface element according to the operational input.

7. 7. The method of claim 1, further comprising: moving the first user interface element to a location in the three-dimensional environment associated with the user's hand when the selection input is received, in accordance with the determination that the user's hand is not within the threshold distance from the first user interface element and in accordance with a determination that the plurality of user inputs includes a predetermined gesture by the user's hand.

8. one or more processors; Memory and and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions that: presenting, via the display, a three-dimensional environment including the first user interface element; receiving a plurality of user inputs, including selection inputs and operation inputs, while presenting the three-dimensional environment; manipulating the first user interface element in accordance with the manipulation input in accordance with a determination that a hand of a user of the electronic device is within a threshold distance from the first user interface element when the selection input is received; in response to determining that the hand of the user of the electronic device is not within the threshold distance from the first user interface element when the selection input is received; In accordance with a determination that a gaze of the user of the electronic device is directed toward the first user interface element, operating the first user interface element in accordance with the operation input; An electronic device for refraining from operating the first user interface element in accordance with the operation input in accordance with a determination that the line of sight of the user of the electronic device is not directed toward the first user interface element.

9. the one or more programs further comprising: and manipulating the second user interface element in accordance with the manipulation input in accordance with a determination that the hand of the user of the electronic device is within the threshold distance from a second user interface element when the selection input is received; in response to determining that the hand of the user of the electronic device is not within the threshold distance from the second user interface element when the selection input is received; In accordance with a determination that the line of sight of the user of the electronic device is directed toward the second user interface element, operating the second user interface element in accordance with the operation input; and refraining from operating the second user interface element in accordance with the operation input in accordance with a determination that the line of sight of the user of the electronic device is not directed toward the second user interface element.

9. The electronic device of claim 8, comprising instructions for:

10. the operation input includes movement of the user's hand, and manipulating the first user interface element in accordance with the manipulation input in accordance with the determination that the hand of the user of the electronic device is within the threshold distance from the first user interface element when the selection input is received includes moving the first user interface element by an amount approximately equal to an amount of the movement of the user's hand within the three-dimensional environment; 10. The electronic device of claim 8 or 9, wherein, in accordance with the determination that the hand of the user of the electronic device is not within the threshold distance from the first user interface element when the selection input is received, operating the first user interface element in accordance with the operation input includes moving the first user interface element an amount unequal to the amount of the movement of the hand of the user.

11. An electronic device as described in any one of claims 8 to 10, wherein the one or more programs further include instructions for, in response to receiving the selection input, changing the orientation of the first user interface element based on the orientation of the user's hand before operating the first user interface element in accordance with the operation input.

12. 12. The electronic device of claim 8, wherein the operational input comprises a rotation of the user's hand, and wherein manipulating the first user interface element in accordance with the operational input comprises rotating the first user interface element.

13. the first user interface element includes a control element, and the one or more programs:

13. The electronic device of claim 8, further comprising instructions for performing an action associated with the control element in response to manipulating the first user interface element in accordance with the operational input.

14. The one or more programs 14. The electronic device of claim 8, further comprising instructions for: moving the first user interface element to a location in the three-dimensional environment associated with the user's hand when the selection input is received, in accordance with the determination that the user's hand is not within the threshold distance from the first user interface element and in accordance with a determination that the plurality of user inputs includes a predetermined gesture by the user's hand.

15. A program having instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: presenting via a display a three-dimensional environment including a first user interface element; receiving a plurality of user inputs, including selection inputs and manipulation inputs, while presenting the three-dimensional environment; manipulating the first user interface element in accordance with the manipulation input in accordance with a determination that a hand of a user of the electronic device is within a threshold distance from the first user interface element when the selection input is received; in response to determining that the hand of the user of the electronic device is not within the threshold distance from the first user interface element when the selection input is received; In accordance with a determination that a gaze of the user of the electronic device is directed toward the first user interface element, operating the first user interface element in accordance with the operation input; and refraining from operating the first user interface element in accordance with the operation input in accordance with a determination that the line of sight of the user of the electronic device is not directed toward the first user interface element.

16. The operation is manipulating the second user interface element in accordance with the manipulation input in accordance with a determination that the hand of the user of the electronic device is within the threshold distance from a second user interface element when the selection input is received; and in response to determining that the hand of the user of the electronic device is not within the threshold distance from the second user interface element when the selection input is received; manipulating the second user interface element in accordance with the manipulation input in accordance with a determination that the line of sight of the user of the electronic device is directed toward the second user interface element; in accordance with determining that the line of sight of the user of the electronic device is not directed toward the second user interface element, refraining from operating the second user interface element in accordance with the operation input; The program of claim 15, further comprising:

17. the operation input includes movement of the user's hand, and manipulating the first user interface element in accordance with the manipulation input in accordance with the determination that the hand of the user of the electronic device is within the threshold distance from the first user interface element when the selection input is received includes moving the first user interface element by an amount approximately equal to an amount of the movement of the user's hand within the three-dimensional environment; 17. The program of claim 15 or 16, wherein, in accordance with the determination that the hand of the user of the electronic device is not within the threshold distance from the first user interface element when the selection input is received, operating the first user interface element in accordance with the operation input includes moving the first user interface element by an amount unequal to the amount of the movement of the user's hand.

18. 18. The program of claim 15, wherein the operation further comprises, in response to receiving the selection input, changing an orientation of the first user interface element based on an orientation of the user's hand before operating the first user interface element according to the operation input.

19. 19. The program of claim 15, wherein the operation input includes a rotation of the user's hand, and operating the first user interface element in accordance with the operation input includes rotating the first user interface element.

20. the first user interface element includes a control element; 20. The program of claim 15, wherein the operation further comprises executing an operation associated with the control element in response to operating the first user interface element according to the operation input.

Images