How to interact with objects in the environment

Abstract

Methods for interacting with objects and user interface elements within a computer-generated environment provide an efficient and intuitive user experience. [Solution] In some embodiments, the user can interact with the object directly or indirectly. In some embodiments, the operation of the virtual object is scaled during indirect operation. In some embodiments, the operation of the virtual object is not scaled during direct operation. In some embodiments, the object can be reset from indirect operation mode to direct operation mode by moving the object to a specific position in the 3D environment in response to a specific gesture.

Description

Technical Field

[0001] The present invention generally relates to methods for interacting with objects within a computer-generated environment.

Background Art

[0002] A computer-generated environment is an environment in which at least some of the objects presented to the user are generated using a computer. A user can interact with the objects displayed in the computer-generated environment by moving the objects, rotating the objects, and so on.

Summary of the Invention

[0003] Some embodiments described in this disclosure are directed to methods for interacting with virtual objects within a computer-generated environment. Some embodiments described in this disclosure are directed to methods for performing direct and indirect manipulation of virtual objects. These interactions provide a more efficient and intuitive user experience. It should be understood that the full description of these embodiments is set forth in the "Drawings" and the "Detailed Description", and this "Summary of the Invention" is not intended to limit the scope of this disclosure in any way.

Brief Description of the Drawings

[0004] For a better understanding of the various embodiments described, reference should be made to the following "Modes for Carrying Out the Invention" in conjunction with the following drawings. Like reference numerals refer to corresponding parts throughout the figures.

[0005] [Figure 1] An electronic device displaying a computer-generated environment according to some embodiments of the present disclosure is shown.

[0006] [Figure 2A] A block diagram of an exemplary architecture for one or more devices according to some embodiments of the present disclosure is shown. [Figure 2B] The following are exemplary block diagrams of architectures relating to one or more devices according to some embodiments of this disclosure.

[0007] [Figure 3] This disclosure describes a method for displaying a three-dimensional environment having one or more virtual objects, according to some embodiments of this disclosure.

[0008] [Figure 4A] This disclosure describes methods for indirectly manipulating virtual objects according to some embodiments of this disclosure. [Figure 4B] This disclosure describes methods for indirectly manipulating virtual objects according to some embodiments of this disclosure. [Figure 4C] This disclosure describes methods for indirectly manipulating virtual objects according to some embodiments of this disclosure. [Figure 4D] This disclosure describes methods for indirectly manipulating virtual objects according to some embodiments of this disclosure.

[0009] [Figure 5A] This disclosure describes methods for directly manipulating virtual objects according to some embodiments of this disclosure. [Figure 5B] This disclosure describes methods for directly manipulating virtual objects according to some embodiments of this disclosure. [Figure 5C] This disclosure describes methods for directly manipulating virtual objects according to some embodiments of this disclosure. [Figure 5D] This disclosure describes methods for directly manipulating virtual objects according to some embodiments of this disclosure.

[0010] [Figure 6A] This disclosure describes a method for moving a virtual object according to some embodiments of this disclosure. [Figure 6B] This disclosure describes a method for moving a virtual object according to some embodiments of this disclosure.

[0011] [Figure 7] This is a flowchart illustrating a method for manipulating a virtual object according to an embodiment of the present disclosure.

[0012] [Figure 8] This flowchart illustrates a method for moving a virtual object by an amount based on the distance of the virtual object to the user, according to some embodiments of this disclosure. [Modes for carrying out the invention]

[0013] The following description of embodiments refers to accompanying drawings that constitute a portion of the embodiments, which illustrate specific embodiments that may be implemented as an option. It should be understood that other embodiments may be used as an option and structural modifications may be implemented as an option without departing from the scope of the disclosed embodiments.

[0014] Humans can interact with and / or perceive the physical environment or physical world without the assistance of electronic devices. The physical environment may include physical features such as physical objects or surfaces. An example of a physical environment is a physical forest, which includes physical plants and animals. Humans can directly perceive and / or interact with the physical environment through various means such as hearing, sight, taste, touch, and smell. In contrast, humans can interact with and / or perceive extended reality (XR) environments, which are simulated whole or partially, using electronic devices. For example, XR environments may include mixed reality (MR) content, augmented reality (AR) content, and virtual reality (VR) content. XR environments are often referred to herein as computer-generated environments. In an XR system, some or a representation of a person's body movements may be tracked, and accordingly, the properties of virtual objects simulated within the XR environment may be adjusted to behave according to at least one physical law. For example, an XR system can detect the movement of a user's head and adjust the graphical and auditory content presented to the user as if such views and sounds were changing within the physical environment. In another example, an XR system can detect the movement of an electronic device (e.g., a mobile phone, tablet, or laptop) presenting an XR environment and adjust the graphical and auditory content presented to that user as if such views and sounds were changing within the physical environment. In some situations, an XR system can adjust the characteristics of the graphical content(single or multiple) in response to other inputs, such as representations of body movements (e.g., voice commands).

[0015] Many different types of electronic devices can enable a user to interact with and / or perceive an XR environment. A non-exclusive list of examples includes heads-up displays (HUDs), head-mounted devices, projection devices, windows or vehicle windshields with built-in display capabilities, displays formed as lenses placed in the user's eyes (e.g., contact lenses), headphones / earphones, input devices with or without haptic feedback (e.g., wearable or handheld controllers), speaker arrays, smartphones, tablets, and desktop / laptop computers. A head-mounted device may have one or more speakers and an opaque display. Other head-mounted devices may be configured to accept an opaque external display (e.g., a smartphone). A head-mounted device may include one or more image sensors for capturing images or videos of the physical environment and / or one or more microphones for capturing sounds of the physical environment. A head-mounted device may have a transparent or translucent display instead of an opaque display. Transparent or translucent displays may have a medium that directs passing light toward the user's eye. Displays may utilize various display technologies such as uLEDs, OLEDs, LEDs, liquid crystal on silicon, laser scanning light sources, digital light projection, or combinations thereof. The medium can include optical waveguides, optical reflectors, holographic media, optical combiners, or combinations thereof. In some implementations, transparent or translucent displays can be selectively controlled to be opaque. Projection devices can utilize retinal projection technology to project images onto the user's retina. Projection devices can also project virtual objects into the physical environment (e.g., as holograms or onto physical surfaces).

[0016] FIG. 1 shows an electronic device 100 configured to display a computer-generated environment according to some embodiments of the present disclosure. In some embodiments, the electronic device 100 is a portable electronic device such as, among other possibilities, a tablet computer, a laptop computer, or a smartphone. An exemplary architecture of the electronic device 100 is described in further detail with reference to FIGS. 2A-2B. FIG. 1 shows the electronic device 100 and the table 104A disposed within the physical environment 102. In some embodiments, the electronic device 100 is configured to capture and / or display an area of the physical environment 102 that includes the table 104A (shown within the field of view of the electronic device 100). In some embodiments, the electronic device 100 is configured to display one or more virtual objects within the computer-generated environment that do not exist within the physical environment 102 but are displayed within the computer-generated environment (e.g., placed on top of or otherwise fixed to a computer-generated representation 104B of the real-world table 104A). In FIG. 1, for example, an object 106 (e.g., a virtual object) that does not exist within the physical environment is optionally displayed on the surface of the table 104B within the computer-generated environment displayed via the device 100 in response to detection of the plane of the table 104A within the physical environment 102. The object 106 is a representative object, and it should be understood that one or more different objects (e.g., of various dimensions such as two-dimensional or three-dimensional objects) can be included in and rendered within the two-dimensional or three-dimensional computer-generated environment. For example, the virtual object can include an application or user interface displayed within the computer-generated environment. Additionally, it should be understood that the three-dimensional (3D) environment (or 3D object) described herein can be a representation of a 3D environment (or 3D object) displayed in a two-dimensional (2D) context (e.g., displayed on a 2D display screen).

[0017] Figures 2A-2B illustrate exemplary block diagrams of architectures related to one or more devices according to some embodiments of the present disclosure. The blocks in FIG. 2A can represent information processing apparatuses for use in a device. In some embodiments, the electronic device 200 is optionally a portable device such as a mobile phone, smartphone, tablet computer, notebook computer, or an auxiliary device capable of communicating with other devices. As shown in FIG. 2A, the device 200 includes various sensors (such as, for example, one or more handtracking sensors 202, one or more location sensors 204, one or more image sensors 206, one or more touch sensing surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other audio sensors), one or more display generation components 214, one or more speakers 216, one or more processors 218, one or more memories 220, and / or a communication circuit configuration 222. One or more communication buses 208 are optionally used for communication between the above-described components of the device 200.

[0018] The communication circuit configuration 222 optionally includes a circuit configuration for communicating with an electronic device, and networks such as the Internet, an intranet, a wired network and / or a wireless network, a cellular network and a wireless local area network (LAN). The communication circuit configuration 222 optionally includes a circuit configuration for communicating using short-range communication such as Near-Field Communication (NFC) and / or Bluetooth (registered trademark).

[0019] The processor(s) 218 ​​optionally includes one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors (DSPs). In some embodiments, memory 220 is a non-temporary computer-readable storage medium (e.g., flash memory, random-access memory, or other volatile or non-volatile memory or storage) that stores computer-readable instructions configured to be executed by the processor(s) 218 ​​to perform the techniques, processes, and / or methods described below. In some embodiments, memory 220 includes two or more non-temporary computer-readable storage mediums. A non-temporary computer-readable storage medium can be any (non-signal) medium that can tangibly store or store computer-executable instructions used by or in association with an instruction execution system, apparatus, or device. In some embodiments, the storage medium is a temporary computer-readable storage medium. In some embodiments, the storage medium is a non-temporary computer-readable storage medium. Non-temporary computer-readable storage media may include, but are not limited to, magnetic storage devices, optical storage devices, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, CDs, DVDs, or optical disks based on Blu-ray® technology, as well as persistent solid-state memory such as flash and solid-state drives.

[0020] The display generation component(s) 214 optionally includes a single display (e.g., a Liquid-Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or other types of displays). In some embodiments, the display generation component(s) 214 includes multiple displays. In some embodiments, the display generation component(s) 214 includes displays, projectors, holographic projectors, retinal projectors, etc., having a touch-sensitive surface (e.g., a touchscreen).

[0021] In some embodiments, device 200 includes one or more touch-sensitive surfaces 209 configured to receive user input (touch input and / or proximity input), such as tap input and swipe input or other gestures. In some embodiments, display-generating components 214 and touch-sensitive surfaces 209 together form a touch-sensitive display (one or more) (e.g., a touchscreen built into device 200, or an external touchscreen of device 200 that can communicate with device 200). It should be understood that device 200 optionally includes or receives input from one or more other physical user interface devices other than the touch-sensitive surfaces, such as a physical keyboard, mouse, stylus, and / or joystick (or any other suitable input device).

[0022] The image sensor(s) 206 optionally includes one or more visible light image sensors, such as Charge Coupled Device (CCD) sensors, and / or Complementary Metal-Oxide-Semiconductor (CMOS) sensors capable of operating to acquire images of physical objects from a real-world environment. The image sensor(s) 206 optionally includes one or more IR or NIR sensors, such as passive or active Infrared (IR) or Near Infrared (NIR) sensors, for detecting infrared or near-infrared light from a real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. The image sensor(s) 206 also optionally includes one or more cameras configured to capture the movement of physical objects in a real-world environment. The image sensor(s) 206 optionally includes one or more depth sensors configured to detect the distance of a physical object from the device 200. In some embodiments, information from one or more depth sensors can enable the device to identify an object in a real-world environment and distinguish it from other objects in the real-world environment. In some embodiments, one or more depth sensors can enable the device to determine the texture and / or topography of an object in a real-world environment.

[0023] In some embodiments, device 200 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment surrounding device 200. In some embodiments, the image sensor(s) 206 includes a first image sensor and a second image sensor. The first and second image sensors cooperate and are configured to optionally capture different information about physical objects in the real-world environment. In some embodiments, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some embodiments, device 200 uses the image sensor(s) 206 to detect the position and orientation of device 200 and / or display generation component(s) 214 in the real-world environment. For example, device 200 uses the image sensor(s) 206 to track the position and orientation of the display generation component(s) 214 relative to one or more stationary objects in the real-world environment.

[0024] In some embodiments, device 200 optionally includes one or more hand tracking sensors 202 and / or one or more eye tracking sensors 212. The hand tracking sensors 202 are configured to track the position / location of the user's hand and / or fingers relative to a computer-generated environment, a display-generating component 214, and / or another coordinate system, and / or the movement of the user's hand and / or fingers. The eye tracking sensors 212 are configured to track the position and movement of the user's line of sight (more commonly, eyes, face, or head) relative to the real world or a computer-generated environment, and / or a display-generating component 214. The user's line of sight may include the direction the eyes are directed, and optionally, the intersections with specific points or regions in space, and / or specific objects. In some embodiments, the hand tracking sensor(s) 202 and / or eye tracking sensor(s) 212 are implemented together with the display generation component(s) 214 (e.g., within the same device). In some embodiments, the hand tracking sensor(s) 202 and / or eye tracking sensor(s) 212 are implemented separately from the display generation component(s) 214 (e.g., in different devices).

[0025] In some embodiments, the hand tracking sensor(s) 202 uses one or more image sensors(s) 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that capture three-dimensional information from the real world, including one or more hands. In some examples, the hand may be resolved with sufficient resolution to distinguish the fingers and their respective positions. In some embodiments, the one or more image sensors(s) 206 are positioned relative to the user so as to define the field of view of the image sensors(s) and the interaction space where the finger / hand positions, orientations, and / or movements captured by the image sensors are used as input (e.g., to distinguish the user's stationary hand from other hands of other people in the real-world environment). Tracking fingers / hands (e.g., gestures) for input can be advantageous in that it provides an input means that does not require the user to touch or hold an input device, and using image sensors enables tracking without requiring the user to wear beacons or sensors etc. on their hands / fingers.

[0026] In some embodiments, the eye-tracking sensor(s)212 includes one or more eye-tracking cameras (e.g., IR cameras) and / or illumination sources (e.g., IR light sources / LEDs) that emit light toward the user's eyes. The eye-tracking cameras may be directed toward the user's eyes so as to receive reflected light from the light sources directly or indirectly through the eyes. In some embodiments, both eyes are tracked separately by their respective eye-tracking cameras and illumination sources, and the line of sight can be determined from the tracking of both eyes. In some embodiments, one eye (e.g., the dominant eye) is tracked by a separate eye-tracking camera / illumination source(s).

[0027] Device 200 optionally includes one or more microphones 213 or other audio sensors. Device 200 uses the one or more microphones 213 to detect sounds from the user and / or the user's real-world environment. In some embodiments, the one or more microphones 213 optionally include an array of microphones that work together (e.g., to identify ambient noise or to locate sound sources in the space of the real-world environment). In some embodiments, the voice and / or voice input may be used for interaction with a user interface or computer-generated environment captured using one or more audio sensors (e.g., microphones), as permitted by the user of the electronic device.

[0028] Device 200 optionally includes one or more location sensors 204 configured to detect the location of device 200 and / or one or more display generating components 214. For example, the location sensors 204 optionally include a GPS receiver that receives data from one or more satellites, enabling device 200 to determine the absolute position of the device in the physical world.

[0029] Device 200 optionally includes motion and / or orientation sensors (one or more) 210 configured to detect the orientation and / or movement of device 200 and / or display generation components (one or more) 214. For example, device 200 uses orientation sensors (one or more) 210 to track changes in the position and / or orientation of device 200 and / or display generation components (one or more) 214 (e.g., relative to physical objects in a real-world environment). Orientation sensors (one or more) 210 optionally include one or more gyroscopes, one or more accelerometers, and / or one or more inertial measurement units (IMUs).

[0030] While the architecture in Figure 2A is illustrative, it should be understood that device 200 is not limited to the components and configurations shown in Figure 2A. For example, a device may have the same or different configurations but fewer components, additional components, or other components. In some embodiments, as shown in Figure 2B, system 250 may be divided among multiple devices. For example, a first device 260 may optionally include a processor(s) 218A that communicates via a communication bus(s) 208A, one or more memories 220A, and a communication circuit configuration 222A. A second device 270 (corresponding to, for example, device 200) optionally includes various sensors (e.g., one or more hand tracking sensors 202, one or more location sensors 204, one or more image sensors 206, one or more touch-sensing surfaces 209, one or more motion and / or orientation sensors 210, one or more eye tracking sensors 212, one or more microphones 213 or other sound sensors), one or more display generating components 214, one or more speakers 216, one or more processors 218B, one or more memories 220B, and / or communication circuit configurations 222B. One or more communication buses 208B are optionally used for communication between the aforementioned components of device 270. The details of the components of devices 260 and 270 are the same as those of the corresponding components of device 200 described above, and will not be repeated here for brevity. The first device 260 and the second device 270 optionally communicate with each other via a wired or wireless connection (for example, via communication circuit configurations 222A to 222B).

[0031] Device 200 or System 250 generally supports applications that can be displayed in a variety of computer-generated environments, such as drawing applications, presentation applications, word processing applications, website creation applications, disk authoring applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo / video management applications, digital camera applications, digital video camera applications, web browsing applications, digital music playback applications, TV channel browsing applications, and / or digital video playback applications.

[0032] A computer-generated environment may be displayed using electronic devices (e.g., electronic device 100, device 200, device 270), including the use of one or more display generation components. The computer-generated environment may optionally include various graphical user interfaces ("Graphical User Interface, GUI") and / or user interface objects.

[0033] In some embodiments, electronic devices can detect or estimate real-world lighting characteristics. Estimating lighting characteristics can provide some understanding of lighting in an environment. For example, estimating lighting characteristics can provide indications of which areas of a real-world environment are bright or dark. Estimating lighting characteristics can also provide indications of the position and / or orientation of light sources (e.g., parametric, directional, point, or area light sources). In some embodiments, lighting characteristics are estimated as an incident light field per voxel indicating brightness, color, and / or direction. For example, lighting characteristics can be parameterized as an image-based lighting (IBL) environment map. It should be understood that other parameterizations of lighting characteristics are also possible. In some examples, lighting characteristics are estimated per pixel using a triangular mesh with lighting characteristics that define lighting for each vertex or face. In addition, it should be understood that estimated lighting characteristics can optionally be derived from an intermediate representation (e.g., an environment map).

[0034] In some embodiments, sensors such as a camera (e.g., one or more image sensors 206) are used to capture images of the real-world environment. By processing the images with a processing circuit configuration (one or more processors 218), the location of light sources can be identified and measured. In some embodiments, light can be determined from reflections and / or shadows projected by light sources in the environment. In some embodiments, (e.g., supervised) deep learning or other artificial intelligence or machine learning is used to estimate lighting characteristics based on the input image(s).

[0035] As described herein, computer-generated environments, including various graphics user interfaces ("GUIs"), may be displayed using electronic devices such as electronic device 100 or device 200, which include one or more display-generating components. The computer-generated environment may include one or more virtual objects. In some embodiments, one or more virtual objects may interact with or be manipulated within a three-dimensional environment. For example, a user may move or rotate a virtual object. As will be described in more detail below, interactions with virtual objects may be direct or indirect, and the device may automatically interpret user input as either direct or indirect manipulation based on context, such as the position of the user's hand and / or the position of the virtual object to be manipulated.

[0036] Figure 3 illustrates a method for displaying a three-dimensional environment 300 having one or more virtual objects, according to some embodiments of the present disclosure. In Figure 3A, an electronic device (e.g., device 100 or 200 described above) is displaying the three-dimensional environment 300. In some embodiments, the three-dimensional environment 300 includes one or more real-world objects (e.g., representations of objects in the physical environment surrounding the device) and / or one or more virtual objects (e.g., representations of objects generated and displayed by the device, not necessarily based on real-world objects in the physical environment surrounding the device). For example, in Figure 3A, the table 302 and frame 304 may be representations of real-world objects in the physical environment surrounding the device. In some embodiments, the table 302 and frame 304 are displayed by a display generation component by capturing one or more images of the table 302 and frame 304 (e.g., in the physical environment surrounding the device) and displaying representations of the table and frame (e.g., photorealistic representations, simplified representations, caricatures, etc.) in the three-dimensional environment 300, respectively. In some embodiments, the table 302 and frame 304 are passively provided by the device via transparent or semi-transparent display so as not to obscure the user's view of the table 302 and frame 304. In Figure 3A, the cube 306 is a virtual object that is displayed on top of the table 302 within the three-dimensional environment 300, but does not exist in the physical environment surrounding the device. In some embodiments, virtual devices such as the cube 306, which is displayed positioned on top of the table 302 in Figure 3, can interact with representations of real-world objects, both when the representations of real-world objects are actively displayed by the device and when they are passively displayed by the device.

[0037] In some embodiments, the table 302 and frame 304 are representations of real-world objects in the environment surrounding the device and therefore cannot be manipulated by the user through the device. For example, since the table 302 exists in the physical environment surrounding the device, in order to move or otherwise manipulate the table 302, the user can move or manipulate the table 302 in the three-dimensional environment 300 by physically moving or manipulating the table 302 in the physical environment surrounding the device. In contrast, since the cube 306 is a virtual object, the cube 306 can be manipulated by the user of the device through the device (for example, without requiring the user to manipulate an object in the physical world surrounding the device), as will be described in more detail below.

[0038] Figures 4A–4D illustrate methods for indirectly manipulating virtual objects according to some embodiments of the present disclosure. In Figure 4A, a device (e.g., device 100 or device 200) displays a three-dimensional environment 400 (similar to, for example, three-dimensional environment 300) including a cube 406 on a table 402 via a display generation component. In some embodiments, the cube 406 is a virtual object similar to the cube 306 described above with respect to Figure 3. Figure 4A shows the cube 406 twice, but it should be understood that the second cube 406 shown near the bottom of the figure (e.g., near the hand 410) is not shown in the three-dimensional environment 400 and is shown in Figure 4A for the purpose of showing the distance of the hand 410 from the cube 406 (e.g., on the table 402) when performing gesture A, as will be described in more detail below. In other words, the three-dimensional environment 400 does not contain two copies of cube 406 (for example, the second cube 406 near hand 410 is a replica of cube 406 on table 402, shown for illustrative purposes, and this replica is not shown in Figures 4B-4D).

[0039] In Figure 4A, hand 410 is the user's hand of the device, and the device can track the position of hand 410 and / or detect gestures made by hand 410 (e.g., via one or more hand tracking sensors). In some embodiments, a representation of hand 410 is displayed in a three-dimensional environment 400, and for example, if hand 410 is held in front of the device, the device can capture an image of hand 410 and display (or passively provide visibility of hand 410) a representation of hand 410 at a corresponding location in the three-dimensional environment. In other embodiments, hand 410 may be a real-world object in a physical environment, passively provided by the device via a transparent or translucent display so as not to obscure the user's view of the hand. As used herein, a reference to a physical object such as a hand may refer to either a representation of that physical object presented on a display, or the physical object itself, such as passively provided by a transparent or translucent display. Thus, as the user moves hand 410, the representation of hand 410 moves accordingly within the three-dimensional environment 400.

[0040] In some embodiments, the user can use their hand 410 to interact with virtual objects in a three-dimensional environment 400 as if the user were interacting with real-world objects in the physical environment surrounding the device. In some embodiments, the user's interaction with the virtual object can refer to either a direct manipulation interaction or an indirect manipulation interaction. In some embodiments, a direct manipulation interaction includes an interaction in which the user uses one or both hands to directly manipulate the virtual object by crossing over it (or entering within a threshold distance of the virtual object). In some embodiments, an indirect manipulation interaction includes an interaction in which the user uses one or both hands to manipulate the virtual object without the hands crossing over it (or entering within a threshold distance of the virtual object).

[0041] Returning to Figure 4A, the device detects (e.g., via one or more hand-tracking sensors) that the hand 410 is performing a first gesture (e.g., "Gesture A") corresponding to a selection input while the gaze 408 is directed towards the virtual object 406. In some embodiments, the gaze 408 is detected via one or more eye-tracking sensors to determine the location or object that the user's eyes are looking at or directed towards. In Figure 4A, when the hand 410 performs the first gesture, the hand 410 is farther from the cube 406 than a threshold distance 412.

[0042] In some embodiments, the distance between the hand 410 and the cube 406 is determined based on the distance between the location of the hand 410 in the physical world and the corresponding location of the cube 406 on the table 402 in the physical world. For example, the cube 406 is displayed at a location in a three-dimensional environment 400 that has a corresponding location in the physical world, and the distance between the corresponding location of the cube 406 in the physical world and the location of the user's hand 410 in the physical world is used to determine whether the hand 410 is farther away from the cube 406 than a threshold distance 412. In some embodiments, the distance can be determined based on the distance between the location of the hand 410 in the three-dimensional environment and the distance between the cube 406 in the three-dimensional environment 400. For example, the representation of the hand 410 is displayed at a separate location in the three-dimensional environment 400, and the distance between the separate position of the hand 410 in the three-dimensional environment 400 and the position of the cube 406 in the three-dimensional environment 400 is used to determine whether the hand 410 is farther away from the cube 406 than a threshold distance 412. For example, if hand 410 is held 1 foot in front of the user (e.g., not extended toward cube 406) and cube 406 is 6 feet away from the user, hand 410 is determined to be 5 feet away from hand 410. In some embodiments, the threshold distance 412 may be 1 inch, 3 inches, 6 inches, 1 foot, 3 feet, etc.

[0043] In some embodiments, the first gesture corresponding to the selection input may be a pinch gesture using two or more fingers of the user or one or both hands (e.g., a pinch between the thumb and index finger of hand 410). In some embodiments, the first gesture corresponding to the selection input may be a pointing gesture or a tap gesture using a finger of hand 410 (e.g., the index finger of hand 410). In some embodiments, the predefined gesture corresponding to the selection input may be anything.

[0044] In some embodiments, the device is configured in an indirect operation mode in which user input is directed to the virtual object the user's gaze is directed to when the input is received, based on the determination that a selection gesture (e.g., a pinch gesture, “Gesture A”) has been performed by hand 410 while hand 410 is more than a threshold distance 412 away from cube 406 (e.g., more than a threshold distance 412 away from any virtual object, arbitrarily). For example, in Figure 4A, gaze 408 is directed to cube 406 (e.g., looking at cube 406, focusing on cube 406) when hand 410 was performing the selection input. Thus, the selection input is performed on cube 406 (e.g., cube 406 is selected as the object of operation). In some embodiments, cube 406 remains selected while hand 410 maintains the selection gesture. While cube 406 remains selected, an operation is performed on cube 406 by an operation gesture of hand 410 (for example, even if the gaze 408 optionally moves away from cube 406).

[0045] Figure 4B illustrates how a virtual object is moved within a three-dimensional environment 400. In Figure 4B, the device detects that the hand 410 is moving to the right (for example, in the "x" axis direction) by a discrete amount 414 while maintaining a selection gesture. In some embodiments, moving the hand 410 to the right by a discrete amount 414 corresponds to a angular movement of the hand 410 by a discrete angle 416. For example, to move the hand 410 by a discrete amount 414, the user rotates their discrete arm by a discrete angle 416. In some embodiments, the discrete angle 416 is the angle formed between a first line extending outward from the device's position to the hand's previous position and a second line extending outward from the device's position to the hand's new position.

[0046] In Figure 4B, in response to the detection that hand 410 has moved to the right by an individual amount 414 while maintaining a selection gesture, cube 406 is similarly moved to the right (e.g., in the "x" axis direction) by a second individual amount 418 within the three-dimensional environment 400. In some embodiments, the second individual amount 418 is different from individual amount 414. In some embodiments, the second individual amount 418 is an individual amount 414 scaled by a scaling factor. In some embodiments, the scaling factor is based on the distance of cube 406 from the user (e.g., the distance of cube 406 from the "camera" in the three-dimensional environment 400, the distance of cube 406 from a location in the three-dimensional environment 400 associated with the user, and / or the location from which the user is viewing the three-dimensional environment 400). In some embodiments, the second individual amount 418 is calculated such that the angular change by cube 406 is the same as the angular change by hand 410. For example, a second individual angle 420 (e.g., the angle formed between a first line extending outward from the device's position to the previous position of the cube 406 and a second line extending outward from the device's position to the new position of the cube 406) is equal to an individual angle 416. Thus, in some embodiments, the scaling factor used for the second individual quantity 414 is calculated based on the distance of the cube 406 from the user and the distance of the hand 410 from the user (e.g., the ratio of the two distances).

[0047] In some embodiments, the movement of the cube 406 can be in any direction based on the movement of the hand 410 (for example, the cube 406 has 6 degrees of freedom), as will be described in more detail below. In some embodiments, the movement of the cube 406 can be fixed to one dimension based on the movement of the hand 410. For example, if the initial movement of the hand 410 is along the x-direction (for example, the horizontal component of the hand 410's movement is greater than other movement components of the hand 410's movement during the first 0.1 seconds, 0.3 seconds, 0.5 seconds, 1 second, or during the first 1 cm, 3 cm, 10 cm, or during the movement), then the movement of the cube 406 is fixed to horizontal movement only until the selection input is finished (for example, the cube 406 moves only horizontally based on the horizontal component of the cube 406's movement, and does not move vertically and / or change in depth even if the hand 410 includes vertical and / or depth movement components and / or moves vertically and / or changes in depth).

[0048] Figure 4C illustrates how a virtual object is rotated within a three-dimensional environment 400. In Figure 4C, the device detects that the hand 410 is rotating by a distinct amount 422 while maintaining a selection gesture. In some embodiments, the rotation of the hand 410 is along the yaw direction (e.g., clockwise, such that the fingers rotate to the right relative to the wrist and the wrist rotates to the left relative to the fingers). In some embodiments, the rotation of the hand 410 is along the roll direction (e.g., the fingers and wrist maintain their respective positions relative to each other, but the hand 410 reveals a portion of the hand 410 that previously faced the other direction (e.g., a portion that was previously obscured and / or facing outward relative to the device)). In some embodiments, a rotation of the hand 410 (e.g., along any orientation) that does not involve lateral movement (e.g., horizontal, vertical, or depth changes) or involves lateral movement less than a threshold amount (e.g., less than 1 inch, less than 3 inches, less than 6 inches, less than 1 foot, etc.) is interpreted as a request to rotate the cube 406.

[0049] In Figure 4C, in response to the detection that the hand 410 has rotated by a separate amount 422 while maintaining a selection gesture, the cube 406 rotates by a second separate amount 424 in accordance with the rotation of the hand 410. In some embodiments, the cube 406 rotates in the same direction as the rotation of the hand 410. For example, if the hand 410 rotates in the yaw direction, the cube 406 rotates in the yaw direction, and if the hand 410 rotates in the roll direction, the cube 406 rotates in the roll direction. In some embodiments, the second separate amount 424 by which the cube 406 rotates is the same as the separate amount 422 by which the hand 410 rotates. For example, if the hand 410 rotates 90 degrees, the cube 406 rotates 90 degrees in the same direction.

[0050] In some embodiments, the second distinct amount 424 by which the cube 406 is rotated is different from the distinct amount 422 of rotation by the hand 410 (e.g., the rotation is damped or amplified). For example, if the cube 406 can only be rotated 180 degrees (e.g., if a property of the cube 406 is that it cannot be turned upside down), the rotation of the cube 406 can be scaled in half (e.g., a 90-degree rotation by the hand 410 rotates the cube 406 by 45 degrees). In another example, if cube 406 can be rotated only 180 degrees, cube 406 will rotate 180 degrees in response to the 180-degree rotation of hand 410, but cube 406 will not rotate in response to further rotations by hand 410 (e.g., rotations exceeding 180 degrees), or will exhibit a rubber-banding effect or resistance to further rotations by hand 410 (e.g., cube 406 will temporarily rotate beyond its maximum amount while hand 410 continues to rotate, but will return to its maximum rotation value when the rotation and / or input ends).

[0051] Figure 4D illustrates how a virtual object can be moved toward or away from the user within a three-dimensional environment 400. In Figure 4D, the device detects that the hand 410 is moving toward the user by a distinct amount 426 while maintaining a selection gesture (e.g., pulling the hand back from a position where it was extended toward the user's body and / or toward the device). Thus, the distance between the hand 410 and the device is reduced (e.g., movement in the z direction).

[0052] In Figure 4D, in response to the detection of movement by hand 410, which is moving toward the user and / or device by a discrete amount 426 while maintaining a selection gesture, cube 406 is moved toward the user by a second discrete amount 428 (e.g., closer to the "camera" of the 3D environment 400). In some embodiments, the amount by which cube 406 moves (e.g., the second discrete amount 428) is the same as the amount of movement by hand 410 (e.g., the discrete amount 426) and optionally along the same direction as hand 410. In some embodiments, the amount by which cube 406 moves (e.g., the second discrete amount 428) is different from the amount of movement by hand 410 (e.g., the discrete amount 426) and optionally along the same direction as hand 410. In some embodiments, the amount by which cube 406 moves is based on the distance of cube 406 from the user and / or the distance of hand 410 from the user. For example, if the cube 406 is farther away from the user, the cube 406 will move more for the same amount of movement by the hand 410 than if the cube 406 is closer to the user. For example, if the hand 410 moves 6 inches toward the user (e.g. toward the device, toward the device's camera), if the cube 406 is farther away from the user, the cube 406 can move 2 feet closer, but if the cube 406 is closer to the user, the cube 406 can move 6 inches closer.

[0053] In some embodiments, the amount of movement by the cube 406 is scaled based on the ratio of the distance the cube 406 is from the user and / or device to the distance the hand 410 is from the user and / or device when the selection input (e.g., a pinch gesture) is first received. For example, if the hand 410 is 2 feet away from the user (e.g., 2 feet from the user's eyes, 2 feet from the device, and 2 feet from the device's camera) and the cube 406 is 10 feet away from the user (e.g., 10 feet from the user's eyes, 10 feet from the device, and 10 feet from the device's camera), then the scaling factor is 5 (e.g., the distance of the cube 406 divided by the distance of the hand 410). Thus, a 1-inch movement of the hand 410 along the z-axis (e.g., towards or away from the user) causes a 5-inch movement of the cube 406 along the same direction (e.g., towards or away from the user). Therefore, the cube 406 moves closer to the user as the user brings the hand 410 closer to the user, so as the hand 410 moves closer to the user, the cube 406 also moves closer to the user. In this way, the user can use the hand 410 to move the cube 406 from its initial position to the user without requiring the user to perform input multiple times. In some embodiments, the cube 406 is moved to the user's location. In some embodiments, the cube 406 is moved to the hand 410's location so that the cube 406 is in contact with the hand 410 or within a threshold distance of the hand 410 (e.g., 1 inch, 3 inches, 6 inches, etc.). In some embodiments, once the cube 406 is moved to the hand 410's location, the user can use the hand 410 to directly manipulate the cube 406, as will be described in more detail below with reference to Figures 5A-5D and 6A-6B.

[0054] In some embodiments, instead of scaling movement based on distance from the user (e.g., cube 406 and / or hand 410), movement is based on distance from a location that is a predetermined distance in front of the user (e.g., a predetermined reference location that is optionally the user's location or a location in front of the user). For example, the reference location could be the user's location, the user's face location, the device location (e.g., as described above), or 3 inches in front of the user (or the user's face, or device), 6 inches in front of the user (or the user's face, or device), 1 foot in front, 3 feet in front, etc. Thus, by using a reference location that is not exactly the user's location, the user can carry cube 406 from a distant location to the user and / or hand 410 by moving hand 410 to a reference location that is just in front of the user (e.g., without requiring the user to move hand 410 to the user's location, which could be an awkward gesture).

[0055] In some embodiments, the above scaling of the movement of cube 406 applies to both movement toward and away from the user. In some embodiments, the above scaling applies only to movement toward the user (e.g., along the z-axis), and movement away from the user is scaled differently (e.g., scaled one-to-one with the movement of hand 410). In some embodiments, the above scaling applies to movement along a particular direction based on the context and / or type of the element being manipulated. For example, if the user is moving a virtual object in a direction not intended by the designer of the 3D environment, the movement of the virtual object may be attenuated (e.g., scaled smaller), but if the user is moving a virtual object in a direction intended by the designer, the movement of the virtual object may be amplified (e.g., scaled larger). Thus, the scaling factor may differ based on the direction of movement to provide the user with feedback on whether a particular direction of movement is coincidental or intended.

[0056] It should be understood that the movement of the virtual object described above is not limited to only one type of operation at a time, or only movement along one axis at a time. For example, the user can move the virtual object (e.g., cube 406) in both the x, y directions (e.g., as in Figure 4B) and the z direction (e.g., changing the depth as in Figure 4D) while simultaneously rotating the virtual object (e.g., as shown in Figure 4C). Thus, the device can determine different movement and / or rotation components of hand 410 and perform appropriate operations on the virtual object. For example, if hand 410 moves to the left while simultaneously moving closer to the user (e.g., while maintaining selection of cube 406), the device can move cube 406 to the left as described above with respect to Figure 4B, while simultaneously moving cube 406 closer to the user as described above with respect to Figure 4D. Similarly, if hand 410 moves to the left while simultaneously rotating, the device can move cube 406 to the left as described above with respect to Figure 4B, while simultaneously rotating cube 406 as described above with respect to Figure 4C.

[0057] Therefore, as described above, during the execution of an indirect operation, the direction, magnitude, and / or speed of the operation may depend on the direction, magnitude, and / or speed of the user's hand movement. For example, during a move operation, if the user's hand moves to the right, the virtual object being manipulated moves to the right; if the user's hand moves to the left, the virtual object moves to the left; if the user's hand moves forward (e.g., away from the user), the virtual object moves forward (e.g., away from the user), and so on. Similarly, if the hand moves quickly, the virtual object moves quickly at will; if the hand moves slowly, the virtual object moves slowly at will. Also, as described above, the amount of movement depends on the amount of hand movement (e.g., it is optionally scaled based on the distance from the user, as described above). In some embodiments, during the execution of a rotation operation, the direction, magnitude, and / or speed of rotation depend on the direction, magnitude, and / or speed of rotation of the user's hand, similar to what is described above for move operations.

[0058] Figures 5A–5D illustrate methods for directly manipulating virtual objects according to some embodiments of the present disclosure. In Figure 5A, the device displays a three-dimensional environment 500 (similar to, for example, three-dimensional environments 300 and 400) including a cube 506 on a table 502 via a display generation component. In some embodiments, the cube 506 is a virtual object similar to the cubes 306 and 406 described above with respect to Figures 3 and 4A–4D. As described above with respect to Figure 4A, Figure 5A shows the cube 506 twice, but it should be understood that the second cube 506, shown near the bottom of the figure (e.g., near the hand 510), is not shown in the three-dimensional environment 500 and is shown in Figure 5A for the purpose of showing the distance of the hand 510 from the cube 506 (e.g., on the table 502) when performing gesture A, as will be described in more detail below. In other words, the three-dimensional environment 500 does not contain two copies of cube 506 (for example, the second cube 506 near hand 510 is a duplicate of cube 506 on table 502, shown for illustrative purposes, and the duplicate is not shown in Figures 5B-5D).

[0059] As described above, direct manipulation is an interaction with a virtual object in which a user crosses the virtual object using one or more hands when manipulating the virtual object. For example, grasping a virtual object in the same way as grasping a physical object and moving the hand grasping the virtual object is an example of moving a virtual object via direct manipulation. In some embodiments, whether a user is performing a direct or indirect manipulation action on a virtual object depends on whether the user's hand is within a threshold distance from the virtual object being manipulated. For example, if the user's hand is in contact with the virtual object (e.g., at least a portion of the user's hand is in a location in physical space such that a portion of the hand appears to be in contact with or crossing the virtual object in a three-dimensional environment), the user is directly interacting with the virtual object. In some embodiments, the device may interpret a user's interaction as direct manipulation if the user's hand is within a threshold distance 512 from the virtual object being manipulated (e.g., within 1 inch, 6 inches, 1 foot, 3 feet, etc.). In some embodiments, user input when the hand 510 is within a threshold distance 512 of the virtual object is directed towards the virtual object. For example, if hand 510 is within a threshold distance 512 of one virtual object, user input is directed to that virtual object (optionally, regardless of whether the user's gaze is directed towards that virtual object). If hand 510 is within a threshold distance 512 of two virtual objects, user input may be directed to the closer virtual object, or the virtual object closer to the part of hand 510 performing the input (for example, closer to the pinch location if the selection input is a pinch gesture), or the virtual object to which the user's gaze is directed. If hand 510 is not within a threshold distance 512 of any virtual object, the device can determine whether the user is performing indirect manipulation of a virtual object (for example, if the user's gaze is directed towards a particular virtual object), as described above with respect to Figures 4A to 4D.

[0060] In Figure 5A, the device detects that hand 510 is performing a gesture corresponding to a selection input (e.g., "Gesture A," pinch gesture, tap gesture, poke gesture, etc.) when hand 510 is within threshold distance 512 of cube 506. In some embodiments, when hand 510 is within threshold distance 512 of cube 506 (and optionally, when hand 510 is not within threshold distance 512 of any other virtual object), cube 506 is selected as an input target so that further user input (e.g., object manipulation input, etc.) can be performed on cube 506. In Figure 5A, cube 506 is selected as an input target even though the user's line of sight 514 is directed towards table 502 when the selection input is performed. Thus, in some embodiments, the user can interact with the virtual object through direct manipulation of the virtual object without requiring the user to look at the virtual object.

[0061] In Figure 5B, depending on the selection of cube 506 as the input object, in some embodiments, cube 506 is automatically rotated by a separate amount 516 such that cube 506 aligns with one or more axes and / or one or more surfaces of an object. For example, the orientation of cube 506 is snapped to the nearest axis such that at least one boundary of cube 506 aligns with the x-axis (e.g., perfectly horizontal), y-axis (e.g., perfectly vertical), or z-axis (e.g., perfectly flat). In some embodiments, cube 506 is automatically snapped to an upward orientation (e.g., aligned with gravity and / or other objects in the environment). In some embodiments, depending on the selection of cube 506 as the input object, cube 506 is snapped to the same orientation as hand 510. For example, if hand 510 is angled at 30 degrees (e.g., as shown in Figure 5B), cube 506 can be snapped to an orientation rotated 30 degrees. In some embodiments, the cube 506 does not change its orientation in response to being selected as the target of the input, but maintains the orientation it had when the selection input was received (for example, as shown in Figure 5A). In some embodiments, the cube 506 is automatically snapped to the orientation of the surface of the table 502 (for example, so that the bottom surface of the cube 506 is coplanar with the top surface of the table 502).

[0062] Figure 5C illustrates how a virtual object is moved within a three-dimensional environment 500. In Figure 5C, the device detects that the hand 510 is moving to the right (e.g., in the "x" axis direction) by a distinct amount 518 while maintaining a selection gesture (e.g., a pinch gesture, pointing gesture, tap gesture, etc.). In response to the detection of the hand 510 moving to the right, the device optionally moves the cube 506 to the right by a second distinct amount 520. In some embodiments, the cube 506 moves by the same amount as the hand 510 so that the relative distance and / or relative position between the cube 506 and the hand 510 is maintained. For example, if the cube 506 was 3 inches in front of the hand 510 when the selection input is received, the cube 506 moves with the movement of the hand 510 in response to user input (and optionally while user input is being received) and remains 3 inches in front of the hand 510. In some embodiments, the movement of the cube 506 along the x and y directions is scaled one-to-one with the movement of the hand 510. Thus, in some embodiments, the movement of the cube 506 simulates the hand 510 physically holding and moving the cube 506, where the cube 506 moves in the same direction, by the same amount, and at the same speed as the hand 510 (for example, during indirect manipulation, the cube 506 may optionally move more or less than the movement of the hand 510, as described above with reference to Figure 4B). In some embodiments, the movement of the cube 506 during direct manipulation is not fixed to a specific direction of movement, but can move in any direction (e.g., 6 degrees of freedom) based on the hand's movement (for example, during the execution of some embodiments of indirect manipulation, the movement of the virtual object is fixed to one direction of movement, such as the x, y, or z axis, and the hand's movement in other directions is filtered out, ignored, or otherwise does not move the virtual object in those other directions).

[0063] Figure 5D illustrates how a virtual object can be moved towards or away from the user within a three-dimensional environment 500. In Figure 5D, the device detects that the hand 510 is moving forward by a distinct amount 522 (e.g., away from the user and / or the device along the z-direction) while maintaining a selection gesture (e.g., a pinch gesture, pointing gesture, tap gesture, etc.). In response to the detection that the hand 510 is moving further away, the device optionally moves the cube 506 further away by a second distinct amount 524. In some embodiments, the cube 506 moves by the same amount as the hand 510 so that the relative distance and / or relative position between the cube 506 and the hand 510 is maintained. Therefore, changes in the distance of the cube 506 from the user and / or device (e.g., in the direction away from the user and in the direction toward the user) are optionally scaled one-to-one with the movement of the hand 510 (e.g., during indirect manipulation, movement toward and / or away from the user is optionally not scaled one-to-one with the movement of the hand 510).

[0064] In some embodiments, when the hand 510 is rotated while performing direct manipulation of the cube 506, the cube 506 rotates accordingly (optionally exhibiting the same or similar behavior as described above with respect to Figure 4C).

[0065] Therefore, as described above, when a user is performing a direct operation on a virtual object, the movement of the virtual object is optionally scaled one-to-one with the movement of the hand performing the selection input, but when a user is performing an indirect operation on a virtual object, the movement of the virtual object is not necessarily scaled one-to-one with the movement of the hand performing the selection input. In some embodiments, rotation input is scaled by the same amount regardless of whether the operation is direct or indirect. In some embodiments, whether a user is performing a direct or indirect operation input is based on whether the user's hand is within a threshold distance of the virtual object when the selection input (e.g., a selection gesture) is received.

[0066] Therefore, as described above, during the execution of a direct operation, the direction, magnitude, and / or speed of the operation may depend on the direction, magnitude, and / or speed of the user's hand movement. For example, during a move operation, if the user's hand moves to the right, the virtual object being manipulated moves to the right; if the user's hand moves to the left, the virtual object moves to the left; if the user's hand moves forward (e.g., away from the user), the virtual object moves forward (e.g., away from the user), and so on. Similarly, if the hand moves quickly, the virtual object moves quickly at an optional rate, and if the hand moves slowly, the virtual object moves slowly at an optional rate. Also, as mentioned above, the amount of movement is scaled one-to-one with the amount of hand movement (in contrast to being scaled by distance as described above in Figures 4A-4D, for example). In some embodiments, during the execution of a rotation operation, the direction, magnitude, and / or speed of rotation depend on the direction, magnitude, and / or speed of rotation of the user's hand, similar to what is described above for move operations.

[0067] Figures 6A–6B illustrate a method for moving a virtual object according to some embodiments of the present disclosure. In Figure 6A, the device displays a three-dimensional environment 600 (similar to, for example, three-dimensional environments 300, 400, and 500) including a cube 606 on a table 602 via a display generation component. In some embodiments, the cube 606 is a virtual object similar to the cubes 306, 406, and 506 described above with respect to Figures 3, 4A–4D, and 5A–5D. As described above with respect to Figures 4A and 5A, Figure 6A shows the cube 606 twice, but it should be understood that the second cube 606 shown near the bottom of the figure (e.g., near the hand 610) is not shown in the three-dimensional environment 600 and is shown in Figure 6A for the purpose of showing the distance of the hand 610 from the cube 606 (e.g., on the table 602) when performing gesture B, as will be described in more detail below. In other words, the three-dimensional environment 600 does not contain two copies of cube 606 (for example, the second cube 606 near hand 610 is a duplicate of cube 606 on table 602, shown for illustrative purposes, and the duplicate is not shown in Figure 6B).

[0068] In Figure 6A, the device detects that the hand 610 has performed a separate gesture (e.g., "gesture B") between cubes where the hand is more than a threshold distance 612 from 606. In some embodiments, the separate gesture includes a pinch gesture (e.g., between the thumb and index finger of the hand, or between any two or more fingers of one or both hands, as described above with respect to "gesture A"). In some embodiments, the separate gesture includes a pinch gesture followed by a predetermined movement and / or rotation of the hand 610 while maintaining the pinch gesture (e.g., gesture B includes gesture A followed by a separate movement by the hand 610). For example, a tag gesture by the hand 610 (e.g., an upward rotation of the hand 610 such that the fingers and / or pinch location move closer to the user and / or rotate toward the user, while optionally maintaining the wrist position). In some embodiments, a separate gesture includes a pinch gesture followed by a movement of hand 610 to move cube 606 from a distant location to hand 610's location, by moving hand 610 to the user's location or a predetermined reference location in front of the user (as described above with reference to, for example, Figure 4D). In some embodiments, a separate gesture corresponds to a request to move cube 606 to a location for direct manipulation (e.g., a location associated with hand 610). In some embodiments, since the separate gesture is an indirect manipulation input (e.g., hand 610 is more than a threshold distance 612 from cube 606), the device uses line of sight 614 to determine that the user's input is directed towards cube 606. It should be understood that a separate gesture can be any predetermined gesture (e.g., including, but not limited to, the selection of selectable options for snapping cube 606 to hand 610's location) that corresponds to a request to move cube 606 to a location for direct manipulation.

[0069] In some embodiments, in response to the detection of a separate gesture (e.g., gesture B) by the hand 610 while the line of sight 615 is directed at the cube 606, the device moves the cube 606 to a location associated with the hand 610, as shown in Figure 6B. In some embodiments, the separate gesture includes a pinch gesture, and the cube 606 is moved to a pinch location (e.g., a portion of the cube 606 is positioned at the pinch location and the hand 610 appears to be pinching a portion of the cube 606), or to a location within a predetermined distance from the pinch (e.g., 1 inch, 3 inches, 6 inches, etc.). Thus, after moving the cube 606 to the pinch location, the user can perform direct operations on the cube 606 by maintaining the pinch gesture (e.g., maintaining the selection input) and performing direct operation gestures (e.g., lateral movement, forward / backward movement, rotation, etc.) similar to those described above with respect to Figures 5A-5D. In some embodiments, moving the cube 606 to a pinch location allows the user to manipulate objects in a location within the three-dimensional environment 600 that might otherwise be far away and out of reach of the user's hands, using direct manipulation inputs.

[0070] The above diagrams and explanations illustrate movement or rotation along a specific direction, but these are merely illustrative examples. It should be understood that virtual objects can exhibit the same or similar behavior for movement or rotation along any direction. For example, a virtual object can be moved to the left and respond to user input similarly to the example above regarding moving the virtual object to the right. Similarly, a virtual object can be rotated counterclockwise and respond to user input similarly to the example above regarding rotating the virtual object clockwise.

[0071] While the above diagrams and explanations illustrate the manipulation of virtual objects, it should be understood that the above methods may also be applicable to any type of user interface element or control element. For example, buttons, sliders, dials, knobs, etc., can be moved or rotated according to the direct or indirect manipulation methods described above.

[0072] Figure 7 is a flowchart illustrating a method 700 for manipulating a virtual object according to an embodiment of the present disclosure. Method 700 is optionally performed on an electronic device such as device 100, device 200, when displaying selectable options on a plane, as described above with reference to Figures 3A-3C, 4A-4B, 5A-5B, and 6A-6B. Some operations in method 700 are optionally combined (e.g., with each other or with operations in method 800), and / or the order of some operations is optionally changed. As described below, method 700 provides a method for manipulating a virtual object according to an embodiment of the present disclosure (e.g., as described above with reference to Figures 3-6B).

[0073] In some embodiments, a display generating component (e.g., a display built into an electronic device (optionally, such as a touchscreen display) and / or an external display such as a monitor, projector, or television) and one or more input devices (e.g., a touchscreen, an external mouse, an optional built-in or external trackpad, an optional built-in or external touchpad, an external remote control device, a separate mobile device (e.g., separate from the electronic device), an external handheld device, an external controller, a camera (e.g., such as a visible light camera), a depth sensor and / or motion sensor (e.g., such as a hand tracking sensor or hand motion sensor), and an electronic device (e.g., a computer such as device 100 and / or device 200) that can communicate with the display generating component presents a computer-generated environment including a first user interface element, such as a computer-generated environment 300 including the cube 306 in Figure 3 (702).

[0074] In some embodiments, while presenting a computer-generated environment, the electronic device receives multiple user inputs (e.g., sequences) including selection inputs and operation inputs, such as a hand 410 performing a gesture corresponding to a selection input in Figure 4A (e.g., gesture A), and a hand 410 moving while maintaining a gesture in Figures 4B-4D (704).

[0075] In some embodiments, upon receiving a selection input, the electronic device determines that a representation of the user's hand is within a threshold distance from the first user interface element, such as a hand 510 within a threshold distance 512 from the cube 506 in Figure 5A. In this determination, the electronic device operates the first user interface element in accordance with an operation input, such as the movement of the cube 506 in accordance with the movement of the hand 510 in Figures 5C-5D (706). In some embodiments, operating the first user interface includes movement, rotation, resizing, or any other preferred operation. In some embodiments, the threshold distance is 1 inch, 3 inches, 6 inches, 1 foot, 3 feet, etc.

[0076] In some embodiments, when a selection input is received, for example, in Figure 4A, the hand 410 is farther away from the cube 406 than the threshold distance 412, according to the determination that the representation of the user's hand of the electronic device is not within the threshold distance from the first user interface element (708), for example, in Figure 4A, when the hand 410 was performing a selection input (e.g., "Gesture A"), the gaze 408 was directed towards the cube 406, so that in Figures 4B to 4D, the cube 406 is operated according to the movement of the hand 410, according to the determination that the user's gaze of the electronic device is not within the threshold distance from the first user interface element. In accordance with the determination that the gaze is directed towards a face element, the electronic device operates a first user interface element in accordance with the operation input (710). For example, if the gaze 408 was not directed towards the cube 406 when the hand 410 was performing a selection input, the electronic device refrains from operating the first user interface element in accordance with the operation input, in accordance with the determination that the user's gaze is not directed towards the first user interface element, so that the cube 406 is optionally not operated in accordance with the movement of the hand 410 (712). In some embodiments, if the gaze was directed towards another object when the selection input was received, the other object is operated in accordance with the movement of the hand 410. In some embodiments, non-virtual objects are not operable in accordance with the movement of the hand 410 if the gaze is directed towards an object that is not a virtual object (e.g., a representation or depiction of a real-world object). (e.g., the user input is optionally discarded or ignored, and / or a notification is displayed to the user indicating that the object is not operable.)

[0077] In some embodiments, when a selection input is received, the electronic device operates the second user interface element according to an operation input, based on a determination that the representation of the user's hand in the electronic device is within a threshold distance from the second user interface element. For example, if the user's hand is within a threshold distance of any virtual object (e.g., so that subsequent movement of the hand triggers operation of a particular virtual object), the particular virtual object closest to the hand and / or closest to the pinch point of the hand is selected as the target of the input.

[0078] In some embodiments, when a selection input is received, the electronic device operates the second user interface element according to the operation input, based on the determination that the user's gaze is directed towards the second user interface element, according to the determination that the user's gaze is directed towards the second user interface element, and refrains from operating the second user interface element according to the determination that the user's gaze is not directed towards the second user interface element. For example, if the user's hand is not within the threshold distance of any virtual object, the object to which the user's gaze is directed is the object selected as the target of the input in response to the detection of a selection input. In some embodiments, if the gaze is directed towards a first virtual object, the first virtual object is selected as the target of the operation, but if the gaze is directed towards a second virtual object, the second virtual object is selected as the target of the operation. As described herein, the determination of whether the user's gaze is directed towards a particular object or location is based on one or more gaze tracking sensors. In some embodiments, if the user's gaze is directed towards a specific location in the physical world that maps to a particular location in the three-dimensional environment (e.g., a corresponding location), then the user's gaze is considered to be directed towards the corresponding location in the three-dimensional environment (for example, if a virtual object is at its corresponding location in the three-dimensional environment, the user's gaze is interpreted as being directed towards that virtual object).

[0079] In some embodiments, the operation input includes the movement of the user's hand, such as the horizontal movement of the hand 410 in Figure 4B and the movement toward the user in Figure 4D. In some embodiments, when a selection input is received, operating the first user interface element in accordance with the operation input, according to the determination that the representation of the user's hand on the electronic device is within a threshold distance from the first user interface element, includes moving the first user interface element by an amount equal to the amount of the user's hand movement, for example, so that the cube 506 moves to the right by the same amount as the rightward movement of the hand 510 in Figure 5C. In some embodiments, when a selection input is received, operating the first user interface element in accordance with the operation input, according to the determination that the representation of the user's hand on the electronic device is not within a threshold distance from the first user interface element, includes moving the first user interface element by an amount not equal to the amount of the user's hand movement, so that the cube 406 moves to the right by more than the amount of the rightward movement of the hand 410 in Figure 4B.

[0080] In some embodiments, upon receiving a selection input and before operating the first user interface element according to an operation input, the electronic device reorients the first user interface element based on the orientation of the user's hand, for example, so that the cube 516 in Figure 5B snaps into a specific orientation based on the orientation of the hand 510, optionally. In some embodiments, the cube 516 snaps into its "upward" orientation. In some embodiments, the cube 516 snaps to the nearest axis. In some embodiments, the cube 516 snaps into the same orientation as the hand 510 (for example, if the hand 510 is held at an angle, the cube 516 snaps to the same angle).

[0081] In some embodiments, the input includes the rotation of the user's hand, and manipulating the first user interface element according to the input includes rotating the first user interface element, such as the rotation of the cube 406 according to the rotation of the hand 410 in Figure 4C. In some embodiments, the virtual object is rotated in the same direction and by the same amount as the hand rotation. For example, if the hand rotates in the yaw direction, the virtual object rotates in the yaw direction, and if the hand rotates in the pitch direction, the virtual object rotates in the pitch direction, and so on. Similarly, if the hand rotates 30 degrees, the virtual object optionally rotates 30 degrees. In some embodiments, the user can perform both rotation and movement operations simultaneously by performing both rotation and movement of the user's hand while maintaining a selection input.

[0082] In some embodiments, the first user interface element includes a control element such as a button, slider, dial, or any other suitable control element. In some embodiments, the electronic device performs an action associated with the control element in response to the operation of the first user interface element in accordance with the operation input. For example, the user may operate the control element in the same way as described above with respect to virtual objects, and by operating the control element, one or more functions associated with the control element may be optionally performed. For example, sliding a volume slider may change the volume accordingly.

[0083] In some embodiments, when a selection input is received, and a determination is made that the user's hand representation is not within a threshold distance from the first user interface element, the system detects a predetermined gesture (e.g., "Gesture B") corresponding to a request to move the cube 606 to a position for direct manipulation in Figure 6A (e.g., a remote request for direct manipulation), and the system then moves the first user interface element to a location in the computer-generated environment associated with the user's hand representation, according to the determination that the multiple user inputs include a predetermined gesture by the user's hand, so that the cube 606 is directed towards the user in Figure 6B and optionally moved to or near the pinch location by the hand 610. Thus, by performing a specific gesture, the user can move an object to the hand location (or within a threshold distance of the hand location) (e.g., throw it towards it) so that the user can perform a direct manipulation action on the object. In this way, the user can directly manipulate the object without relying on indirect manipulation actions and without the user having to walk towards the object. In some embodiments, after completing an operation, such as after detecting the end of a selection input (e.g., the end of a pinch gesture, the end of gesture B, and / or detection of another gesture in response to a request to return the virtual object to its original position), the cube 606 is returned to its original position before the user input (optionally, maintaining any operations performed while it was being held by the user, such as rotation). In some embodiments, after completing an operation, such as after detecting the end of a selection input, the cube 606 remains at the location where it was when the selection input was ended (e.g., the cube 606 does not return to its original position but remains at the position where the user placed it).

[0084] Figure 8 is a flowchart illustrating Method 800, according to some embodiments of the present disclosure, which moves a virtual object by an amount based on the distance of the virtual object to the user. Method 800 is optionally performed on an electronic device such as Device 100, Device 200, when displaying selectable options on a surface, as described above with reference to Figures 3A-3C, 4A-4B, 5A-5B, and 6A-6B. Some operations in Method 800 are optionally combined (e.g., with each other or with operations in Method 700), and / or the order of some operations is optionally changed. As described below, Method 800 provides a method, according to embodiments of the present disclosure, for moving a virtual object by an amount based on the distance of the virtual object to the user (e.g., as described above with reference to Figures 3-6B).

[0085] In some embodiments, a display generating component (e.g., a display built into an electronic device (optionally, such as a touchscreen display) and / or an external display such as a monitor, projector, or television) and one or more input devices (e.g., a touchscreen, an external mouse, an optional built-in or external trackpad, an optional built-in or external touchpad, an external remote control device, a separate mobile device (e.g., separate from the electronic device), an external handheld device, an external controller, a camera (e.g., such as a visible light camera), a depth sensor and / or motion sensor (e.g., such as a hand tracking sensor, a hand motion sensor), and an electronic device (e.g., a computer such as device 100 and / or device 200) that can communicate with the display generating component presents a computer-generated environment including a first user interface element, such as a computer-generated environment 300 including the cube 306 in Figure 3 (802).

[0086] In some embodiments, while presenting a computer-generated environment, the electronic device receives user input including a movement component directed towards a first user interface element, such as the rightward movement of hand 410 in Figure 4B (804). In some embodiments, according to a determination that the electronic device is in a first operating mode, the electronic device moves the first user interface element by a first amount according to the movement component, such as moving cube 506 by an amount 520 while in direct operating mode in Figure 5C (806). In some embodiments, according to a determination that the electronic device is in a second operating mode different from the first operating mode, the electronic device moves the first user interface element by a second amount greater than the first amount according to the movement component, such as moving cube 406 by an amount 418 while in indirect operating mode in Figure 4B (808).

[0087] In some embodiments, the first operating mode is a direct operating mode in which, when user input is received, the representation of the user's hand in the electronic device is within the threshold distance of the first user interface element, such that the hand 510 in Figure 5A is within the threshold distance 512 of the cube 406; and the second operating mode is an indirect operating mode in which, when user input is received, the representation of the user's hand is not within the threshold distance of the first user interface element, such that the hand 410 in Figure 4A is further away than the threshold distance 412 of the cube 506.

[0088] In some embodiments, the first quantity is the same as the movement of the user input's movement component, as shown in Figure 5C, and the second quantity is a different quantity from the movement of the user input's movement component, as shown in Figure 4B.

[0089] In some embodiments, the second quantity is the amount of movement of the user input's movement component scaled by a scaling factor, such as the movement of cube 406 scaled by a scaling factor based on the distance of cube 406 from the user and / or the distance of hand 410 from the user in Figure 4B.

[0090] In some embodiments, a first scaling factor is set as the scaling factor according to the determination that the movement of the moving component is in a first direction relative to the user of the electronic device, and a second scaling factor different from the first scaling factor is set as the scaling factor according to the determination that the movement of the moving component is in a second direction different from the first direction relative to the user. For example, if the object is moving away from the user, the scaling factor is optionally not based on the distance of the object from the user and / or the distance of the user's hand (e.g., optionally, the scaling factor is 1), but if the object is moving toward the user, the scaling factor is optionally based on the distance of the object from the user and / or the distance of the user's hand, as shown in Figure 4D (e.g., optionally, the scaling factor is greater than 1).

[0091] In some embodiments, the second scaling factor is based at least on the distance of the first user interface element from a predetermined reference location in the computer-generated environment (e.g., the location of the user's head in the electronic device, the location of the user in the electronic device, the location of the electronic device, and the location in the three-dimensional environment corresponding to any of the above 1 inch forward, 3 inches forward, 6 inches forward, 1 foot forward, and 3 feet forward), as shown in Figure 4B, and the distance of the representation of the user's hand from the predetermined reference location (e.g., the distance from the location in the three-dimensional environment corresponding to the user's hand to the location of the user's head in the electronic device, the location of the user in the electronic device, the location of the electronic device, and the location corresponding to any of the above 1 inch forward, 3 inches forward, 6 inches forward, 1 foot forward, and 3 feet forward).

[0092] In some embodiments, the movement component of user input includes a lateral movement component parallel to the user of the electronic device (e.g., horizontal and / or vertical movement while maintaining the same distance from the user), as shown in Figure 4B. In some embodiments, the movement angle of a second amount relative to the user of the electronic device is the same as the movement angle of the lateral movement component of user input relative to the user of the electronic device, such that the cube 406 is moved to the right by an amount such that the change angle 420 of the movement of the hand 410 due to the rightward movement 414 of the hand 410 is the same as the change angle 416. Thus, in some embodiments, the scaling factor for lateral movement is proportional to the ratio of the distance from the user to the distance of the hand from the user to the distance of the object from the user.

[0093] The above is written with reference to specific embodiments for illustrative purposes. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to any specific form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the invention and its practical applications, thereby enabling other persons skilled in the art to best use the invention and the various described embodiments with various modifications suitable for any particular application that may be conceived.

Claims

1. In an electronic device capable of communicating with a display, The computer-generated environment including the first user interface element is presented via the aforementioned display, While the aforementioned computer-generated environment is being presented, multiple user inputs, including selection inputs and operation inputs, When the selection input is received, the first user interface element is operated according to the operation input, based on the determination that the user's hand of the electronic device is within a threshold distance from the first user interface element. When the selection input is received, according to the determination that the user's hand of the electronic device is not within the threshold distance from the first user interface element, The determination that the user's gaze is directed towards the first user interface element of the electronic device, and the operation of the first user interface element according to the operation input, A method comprising: determining that the user of the electronic device is not directing their gaze toward the first user interface element, and therefore refraining from operating the first user interface element in accordance with the operation input.

2. When the selection input is received, the electronic device operates the second user interface element according to the operation input, based on the determination that the user's hand is within the threshold distance from the second user interface element. When the selection input is received, according to the determination that the user's hand of the electronic device is not within the threshold distance from the second user interface element, The determination that the user's gaze is directed towards the second user interface element of the electronic device, and the operation of the second user interface element according to the operation input, In accordance with the determination that the user's gaze is not directed towards the second user interface element of the electronic device, the operation of the second user interface element in accordance with the operation input is postponed. The method according to claim 1, further comprising:

3. The aforementioned operation input includes the movement of the user's hand, When the selection input is received, the determination that the user's hand of the electronic device is within the threshold distance from the first user interface element, and the operation of the first user interface element according to the operation input, includes moving the first user interface element by an amount approximately equal to the amount of movement of the user's hand in the computer-generated environment, The method according to claim 1 or 2, wherein, when the selection input is received, the determination that the user's hand of the electronic device is not within the threshold distance from the first user interface element, the operation of the first user interface element in accordance with the operation input includes moving the first user interface element by an amount not equal to the amount of the user's hand movement.

4. The method according to any one of claims 1 to 3, further comprising changing the orientation of the first user interface element based on the orientation of the user's hand before operating the first user interface element in accordance with the operation input, in response to receiving the selection input.

5. The method according to any one of claims 1 to 4, wherein the operation input includes the rotation of the user's hand, and operating the first user interface element in accordance with the operation input includes rotating the first user interface element.

6. The first user interface element includes a control element, and the method is The method according to any one of claims 1 to 5, further comprising performing an action associated with the control element in response to operating the first user interface element in accordance with the operation input.

7. The method according to any one of claims 1 to 6, further comprising moving the first user interface element to a location in the computer-generated environment associated with the user's hand, in accordance with the determination that when the selection input is received, the user's hand is not within the threshold distance from the first user interface element, and in accordance with the determination that the plurality of user inputs include a predetermined gesture by the user's hand.

8. One or more processors, Memory and An electronic device having one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions, the instructions are A computer-generated environment including a first user interface element is presented via the display. While the aforementioned computer-generated environment is being presented, multiple user inputs, including selection inputs and operation inputs, are received. When the selection input is received, the first user interface element is operated according to the operation input, based on the determination that the user's hand of the electronic device is within a threshold distance from the first user interface element. When the selection input is received, according to the determination that the user's hand of the electronic device is not within the threshold distance from the first user interface element, In accordance with the determination that the user's gaze is directed towards the first user interface element of the electronic device, the first user interface element is operated according to the operation input. An electronic device that, in accordance with the determination that the user's gaze is not directed toward the first user interface element, refrains from operating the first user interface element in accordance with the operation input.

9. A non-temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device... A computer-generated environment including a first user interface element is presented via the display. While the aforementioned computer-generated environment is being presented, multiple user inputs, including selection inputs and operation inputs, are received. When the selection input is received, the device determines that the user's hand is within a threshold distance from the first user interface element, and then operates the first user interface element according to the operation input. When the selection input is received, the electronic device is made to operate the first user interface element according to the operation input, based on the determination that the user's hand is not within the threshold distance from the first user interface element, and based on the determination that the user's gaze is directed towards the first user interface element. A non-temporary computer-readable storage medium that, in accordance with the determination that the user of the electronic device is not directing their gaze toward the first user interface element, causes the user to refrain from operating the first user interface element in accordance with the operation input.

10. One or more processors, An electronic device having memory, A means for presenting a computer-generated environment including a first user interface element via a display, While the aforementioned computer-generated environment is being presented, means for receiving multiple user inputs, including selection inputs and operation inputs, When the selection input is received, means for operating the first user interface element according to the operation input, based on the determination that the user's hand of the electronic device is within a threshold distance from the first user interface element, When the selection input is received, according to the determination that the user's hand of the electronic device is not within the threshold distance from the first user interface element, In accordance with the determination that the user's gaze is directed towards the first user interface element of the electronic device, the first user interface element is operated according to the operation input. An electronic device having means for refraining from operating the first user interface element in accordance with the operation input, based on the determination that the user's gaze is not directed toward the first user interface element of the electronic device.

11. An information processing device for use in an electronic device, wherein the information processing device is A means for presenting a computer-generated environment including a first user interface element via a display, While the aforementioned computer-generated environment is being presented, means for receiving multiple user inputs, including selection inputs and operation inputs, When the selection input is received, means for operating the first user interface element according to the operation input, based on the determination that the user's hand of the electronic device is within a threshold distance from the first user interface element, When the selection input is received, according to the determination that the user's hand of the electronic device is not within the threshold distance from the first user interface element, In accordance with the determination that the user's gaze is directed towards the first user interface element of the electronic device, the first user interface element is operated according to the operation input. An information processing device having means for refraining from operating the first user interface element in accordance with the operation input, based on the determination that the user's gaze is not directed towards the first user interface element of the electronic device.

12. One or more processors, Memory and An electronic device having one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions, the instructions include instructions that perform any one of the methods according to claims 1 to 7.

13. A non-temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device is caused to execute any one of the methods according to claims 1 to 7.

14. One or more processors, Memory and An electronic device having means for carrying out any one of the methods described in claims 1 to 7.

15. An information processing device for use in an electronic device, wherein the information processing device is An information processing apparatus having means for performing any one of the methods described in claims 1 to 7.

16. In an electronic device capable of communicating with a display, The computer-generated environment including the first user interface element is presented via the aforementioned display, While the computer-generated environment is being presented, user input including a moving component directed towards the first user interface element is received, Based on the distance between the user's hand and the first user interface element of the electronic device when the user input is received, the determination that the electronic device is in a first operating mode is made, and the first user interface element is moved by a first amount according to the movement component. In accordance with the determination that the electronic device is in a second operating mode different from the first operating mode, the first user interface element is moved by a second amount greater than the first amount according to the movement component. Methods that include...

17. The method according to claim 16, wherein when the user input is received, the first operation mode is a direct operation mode when the user's hand is within the threshold distance of the first user interface element of the electronic device, and the second operation mode is an indirect operation mode when the user input is received, when the user's hand is not within the threshold distance of the first user interface element.

18. The method according to claim 16 or 17, wherein the first amount is approximately the same as the movement of the movement component of the user input, and the second amount is different from the movement of the movement component of the user input.

19. The method according to any one of claims 16 to 18, wherein the second amount is the amount of movement of the movement component of the user input scaled by a scaling factor.

20. According to the determination that the movement of the moving component is in a first direction relative to the user of the electronic device, the first scaling factor is set as the scaling factor. The method according to claim 19, wherein, according to the determination that the movement of the moving component is in a second direction different from the first direction with respect to the user, a second scaling coefficient different from the first scaling coefficient is set as the scaling coefficient.

21. The method according to claim 20, wherein the second scaling factor is based at least on the distance of the first user interface element from a predetermined reference location in the computer-generated environment and the distance of the user's hand from the predetermined reference location.

22. The movement component of the user input includes a lateral movement component of the electronic device parallel to the user, The method according to any one of claims 16 to 21, wherein the angle of movement of the second amount of the electronic device relative to the user is substantially the same as the angle of movement of the lateral movement component of the user input of the electronic device relative to the user.

23. One or more processors, Memory and An electronic device having one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions, the instructions are A computer-generated environment including a first user interface element is presented via the display. While the computer-generated environment is being presented, user input including a moving component directed towards the first user interface element is received. Based on the distance between the user's hand and the first user interface element of the electronic device when the user input is received, the determination that the electronic device is in a first operating mode is made, and the first user interface element is moved by a first amount according to the movement component. An electronic device that includes a command to move the first user interface element by a second amount greater than the first amount, according to the movement component, in accordance with the determination that the electronic device is in a second operating mode different from the first operating mode.

24. A non-temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device... A computer-generated environment including a first user interface element is presented via the display. While the computer-generated environment is being presented, user input including a movement component directed towards the first user interface element is received. Based on the distance between the user's hand and the first user interface element of the electronic device when the user input is received, a determination is made that the electronic device is in a first operating mode, and the first user interface element is moved by a first amount according to the movement component. A non-temporary computer-readable storage medium that, according to the determination that the electronic device is in a second operating mode different from the first operating mode, moves the first user interface element by a second amount greater than the first amount, according to the movement component.

25. One or more processors, Memory and An electronic device having, A means for presenting a computer-generated environment including a first user interface element via a display, While the computer-generated environment is being presented, means for receiving user input including a moving component directed towards the first user interface element, Means for moving the first user interface element by a first amount according to the movement component, based on a determination that the electronic device is in a first operating mode, which is determined based on the distance between the user's hand and the first user interface element of the electronic device when the user input is received, An electronic device having means for moving the first user interface element by a second amount greater than the first amount, according to the movement component, in accordance with the determination that the electronic device is in a second operating mode different from the first operating mode.

26. An information processing device for use in an electronic device, wherein the information processing device is A means for presenting a computer-generated environment including a first user interface element via a display, While the computer-generated environment is being presented, means for receiving user input including a moving component directed towards the first user interface element, Means for moving the first user interface element by a first amount according to the movement component, based on a determination that the electronic device is in a first operating mode, which is determined based on the distance between the user's hand and the first user interface element of the electronic device when the user input is received, An information processing apparatus having means for moving the first user interface element by a second amount greater than the first amount, according to the movement component, in accordance with the determination that the electronic device is in a second operating mode different from the first operating mode.

27. One or more processors, Memory and An electronic device having one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions, the instructions execute any one of the methods according to claims 16 to 22.

28. A non-temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the non-temporary computer-readable storage medium causes the electronic device to execute any one of the methods according to claims 16 to 22.

29. One or more processors, Memory and An electronic device having means for carrying out any one of the methods described in claims 16 to 22.

30. An information processing device for use in an electronic device, wherein the information processing device is An information processing apparatus having means for performing any one of the methods described in claims 16 to 22.

Images