Method for moving object in three-dimensional environment

Abstract

In some embodiments, an electronic device uses different algorithms to move an object in a three-dimensional environment based on the direction of movement. In some embodiments, the electronic device modifies the size of an object in the three-dimensional environment as the distance between the object and the user's viewpoint changes. In some embodiments, an electronic device selectively prevents movement of an object when the object is in contact with another object in a three-dimensional environment. In some implementations, an electronic device selectively adds another object in a three-dimensional environment to the object based on whether the object is a valid delivery target for the object. In some embodiments, an electronic device facilitates simultaneous movement of multiple objects in a three-dimensional environment. In some embodiments, an electronic device facilitates throwing an object in a three-dimensional environment.

Description

[0001] This application is a divisional application of the invention patent application filed on September 21, 2022, with application number 202280076254.4 and invention title "Method for Moving an Object in a Three-Dimensional Environment".

[0002] Cross-reference to related applications

[0003] This application claims the benefit of U.S. Provisional Application 63 / 261,556, filed September 23, 2021, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] The present invention relates generally to a computer system having a display generation component for presenting a graphical user interface and one or more input devices, including but not limited to electronic devices that facilitate the movement of objects in a three-dimensional environment. Background Technology

[0005] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices for computer systems and other electronic computing devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays) are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include virtual objects (including digital images, videos, text, icons, control elements such as buttons, and other graphics).

[0006] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems where manipulating virtual objects is complex, tedious, and error-prone, impose a significant cognitive burden on users and detract from the immersive experience of virtual / augmented reality environments. Furthermore, these methods take longer than necessary, thus wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the Invention

[0007] Therefore, computer systems with improved methods and interfaces are needed to provide users with computer-generated experiences, making user interaction with the computer system more efficient and intuitive. Such methods and interfaces optionally complement or replace conventional methods for providing users with computer-generated realistic experiences. By helping users understand the relationship between the inputs provided and the device's responses to those inputs, such methods and interfaces reduce the quantity, extent, and / or nature of user input, thus creating a more effective human-computer interface.

[0008] The disclosed systems reduce or eliminate the aforementioned defects and other problems associated with user interfaces used in computer systems having display generation components and one or more input devices. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a laptop, tablet, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touchscreen" or "touchscreen display"). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, in addition to the display generation components, the computer system also has one or more output devices, including one or more haptic output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, and a program or instruction set stored in memory for performing multiple functions. In some implementations, users interact with the GUI through touch and gestures of a stylus and / or fingers on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI or the user's body (as captured by a camera and other motion sensors), and voice input (as captured by one or more audio input devices). In some implementations, functions performed through interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving phone calls, video conferencing, sending and receiving emails, instant messaging, test support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] There is a need for electronic devices with improved methods and interfaces for interacting with objects in a 3D environment. Such methods and interfaces can complement or replace conventional methods for interacting with objects in a 3D environment. These methods and interfaces reduce the amount, extent, and / or nature of user input, resulting in more efficient human-computer interfaces.

[0010] In some implementations, the electronic device uses different algorithms to move objects in a three-dimensional environment based on the direction of movement. In some implementations, the electronic device modifies the size of the object as the distance between the object in the three-dimensional environment and the user's viewpoint changes. In some implementations, the electronic device selectively prevents the object from moving when it comes into contact with another object in the three-dimensional environment. In some implementations, the electronic device selectively adds the object to another object based on whether that object is a valid target for the object. In some implementations, the electronic device facilitates the simultaneous movement of multiple objects in the three-dimensional environment. In some implementations, the electronic device facilitates the throwing of objects in the three-dimensional environment.

[0011] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not exhaustive; in particular, many additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, description, and claims. Furthermore, it should be pointed out that the language used in this specification has been chosen in principle for readability and instruction purposes, and such choice may not be necessary to depict or define the subject matter of the invention. Attached Figure Description

[0012] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0013] Figure 1 This is a block diagram illustrating the operating environment of a computer system for providing a CGR experience according to some implementation schemes.

[0014] Figure 2 This is a block diagram illustrating a controller of a computer system configured to manage and coordinate the user's CGR experience according to some implementation schemes.

[0015] Figure 3 This is a block diagram illustrating a display generation component of a computer system configured to provide a CGR experience to a user, according to some embodiments.

[0016] Figure 4 This is a block diagram illustrating a hand tracking unit of a computer system configured to capture user gesture input according to some embodiments.

[0017] Figure 5 This is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input according to some embodiments.

[0018] Figure 6A This is a flowchart illustrating a gaze tracking pipeline with flash assistance according to some implementation schemes.

[0019] Figure 6B An exemplary environment for an electronic device that provides a CGR experience according to some implementation schemes is shown.

[0020] Figures 7A to 7E Examples are shown of electronic devices using different algorithms to move objects in different directions in a three-dimensional environment according to some implementation schemes.

[0021] Figures 8A to 8K This is a flowchart illustrating methods for moving objects in different directions in a three-dimensional environment using different algorithms according to some implementation schemes.

[0022] Figures 9A to 9E Examples are shown of electronic devices, according to some implementation schemes, dynamically adjusting (or not adjusting) the size of virtual objects in a three-dimensional environment.

[0023] Figures 10A to 10I This is a flowchart illustrating a method for dynamically adjusting (or not adjusting) the size of a virtual object in a three-dimensional environment according to some implementation schemes.

[0024] Figures 11A to 11E An example is shown of an electronic device, according to some implementation schemes, selectively preventing the movement of objects in a three-dimensional environment.

[0025] Figures 12A to 12G This is a flowchart illustrating a method for selectively preventing the movement of objects in a three-dimensional environment according to some implementation schemes.

[0026] Figures 13A to 13D Examples of objects selectively added to a three-dimensional environment by an electronic device according to some implementation schemes are shown.

[0027] Figures 14A to 14H This is a flowchart illustrating a method for selectively adding corresponding objects to an object in a three-dimensional environment according to some implementation schemes.

[0028] Figures 15A to 15D Examples of electronic devices according to some implementation schemes are shown that facilitate the movement and / or placement of multiple virtual objects in a three-dimensional environment.

[0029] Figures 16A to 16J This is a flowchart illustrating a method for facilitating the movement and / or placement of multiple virtual objects in a three-dimensional environment, according to some implementation schemes.

[0030] Figures 17A to 17D An example of an electronic device, according to some implementation schemes, is shown that facilitates the throwing of virtual objects in a three-dimensional environment.

[0031] Figures 18A to 18F This is a flowchart illustrating a method for facilitating the throwing of virtual objects in a three-dimensional environment according to some implementation schemes. Detailed Implementation

[0032] According to some implementations, this disclosure relates to a user interface for providing a computer-generated reality (CGR) experience to a user.

[0033] The systems, methods, and GUIs described in this paper provide improved ways for electronic devices to facilitate interaction with and manipulate objects in a three-dimensional environment.

[0034] In some implementations, the computer system displays a virtual environment within a three-dimensional environment. In some implementations, the virtual environment is displayed via a far-field or near-field process based on the geometry (e.g., size and / or shape) of the three-dimensional environment (e.g., which optionally simulates the real-world environment surrounding the device). In some implementations, the far-field process includes introducing the virtual environment from a position furthest from the user's viewpoint and gradually expanding the virtual environment toward the user's viewpoint (e.g., using information about the position, location, distance, etc., of objects within the environment). In some implementations, the near-field process includes introducing the virtual environment from a position furthest from the user's viewpoint and expanding outward from an initial position, regardless of the distance and / or position of objects within the environment (e.g., expanding the size of the virtual environment relative to the display-generating components).

[0035] In some embodiments, the computer system displays a virtual environment and / or atmospheric effects in a three-dimensional environment. In some embodiments, displaying atmospheric effects includes displaying one or more lighting and / or particle effects in the three-dimensional environment. In some embodiments, in response to detecting movement of the device, portions of the virtual environment are de-emphasized, but optionally the atmospheric effects are not reduced. In some embodiments, in response to detecting rotation of the user's body (e.g., simultaneously with rotation of the device), the virtual environment is moved to a new position in the three-dimensional environment, optionally aligned with the user's body.

[0036] In some implementations, the computer system displays a virtual environment and an application's user interface simultaneously. In some implementations, the application's user interface can be moved into the virtual environment and treated as a virtual object existing within it. In some implementations, the size of the user interface is automatically adjusted based on the distance it travels when moved into the virtual environment. In some implementations, when both the virtual environment and the user interface are displayed, the user can request that the user interface be displayed as an immersive environment. In some implementations, in response to a request to display the user interface as an immersive environment, the previously displayed virtual environment is replaced with the immersive environment of the user interface.

[0037] In some implementations, the computer system displays a three-dimensional environment with one or more virtual objects. In some implementations, in response to detecting movement input in a first direction, the computer system moves the virtual object in a first output direction using a first movement algorithm. In some implementations, in response to detecting movement input in different second directions, the computer system moves the virtual object in a second output direction using different second movement algorithms. In some implementations, during movement, when the first object moves near the second object, the first object becomes aligned with the second object.

[0038] In some implementations, the computer system displays a three-dimensional environment with one or more virtual objects. In some implementations, in response to detecting movement input pointing at an object, the computer system adjusts the size of the object if the object moves toward or away from the user's viewpoint. In some implementations, in response to detecting movement of the user's viewpoint, the computer system does not adjust the size of the object even if the distance between the viewpoint and the object changes.

[0039] In some implementations, the computer system displays a three-dimensional environment with one or more virtual objects. In some implementations, in response to movement of a corresponding virtual object in a corresponding direction containing another virtual object, movement of the corresponding virtual object is prevented when the corresponding virtual object comes into contact with the other virtual object. In some implementations, movement of the corresponding virtual object is prevented because the other virtual object is a valid target for deployment by the corresponding virtual object. In some implementations, the corresponding virtual object moves through the other virtual object in a corresponding direction when its movement through the other virtual object exceeds a corresponding magnitude threshold.

[0040] In some implementations, the computer system displays a three-dimensional environment with one or more virtual objects. In some implementations, in response to a corresponding virtual object moving to another virtual object that is a valid target for its deployment, the corresponding object is added to the other virtual object. In some implementations, in response to a corresponding virtual object moving to another virtual object that is an invalid target for its deployment, the corresponding virtual object is not added to the other virtual object and is moved back to the corresponding position from which it initially moved. In some implementations, in response to a corresponding virtual object moving to a corresponding position in an empty space within the virtual environment, the corresponding virtual object is added to a newly generated virtual object at that position in the empty space.

[0041] In some implementations, the computer system displays a three-dimensional environment with one or more virtual objects. In some implementations, in response to a movement input pointing to multiple objects, the computer system moves the multiple objects together in the three-dimensional environment. In some implementations, in response to detecting the end of the movement input, the computer system places the objects among the multiple objects separately in the three-dimensional environment. In some implementations, the multiple objects are arranged in a stacked arrangement when moved.

[0042] In some implementations, the computer system displays a three-dimensional environment with one or more virtual objects. In some implementations, in response to a throwing input, if a second object is aimed as part of the throwing input, the computer system moves the first object to the second object. In some implementations, if the second object is not aimed as part of the throwing input, the computer system moves the first object in the three-dimensional environment according to the velocity and / or direction of the throwing input. In some implementations, aiming at the second object is based on the user's gaze and / or the direction of the throwing input.

[0043] The processes described below enhance device operability and make the user-device interface more efficient through various technologies (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device), including providing users with improved visual feedback, reducing the amount of input required to perform operations, providing additional control options without cluttering the user interface with additional display controls, performing operations without further user input when a set of conditions have been met, improving privacy and / or security, and / or additional technologies. These technologies also reduce power consumption and extend device battery life by enabling users to use the device faster and more efficiently.

[0044] Figure 1Figures 6 to 6 provide a description of exemplary computer systems for providing a CGR experience to a user (such as those described below with respect to methods 800, 1000, 1200, 1400, 1600, and / or 1800). In some embodiments, such as Figure 1 As shown, a CGR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted display (HMD), a monitor, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a haptic output generator 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, haptic sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, speed sensors, etc.), and optionally one or more peripheral devices 195 (e.g., home appliances, wearable devices, etc.). In some embodiments, one or more of the input devices 125, output devices 155, sensors 190, and peripheral devices 195 are integrated with the display generation component 120 (e.g., in a head-mounted or handheld device).

[0045] In describing the CGR experience, various terms are used to distinguish several related but different environments that a user can sense and / or interact with (e.g., interacting with inputs detected by the computer system 101 that generates the CGR experience, causing the computer system to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101). The following is a subset of these terms:

[0046] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through senses such as sight, touch, hearing, taste, and smell.

[0047] Computer-Generated Reality (or Extended Reality (XR)): Conversely, a computer-generated reality (CGR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In CGR, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are modulated in a manner consistent with at least one physical law. For example, a CGR system can detect a person's head rotation and, in response, modulate the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the modulation of characteristics of virtual objects in the CGR environment can be done in response to a representation of physical motion (e.g., a voice command). A person can use any of their senses to sense and / or interact with CGR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. For example, audio objects can enable audio transparency, which selectively introduces ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, people can sense and / or interact only with audio objects.

[0048] Examples of CGR include virtual reality and mixed reality.

[0049] Virtual Reality: A virtual reality (VR) environment is a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing human avatars are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through the simulation of a person's presence within the computer-generated environment and / or through the simulation of a subgroup of physical movements of a person within the computer-generated environment.

[0050] Mixed Reality: Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments refer to simulated environments designed to incorporate sensory input from the physical environment, or its representations, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between, but not limited to, a purely physical environment as one end and a virtual reality environment as the other. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to present an MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical objects or their representations from the physical environment). For example, a system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0051] Examples of mixed reality include augmented reality and augmented virtual reality.

[0052] Augmented Reality (AR): An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation of the physical environment. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects overlaid on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto the physical environment, such as as holograms or on physical surfaces, allowing a person to perceive the virtual objects superimposed on the physical environment. Augmented reality environments also refer to simulated environments in which the representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system can transform images from one or more sensors to apply a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. Furthermore, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it, such that the modified portions are representative but not realistic versions of the original captured image. Still further, the representation of the physical environment can be transformed by graphically removing or blurring portions of it.

[0053] Augmented Virtual: An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. Sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park could have virtual trees and virtual buildings, but a person's face could be realistically reproduced from an image taken of a physical person. Similarly, virtual objects could adopt the shape or color of a physical object imaged by one or more imaging sensors. Furthermore, virtual objects could use shadows that correspond to the sun's position within the physical environment.

[0054] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same location and / or position within the user's viewpoint, the virtual object remains viewpoint-locked even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the direction forward of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); therefore, the user's viewpoint remains fixed even when the user's gaze shifts without moving the user's head. In embodiments where the computer system has a display generating component (e.g., a display screen) that can be repositioned relative to the user's head, the user's viewpoint is the augmented reality view presented to the user on the display generating component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint, even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or orientation of a viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In an implementation where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, so the virtual object is also referred to as a "head-locked virtual object".

[0055] Environment-locked visual objects: When a computer system displays a virtual object at a location and / or position within the user's viewpoint, the virtual object is environment-locked (or, "world-locked"), the location and / or position being based on a location and / or object in a three-dimensional environment (e.g., a physical or virtual environment) (e.g., selected and / or anchored to that location and / or object with reference to it). As the user's viewpoint moves, the location and / or object in the environment relative to the user's viewpoint changes, causing the environment-locked virtual object to appear at different locations and / or positions within the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user appears at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) so that the tree is now centered to the left in the user's viewpoint (e.g., the tree's position shifts in the user's viewpoint), the environment-locked virtual object locked to the tree appears centered to the left in the user's viewpoint. In other words, the position and / or orientation of an environment-locked virtual object displayed in the user's viewpoint depends on the position to which the virtual object is locked and / or the position and / or orientation of the object in the environment. In some implementations, the computer system uses a stationary frame of reference (e.g., a coordinate system anchored to a fixed position and / or object in the physical environment) to determine the orientation of the environment-locked virtual object displayed in the user's viewpoint. An environment-locked virtual object may be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object), or it may be locked to a movable part of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot), causing the virtual object to move with the viewpoint or that part of the environment to maintain a fixed relationship between the virtual object and that part of the environment.

[0056] In some implementations, environment-locked or viewpoint-locked virtual objects exhibit lazy following behavior, reducing or delaying their movement relative to the movement of a reference point they are following. In some implementations, when exhibiting lazy following behavior, the computer system intentionally delays the movement of the virtual object when movement of the reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 cm and 300 cm from the viewpoint) is detected. For example, when the reference point (e.g., that portion of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up). In some implementations, when the virtual object exhibits lazy following behavior, the device ignores small movements of the reference point (e.g., ignores movements of the reference point below a threshold amount, such as 0 to 5 degrees or 0 to 50 cm). For example, when the reference point (e.g., the part of the environment or viewpoint to which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed to maintain a fixed or substantially fixed position relative to a viewpoint or part of the environment different from the reference point to which the virtual object is locked), and when the reference point (e.g., the part of the environment or viewpoint to which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed to maintain a fixed or substantially fixed position relative to a viewpoint or part of the environment different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., a "lazy following" threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some implementations, maintaining a substantially fixed position of the virtual object relative to a reference point includes displaying the virtual object within a threshold distance (e.g., 1 cm, 2 cm, 3 cm, 5 cm, 15 cm, 20 cm, or 50 cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the position of the reference point).

[0057] Hardware: Many different types of electronic systems enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may combine one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into a physical environment, such as as holograms, or onto a physical surface. In some embodiments, controller 110 is configured to manage and coordinate the user's CGR experience. In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. References below... Figure 2The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device located locally or remotely relative to scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within scene 105. Alternatively, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 105. In some embodiments, the controller 110 is communicatively coupled to a display generation component 120 (e.g., an HMD, a monitor, a projector, a touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, controller 110 is included within the housing (e.g., physical enclosure) of display generation component 120 (e.g., HMD or portable electronic device including display and one or more processors), one or more input devices in input device 125, one or more output devices in output device 155, one or more sensors in sensor 190, and / or one or more peripheral devices in peripheral device 195, or shares the same physical housing or support structure with one or more of the aforementioned devices.

[0058] In some embodiments, the display generation component 120 is configured to provide a CGR experience to a user (e.g., at least the visual components of the CGR experience). In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. The following is relative to... Figure 3 The display generation component 120 is described in more detail. In some embodiments, the functionality of the controller 110 is provided by and / or combined with the display generation component 120.

[0059] According to some implementation schemes, when a user is virtually and / or physically present within scene 105, the display generation component 120 provides the user with a CGR experience.

[0060] In some embodiments, the display generating component is worn on a part of the user's body (e.g., on his / her head, his / her hand, etc.). Thus, the display generating component 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, the display generating component 120 surrounds the user's field of view. In some embodiments, the display generating component 120 is a handheld device (such as a smartphone or tablet) configured to present CGR content, and the user holds the device having a display facing the user's field of view and a camera facing scene 105. In some embodiments, the handheld device is optionally placed within a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generating component 120 is a CGR chamber, housing, or room configured to present CGR content, wherein the user does not wear or hold the display generating component 120. Many user interfaces described with reference to one type of hardware (e.g., a handheld device or a tripod-mounted device) for displaying CGR content can be implemented on another type of hardware (e.g., an HMD or other wearable computing device) for displaying CGR content. For example, a user interface illustrating interaction with CGR content triggered by an interaction occurring in the space in front of a handheld device or a tripod-mounted device can be similarly implemented using an HMD, where the interaction occurs in the space in front of the HMD and the response to the CGR content is displayed via the HMD. Similarly, a user interface illustrating interaction with CGR content triggered by movement of a handheld device or a tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)) can be similarly implemented using an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)).

[0061] Despite Figure 1 The relevant features of the operating environment 100 are shown in this disclosure, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure further relevant aspects of the exemplary embodiments disclosed herein.

[0062] Figure 2This is a block diagram of an example controller 110 according to some implementation schemes. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the implementation schemes disclosed herein. Therefore, as a non-limiting example, in some embodiments, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing kernels, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0063] In some embodiments, one or more communication buses 204 include circuitry for interconnecting and communicating between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0064] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures or subsets thereof, including optional operating system 230 and CGR experience module 240.

[0065] Operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate single or multiple CGR experiences for one or more users (e.g., single CGR experiences for one or more users, or multiple CGR experiences for corresponding groups of one or more users). To this end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

[0066] In some implementations, the data acquisition unit 242 is configured to acquire data from... Figure 1 The data acquisition unit 242 includes at least the display generation component 120, and optionally acquires data (e.g., presentation data, interactive data, sensor data, location data, etc.) from one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data acquisition unit 242 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0067] In some implementations, the tracking unit 244 is configured to map scene 105, and the tracking at least shows the generated component 120 relative to... Figure 1 The tracking unit 244 is used to locate / position of scene 105, and optionally track the position of one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. For this purpose, in various embodiments, the tracking unit 244 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics. In some embodiments, the tracking unit 244 includes a hand tracking unit 243 and / or an eye tracking unit 245. In some embodiments, the hand tracking unit 243 is configured to track the location / position of one or more portions of the user's hand, and / or the position of one or more portions of the user's hand relative to the user's hand. Figure 1 The movement of scene 105 relative to the display generating component 120 and / or relative to a coordinate system (defined relative to the user's hand). The following refers to the movement relative to... Figure 4 The hand tracking unit 243 is described in more detail. In some embodiments, the eye tracking unit 245 is configured to track the user's gaze (or more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hand)) or relative to CGR content displayed via display generation component 120. The following description is relative to... Figure 5 The eye-tracking unit 245 is described in more detail.

[0068] In some implementations, coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripheral device 195. For this purpose, in various implementations, coordination unit 246 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0069] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data transmission unit 248 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0070] Although the data acquisition unit 242, the tracking unit 244 (e.g., including eye tracking unit 243 and hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are shown residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including eye tracking unit 243 and hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may reside in a separate computing device.

[0071] also, Figure 2 This is used more as a functional description of various features that can exist in a specific implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 2 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0072] Figure 3This is a block diagram illustrating an example of generating component 120 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. For the purposes described, as a non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, and / or similar interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional internal and / or external image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0073] In some embodiments, one or more communication buses 304 include circuitry for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include inertial measurement units (IMUs), accelerometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, and / or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0074] In some embodiments, one or more CGR displays 312 are configured to provide a CGR experience to a user. In some embodiments, the one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, the one or more CGR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, HMD 120 includes a single CGR display. Alternatively, HMD 120 may include CGR displays for each of the user's eyes. In some embodiments, the one or more CGR displays 312 are capable of displaying MR and VR content. In some embodiments, the one or more CGR displays 312 are capable of displaying either MR or VR content.

[0075] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand and optionally the user's arm (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward to acquire image data corresponding to the scene the user would see in the absence of HMD 120 (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0076] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330 and CGR rendering module 340.

[0077] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. For this purpose, in various embodiments, CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR mapping generation unit 346, and a data transmission unit 348.

[0078] In some implementations, the data acquisition unit 342 is configured to acquire data from at least... Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For the purposes described, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0079] In some embodiments, the CGR rendering unit 344 is configured to render CGR content via one or more CGR displays 312. For the aforementioned purposes, in various embodiments, the CGR rendering unit 344 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0080] In some implementations, the CGR mapping generation unit 346 is configured to generate a CGR mapping based on media content data (e.g., a 3D mapping of a mixed reality scene or a physical environment in which computer-generated objects may be placed to generate a computer-generated reality mapping). For the aforementioned purposes, in various implementations, the CGR mapping generation unit 346 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0081] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For the stated purpose, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0082] Although the data acquisition unit 342, CGR rendering unit 344, CGR mapping generation unit 346, and data transmission unit 348 are shown residing in a single device (e.g., Figure 1 The data acquisition unit 342, the CGR rendering unit 344, the CGR mapping generation unit 346, and the data transmission unit 348 are located on the display generation component 120, but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR rendering unit 344, the CGR mapping generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0083] also, Figure 3 This is used more as a functional description of various features that may exist in a particular implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separate. For example, Figure 3 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0084] Figure 4 This is a schematic illustration of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1 Controlled by hand tracking unit 243 Figure 2 To track the location / position of one or more parts of a user's hand, and / or the location of one or more parts of the user's hand relative to... Figure 1The movement of the hand tracking device 140 is relative to a portion of the user's surrounding physical environment, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system (defined relative to the user's hand). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0085] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures at least three-dimensional scene information including the human user's hand 406. The image sensor 404 captures images of the hand at sufficient resolution to distinguish the fingers and their corresponding positions. The image sensor 404 typically captures images of other parts of the user's body, or possibly all parts of the body, and may have scaling capabilities or be a dedicated sensor with increased magnification to capture images of the hand at the desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of scene 105, or serves as the image sensor for capturing the physical environment of scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a way that uses the field of view of the image sensor 404 or a portion thereof to define an interaction space in which hand movements captured by the image sensor are considered input to the controller 110.

[0086] In some implementations, image sensor 404 outputs a sequence of frames containing 3D mapping data (and, in addition, possibly color image data) to controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided via an application programming interface (API) to an application running on the controller, which in turn drives display generation component 120. For example, a user can interact with software running on controller 110 by moving his hand 408 and changing his hand gestures.

[0087] In some embodiments, image sensor 404 projects a speckle pattern onto a scene including hand 406 and captures an image of the projected pattern. In some embodiments, controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) via triangulation based on the lateral offset of the specks in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. This method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from image sensor 404. In this disclosure, it is assumed that image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis such that the depth coordinates of points in the scene correspond to the z-component measured by the image sensor. Alternatively, hand tracking device 440 may use other 3D mapping methods, such as stereo imaging or time-of-flight measurement, based on single or multiple cameras or other types of sensors.

[0088] In some implementations, hand tracking device 140 captures and processes time-series depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on a processor in image sensor 404 and / or controller 110 processes the 3D mapping data to extract image block descriptors of the hand from these depth maps. The software may match these descriptors with image block descriptors stored in database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and fingertips.

[0089] The software can also analyze the trajectories of the hand and / or fingers across multiple frames in a sequence to identify gestures. The pose estimation function described herein can be alternated with motion tracking, such that patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to find pose changes occurring in the remaining frames. Pose, motion, and gesture information is provided to an application running on controller 110 via the aforementioned API. This application can, for example, move and modify the image presented on display generation unit 120 in response to the pose and / or gesture information, or perform other functions.

[0090] In some implementations, gestures include air gestures. An air gesture is a gesture detected without the user touching an input element that is part of the device (e.g., computer system 101, one or more input devices 125 and / or hand tracking device 140) (or independent of an input element that is part of the device) and based on the detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers and / or one or more legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., including a tapping gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shaking gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0091] In some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with a CGR or XR environment (e.g., a virtual or mixed reality environment) performed by the movement of a user's fingers relative to other fingers or portions of the user's hand. In some embodiments, air gestures are gestures detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and based on detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or portion of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tapping gesture that involves the hand moving a predetermined amount and / or speed in a predetermined posture, or a shaking gesture that involves a predetermined speed or amount of rotation of a part of the user's body)).

[0092] In some implementations where the input gesture is an air gesture (e.g., where the input device provides information to the computer system about which user interface element is the target of the user input in the absence of physical contact, such as contact with a user interface element displayed on a touchscreen, or contact with a mouse or touchpad to move the cursor to the user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Therefore, in implementations involving air gestures, for example, the input gesture is combined with (e.g., simultaneously) movement of the user's fingers and / or hand to detect attention (e.g., gaze) toward a user interface element to perform pinch and / or tap input, as described below.

[0093] In some implementations, input gestures directed to a user interface object are performed, either directly or indirectly, by referencing the user interface object. For example, user input is performed directly on the user interface object based on the user's hand being placed at a location corresponding to the position of the user interface object in the three-dimensional environment (e.g., determined based on the user's current viewpoint). In some implementations, when user attention to the user interface object (e.g., gazing) is detected, input gestures are performed indirectly on the user interface object based on the user's hand not being at a position corresponding to the user interface object's position in the three-dimensional environment while the user is performing the input gesture. For example, for direct input gestures, the user can guide their input to the user interface object by initiating a gesture at or near a location corresponding to the display position of the user interface object (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge or center of the option). For indirect input gestures, the user can guide their input to the user interface object by focusing on it (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location not corresponding to the display position of the user interface object).

[0094] In some implementations, the input gestures (e.g., air gestures) used in the various examples and implementations described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0095] In some implementations, pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes the movement of two or more fingers of the hand to contact each other, i.e., optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact. A long pinch gesture as an air gesture includes the movement of two or more fingers of the hand to contact each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact is detected. For example, a long pinch gesture includes a user holding a pinch gesture (e.g., where two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some implementations, a double pinch gesture as an air gesture includes two (e.g., more) pinch inputs (e.g., performed by the same hand) that are detected consecutively with each other immediately (e.g., within a predefined time period). For example, a user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts the contact between two or more fingers), and performs a second pinch input within a predefined time period after releasing the first pinch input (e.g., within 1 second or 2 seconds).

[0096] In some embodiments, pinch and drag gestures as air gestures include pinch gestures (e.g., pinching gestures or long pinch gestures) performed in conjunction with (e.g., following) a drag input that changes the position of the user's hand from a first position (e.g., the start position of the drag) to a second position (e.g., the end position of the drag). In some embodiments, the user holds the pinch gesture while performing the drag input and releases the pinch gesture (e.g., opening two or more of their fingers) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and drag input are performed by the same hand (e.g., the user pinches two or more fingers together to touch each other and uses the drag gesture to move the same hand to the second position in the air). In some implementations, pinch input is performed by the user's first hand, and drag input is performed by the user's second hand (e.g., while the user continues pinch input with their first hand, their second hand moves in the air from a first position to a second position). In some implementations, input gestures as air gestures include inputs performed using both of the user's hands (e.g., pinch and / or tap inputs). For example, input gestures include two (e.g., more) pinch inputs performed in combination with each other (e.g., simultaneously or within a predefined time period). For example, a first pinch gesture (e.g., pinch input, long pinch input, or pinch and drag input) is performed using the user's first hand, and a second pinch input is performed using the other hand (e.g., the second of the user's two hands). In some implementations, there is movement between the user's two hands (e.g., increasing and / or decreasing the distance or relative orientation between the user's two hands).

[0097] In some embodiments, a tap input performed as an air gesture (e.g., pointing at a user interface element) includes movement of a user's finger toward the user interface element, movement of the user's hand toward the user interface element (optionally, the user's finger extends toward the user interface element), downward movement of the user's finger (e.g., mimicking a mouse click or a tap on a touchscreen), or other predefined movements of the user's hand. In some embodiments, the tap input performed as an air gesture is detected based on the movement characteristics of the finger or hand performing the tap gesture movement, which is the finger or hand moving away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by the end of the movement. In some embodiments, the end of the movement is detected based on changes in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of finger or hand movement, and / or a reversal of the acceleration direction of finger or hand movement).

[0098] In some implementations, the user's attention is determined to be directed to a portion of the 3D environment based on the detection of a gaze directed to that portion of the 3D environment (optionally, no other conditions are required). In some implementations, the user's attention is determined to be directed to that portion of the 3D environment based on the detection of a gaze directed to that portion of the 3D environment using one or more additional conditions, such as requiring the gaze to be directed to that portion of the 3D environment for at least a threshold duration (e.g., dwell time) and / or requiring the gaze to be directed to that portion of the 3D environment when the user's viewpoint is within a distance threshold from that portion of the 3D environment, so that the device determines that the user's attention is directed to that portion of the 3D environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the 3D environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0099] In some implementations, the detection of the readiness configuration of a user or a portion of a user is performed by a computer system. The detection of the readiness configuration of the hand is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the readiness state of the hand is determined based on whether it has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart in preparation for a pinch or grasping gesture, or a pre-tap shape with one or more fingers extended and the back of the hand facing the user), whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15cm, 20cm, 25cm, 30cm, or 50cm from the body), and / or whether the hand has moved in a particular manner (e.g., moving towards an area in front of the user above the user's waist and below the user's head, or moving away from the user's body or legs). In some implementations, the readiness state is used to determine whether an interactive element of the user interface responds to attentional (e.g., gaze) input.

[0100] In some embodiments, the software may be downloaded to controller 110 electronically, for example, via a network, or alternatively, may be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, database 408 is also stored in memory associated with controller 110. Alternatively or additionally, some or all of the described functions of the computer may be implemented in dedicated hardware, such as custom or semi-custom integrated circuits or programmable digital signal processors (DSPs). Although in Figure 4The controller 110 is shown, but for example, as a separate unit from the image sensor 440, some or all of the controller's processing functions may be performed by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or by other devices associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing function of the image sensor 404 may also be integrated into a computer or other computerized device controlled by the sensor output.

[0101] Figure 4 It also includes a schematic diagram of a depth map 410 captured by image sensor 404 according to some embodiments. As described above, the depth map comprises a matrix of pixels with corresponding depth values. Pixel 412 corresponding to hand 406 has been segmented from the background and wrist in this map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z-distance from image sensor 404), where gray shadows become darker as depth increases. Controller 110 processes these depth values ​​to identify and segment components of the image that have human hand features (i.e., a group of adjacent pixels). These features may include, for example, overall size, shape, and frame-to-frame motion from the depth map sequence.

[0102] Figure 4 The diagram also schematically illustrates the hand skeleton 414 ultimately extracted by the controller 110 from the depth map 410 of the hand 406 according to some embodiments. Figure 4 In this configuration, the skeleton 414 is overlaid on the hand background 416, which has already been segmented from the original depth map. In some embodiments, key feature points of the hand, and optionally those on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, fingertips, the center of the palm, the end of the hand that connects to the wrist, etc.), are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points across multiple image frames to determine, according to some embodiments, the gesture performed by the hand or the current state of the hand.

[0103] Figure 5 An eye-tracking device 130 is shown. Figure 1 Exemplary implementations of ). In some implementations, the eye-tracking device 130 comprises an eye-tracking unit 245 ( Figure 2The eye-tracking device 130 is used to track the positioning and movement of a user's gaze relative to scene 105 or relative to CGR content displayed via display generation component 120. In some embodiments, the eye-tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as a headset, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating CGR content for the user to view and components for tracking the user's gaze relative to the CGR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or a CGR chamber, the eye-tracking device 130 is optionally a separate device from the handheld device or CGR chamber. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or not head-mounted. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally used in conjunction with head-mounted display generation components. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally part of non-head-mounted display generation components.

[0104] In some embodiments, the display generation unit 120 uses display mechanisms (e.g., a left near-eye display panel and a right near-eye display panel) to display frames including left and right images in front of the user's eyes, thereby providing the user with a 3D virtual view. For example, the head-mounted display generation unit may include left and right optical lenses (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation unit may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation unit may have a transparent or semi-transparent display on which virtual objects are displayed, allowing the user to view the physical environment directly through the transparent or semi-transparent display. In some embodiments, the display generation unit projects virtual objects onto the physical environment. The virtual objects may be projected, for example, onto a physical surface or as holograms, allowing an individual to observe virtual objects superimposed on the physical environment using the system. In this case, separate display panels and image frames for the left and right eyes may not be necessary.

[0105] like Figure 5As shown, in some embodiments, gaze tracking device 130 includes at least one eye-tracking camera (e.g., an infrared (IR) or near-infrared (NIR) camera) and an illumination source (e.g., an array or ring of IR or NIR light sources, such as LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be pointed at the user's eye to receive IR or NIR light reflected directly from the eye, or alternatively, it may be pointed at "hot" mirrors located between the user's eye and the display panel, which reflect the IR or NIR light from the eye back to the eye-tracking camera while allowing visible light to pass through. Gaze tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps), analyzes these images to generate gaze tracking information, and transmits the gaze tracking information to controller 110. In some embodiments, the user's two eyes are tracked separately using corresponding eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked using corresponding eye-tracking cameras and illumination sources.

[0106] In some implementations, a device-specific calibration procedure is used to calibrate the eye-tracking device 130 to determine parameters for the eye-tracking device in a specific operating environment 100, such as the 3D geometry and parameters of the LEDs, camera, thermal mirror (if present), eye lenses, and display. The device-specific calibration procedure can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration procedure can be automated or manual. According to some implementations, a user-specific calibration procedure may include estimations of eye parameters for a specific user, such as pupil position, foveal position, optical axis, visual axis, interocular distance, etc. According to some implementations, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, a flash-assisted method can be used to process the images captured by the eye-tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0107] like Figure 5As shown, the eye-tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system. The gaze tracking system includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-infrared (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an array or ring of IR or NIR light sources, such as NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye-tracking camera 540 may be pointed toward a mirror 550 located between the user's eye 592 and a display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, projector, etc.). These mirrors reflect the IR or NIR light from the eye 592 while allowing visible light to pass through. Figure 5 (as shown in the top portion), or alternatively, it can be pointed towards the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion), Figure 5 (As shown in the bottom part).

[0108] In some implementations, controller 110 renders AR or VR frames 562 (e.g., left and right frames for the left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye-tracking camera 540 for various purposes, such as processing frame 562 for display. Controller 110 optionally estimates the user's gaze point on display 510 based on the gaze tracking input 542 obtained from eye-tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated based on gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0109] The following describes several possible use cases for the user's current gaze direction and is not intended to be limiting. As an example use case, controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, controller 110 may generate virtual content at a higher resolution in the concave region determined according to the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content in the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content in the view based at least partially on the user's current gaze direction. As another exemplary use case in an AR application, controller 110 may guide an external camera used to capture the physical environment for CGR experiences to focus in the determined direction. The external camera's autofocus mechanism may then focus on an object or surface in the environment that the user is currently looking at on display 510. As another exemplary use case, eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of eye lens 520 so that the virtual object the user is currently looking at has appropriate convergence / divergence to match the convergence of the user's eyes 592. The controller 110 can use gaze tracking information to guide the eye lens 520 to adjust its focus so that the nearby object that the user is looking at appears at the correct distance.

[0110] In some embodiments, the eye-tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens 520), an eye-tracking camera (e.g., eye-tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) mounted in a wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eyes 592. In some embodiments, the light source may be arranged in a ring or circle around each lens in the head-mounted device, such as... Figure 5 As shown in the diagram. In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0111] In some embodiments, the display 510 emits light in the visible light range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. It should be noted that the position and angle of the eye-tracking camera 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, cameras 540 with a wider field of view (FOV) and cameras 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, cameras 540 operating at one wavelength (e.g., 850 nm) and cameras 540 operating at different wavelengths (e.g., 940 nm) may be used on each side of the user's face.

[0112] like Figure 5 The implementation of the gaze tracking system shown can be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide users with computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experiences.

[0113] Figure 6A A flash-assisted gaze tracking pipeline according to some embodiments is illustrated. In some embodiments, the gaze tracking pipeline uses a flash-assisted gaze tracking system (e.g., such as...) Figure 1 and Figure 5 The eye-tracking device 130 shown is used to implement this. The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and flash in the current frame. When not in tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and flash in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in tracking state.

[0114] like Figure 6A As shown, the gaze-tracking camera captures left and right images of the user's left and right eyes. The captured images are then fed into a gaze-tracking pipeline for processing to begin at 610. As indicated by the arrow returning to element 600, the gaze-tracking system can continue capturing images of the user's eyes, for example, at a rate of 60 to 120 frames per second. In some embodiments, each set of captured images can be fed into the pipeline for processing. However, in some embodiments or under certain conditions, not all captured frames are processed by the pipeline.

[0115] At 610, for the currently captured image, if the tracking state is yes, the method proceeds to element 640. At 610, if the tracking state is no, the image is analyzed to detect the user's pupil and flash, as indicated at 620. At 630, if the pupil and flash are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0116] At 640, if proceeding from element 410, the current frame is analyzed to track the pupil and flashes in part based on previous information from the previous frame. At 640, if proceeding from element 630, the tracking state is initialized based on the pupils and flashes detected in the current frame. The processing result at element 640 is checked to verify that the tracking or detection result is credible. For example, the result may be checked to determine whether a sufficient number of pupils and flashes were successfully tracked or detected in the current frame for performing gaze estimation. At 650, if the result is not credible, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is credible, the method proceeds to element 670. At 670, the tracking state is set to YES (if not already yes), and the pupil and flash information is passed to element 680 to estimate the user's gaze point.

[0117] Figure 6A This is intended as an example of an eye-tracking technology that can be used in a particular specific implementation. As those skilled in the art will recognize, in various implementations, other existing or future-developed eye-tracking technologies may be used in place of or in combination with the flash-assisted eye-tracking technology described herein in a computer system 101 for providing a CGR experience to a user.

[0118] Figure 6B An exemplary environment for an electronic device 101 providing a CGR experience according to some implementation schemes is shown. Figure 6B In this context, the real-world environment 602 includes electronic device 101, user 608, and real-world objects (e.g., table 604). Figure 6BAs shown, electronic device 101 is optionally mounted on a tripod or otherwise fixed in a real-world environment 602, such that one or more hands of user 608 are free (e.g., user 608 optionally does not hold device 101 with one or more hands). As described above, device 101 optionally has one or more sets of sensors positioned on different sides of device 101. For example, device 101 optionally includes sensor set 612-1 and sensor set 612-2, which are located on the “rear” and “front” sides of device 101, respectively (e.g., they are capable of capturing information from the respective sides of device 101). As used herein, the front side of device 101 is the side facing user 608, and the rear side of device 101 is the side facing away from user 608.

[0119] In some implementations, sensor group 612-2 includes an eye-tracking unit (e.g., as referenced above). Figure 2 The eye-tracking unit 245 includes one or more sensors for tracking the user's eyes and / or gaze, enabling the eye-tracking unit to "look" at the user 608 and track the user 608's eyes in the manner previously described. In some embodiments, the eye-tracking unit of device 101 is capable of capturing the movement, orientation, and / or gaze of the user 608's eyes and treating the movement, orientation, and / or gaze as input.

[0120] In some implementations, sensor group 612-1 includes a hand tracking unit (e.g., as referenced above). Figure 2 The hand tracking unit 243 is capable of tracking one or more hands of a user 608 held on the "rear" side of the device 101, such as... Figure 6B As shown. In some embodiments, a hand tracking unit is optionally included in sensor group 612-2, such that user 608 can additionally or alternatively hold one or more hands on the "front" side of device 101 while device 101 tracks the position of one or more hands. As described above, the hand tracking unit of device 101 is capable of capturing the movement, position, and / or gestures of one or more hands of user 608 and treating said movement, position, and / or gestures as input.

[0121] In some embodiments, sensor group 612-1 optionally includes one or more sensors configured to capture images of a real-world environment 602 including table 604 (e.g., such as those referenced above). Figure 4The image sensor 404 described above. As described above, device 101 is capable of capturing an image of a portion (e.g., some or all) of the real-world environment 602 and presenting the captured portion of the real-world environment 602 to a user via one or more display generation components of device 101 (e.g., the display of device 101, which is optionally located on the user-facing side of device 101, opposite the side of device 101 facing the captured portion of the real-world environment 602).

[0122] In some implementations, a portion of the captured real-world environment 602 is used to provide a CGR experience to the user, such as a mixed reality environment in which one or more virtual objects are overlaid on a representation of the real-world environment 602.

[0123] Therefore, this description describes some embodiments of a three-dimensional environment (e.g., a CGR environment) that includes representations of real-world objects and virtual objects. For example, the three-dimensional environment optionally includes a representation of a table existing in a physical environment, which is captured and displayed in the three-dimensional environment (e.g., actively displayed via a camera and display of an electronic device or passively displayed via a transparent or translucent display of an electronic device). As previously described, the three-dimensional environment is optionally a mixed reality system, wherein the three-dimensional environment is based on a physical environment captured by one or more sensors of the device and displayed via a display generation component. As a mixed reality system, the device is optionally capable of selectively displaying portions and / or objects of the physical environment such that the corresponding portions and / or objects of the physical environment appear as if they exist in the three-dimensional environment displayed by the electronic device. Similarly, the device is optionally capable of displaying virtual objects in the three-dimensional environment to appear as if the virtual objects exist in the real world (e.g., the physical environment) by placing virtual objects at corresponding locations in the three-dimensional environment that have corresponding locations in the real world. For example, the device optionally displays a vase such that the vase appears as if a real vase were placed on top of a table in the physical environment. In some embodiments, each location in the three-dimensional environment has a corresponding location in the physical environment. Therefore, when a device is described as displaying a virtual object at a corresponding location relative to a physical object (e.g., such as at or near a user's hand or at or near a physical table), the device displays the virtual object at a specific location in the three-dimensional environment such that it appears as if the virtual object were at or near a physical object in the physical world (e.g., the virtual object is displayed in the three-dimensional environment at the location corresponding to the location of the virtual object in the physical environment if the virtual object were a real object at that specific location).

[0124] In some implementations, real-world objects that exist in a physical environment and are displayed in a 3D environment can interact with virtual objects that exist only in the 3D environment. For example, the 3D environment may include a table and a vase placed on top of the table, where the table is a view (or representation) of a physical table in the physical environment, and the vase is a virtual object.

[0125] Similarly, a user may optionally be able to interact with virtual objects in a 3D environment using one or both hands as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the device may optionally capture one or more of the user's hands and display a representation of the user's hands in the 3D environment (e.g., in a manner similar to displaying real-world objects in the 3D environment described above), or in some embodiments, the user's hands may be seen via the display generating component, through the ability to see the physical environment through the user interface, due to the transparency / semi-transparency of a portion of the projection of the user interface onto a transparent / semi-transparent surface or onto the user's eyes or into the user's field of view. Thus, in some embodiments, the user's hands are displayed at corresponding locations in the 3D environment and are treated as if they were objects in the 3D environment that could interact with virtual objects in the 3D environment as if these virtual objects were real physical objects in the physical environment. In some embodiments, the user may be able to move his or her hand so that the representation of the hand in the 3D environment moves in conjunction with the movement of the user's hand.

[0126] In some embodiments described below, the device optionally determines an “effective” distance between a physical object in the physical world and a virtual object in a three-dimensional environment, for example, to determine whether the physical object is interacting with the virtual object (e.g., whether a hand is touching, grasping, holding, or holding the virtual object, or is within a threshold distance of the virtual object). For example, the device determines the distance between a user’s hand and a virtual object when determining whether and / or how the user is interacting with the virtual object. In some embodiments, the device determines the distance between a user’s hand and a virtual object by determining the distance between the position of the hand in the three-dimensional environment and the position of the virtual object of interest in the three-dimensional environment. For example, if the user’s one or more hands are located at a specific location in the physical world, the device optionally captures the one or more hands and displays the one or more hands at a specific corresponding location in the three-dimensional environment (e.g., the location where the hand would be displayed in the three-dimensional environment if it is a virtual hand and not a physical hand). Optionally, the position of the hand in the three-dimensional environment is compared with the position of the virtual object of interest in the three-dimensional environment to determine the distance between the user’s one or more hands and the virtual object. In some implementations, the device optionally determines the distance between a physical object and a virtual object by comparing their positions in the physical world (e.g., rather than comparing their positions in a three-dimensional environment). For example, when determining the distance between a user's one or more hands and a virtual object, the device optionally determines the corresponding physical location of the virtual object (e.g., the location the virtual object would be in the physical world if it were a physical object, not a virtual object), and then determines the distance between that corresponding physical location and the user's one or more hands. In some implementations, the same technique is optionally used to determine the distance between any physical object and any virtual object. Thus, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, the device optionally performs any of the techniques described above to map the position of the physical object to the three-dimensional environment and / or map the position of the virtual object to the physical world.

[0127] In some implementations, the same or similar techniques are used to determine where and what the user's gaze is directed at, and / or where and what the physical stylus held by the user is pointing at. For example, if the user's gaze is directed at a specific location in the physical environment, the device optionally determines the corresponding location in the 3D environment, and if a virtual object is located at that corresponding virtual location, the device optionally determines that the user's gaze is directed at that virtual object. Similarly, the device may optionally be able to determine the direction in which the physical stylus is pointing in the physical world based on the orientation of the physical stylus. In some implementations, based on this determination, the device determines a corresponding virtual location in the 3D environment corresponding to the location pointed at by the stylus in the physical world, and optionally determines that the stylus is pointing at the corresponding virtual location in the 3D environment.

[0128] Similarly, the embodiments described herein may refer to the position of a user (e.g., a user of a device) in a three-dimensional environment and / or the position of the device in a three-dimensional environment. In some embodiments, the user of the device is holding, wearing, or otherwise located at or near the electronic device. Thus, in some embodiments, the device's position serves as a proxy for the user's position. In some embodiments, the position of the device and / or the user in the physical environment corresponds to a corresponding position in the three-dimensional environment. In some embodiments, the corresponding position is the location from which a "camera" or "view" of the three-dimensional environment extends. For example, the device's position would be its position in the physical environment (and its corresponding position in the three-dimensional environment) from which, if the user stands facing the corresponding portion of the physical environment displayed by the display generation component, the user will see from that position objects in the physical environment at the same location, orientation, and / or size (e.g., in an absolute sense and / or relative to each other) as when the objects are displayed by the device's display generation component. Similarly, if the virtual objects displayed in a 3D environment are physical objects in the physical environment (e.g., physical objects placed in the physical environment at the same location as these virtual objects in the 3D environment, and physical objects in the physical environment having the same size and orientation as in the 3D environment), then the device and / or user's position is the position in the physical environment where the user will see these virtual objects in the same location, orientation, and / or size (e.g., in an absolute sense and / or relative to each other and real-world objects) as when the virtual objects are displayed by the device's display generation components.

[0129] In this disclosure, various input methods are described in relation to interaction with a computer system. When an example is provided using one input device or method, and another example is provided using another input device or method, it should be understood that each example is compatible with and optionally utilizes the input device or method described with respect to the other example. Similarly, various output methods are described in relation to interaction with a computer system. When an example is provided using one output device or method, and another example is provided using another output device or method, it should be understood that each example is compatible with and optionally utilizes the output device or method described with respect to the other example. Similarly, various methods are described in relation to interaction with a virtual or mixed reality environment via a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and optionally utilizes the methods described with respect to the other example. Therefore, this disclosure discloses embodiments that are combinations of features of a plurality of examples without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.

[0130] Furthermore, in methods described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the method may be repeated in multiple repetitions such that, during the repetitions, all conditions determining the steps in the method are satisfied in different repetitions of the method. For example, if a method requires performing a first step (if a condition is satisfied) and a second step (if a condition is not satisfied), those skilled in the art will know that the stated steps are repeated until both conditions are satisfied and not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing discretionary operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions determining the steps in the method are satisfied. Those skilled in the art will also understand that, similar to methods having discretionary steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all discretionary steps have been performed.

[0131] User interface and related processes

[0132] Now turn attention to implementations of user interfaces (“UIs”) and associated processes that can be implemented on computer systems (such as portable multifunction devices or head-mounted devices) having display generation components, one or more input devices, and (optionally) one or more cameras.

[0133] Figures 7A to 7E Examples are shown of electronic devices using different algorithms to move objects in different directions in a three-dimensional environment according to some implementation schemes.

[0134] Figure 7A The electronic device 101 is shown via a display generation component (e.g., Figure 1 The display generation component 120 displays the three-dimensional environment 702 from the viewpoint of the user 726 shown in the top view (e.g., facing the rear wall of the physical environment where the device 101 is located). (Refer to the above) Figure 1 As depicted in Figure 6, the electronic device 101 optionally includes a display generation component (e.g., a touchscreen) and multiple image sensors (e.g., ...). Figure 3 Image sensor 314). The image sensor optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the electronic device 101 can use to capture one or more images of the user or a portion of the user (e.g., one or more of the user's hands) when the user interacts with the electronic device 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display including display generating components for displaying the user interface or a three-dimensional environment to the user, and sensors for detecting the physical environment and / or movement of the user's hands (e.g., external sensors facing outward from the user) and / or sensors for detecting the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0135] like Figure 7A As shown, device 101 captures one or more images of the physical environment surrounding device 101 (e.g., operating environment 100) (including one or more objects in the physical environment surrounding device 101). In some embodiments, device 101 displays a representation of the physical environment in a three-dimensional environment 702. For example, the three-dimensional environment 702 includes a representation 722a of a coffee table (corresponding to table 722b in a top view), which optionally represents a physical coffee table in the physical environment, and the three-dimensional environment 702 includes a representation 724a of a sofa (corresponding to sofa 724b in a top view), which optionally represents a physical sofa in the physical environment.

[0136] exist Figure 7AIn this 3D environment 702, virtual objects 706a (corresponding to object 706b in the top view) and 708a (corresponding to object 708b in the top view) are also included. Virtual object 706a is optionally located at a distance relatively close to the viewpoint of user 726, and virtual object 708a is optionally located at a distance relatively far from the viewpoint of user 726. Virtual objects 706a and / or 708a are optionally one or more of the following: an application user interface (e.g., a messaging user interface, a content browsing user interface, etc.), a 3D object (e.g., a virtual clock, a virtual ball, a virtual car, etc.), or any other element displayed by device 101 that is not included in the physical environment of device 101.

[0137] In some embodiments, device 101 uses different algorithms to control the movement of an object in different directions within the three-dimensional environment 702; for example, different algorithms for moving the object toward or away from the viewpoint of user 726, or different algorithms for moving the object vertically or horizontally within the three-dimensional environment 702. In some embodiments, device 101 utilizes sensors (e.g., sensor 314) to detect one or more of the absolute and / or relative positions (e.g., relative to each other and / or relative to objects moving within the three-dimensional environment 702) of one or more of the user's hand 705b providing motion input, the shoulder 705a of the user 726 corresponding to hand 705b (e.g., the right shoulder if the right hand is providing motion input), or the object to which the motion input is directed. In some embodiments, the movement of the object is based on the detection quantities described above. Reference method 800 provides details on how the aforementioned detection quantities can optionally be used by device 101 to control the movement of the object.

[0138] exist Figure 7A In this embodiment, hand 703a is providing motion input pointing towards object 708a, and hand 703b is providing motion input pointing towards object 706a. Hand 703a optionally provides input for moving object 708a closer to the viewpoint of user 726, and hand 703b optionally provides input for moving object 706a further away from the viewpoint of user 726. In some embodiments, such motion input includes movement of the hand toward or away from the user 726's body when the user's hand is in a pinched hand shape (e.g., when the tips of the thumb and forefinger touch). For example, according to Figures 7A to 7B The device 101 optionally detects hand 703a moving toward the user 726's body when in a pinched hand shape, and the device 101 optionally detects hand 703b moving away from the user 726's body when in a pinched hand shape. It should be understood that, although in Figures 7A to 7EMultiple hands and corresponding inputs are shown, but such hands and inputs do not need to be detected simultaneously by device 101; instead, in some embodiments, device 101 responds independently to such hands and / or inputs in response to the independent detection of the hands and / or inputs shown and described.

[0139] In response to Figures 7A to 7B Upon detecting motion input, device 101 moves objects 706a and 708a accordingly within the three-dimensional environment 702, such as... Figure 7B As shown. In some embodiments, for a given amount of movement of the user's hand, if the input is an input moving the object toward the user's viewpoint, the device 101 moves the target object more, and if the input is an input moving the object away from the user's viewpoint, the device moves the target object less. In some embodiments, this difference is to prevent the object from moving in the 3D environment 702 far enough from the user's viewpoint that the object can no longer be reasonably interacted with from the user's current viewpoint (e.g., too small in the user's field of view for reasonable interaction). Figures 7A to 7B In this context, hands 703a and 703b optionally have the same amount of movement, but in different directions, as previously described. In response to a given amount of movement of hand 703a toward the body of user 726, device 101 has moved object 708a from its original position to almost the entire distance from the viewpoint of user 726, as... Figure 7B As shown in the top view. In response to the same given amount of movement of hand 703b away from the body of user 726, device 101 has moved the viewpoint of object 706a away from user 726 by a distance less than the distance covered by object 708a, as shown in the top view. Figure 7B As shown in the top view.

[0140] Furthermore, in some embodiments, device 101 controls the size of an object based on its distance from the viewpoint of user 726 within the 3D environment 702, to avoid the object consuming a large portion of the user 726's field of view from its current viewpoint. Therefore, in some embodiments, objects are associated with appropriate or optimal sizes based on their current distances from the user 726's viewpoint, and device 101 automatically adjusts the size of the objects to conform to their appropriate or optimal sizes. However, in some embodiments, device 101 adjusts the size of the object only after detecting user input for moving the object. For example, in... Figure 7A In this context, object 706a is optionally larger than an appropriate or optimal size for its current distance from the viewpoint of user 726, but device 101 has not automatically shrunk object 706a to that appropriate or optimal size. In response to detecting input provided by hand 703b for moving object 706a in the 3D environment 702, device 101 optionally reduces the size of object 706a in the 3D environment 702, such as... Figure 7B The top view is shown in Figure 8. The reduced size of object 706a optionally corresponds to the current distance of object 706a from the viewpoint of user 726. Additional details regarding controlling the size of these objects based on their distance from the user's viewpoint are described with reference to a series of figures in Figure 8 and method 900.

[0141] In some implementations, device 101 applies varying amounts of noise reduction to the movement of objects, depending on the distance of these objects from the viewpoint of user 726. For example, in Figure 7C In this context, device 101 is displaying a three-dimensional environment 702, which includes objects 712a (corresponding to object 712b in the top view), 716a (corresponding to object 716b in the top view), and 714a (corresponding to object 714b in the top view). Objects 712a and 716a are optionally two-dimensional objects (e.g., similar to objects 706a and 708a), and object 714a is optionally a three-dimensional object (e.g., a cube, a three-dimensional model of a car, etc.). Figure 7C In this context, object 712a is farther from the viewpoint of user 726 than object 716a. Hand 703e is optionally currently providing movement input to object 716a, and hand 703c is optionally currently providing movement input to object 712a. Hands 703e and 703c optionally have the same amount of noise at their respective positions (e.g., the same amount of shaking, trembling, or vibration of the hand reflected in bidirectional arrows 707c and 707e with the same length / magnitude). Because device 101 optionally applies more noise reduction to the resulting movement of object 712a controlled by hand 703c than to the resulting movement of object 716a controlled by hand 703e (e.g., because object 712a is farther from the viewpoint of user 726 than object 716a), the noise in the position of hand 703c optionally causes less movement of object 712a than the noise in the position of hand 703e causes movement of object 716a. This difference in noise reduction during movement is optionally reflected in the double-headed arrow 709a, which has a smaller length / magnitude than the double-headed arrow 709b.

[0142] In some implementations, in addition to using different algorithms to move objects toward and away from the user's viewpoint, or alternatively, device 101 uses different algorithms to move objects horizontally and vertically in the three-dimensional environment 702. For example, in Figure 7CIn this embodiment, hand 703d provides an upward vertical movement input pointing to object 714a, and hand 703c provides a rightward horizontal movement input pointing to object 712a. In some embodiments, in response to a given amount of hand movement, when the movement input is a vertical movement input, device 101 moves the object more in the three-dimensional environment 702 compared to when the movement input is a horizontal movement input. In some embodiments, this difference is intended to reduce the strain that a user may feel when moving their hands, as vertical hand movement can be more difficult than horizontal hand movement (e.g., due to gravity and / or anatomical reasons). For example, in Figure 7C In this context, the amount of movement of hands 703c and 703d is optionally the same. In response, such as... Figure 7D As shown, device 101 moved object 714a vertically in three-dimensional environment 702 by a greater amount than it moved object 712a horizontally in three-dimensional environment 702.

[0143] In some implementations, when an object moves within a three-dimensional environment 702 (e.g., in response to indirect hand input, which optionally occurs when the hand is more than a threshold distance (e.g., 1, 3, 6, 12, 24, 36, 48, 60, or 72 cm) from the object being controlled by the hand, as described in more detail with reference to method 800), its orientation is optionally controlled differently depending on whether the object being moved is a two-dimensional or three-dimensional object. For example, when a two-dimensional object moves within a three-dimensional environment, device 101 optionally adjusts the object's orientation such that it remains perpendicular to the viewpoint of user 726. For example, in Figures 7C to 7D In this embodiment, when object 712a is moved horizontally, device 101 automatically adjusts the orientation of object 712a so that it remains perpendicular to the viewpoint of user 726 (e.g., as shown in a top view of the three-dimensional environment). In some embodiments, the perpendicularity of the orientation of object 712a relative to the viewpoint of user 726 is optionally maintained for movement in any direction within the three-dimensional environment 702. Conversely, when a three-dimensional object moves within the three-dimensional environment, device 101 optionally maintains the relative orientation of a particular surface of the object relative to the object or surface within the three-dimensional environment 702. For example, in Figures 7C to 7D In this embodiment, when object 714a moves vertically, device 101 has controlled the orientation of object 714a such that the bottom surface of object 714a remains parallel to the floor in the physical environment and / or the three-dimensional environment 702. In some embodiments, the parallel relationship between the bottom surface of object 714a and the floor in the physical environment and / or the three-dimensional environment 702 is optionally maintained for movement in any direction in the three-dimensional environment 702.

[0144] In some implementations, as described more in reference to method 800, an object moved via direct motion input optionally rotates freely (e.g., pitch, yaw, and / or roll) according to a corresponding rotational input provided by the hand (e.g., rotation of the hand about one or more axes during the motion input), which optionally occurs when the hand is less than a threshold distance (e.g., 1, 3, 6, 12, 24, 36, 48, 60, or 72 cm) from the object controlled by the hand. This free rotation of the object optionally opposes the controlled orientation of the object described above with reference to indirect motion input. For example, in Figure 7C In this process, hand 703e optionally provides direct movement input to object 716a. Figures 7C to 7D The hand 703e can optionally move to the left to move the object 716a to the left in the three-dimensional environment 702, and also provides rotational (e.g., pitch, yaw, and / or roll) input to change the orientation of the object 716a, such as... Figure 7D As shown. Figure 7D As shown, object 716a has been rotated according to the rotation input provided by hand 703e, and object 716a does not maintain a viewpoint perpendicular to user 726.

[0145] However, in some implementations, upon detection of the end of direct movement input (e.g., upon detection of release of a pinched hand shape made by the hand, or upon detection that the user's hand has moved further from the controlled object than a threshold distance), device 101 automatically adjusts the orientation of the controlled object according to the aforementioned rules applied to the type of the controlled object (e.g., two-dimensional or three-dimensional). For example, in Figure 7E In this process, hand 703e has cast object 716a into an empty space in the 3D environment 702. In response, device 101 has automatically adjusted the orientation of object 716a to be perpendicular to the viewpoint of user 726 (e.g., different from the orientation of the object when it was cast), such as... Figure 7E As shown.

[0146] In some implementations, when an object approaches another object or surface, device 101 automatically adjusts the object's orientation to correspond to the other object or surface (e.g., regardless of the orientation rules for two-dimensional and three-dimensional objects described above). For example, from Figures 7D to 7E Hand 703d has provided indirect movement input to move object 714a within a threshold distance (e.g., 0.1, 0.5, 1, 3, 6, 12, 24, 36, or 48 cm) of the surface of a sofa representation 724a, which optionally is a representation of a physical sofa in the physical environment of device 101. In response, in Figure 7EIn this process, device 101 has adjusted the orientation of object 714a to correspond to and / or be parallel to the surface of the approaching representation 724a (e.g., this optionally causes the bottom surface of object 714a to no longer remain parallel to the floor in the physical environment and / or three-dimensional environment 702). Similarly, from Figures 7D to 7E The hand 703c has provided indirect movement input to move object 712a to within a threshold distance (e.g., 0.1, 0.5, 1, 3, 6, 12, 24, 36, or 48 cm) of the surface of virtual object 718a. In response, in Figure 7E In this process, device 101 has adjusted the orientation of object 712a to correspond to and / or be parallel to the surface of the approaching object 718a (e.g., this optionally causes the orientation of object 712a to no longer remain perpendicular to the viewpoint of user 726).

[0147] Furthermore, in some implementations, when a corresponding object moves within a threshold distance of the surface of an object (e.g., physical or virtual), device 101 displays a badge on the corresponding object indicating whether the object is a valid or invalid delivery target. For example, a valid delivery target for the corresponding object is a target to which the corresponding object can be added and / or a target that can contain the corresponding object, and an invalid delivery target for the corresponding object is a target to which the corresponding object cannot be added and / or a target that cannot contain the corresponding object. Figure 7E In this context, object 718a is a valid delivery target for object 712a; therefore, device 101 displays a badge 720 overlaid on the upper right corner of object 712a, indicating that object 718a is a valid delivery target for object 712a. Additional details regarding valid and invalid delivery targets, as well as the associated indications displayed and other responses of device 101, are described with reference to methods 1000, 1200, 1400, and / or 1600.

[0148] Figures 8A to 8K This is a flowchart illustrating a method 800 for moving objects in different directions in a three-dimensional environment using different algorithms according to some embodiments. In some embodiments, method 800 is implemented in a computer system (e.g., Figure 1 The computer system 101, such as a tablet computer, smartphone, wearable computer, or head-mounted device, performs the operation. This computer system includes a display generation component (e.g., Figure 1 , Figure 3 and Figure 4The display generating component 120 (e.g., a head-up display, monitor, touchscreen, projector, etc.) and one or more cameras (e.g., cameras pointing downwards at the user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras) or cameras pointing forward from the user's head). In some embodiments, method 800 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., ...). Figure 1 The control unit 110 in A executes instructions to manage the process. Some operations in method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0149] In some embodiments, method 800 is performed at an electronic device (e.g., 101) communicating with a display generating component (e.g., 120) and one or more input devices (e.g., 314). For example, a mobile device (e.g., a tablet, smartphone, media player, or wearable device) or a computer. In some embodiments, the display generating component is a display integrated with the electronic device (optionally a touchscreen display), an external display such as a monitor, projector, television, or a hardware component (optionally integrated or external) for projecting a user interface or making the user interface visible to one or more users. In some embodiments, the one or more input devices include the ability to receive user input (e.g., capture user input, detect user input, etc.) and transmit information associated with that user input to the electronic device. Examples of input devices include touchscreens, mice (e.g., external), touchpads (optionally integrated or external), remote control devices (e.g., external), another mobile device (e.g., separate from the electronic device), handheld devices (e.g., external), controllers (e.g., external), cameras, depth sensors, eye-tracking devices, and / or motion sensors (e.g., hand-tracking devices, hand motion sensors), etc. In some embodiments, the electronic device communicates with a hand-tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touchscreens, touchpads)). In some embodiments, the hand-tracking device is a wearable device, such as a smart glove. In some embodiments, the hand-tracking device is a handheld input device, such as a remote control or stylus.

[0150] In some implementations, when a first object is displayed in a three-dimensional environment via a display generation component, such as Figure 7AThe object 706a or 708a (e.g., the environment corresponding to the physical environment surrounding the display-generated component) (e.g., a window of an application displayed in a three-dimensional environment, a virtual object displayed in a three-dimensional environment (e.g., a virtual clock, a virtual table, etc.). In some embodiments, a three-dimensional environment is generated, displayed, or otherwise made viewable by the electronic device (e.g., a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment, etc.)), and when the first object is selected for movement in the three-dimensional environment, the electronic device detects (802a) a first input via one or more input devices corresponding to the movement of a corresponding part of the user's body in the physical environment where the display-generated component is located (e.g., movement of the user's hand and / or arm), such as Figure 7A The input from hand 703a, 703b (when the user's hand is greater than a threshold distance (e.g., 0.2, 0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm) and the user's gaze is directed at the first object, the user's index finger and thumb form a pinch gesture, and then the hand moves in the pinched hand shape; or when the user's hand is less than a threshold distance from the first object, the user's index finger and thumb pinch, and then the hand moves in the pinched hand shape regardless of the user's gaze position). In some embodiments, the first input has one or more of the characteristics of the input described in reference methods 1000, 1200, 1400, and / or 1600.

[0151] In some implementations, in response to detecting a first input (802b), based on determining that the first input includes movement of a corresponding part of the user's body in the physical environment in the direction of the first input, such as for Figure 7A In the hand 703b (e.g., movement of the hand in the first input includes (or only includes) movement of the hand away from the user's viewpoint in the three-dimensional environment), the electronic device moves the first object in the first output direction in the three-dimensional environment (802c) according to the movement of a corresponding part of the user's body in the first input direction in the physical environment, such as... Figure 7B The movement of object 706a in the three-dimensional environment (e.g., moving a first object away from the user's viewpoint in a three-dimensional environment based on the movement of the user's hand), wherein the movement of the first object in the first output direction has a first relationship with the movement of a corresponding part of the user's body in the physical environment in the first input direction (e.g., the amount and / or manner of the first object moving away from the first user's viewpoint is controlled by a first algorithm that converts the movement of the user's hand (e.g., away from the user's viewpoint) into the movement of the first object (e.g., away from the user's viewpoint), the exemplary details of which will be described later).

[0152] In some implementations, the determination of the first input includes the movement of a corresponding part of the user's body in a second input direction different from the first input direction, such as for Figure 7A The hand 703a in the first input (e.g., movement of the hand in the first input includes (or only includes) movement of the hand toward the user's viewpoint in the three-dimensional environment), the electronic device moves the first object in the three-dimensional environment in a second output direction different from the first output direction based on the movement of a corresponding part of the user's body in the second input direction in the physical environment, such as Figure 7B The movement of object 708a in the first object has a second relationship, different from the first relationship, with the movement of a corresponding part of the user's body in the physical environment in the second input direction. For example, the amount and / or manner in which the first object moves toward the first user's viewpoint is controlled by a second algorithm, different from the first algorithm, which translates movement of the user's hand (e.g., toward the user's viewpoint) into movement of the first object (e.g., toward the user's viewpoint), exemplary details of which will be described later. In some embodiments, the translation of hand movement into object movement in the first and second algorithms is different (e.g., different amounts of object movement for a given amount of hand movement). Therefore, in some embodiments, the movement of the first object in different directions (e.g., horizontal relative to the user's viewpoint, vertical relative to the user's viewpoint, away from the user's viewpoint, toward the user's viewpoint, etc.) is controlled by different algorithms that differently translate movement of the user's hand into movement of the first object in the three-dimensional environment. Using different algorithms to control the movement of objects in the three-dimensional environment for different movement directions allows the device to leverage algorithms more suited to the movement directions in question to facilitate improved positional control of objects in the three-dimensional environment, thereby reducing errors in use and improving user-device interaction.

[0153] In some implementations, the amount of movement of the first object (e.g., object 706a) in the first output direction (e.g., the distance the first object moves in the first output direction in a three-dimensional environment, such as moving away from...) Figure 7B The viewpoint in the image is independent of the movement speed of the corresponding part of the user's body (e.g., hand 703b) in the first input direction (804a). (e.g., the amount of movement of the first object in the three-dimensional environment is based on one or more of the following factors: the amount of movement of the user's hand when providing input in the first input direction, the distance between the user's hand and the user's shoulder when providing input in the first input direction, and / or various factors described below, but optionally not based on the speed of the user's hand movement during input in the first input direction).

[0154] In some implementations, the amount of movement of the first object (e.g., object 708a) in the second output direction (e.g., the distance the first object moves in the second output direction in the three-dimensional environment, such as...) Figure 7B The movement speed of the first object in the second input direction (804b) is independent of the speed of movement of the corresponding part of the user's body (e.g., hand 703a) in the second input direction (as directed towards the viewpoint). For example, the amount of movement of the first object in the three-dimensional environment is based on one or more of the following factors: the amount of movement of the user's hand when providing input in the second input direction, the distance between the user's hand and the user's shoulder when providing input in the second input direction, and / or various factors described below, but optionally not based on the speed of movement of the user's hand during input in the second input direction. In some embodiments, even if the distance the object moves in the three-dimensional environment is independent of the speed of movement of the corresponding part of the user, the speed at which the first object moves through the three-dimensional environment is based on the speed of movement of the corresponding part of the user. Making the amount of movement of the object independent of the speed of the corresponding part of the user avoids the situation where a sequential input used to move the object, which originally corresponds to returning the object to its original position before receiving input (if the amount of movement of the object is independent of the speed of movement of the corresponding part of the object), causes the object not to move back to its original position (e.g., because the sequential input is provided with different speeds of the corresponding parts of the user), thereby improving user-device interaction.

[0155] In some implementations, the first output direction (e.g., the first input direction and) is a horizontal direction in the three-dimensional environment relative to the user's viewpoint (806a), such as relative to Figure 7C and Figure 7D The object 712a in the image (e.g., the electronic device is displaying a viewpoint of a three-dimensional environment associated with the user of the electronic device, and the first input direction and / or the first output direction corresponds to inputs and / or outputs for moving the first object in a horizontal direction (e.g., substantially parallel to the floor plane of the three-dimensional environment). For example, the first input direction corresponds to the movement of the user's hand substantially parallel to the floor (and / or substantially perpendicular to gravity) in the physical environment of the electronic device and / or the display generating component, and the first output direction corresponds to the movement of the first object substantially parallel to the floor plane in the three-dimensional environment displayed by the electronic device.

[0156] In some implementations, the second input direction (e.g., the second input direction) and the second output direction are perpendicular directions in the three-dimensional environment relative to the user's viewpoint (806b), such as relative to... Figure 7C and Figure 7DObject 714a in the diagram. For example, the second input direction and / or the second output direction correspond to inputs and / or outputs for moving the first object in a vertical direction (e.g., substantially perpendicular to the floor plane of the three-dimensional environment). For example, the second input direction corresponds to the movement of a user's hand substantially perpendicular to the floor (and / or substantially parallel to gravity) in the physical environment of the electronic device and / or the display generating component, and the second output direction corresponds to the movement of the first object substantially perpendicular to the floor plane in the three-dimensional environment displayed by the electronic device. Thus, in some embodiments, for a given amount of movement of a corresponding part of the user, the amount of movement of the first object is different for vertical movement and horizontal movement of the first object. Providing different amounts of movement for vertical and horizontal movement inputs allows the device to utilize algorithms more suited to the movement directions in question to facilitate improved positional control of objects in the three-dimensional environment (e.g., because vertical hand movement may be more difficult for the user to perform than horizontal hand movement due to gravity), thereby reducing errors in use and improving user-device interaction.

[0157] In some implementations, the movement of a corresponding part of a user's body in the physical environment in a first input direction and a second input direction has a first magnitude (808a), such as Figure 7A The movement values ​​of the middle hand parts 703a and 703b are the same (e.g., the user's hand moves 12cm in the physical environment of the electronic device and / or display generating component), and the movement of the first object in the first output direction has a second value (808b) greater than the first value, such as relative to Figure 7B As shown in object 706a (for example, if the input is an input that causes the first object to move horizontally, then the first object moves horizontally in the three-dimensional environment by 12 cm multiplied by a first multiplier, such as 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2, 4, 6, or 10), and the movement of the first object in the second output direction has a third value (808c) that is greater than the first value and different from the second value, such as relative to... Figure 7BAs shown in object 708a (e.g., if the input is one that causes the first object to move vertically, then the first object moves vertically in the three-dimensional environment by 12 cm multiplied by a second multiplier greater than the first multiplier, such as 1.2, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2, 4, 6, or 10). Therefore, in some embodiments, for a given amount of hand movement, an input for moving the first object vertically (e.g., up or down) results in the first object moving more in the three-dimensional environment compared to an input for moving the first object horizontally (e.g., left or right). It should be understood that the aforementioned multipliers are optionally applied to the entire hand movement or only to the component of the hand movement in the corresponding direction (e.g., horizontal or vertical). Providing different amounts of movement for vertical and horizontal movement inputs allows devices to leverage algorithms better suited to the direction of movement in question to facilitate improved positional control of objects in a 3D environment (e.g., because vertical hand movement may be more difficult for a user to perform than horizontal hand movement due to gravity), thereby reducing errors in use and improving user-device interaction.

[0158] In some implementations, the first relationship is based on the offset between a second corresponding part of the user's body (e.g., a shoulder, such as 705a) and a corresponding part of the user's body (e.g., a hand corresponding to the shoulder, such as 705b), and the second relationship is based on the offset between the second corresponding part of the user's body (e.g., a shoulder) and a corresponding part of the user's body (e.g., a hand corresponding to the shoulder) (810). In some implementations, the offset and / or separation and / or distance and / or angular offset between the user's hand and the user's corresponding shoulder are factors in determining the movement of the first object away from and / or toward the user's viewpoint. For example, the offset between the user's shoulder and hand relative to the movement of the first object away from the user's viewpoint is optionally recorded at the start of the first input (e.g., at the moment when the pinching of the index finger and thumb of the hand is detected, such as when the fingertips of the thumb and index finger are detected together and touching, before the movement of the hand in the pinching hand shape is detected), and this offset corresponds to a factor to be multiplied with the movement of the hand to determine how much the first object moves away from the user's viewpoint. In some implementations, the factor has an offset value of 1 between the shoulder and hand from 0 to 40 cm (or 5, 10, 15, 20, 30, 50, or 60 cm), then linearly increases from 40 cm (or 5, 10, 15, 20, 30, 50, or 60 cm) to 60 cm (or 25, 30, 35, 40, 50, 70, or 80 cm), and then continues to linearly increase at a greater rate from 60 cm (or 25, 30, 35, 40, 50, 70, or 80 cm). The offset between the user's shoulder and hand relative to the movement of the first object toward the user's viewpoint is optionally recorded at the start of the first input (e.g., at the moment when a pinching of the index finger and thumb of the hand is detected, before the movement of the hand in the pinching hand shape is detected), and this offset is halved. The movement of the hand from the initial offset to half the offset is optionally set to correspond to the movement of the first object from its current position to the user's viewpoint. By utilizing hand-to-shoulder offsets when determining object movement, a comfortable and consistent object movement response is provided given different starting offsets, thereby reducing errors in use and improving user-device interaction.

[0159] In some implementations, (e.g., the first input direction and) the first output direction correspond to movement away from the user's viewpoint in the three-dimensional environment (812a), such as Figure 7BAs shown in object 706a. For example, the electronic device is displaying a viewpoint of a three-dimensional environment associated with the user of the electronic device, and a first input direction and / or a first output direction corresponds to inputs and / or outputs for moving a first object away from the user's viewpoint (e.g., substantially parallel to the floor plane of the three-dimensional environment and / or substantially parallel to the orientation of the user's viewpoint in the three-dimensional environment). For example, the first input direction corresponds to the movement of the user's hand away from the user's body in the physical environment of the electronic device and / or the display generating component, and the first output direction corresponds to the movement of the first object away from the user's viewpoint in the three-dimensional environment displayed by the electronic device.

[0160] In some implementations, the second input direction (and) the second output direction corresponds to movement in the three-dimensional environment toward the user's viewpoint (812b), such as Figure 7B The object 708a is shown in the diagram. For example, the second input direction and / or the second output direction correspond to inputs and / or outputs for moving the first object closer to the user's viewpoint (e.g., substantially parallel to the floor plane of the three-dimensional environment and / or substantially parallel to the user's viewpoint orientation in the three-dimensional environment). For example, the second input direction corresponds to the movement of the user's hand toward the user's body in the physical environment of the electronic device and / or the display generating component, and the second output direction corresponds to the movement of the first object toward the user's viewpoint in the three-dimensional environment displayed by the electronic device.

[0161] In some implementations, the movement of the first object in the first output direction is the movement of a corresponding part of the user in the first input direction (e.g., Figure 7A The movement of the hand 703b is increased (e.g., multiplied) by a first value, which is based on the distance (812c) between a portion of the user (e.g., the user's hand, the user's elbow, the user's shoulder) and the position corresponding to the first object. For example, the amount by which the first object moves in the first output direction in a three-dimensional environment is defined by multiplying the amount of movement of the user's hand in the first input direction by the first value. In some embodiments, the first value is based on the distance between a specific portion of the user (e.g., the user's hand, shoulder, and / or elbow) and the position of the first object. For example, in some embodiments, the first value increases as the distance between the first object and the shoulder corresponding to the user's hand providing input for moving the object increases, and decreases as the distance between the first object and the user's shoulder decreases. Further details of the first value will be provided below.

[0162] In some implementations, the movement of the first object in the second output direction is the movement of a corresponding part of the user in the second input direction (e.g., Figure 7AThe movement of the hand 703a in the three-dimensional environment is increased (e.g., multiplied) by a second value different from the first value. This second value is based on the distance (812d) between the user's viewpoint and the position corresponding to the first object in the three-dimensional environment (e.g., and not based on the distance between that part of the user (e.g., the user's hand, the user's elbow, the user's shoulder) and the position corresponding to the first object). For example, the amount of movement of the first object in the three-dimensional environment in the second output direction is defined by multiplying the amount of movement of the user's hand in the second input direction by the second value. In some embodiments, the second value is based on the distance between the user's viewpoint and the position of the first object when the corresponding part of the user begins to move in the second input direction (e.g., when the user's gaze is directed towards the first object and the user's hand is detected to perform a thumb and forefinger pinch gesture). For example, in some embodiments, the second value increases as the distance between the first object and the user's viewpoint increases and decreases as the distance between the first object and the user decreases. Further details of the second value are provided below. Providing different multipliers for movement away from and toward the user's viewpoint allows the device to leverage algorithms better suited to the direction of movement in question to facilitate improved positional control of objects in a 3D environment (e.g., because the maximum movement toward the user's viewpoint is known (e.g., limited to movement toward the user's viewpoint), while the maximum movement away from the user's viewpoint may be unknown), thereby reducing errors in use and improving user-device interaction.

[0163] In some implementations, the first value moves with the corresponding part of the user in the first input direction (e.g., Figure 7A The second value changes as the user's corresponding part moves in the second input direction (e.g., the movement of the middle hand 703b), and / or the second value changes as the user's corresponding part moves in the second input direction (e.g., the movement of the middle hand 703b). Figure 7AThe value changes (814) as the user's hand moves (703a). In some embodiments, the first value is a function of the distance between the first object and the user's shoulder (e.g., the first value increases as this distance increases) and the distance between the user's hand and the user's shoulder (e.g., the first value increases as this distance increases). Thus, in some embodiments, as the user's hand moves in the first input direction (e.g., away from the user), the distance between the first object and the shoulder increases (e.g., in response to the movement of the user's hand) and the distance between the user's hand and the user's shoulder increases (e.g., as a result of the user's hand moving away from the user's body); therefore, the first value increases. In some embodiments, the second value is additionally or alternatively a function of the distance between the first objects and the distance between the user's hand and the user's shoulder when the initial pinch performed by the user's hand causes the user's hand to move in the second input direction. Thus, in some embodiments, when the user's hand moves in the second input direction (e.g., toward the user's body), the distance between the user's hand and the user's shoulder decreases (e.g., as a result of the user's hand moving toward the user's body); therefore, the second value decreases. Dynamic multipliers for movement away from and toward the user's viewpoint provide precise positional control within certain ranges of an object's movement, while also providing the ability to move objects a great distance within other ranges of movement, thereby reducing errors in use and improving user-device interaction.

[0164] In some implementations, the first value moves with the corresponding part of the user in the first input direction (e.g., Figure 7A The value changes in a first manner as the middle hand 703b moves, and the second value changes as the corresponding part of the user moves in the second input direction (e.g., ...). Figure 7A The movement of the hand 703a is altered in a second manner (816) different from the first manner. For example, the change in the first value as a function of the distance between the first object and the user's hand and / or between the user's hand and the user's shoulder is different from the change in the second value as a function of the distance between the user's hand and the user's shoulder (e.g., changing to a larger or smaller value and / or changing in the opposite direction (e.g., increasing or decreasing)). Providing different multipliers for movements away from and towards the user's viewpoint takes into account the difference between the user input (e.g., hand movement) required to move the object away from the user's viewpoint to a potentially unknown distance and the user input (e.g., hand movement) required to move the object towards the user's viewpoint to the maximum distance (e.g., limited to movement all the way to the user's viewpoint), thereby reducing errors in use and improving user-device interaction.

[0165] In some implementations, the first value is moved in the corresponding part of the user in the first input direction (e.g., Figure 7AThe second value remains constant during a given portion of the movement of the middle hand 703b, and the second value remains constant during the movement of the corresponding portion of the user's hand in the second input direction (e.g., Figure 7A The movement of the hand 703a does not remain constant (818) during any given portion of the movement. For example, the first value is optionally constant within a first distance range between the first object and the user's hand and / or between the user's hand and the user's shoulder (e.g., at relatively low distances, such as below 5, 10, 20, 30, 40, 50, 60, 80, 100, or 120 cm), and optionally increases linearly as a function of distance for distances greater than the first distance range. In some embodiments, after a threshold distance greater than the first distance range (e.g., after 30, 40, 50, 60, 80, 100, 120, 150, 200, 300, 400, 500, 750, or 1000 cm), the first value is locked as a constant value greater than its value below the threshold distance. Conversely, the second value optionally varies continuously and / or exponentially and / or logarithmically with the change in distance between the user's hand and shoulder (e.g., decreasing as the distance between the user's hand and shoulder decreases). Providing different multipliers for movements away from and towards the user's viewpoint takes into account the difference in user input (e.g., hand movement) required to move an object away from the user's viewpoint to a potentially unknown distance and user input (e.g., hand movement) required to move an object towards the user's viewpoint to a maximum distance (e.g., limited to movement all the way to the user's viewpoint), thereby reducing errors in use and improving user-device interaction.

[0166] In some implementations, the first and second multipliers are based on the ratio of distance (e.g., between the user's shoulder and the corresponding part of the user's arm) to the length of the user's arm (820). For example, the first value is optionally the result of multiplying two factors. The first factor is optionally the distance between the first object and the user's shoulder (e.g., optionally as a percentage or ratio of the total arm length corresponding to the hand providing the motion input), and the second factor is optionally the distance between the user's hand and the user's shoulder (e.g., optionally as a percentage or ratio of the total arm length corresponding to the hand providing the motion input). The second value is optionally the result of multiplying two factors. The first factor is optionally the distance between the first object and the user's viewpoint when the initial pinch gesture performed by the user's hand results in motion input provided by the user's hand (e.g., this first factor is optionally constant), and is optionally provided as a percentage or ratio of the total arm length corresponding to the hand providing the motion input, and the second factor is optionally the distance between the user's shoulder and the user's hand (e.g., optionally as a percentage or ratio of the total arm length corresponding to the hand providing the motion input). The second factor optionally includes determining the distance between the shoulder and the user's hand at the initial pinch gesture performed by the user's hand, and defining a movement of the user's hand to half of that initial distance that would cause the first object to move from its initial / current position to the position at the user's viewpoint. Defining the movement multiplier as a percentage or ratio based on the user's arm length (e.g., rather than an absolute distance) allows for predictable and consistent device responses to inputs from users with different arm lengths, thereby reducing errors in use and improving user-device interaction.

[0167] In some implementations, as described herein, the electronic device utilizes different algorithms to control the movement of a first object away from or towards the user's viewpoint (e.g., corresponding to the movement of the user's hand away from or towards the user's body, respectively). In some implementations, the electronic device continuously or periodically detects hand movement (e.g., when the hand is in a pinched shape), averages a specific number of hand movement detection frames (e.g., 2, 3, 5, 10, 20, 40, or 70 frames), and determines whether the user's hand movement corresponds to the first object moving away from or towards the user's viewpoint during these averaged frames. When the electronic device makes these determinations, it switches between utilizing a first algorithm (e.g., an algorithm for moving the first object away from the user's viewpoint) or a second algorithm (e.g., an algorithm for moving the first object towards the user's viewpoint), both of which map the user's hand movement to the movement of the first object in a three-dimensional environment. The electronic device optionally switches dynamically between these two algorithms based on the most recent average of the detected user's hand movement.

[0168] Relative to the first algorithm, two factors are optionally determined at the start of the movement input (e.g., when the user's thumb and forefinger pinch) and / or at the start of movement of the user's hand away from the user's body, and these two factors are optionally updated as their components change and / or multiplied by the magnitude of the hand movement to define the magnitude of the object's movement. The first factor is optionally an object-to-shoulder factor, corresponding to the distance between the moving object and the user's shoulder. For distances between the shoulder and the object ranging from 0 cm to a first distance threshold (e.g., 5 cm, 10 cm, 20 cm, 30 cm, 40 cm, 50 cm, 60 cm, or 100 cm), the first factor optionally has a value of 1, and optionally has a value that increases linearly as a function of distance until a maximum factor value (e.g., 2, 3, 4, 5, 6, 7.5, 8, 9, 10, 15, or 20). The second factor is optionally a shoulder-to-hand factor, which has a value of 1 for the distance between the shoulder and hand from 0 cm to a first distance threshold (e.g., 5 cm, 10 cm, 20 cm, 30 cm, 40 cm, 50 cm, 60 cm, or 100 cm, optionally the same as or different from the first distance threshold in the first factor), has a value that increases linearly at a first rate as a function of the distance from the first distance threshold to a second distance threshold (e.g., 7.5 cm, 15 cm, 30 cm, 45 cm, 60 cm, 75 cm, 90 cm, or 150 cm), and then has a value that increases linearly at a second rate greater than the first rate as a function of the distance continuing from the second distance threshold. In some embodiments, the first and second factors are multiplied and multiplied by the amount of hand movement to determine the movement of the object away from the user's viewpoint in the three-dimensional environment. In some implementations, the electronic device applies a maximum value of the object's movement away from the user for a given movement of the user's hand away from the user (e.g., a movement of 1, 3, 5, 10, 30, 50, 100, or 200 meters), and thus applies an upper bound function with the maximum value to the result of the above multiplication.

[0169] In contrast to the second algorithm, a third factor is optionally determined at the start of the motion input (e.g., when the user's thumb and forefinger pinch together) and / or at the start of the movement of the user's hand toward the user's body. The third factor is optionally updated as its components change and / or multiplied by the magnitude of the hand's movement to define the resulting magnitude of the object's movement. The third factor is optionally a shoulder-to-hand factor. An initial distance ("initial distance") between the user's shoulder and hand is optionally determined and recorded at the start of the motion input (e.g., when the user's thumb and forefinger pinch together) and / or at the start of the movement of the user's hand toward the user's body. The value of the third factor is defined by a function that maps the hand's movement, which is half of the "initial distance," to the movement of the object from its current position in the 3D environment up to the user's viewpoint (or to a position corresponding to the initial position of the user's hand when the user's forefinger and forefinger pinch together, or from that position offset by a predetermined amount such as 0.1 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 20 cm, or 50 cm). In some implementations, the function has relatively high values ​​for relatively high shoulder-to-hand distances and relatively low values ​​(e.g., 1 and above) for relatively low shoulder-to-hand distances. In some implementations, the function is a curve concave towards lower factor values.

[0170] In some implementations, the aforementioned distances and / or distance thresholds in the first and / or second algorithms are optionally additionally expressed as relative values ​​(e.g., a percentage of the user's arm length) rather than absolute distances.

[0171] In some implementations, (e.g., a first value and / or) a second value is based on the corresponding portion of the user's data when the first object is selected for movement (e.g., ...). Figure 7A The position (822) of the hand (703a or 703b) (e.g., when an initial pinch gesture performed by the user's hand is detected while the user's gaze is directed at the first object, resulting in movement of the user's hand for moving the first object). For example, as described above, one or more factors defining the second value are based on measurements performed by the electronic device corresponding to the distance between the first object and the user's viewpoint at the time of the initial pinch gesture performed by the user's hand and / or the distance between the user's shoulder and the user's hand at the time of the initial pinch gesture performed by the user's hand. Defining one or more of the movement multipliers based on the arm position when selecting the first object for movement allows the device to facilitate the same amount of movement of the first object for various (e.g., multiple, any) arm positions that initiate the movement input, rather than causing some movements of the first object to be unachievable based on the user's initial arm position when selecting the first object for movement, thereby reducing errors in use and improving user-device interaction.

[0172] In some implementations, when a first object is selected for movement in a three-dimensional environment, the electronic device detects (824a), via one or more input devices, the corresponding movement of a user's corresponding portion in a direction relative to the user's viewpoint level in the three-dimensional environment, such as... Figure 7C The movement 707c of the middle hand 703c. For example, the user's hand moves left and right relative to gravity. In some embodiments, the hand movement corresponds to noise in the user's hand movement (e.g., the user's hand shaking or trembling). In some embodiments, the hand movement has a velocity and / or acceleration less than a corresponding velocity and / or acceleration threshold. In some embodiments, in response to detecting a lateral hand movement with a velocity and / or acceleration greater than the aforementioned velocity and / or acceleration threshold, the electronic device does not apply the aforementioned noise reduction to such movement, but instead moves the first object according to the lateral movement of the user's hand without applying noise reduction.

[0173] In some embodiments, in response to detecting a corresponding movement of a corresponding part of the user's hand, the electronic device updates (824b) the position of the first object in a three-dimensional environment, as described in reference object 712a, based on the noise reduction of the corresponding movement of the corresponding part of the user's hand. In some embodiments, the electronic device moves the first object in the three-dimensional environment based on the magnitude, frequency, velocity, and / or acceleration of the noise reduction of the user's hand movement (e.g., using a 1 EUR filter), rather than based on the magnitude, frequency, velocity, and / or acceleration of the non-noise reduction of the user's hand movement. In some embodiments, the magnitude, frequency, velocity, and / or acceleration of the movement of the first object by the electronic device is less than the magnitude, frequency, velocity, and / or acceleration of the movement of the first object without the application of noise reduction. In some embodiments, the electronic device applies this noise reduction to lateral hand movements (and / or its lateral component), but not to vertical hand movements (and / or its vertical component) and / or hand movements toward or away from the user's viewpoint (and / or components of the hand movement). To reduce noise during lateral hand movement, the noise from object movement that may be more likely to occur during left-right and / or lateral hand movements is reduced, thereby reducing errors in use and improving user-device interaction.

[0174] In some implementations, the location corresponding to the first object is determined as a first distance from the corresponding part of the user when a corresponding movement of the corresponding part of the user is detected, such as... Figure 7C The object 712a (e.g., the first object is at a first distance from the user's hand during a corresponding lateral movement of the hand) adjusts the corresponding movement of the corresponding part of the user based on a first noise reduction amount to generate an adjusted movement (826a) for updating the position of the first object in the three-dimensional environment, such as a reference. Figure 7CThe object 712a described in the text, and based on determining that the position corresponding to the first object is a second distance less than the first distance from the corresponding part of the user when the corresponding movement of the corresponding part of the user is detected, such as Figure 7C The object 716a (e.g., the first object is at a second distance from the user's hand during a corresponding lateral movement of the hand) adjusts the corresponding movement of the corresponding part of the user based on a second noise reduction amount less than a first amount. This second noise reduction amount is used to generate an adjusted movement (826b) for updating the position of the first object in the three-dimensional environment, such as a reference. Figure 7C As described in object 716a. Therefore, in some embodiments, when the movement is directed towards an object farther from the user's hand, the electronic device applies more noise reduction to left-right and / or lateral movements of the user's hand compared to when the movement is directed towards an object closer to the user's hand. Applying different amounts of noise reduction based on the object's distance from the hand and / or the user's viewpoint allows for less filtered / more direct response when the object is at a distance where changes in movement input are more easily perceived, and more filtered / less direct response when the object is at a distance where changes in movement input are less easily perceived, thereby reducing errors in use and improving user-device interaction.

[0175] In some implementations, when a first object is selected for movement and during a first input, the electronic device controls multiple orientation portions corresponding to the first input, such as relative to... Figures 7C to 7D Object 716a controls (828) the orientation of a first object in a three-dimensional environment in multiple directions (e.g., pitch, yaw, or roll). For example, when a user's hand is providing motion input to the first object, input from the user's hand to change the orientation of the first object causes the first object to change its orientation accordingly. For example, when a hand is directly manipulating the first object (e.g., the hand is closer to the first object than a threshold distance (e.g., 0.2, 0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm) during the first input) to rotate the first object, tilt the object, etc., input from the hand causes the electronic device to rotate, tilt, etc., based on such input. In some embodiments, such input includes rotation of the hand, tilting of the hand, etc., when the hand is providing motion input to the first object. Therefore, in some embodiments, during the motion input, the first object is freely tilted, rotated, etc., based on the motion input provided by the hand. Changing the orientation of the first object based on the orientation provided by the user's hand during mobile input allows the first object to respond more fully to the input provided by the user, thereby reducing errors in use and improving user-device interaction.

[0176] In some implementations, when controlling the orientation of the first object in multiple directions (e.g., when the electronic device allows input from a user's hand to control the pitch, yaw, and / or roll of the first object), the electronic device detects (830a) a threshold distance (e.g., 0.1, 0.5, 1, 3, 5, 10, 20, 40, or 50 cm) of the first object on a surface in a three-dimensional environment, such as within... Figure 7E Object 712a is relative to object 718a or object 714a is relative to representation 724a. For example, movement input provided by a user's hand moves the first object within a threshold distance of a virtual or physical surface in a three-dimensional environment. For example, a virtual surface is optionally the surface of a virtual object in a three-dimensional environment (e.g., the top of a virtual table that does not exist in the physical environment where the generated components and / or electronic devices are displayed). A physical surface is optionally the surface of a physical object in the physical environment of an electronic device, and its representation is displayed by the electronic device in a three-dimensional environment (e.g., via digital pass-through or physical pass-through, such as by displaying a transparent portion of the generated components), such as the top of a physical table in a physical environment.

[0177] In some implementations, in response to detecting that a first object is within a threshold distance of a surface in a three-dimensional environment, the electronic device updates (830b) one or more orientations of the first object in the three-dimensional environment to an orientation based on that surface, such as a reference. Figure 7E The objects 712a and 714a described herein (e.g., and not based on the multiple orientation control portions of the first input). In some embodiments, the orientation of the first object is changed to an orientation defined by the surface. For example, if the surface is a wall (e.g., a physical wall or a virtual wall), the orientation of the first object is optionally updated to be parallel to the wall, even if no hand input is provided to change the orientation of the first object (e.g., parallel to the wall). If the surface is a desktop (e.g., a physical desktop or a virtual desktop), the orientation of the first object is optionally updated to be parallel to the desktop, even if no manual input is provided to change the orientation of the first object (e.g., parallel to the desktop). Updating the orientation of the first object to be based on the orientation of a nearby surface provides a quick way to position the first object relative to the surface in a complementary manner, thereby reducing errors in use and improving user-device interaction.

[0178] In some implementations, when the orientation of the first object is controlled in multiple directions (e.g., when the electronic device allows input from a user's hand to control the pitch, yaw, and / or roll of the first object), the electronic device detects (832a) that the first object is no longer selected for movement, such as relative to... Figure 7EThe object 716a in the image (e.g., detecting that the user's hand is no longer in a pinched hand position where the index fingertip is touching the thumbtip and / or that the user's hand has performed a gesture of moving the index fingertip away from the thumbtip). In some embodiments, in response to detecting that the first object is no longer selected for movement, the electronic device updates (832b) one or more orientations of the first object in the three-dimensional environment to a default orientation of the first object in the three-dimensional environment, such as a reference. Figure 7E The object described in 716a is as follows. For example, in some embodiments, the default orientation of an object is defined by the three-dimensional environment, such that the object has a default orientation in the three-dimensional environment without user input to change the object's orientation. For example, the default orientation of a three-dimensional object in the three-dimensional environment is optionally such that the bottom surface of the object should be parallel to the floor of the three-dimensional environment. During movement input, the user's hand may optionally provide orientation change input that causes the bottom surface of the object to be no longer parallel to the floor. However, upon detection of the end of movement input, the electronic device may optionally update the object's orientation such that the bottom surface of the object remains parallel to the floor even without hand input to change the object's orientation. Two-dimensional objects in the three-dimensional environment may optionally have different default orientations. In some embodiments, the default orientation of a two-dimensional object is an orientation in which the surface normal of the object is parallel to the orientation of the user's viewpoint in the three-dimensional environment. During movement input, the user's hand may optionally provide orientation change input that causes the surface normal of the object to be no longer parallel to the orientation of the user's viewpoint. However, upon detection of the end of movement input, the electronic device may optionally update the object's orientation such that the surface normal of the object remains parallel to the orientation of the user's viewpoint even without hand input to change the object's orientation. Updating the orientation of the first object to the default orientation ensures that the object does not end up with an unavailable orientation over time, thereby reducing errors in use and improving user-device interaction.

[0179] In some implementations, the first input is in the user's corresponding section (e.g., Figure 7CThe hand (703e) in the image occurs (834a) when it is within a threshold distance (e.g., 0.2, 0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm) corresponding to the position of the first object (e.g., during which the electronic device allows input from the user's hand to control the pitch, yaw, and / or roll of the first object; the first input is a direct manipulation input from the hand pointing at the first object). In some embodiments, when the first object is selected for movement and during a second input corresponding to the movement of the first object in a three-dimensional environment, wherein during the second input the corresponding part of the user is farther from the position corresponding to the first object than the threshold distance (834b) (e.g., the second input is an indirect manipulation input from the hand pointing at the first object), based on determining that the first object is a two-dimensional object (e.g., the first object is an application window / user interface, a representation of a picture, etc.), the electronic device moves the first object in a three-dimensional environment according to the second input (834c) while the orientation of the first object relative to the user's viewpoint in the three-dimensional environment remains constant, such as a reference. Figures 7C to 7D The object 712a described in the text (e.g., for a two-dimensional object, the indirect movement input optionally causes the position of the two-dimensional object to change in a three-dimensional environment according to the input, but the orientation of the two-dimensional object is controlled by an electronic device so that the surface normal of the two-dimensional object remains parallel to the orientation of the user's viewpoint).

[0180] In some implementations, based on the determination that the first object is a three-dimensional object, the electronic device moves (834d) the first object in a three-dimensional environment according to a second input, while the orientation of the first object relative to a surface in the three-dimensional environment remains constant, such as a reference surface. Figures 7C to 7D The object 714a described in the text (e.g., for a three-dimensional object, indirect movement input optionally causes the position of the three-dimensional object to change in the three-dimensional environment according to the input, but the orientation of the three-dimensional object is controlled by an electronic device such that the normal of the bottom surface of the three-dimensional object remains perpendicular to the floor in the three-dimensional environment). Controlling the orientation of the object during movement input ensures that the object does not end up with an unusable orientation over time, thereby reducing errors in use and improving user-device interaction.

[0181] In some implementations, when a first object is selected for movement and during the first input (836a), the electronic device moves (836b) the first object in a three-dimensional environment according to the first input, while maintaining the orientation of the first object relative to the user's viewpoint in the three-dimensional environment, such as a reference. Figures 7C to 7D The object 712a is described in the example. For example, during indirect movement manipulation of a two-dimensional object, when the object is moved to different locations in a three-dimensional environment, the orientation of the object's surface normals remains parallel to the user's viewpoint.

[0182] In some implementations, after moving the first object while maintaining its orientation relative to the user's viewpoint, the electronic device detects (836c) that the first object is within a threshold distance (e.g., 0.1, 0.5, 1, 3, 5, 10, 20, 40, or 50 cm) of a second object in the three-dimensional environment, such as... Figure 7E Object 712a is relative to object 718a. For example, the first object is moved to within a threshold distance of a physical or virtual surface in the three-dimensional environment, as previously described, such as the surface of an application window / user interface, a wall, a table, a floor, etc.

[0183] In some implementations, in response to detecting that the first object is within a threshold distance of the second object, the electronic device updates (836d) the orientation of the first object in the three-dimensional environment based on the orientation of the second object, independent of the orientation of the first object relative to the user's viewpoint, such as a reference. Figure 7E As described in object 712a. For example, when the first object is moved within a threshold distance of the second object, the orientation of the first object is no longer based on the user's viewpoint, but is updated to be defined by the second object. For example, even without providing hand input to change the orientation of the first object, the orientation of the first object is updated to be parallel to the surface of the second object. Updating the orientation of the first object to be based on the orientation of nearby objects provides a fast way to position the first object relative to that object in a complementary manner, thereby reducing errors in use and improving user-device interaction.

[0184] In some embodiments, the first object has a first size in the three-dimensional environment (e.g., size in the three-dimensional environment) (838a) before being selected for movement in the three-dimensional environment. In some embodiments, in response to detecting the selection of the first object for movement in the three-dimensional environment (e.g., in response to detecting a user's hand performing a pinching gesture while the user's gaze is directed at the first object), the electronic device scales (838b) the first object to give it a second size in the three-dimensional environment that differs from the first size, wherein the second size is based on the distance between the position of the first object and the user's viewpoint in the three-dimensional environment corresponding to the detection of the selection of the first object for movement, such as relative to objects 706a and / or 708a. For example, objects optionally have a defined size that is optimal or ideal for their current distance from the user's viewpoint (e.g., to ensure that objects are still interactive with the user at their current distance from the user). In some embodiments, upon detecting the initial selection of an object for movement, the electronic device changes the size of the object to this optimal or ideal size based on the current distance between the object and the user's viewpoint, even if no input to change the size of the first object is received from the hand. Reference method 1000 describes additional details of such rescaling of objects and / or distance-based dimensions. Updating the size of a first object based on its distance from the user's viewpoint provides a fast way to ensure that the object is interactive with the user, thereby reducing errors in use and improving user-device interaction.

[0185] In some implementations, a first object is selected for movement in a 3D environment in response to the detection of a second input, which includes: when the user's gaze is directed at the first object, a corresponding part of the user (e.g., hand 703a or 703b) performs a first gesture, and then maintains the first shape for a threshold time period (840) (e.g., 0.2, 0.4, 0.5, 1, 2, 3, 5, or 10 seconds). For example, if the first object is currently in unoccupied space in the 3D environment (e.g., not included in a container or application window), the first object is selected for movement optionally in response to a pinch-to-hand gesture performed by the user's hand when the user's gaze is directed at the first object, and then the user's hand maintains the pinch-to-hand shape for the threshold time period. After the second input, movement of the hand while maintaining the pinch-to-hand shape optionally causes the first object to move in the 3D environment according to the movement of the hand. Selecting a first object for movement in response to a gaze + long pinch-to-hand gesture provides a quick way to select a first object for movement while avoiding unintentional selection of the object for movement, thereby reducing errors in use and improving user-device interaction.

[0186] In some implementations, a first object is selected for movement in a three-dimensional environment in response to the detection of a second input, which includes a movement (842) greater than a movement threshold (e.g., 0.1, 0.3, 0.5, 1, 2, 3, 5, 10, or 20 cm) of a corresponding portion of the user (e.g., the user's hand providing the movement input, such as hand 703a) toward the user's viewpoint in the three-dimensional environment. For example, if the first object is included within another object (e.g., an application window), the first object is selected for movement optionally in response to a pinch gesture performed by the user's hand when the user's gaze is directed toward the first object, and then the user's hand moves toward the user corresponding to the movement of the first object toward the user's viewpoint. In some implementations, the movement of the hand toward the user needs to correspond to a movement greater than the movement threshold; otherwise, the first object is optionally not selected for movement. After the second input, movement of the hand while maintaining the pinched hand shape optionally causes the first object to move in the three-dimensional environment according to the movement of the hand. Responding to gaze + pinch + flick gestures to select the first object to move provides a quick way to select the first object to move while avoiding unintentional selection of the object to move, thereby reducing errors in use and improving user-device interaction.

[0187] In some implementations, when the first object is selected for movement and during the first input (844a), the electronic device detects (844b) that the first object is within a threshold distance (e.g., 0.1, 0.5, 1, 3, 5, 10, 20, 40, or 50 cm) of the second object in the three-dimensional environment. For example, the first object is moved within the threshold distance of a physical or virtual surface in the three-dimensional environment, such as, as previously described, application windows / user interfaces, walls, tables, floor surfaces, etc. In some implementations, in response to detecting that the first object is within the threshold distance of the second object (844c), based on determining that the second object is a valid delivery target for the first object (e.g., the second object may contain or receive the first object, such as the second object being a messaging user interface of a messaging application, and the first object being a representation of something that can be sent to another user via the messaging application, such as a representation of a photograph, a representation of a video, a representation of text content, etc.), the electronic device displays (844d) via a display generation component a first visual indication indicating that the second object is a valid delivery target for the first object, such as a reference. Figure 7E The object 712a described in the text (e.g., and without displaying the second visual indication described below). For example, displaying a badge (e.g., a circle including a + symbol) indicating that the second object is a valid delivery target, overlaid on the upper right corner of the first object, and releasing it at the current position of the first object would cause the first object to be added to the second object.

[0188] In some implementations, based on the determination that the second object is not a valid delivery target for the first object (e.g., the second object cannot contain or receive the first object, such as the second object being a messaging user interface of a messaging application and the first object being a user interface of another application or a three-dimensional object), the electronic device displays (844e) a second visual indication via a display generating component indicating that the second object is not a valid delivery target for the first object, such as displaying this indication instead of... Figure 7E The indicator 720 (e.g., and not displaying the aforementioned first visual indicator). For example, displaying a badge (e.g., a circle including an X symbol) overlaid on the upper right corner of the first object indicating that the second object is not a valid delivery target, and releasing it at the current position of the first object will not cause the first object to be added to the second object. Indicating whether the second object is a valid delivery target for the first object quickly conveys the result of placing the first object at its current position, thereby reducing errors in use and improving user-device interaction.

[0189] It should be understood that the specific order in which the operations in method 800 are described is merely exemplary and not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein.

[0190] Figures 9A to 9E Examples are shown of electronic devices, according to some implementation schemes, dynamically adjusting (or not adjusting) the size of virtual objects in a three-dimensional environment.

[0191] Figure 9A The electronic device 101 is shown via a display generation component (e.g., Figure 1 The display generation component 120 displays the three-dimensional environment 902 from the viewpoint of the user 926 shown in the top view (e.g., facing the rear wall of the physical environment where the device 101 is located). (Refer to the above) Figure 1 As depicted in Figure 6, the electronic device 101 optionally includes a display generation component (e.g., a touchscreen) and multiple image sensors (e.g., ...). Figure 3Image sensor 314). The image sensor optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the electronic device 101 can use to capture one or more images of the user or a portion of the user (e.g., one or more of the user's hands) when the user interacts with the electronic device 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display including display generating components for displaying the user interface or a three-dimensional environment to the user, and sensors for detecting the physical environment and / or movement of the user's hands (e.g., external sensors facing outward from the user) and / or sensors for detecting the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0192] like Figure 9A As shown, device 101 captures one or more images of the physical environment surrounding device 101 (e.g., operating environment 100) (including one or more objects in the physical environment surrounding device 101). In some embodiments, device 101 displays a representation of the physical environment in a three-dimensional environment 902. For example, the three-dimensional environment 902 includes a representation 924a of a sofa (corresponding to sofa 924b in a top view), which optionally is a representation of a physical sofa in the physical environment.

[0193] exist Figure 9A In the 3D environment 902, virtual objects 906a (corresponding to object 906b in the top view), 908a (corresponding to object 908b in the top view), 910a (corresponding to object 910b in the top view), 912a (corresponding to object 912b in the top view), and 914a (corresponding to object 914b in the top view) are also included. Figure 9AIn this context, objects 906a, 910a, 912a, and 914a are two-dimensional objects, while object 908a is a three-dimensional object (e.g., a cube). Virtual objects 906a, 908a, 910a, 912a, and 914a are optionally one or more of the following: an application user interface (e.g., a messaging user interface, a content browsing user interface, etc.), a three-dimensional object (e.g., a virtual clock, a virtual ball, a virtual car, etc.), or any other element displayed by device 101 that is not included in the physical environment of device 101. In some embodiments, object 906a is a user interface for playing back content (e.g., a video player) and is displayed together with a control user interface 907a (corresponding to object 907b in the top view). The control user interface 907a optionally includes one or more selectable options for controlling the playback of content being presented in object 906a. In some embodiments, the control user interface 907a is displayed below and / or slightly in front of object 906a (e.g., closer to the user's viewpoint). In some embodiments, object 908a is displayed together with gripper bar 916a (corresponding to object 916b in the top view). Gripper bar 916a is optionally an element to which user-supplied input is directed to control the position of object 908a in the three-dimensional environment 902. In some embodiments, input is directed to object 908a (not to gripper bar 916a) to control the position of object 908a in the three-dimensional environment. Thus, in some embodiments, the presence of gripper bar 916a indicates that object 908a can be independently positioned in the three-dimensional environment 902, as described in more detail with reference to method 1600. In some embodiments, gripper bar 916a is displayed below and / or slightly in front of object 908a (e.g., closer to the user's viewpoint).

[0194] In some implementations, device 101 dynamically scales the size of objects as the distance between objects in the 3D environment 902 and the user's viewpoint changes. Whether device 101 scales objects and / or by how much it scales the objects optionally depends on the type of object moving in the 3D environment 902 (e.g., two-dimensional or three-dimensional). For example, in Figure 9AIn this embodiment, hand 903c provides motion input to object 906a to move object 906a further from the viewpoint of user 926 in the 3D environment 902, hand 903b provides motion input to object 908a (e.g., pointing to the grip bar 916a) to move object 908a further from the viewpoint of user 926 in the 3D environment 902, and hand 903a provides motion input to object 914a to move object 914a further from the viewpoint of user 926 in the 3D environment 902. In some embodiments, such motion input includes movement of the hand toward or away from the user 926's body when the user's hand is in a pinched hand shape (e.g., when the tips of the thumb and forefinger touch). For example, according to Figures 9A to 9B Device 101 optionally detects body movement of hands 903a, 903b, and / or 903c away from user 926 while in a pinched hand shape. It should be understood that, although in Figures 9A to 9E Multiple hands and corresponding inputs are shown, but such hands and inputs do not need to be detected simultaneously by device 101; instead, in some embodiments, device 101 responds independently to such hands and / or inputs in response to the independent detection of the hands and / or inputs shown and described.

[0195] In response to detection Figure 9A Based on the input, device 101 moves objects 906a, 908a, and 914a away from the viewpoint of user 926, such as Figure 9B As shown. For example, device 101 has moved object 906a further away from the viewpoint of user 926. In some embodiments, to maintain interactivity with the object as it moves further away from the user's viewpoint (e.g., by avoiding an unreasonably small display size), device 101 increases the size of the object in the 3D environment 902 (e.g., and similarly, decreases the size of the object in the 3D environment 902 as it moves closer to the user's viewpoint). However, to avoid user confusion and / or disorientation, device 101 optionally increases the size of the object by an amount that ensures the object is displayed at a continuously smaller size as it moves further away from the user's viewpoint, but the decrease in the display size of the object is optionally less than the decrease would be if device 101 did not increase the size of the object in the 3D environment 902. Furthermore, in some embodiments, device 101 applies this dynamic scaling to 2D objects but not to 3D objects.

[0196] Therefore, for example in Figure 9B In the process, as object 906a has moved further from the viewpoint of user 926, device 101 has increased the size of object 906a in the 3D environment 902 (e.g., as in a top view, object 906b and...). Figure 9AThe size of object 906b is relative to the increased size indicated by the previous size, but the size of object 906a has been increased in a sufficiently small manner to ensure that the area of ​​the field of view consumed by object 906a in the 3D environment 902 has been reduced from the previous size. Figure 9A Reduce to Figure 9B In this way, the interactivity of object 906a is optionally maintained as the object moves further away from the viewpoint of user 926, while avoiding user confusion and / or disorientation, which is optionally caused by the display size of object 906a not decreasing as object 906a moves further away from the viewpoint of user 926.

[0197] In some implementations, controls displayed alongside object 906a, such as system controls, are not scaled or are scaled differently by device 101 compared to object 906a. For example, in Figure 9B In response to a movement input pointing to object 906a, the control user interface 907a moves away from the viewpoint of user 926 along with object 906a. However, in Figure 9B In this context, device 101 has sufficiently increased the size of the control user interface 907a in the three-dimensional environment 902 (e.g., as reflected in the top view) so that the display size of the control user interface 907a (e.g., the portion of the field of view it consumes) remains constant as the object 906a and the control user interface 907a move away from the viewpoint of user 926. For example, in Figure 9A In the middle, the control user interface 907a has a width approximately the same as that of object 906a, but in Figure 9B In this embodiment, device 101 has sufficiently increased the size of the control user interface 907a, such that the width of the control user interface 907a is greater than the width of the object 906a. Therefore, in some implementations, when the object 906a and the control user interface 907a are moved away from the viewpoint of user 926, device 101 increases the size of the control user interface 907a by more than it increases the size of the object 906a, and when the object 906a and the control user interface 907a are moved closer to the viewpoint of user 926, device 101 optionally similarly decreases the size of the control user interface 907a by more than it decreases the size of the object 906a. Device 101 optionally similarly scales the grip bar 916a associated with object 908a, such as... Figure 9B As shown.

[0198] However, in some implementations, device 101 does not scale 3D objects as they move away from or towards the viewpoint of user 926 in 3D environment 902. Device 101 optionally does this such that the 3D objects mimic the appearance and / or behavior of physical objects as they move toward or away from the user in the physical environment. For example, as reflected in the top view of 3D environment 902, object 908b in Figure 9B China maintains its position with Figure 9A The same size as in. Therefore, from Figures 9A to 9B The display size of object 908b has been reduced significantly compared to the display size of object 906a. Therefore, for the same amount of movement of objects 906a and 908a away from the viewpoint of user 926, from Figures 9A to 9B The reduction in the portion of the user's 926 field of view consumed by object 906a is optionally less than the reduction in the portion of the user's 926 field of view consumed by object 908a.

[0199] exist Figure 9B In this context, object 910a is associated with a drop area 930 for adding objects to object 910a. Drop area 930 is optionally a spatial volume (e.g., a cube or prism) in the 3D environment 902 adjacent to and / or in front of object 910a. The boundaries or volume of drop area 930 are optionally not displayed in the 3D environment 902; in some embodiments, the boundaries or volume of drop area 930 are displayed in the 3D environment 902 (e.g., via outlines, volume highlighting, volume shading, etc.). When an object is moved into drop area 930, device 101 optionally scales the object based on drop area 930 and / or the object associated with drop area 930. For example, in... Figure 9BIn this process, object 914a has been moved into delivery area 930. Therefore, device 101 has scaled down object 914a to fit within delivery area 930 and / or object 910a. The amount of scaling performed by device 101 on object 914a may optionally differ from the scaling of object 914a performed by device 101 based on the distance of object 914a from the viewpoint of user 926 (e.g., as described with reference to object 906a) (e.g., different magnitude and / or different direction). Therefore, in some embodiments, the scaling of object 914a may optionally be based on the size of object 910a and / or delivery area 930, and may optionally not be based on the distance of object 914a from the viewpoint of user 926. Furthermore, in some embodiments, device 101 displays a badge or indication 932 overlaid on the upper right portion of object 914a, indicating whether object 910a is a valid delivery target for object 914a. For example, if object 910a is a picture frame container and object 914a is a representation of an image, then object 910a is optionally a valid target for object 914a, and indication 932 optionally indicates the same amount; conversely, if object 914a is an application icon, then object 910a is optionally an invalid target for object 914a, and indication 932 optionally indicates the same amount. Additionally, in Figure 9B In response to object 914a moving into delivery area 930, device 101 has adjusted the orientation of object 914a to be aligned with object 910a (e.g., parallel to object 910a). If object 914a is not moved into delivery area 930, object 914a will optionally have a different orientation (e.g., corresponding to...). Figure 9A Orientation of object 914a in the middle), such as the reference Figure 9C Discussed.

[0200] In some implementations, if object 914a is removed from delivery area 930, device 101 automatically scales object 914a to a size based on the distance of object 914a from the viewpoint of user 926 (e.g., as described with reference to object 906a) and / or restores the orientation of object 914a to the orientation it would have without becoming aligned with object 910a, such as Figure 9C As shown in the diagram. In some embodiments, if object 910a is an invalid delivery target for object 914a, device 101 displays object 914a at the same size and / or orientation, even if object 914a has moved into and / or is close to object 910a. For example, in Figure 9C In the middle, object 914a has been removed from the delivery area 930 (e.g., in response to...). Figure 9B(Movement input from hand 903a) and / or object 910a is not a valid delivery target for object 914a. Therefore, device 101 displays object 914a in the three-dimensional environment 902 with a size based on the distance of object 914a from the viewpoint of user 926 (e.g., not based on object 910a), which optionally is larger than Figure 9B The dimensions of object 914a. Furthermore, device 101 additionally or alternatively displays object 914a with an orientation optionally based on orientation input from hand 903a and optionally not based on the orientation of object 910a, which orientation is optionally related to... Figure 9B The orientation of object 914a is different.

[0201] from Figures 9B to 9C The hand 903c also provides further movement input to the object 906a, allowing the device 101 to further move the object 906a away from the viewpoint of the user 926. Therefore, the device 101 has further increased the size of the object 906a (e.g., as shown in the top view of the 3D environment 902) while not exceeding the limitations of this scaling, as it relates to the display size of the object 906a, as previously described. The device 101 also optionally further scales the control user interface 907a (e.g., as shown in the top view of the 3D environment 902) to maintain the display size of the control user interface 907a, as previously described.

[0202] In some implementations, when distance changes are caused by input (e.g., from user 926) for moving objects in the 3D environment 902, device 101 scales the objects based on their distance from the user 926's viewpoint. However, when these distance changes are caused by movement of the user 926's viewpoint within the 3D environment 902 (e.g., opposite to the movement of the objects in the 3D environment 902), device 101 does not scale the objects based on their distance from the user 926's viewpoint. For example, in Figure 9D In this context, user 926's viewpoint has moved toward objects 906a, 908a, and 910a in the 3D environment 902 (e.g., corresponding to the user's movement toward the back wall of a room in the physical environment). In response, device 101 updates the display of the 3D environment 902 to reflect the updated viewpoint from user 926. Figure 9D As shown.

[0203] As shown in the top view of the 3D environment 902, device 101 did not [follow any other condition]. Figure 9DThe objects 906a, 908a, or 910a in the 3D environment 902 are scaled as the viewpoint of user 926 moves. The display size of objects 906a, 908a, and 908c increases as the distance between the viewpoint of user 926 and objects 906a, 908a, and 910a decreases.

[0204] However, despite device 101 in Figure 9D There is no scaling object 908a in the display, but device 101 has already scaled the gripper bar 916a based on the reduced distance between the gripper bar 916a and the viewpoint of user 926 (e.g., reducing its size, as reflected in the top view of the 3D environment 902) (e.g., to maintain the display size of the gripper bar 916a). Therefore, in some embodiments, in response to a movement of the viewpoint of user 926, device 101 does not scale non-system objects (e.g., application user interfaces, representations of content such as pictures or movies), but instead scales system objects (e.g., gripper bar 916a, control user interface 907a, etc.) based on the distance between the viewpoint of user 926 and these system objects.

[0205] In some implementations, in response to device 101 detecting motion input pointing to an object that currently has a size not based on the distance between the object and the viewpoint of user 926, device 101 scales the object to have a size based on the distance between the object and the viewpoint of user 926. For example, in Figure 9D In the process, device 101 detects that hand 903b provides movement input pointing towards object 906a. In response, Figure 9E In this process, device 101 has scaled down object 906a (e.g., as reflected in a top view of 3D environment 902) to a size based on the current distance between user 926's viewpoint and object 906a.

[0206] Similarly, in response to device 101 detecting that an object has been removed from a container object (e.g., a delivery target), wherein the object's size is based on the container object rather than on the distance between the object and the user 926's viewpoint, device 101 scales the object to have a size based on the distance between the object and the user 926's viewpoint. For example, in Figure 9D In this context, object 940a is included within object 910a and has dimensions based on object 910a (e.g., as previously described with reference to objects 914a and 910a). Figure 9D In this process, device 101 detects a movement input from hand 903a pointing at object 940a (e.g., from the viewpoint of user 926) for removing object 940a from object 910a. In response, as... Figure 9EAs shown, device 101 has magnified object 940a (e.g., as reflected in the top view of 3D environment 902) to a size based on the current distance between user 926's viewpoint and object 940a.

[0207] Figures 10A to 10I This is a flowchart illustrating a method 1000 for dynamically adjusting (or not adjusting) the size of a virtual object in a three-dimensional environment according to some embodiments. In some embodiments, method 1000 is implemented in a computer system (e.g., Figure 1 The computer system 101, such as a tablet computer, smartphone, wearable computer, or head-mounted device, performs the operation. This computer system includes a display generation component (e.g., Figure 1 , Figure 3 and Figure 4 The display generating component 120 (e.g., a head-up display, monitor, touchscreen, projector, etc.) and one or more cameras (e.g., cameras pointing downwards at the user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras) or cameras pointing forward from the user's head). In some embodiments, method 1000 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., ...). Figure 1 The control unit 110 in A executes instructions to manage the process. Some operations in method 1000 may be optionally combined, and / or the order of some operations may be optionally changed.

[0208] In some embodiments, method 1000 is performed at an electronic device (e.g., 101) communicating with a display generating component (e.g., 120) and one or more input devices (e.g., 314). For example, a mobile device (e.g., a tablet, smartphone, media player, or wearable device) or a computer. In some embodiments, the display generating component is a display integrated with the electronic device (optionally a touchscreen display), an external display such as a monitor, projector, television, or a hardware component (optionally integrated or external) for projecting a user interface or making the user interface visible to one or more users. In some embodiments, the one or more input devices include those capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information associated with that user input to the electronic device. Examples of input devices include touchscreens, mice (e.g., external), touchpads (optionally integrated or external), remote control devices (e.g., external), another mobile device (e.g., separate from the electronic device), handheld devices (e.g., external), controllers (e.g., external), cameras, depth sensors, eye-tracking devices, and / or motion sensors (e.g., hand-tracking devices, hand motion sensors), etc. In some embodiments, the electronic device communicates with a hand-tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touchscreens, touchpads)). In some embodiments, the hand-tracking device is a wearable device, such as a smart glove. In some embodiments, the hand-tracking device is a handheld input device, such as a remote control or stylus.

[0209] In some embodiments, the electronic device displays (1002a) a three-dimensional environment via a display generation component, the three-dimensional environment including a first object (e.g., a three-dimensional virtual object such as a car model, or a two-dimensional virtual object such as a user interface of an application on the electronic device) at its first location, such as Figure 9A An object 906a (e.g., optionally generating, displaying, or otherwise enabling a three-dimensional environment to be viewed by an electronic device (e.g., a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment, etc.)) is present in the three-dimensional environment, wherein the first object has a first size in the three-dimensional environment and occupies a first amount of the field of view from the corresponding viewpoint (e.g., the display size of the first object and / or the angular size of the first object relative to the current position of the user's viewpoint), such as Figure 9AThe size and display size of object 906a in the three-dimensional environment (e.g., the user's viewpoint of the electronic device in a three-dimensional environment). For example, the size of the first object in the three-dimensional environment optionally defines the space / volume occupied by the first object in the three-dimensional environment and is not a function of the distance the first object is from the user's viewpoint into the three-dimensional environment. The size of the first object displayed via the display generation component (e.g., the amount of display area and / or field of view occupied by the first object via the display generation component) is optionally based on the size of the first object in the three-dimensional environment and the distance the first object is from the user's viewpoint into the three-dimensional environment. For example, a given object of a given size in the three-dimensional environment may optionally be displayed via the display generation component at a relatively large size (e.g., occupying a relatively large portion of the field of view from the corresponding viewpoint) when relatively close to the user's viewpoint, and may optionally be displayed via the display generation component at a relatively small size (e.g., occupying a relatively small portion of the field of view from the corresponding viewpoint) when relatively far from the user's viewpoint.

[0210] In some implementations, when a three-dimensional environment is displayed, which includes a first object at its first location, the electronic device receives (1002b) a first input corresponding to a request to move the first object away from the first location in the three-dimensional environment, such as... Figure 9A The input originates from the hand 903c pointing at the object 906a. For example, when the user's hand is greater than a threshold distance (e.g., 0.2, 0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm) from the first object, the user's index finger and thumb form a pinch gesture while the user's gaze is directed at the first object, and then the hand moves in the pinched hand shape; or when the user's hand is less than the threshold distance from the first object, the user's index finger and thumb pinch together, and then the hand moves in the pinched hand shape regardless of the user's gaze position. The movement of the hand optionally moves away from the user's viewpoint, which optionally corresponds to a request to move the first object away from the user's viewpoint in a three-dimensional environment. In some embodiments, the first input has one or more of the characteristics of the input described with reference to methods 800, 1200, 1400, and / or 1600.

[0211] In some implementations, in response to receiving a first input (1002c), based on determining that the first input corresponds to a request to move a first object away from the corresponding viewpoint (1002d), such as Figure 9A The input from the hand 903c pointing to the object 906a (e.g., a request to move the first object away from a position in the 3D environment from which the electronic device displays the 3D environment) causes the electronic device to move the first object away from the corresponding viewpoint from a first position (1002e) to a second position in the 3D environment, where the second position is farther from the corresponding viewpoint than the first position, such as... Figure 9BAs shown in object 906a (e.g., if the first input corresponds to an input that moves the first object from a position 10 meters from the user's viewpoint to a position 20 meters from the user's viewpoint, then the first object in the three-dimensional environment is moved from a position 10 meters from the user's viewpoint to a position 20 meters from the user's viewpoint). In some embodiments, the electronic device scales (1002f) the first object such that when the first object is in the second position, the first object has a second size in the three-dimensional environment that is larger than the first size, such as... Figure 9B The object 906a in the image (e.g., the size of the first object in the 3D environment increases as the first object moves further away from the user's viewpoint, and optionally decreases as the first object moves closer to the user's viewpoint) occupies a second quantity of the field of view from the corresponding viewpoint, wherein the second quantity is smaller than the first quantity, such as... Figure 9B As shown in object 906a. For example, increasing the size of the first object in the three-dimensional environment as it moves away from the user's viewpoint would make the display area occupied by the first object via the display generating components remain the same or increase by a smaller amount as the first object moves away from the user's viewpoint. In some embodiments, the size of the first object in the three-dimensional environment increases as the first object moves away from the user's viewpoint to maintain the user's ability to interact with the first object, if the first object is not magnified, the first object may optionally become too small to interact with as it moves away from the user's viewpoint. However, to avoid the perception that the first object has not actually moved further away from the user's viewpoint (e.g., this would optionally occur where the display size of the first object remains the same or increases as the first object moves further away from the user's viewpoint), the amount of scaling of the first object performed by the electronic device is small enough to ensure that the display size of the first object (e.g., the amount of field of view occupied by the first object from the corresponding viewpoint) decreases as the first object moves further away from the user's viewpoint. The first object is scaled based on its distance from the user's viewpoint, while maintaining the first object's display or angular size by increasing (when the first object moves toward the user's viewpoint) or decreasing (when the first object moves away from the user's viewpoint). This ensures continued interactivity of the first object within the range of distances from the user's viewpoint, while preventing the first object from becoming disoriented in the 3D environment, thereby improving user-device interaction.

[0212] In some implementations, upon receiving a first input and based on determining that the first input corresponds to a request to move a first object away from the corresponding viewpoint, the first object is continuously scaled to an increased size (e.g., larger than a first size) as it moves further from the corresponding viewpoint (1004), such as a reference. Figure 9BAs described in object 906a. Therefore, in some embodiments, the size change of the first object occurs continuously as the distance of the first object from the corresponding viewpoint changes (e.g., the first object shrinks as it gets closer to the corresponding viewpoint, or the first object is magnified as it moves further away from the corresponding viewpoint). In some embodiments, the size of the first object that increases as it moves further from the corresponding viewpoint is the size of the first object in the three-dimensional environment, which is optionally an amount different from the size of the first object in the field of view observed by the user from the corresponding viewpoint, as will be described in more detail below. Continuously scaling the first object as the distance between the first object and the corresponding viewpoint changes provides the user with immediate feedback on the movement of the first object, thereby improving user-device interaction.

[0213] In some embodiments, the first object is a first type of object, such as object 906a, which is a two-dimensional object (e.g., the first object is a two-dimensional object in a three-dimensional environment, such as a user interface of a messaging application for sending messages to other users), and the three-dimensional environment also includes a second object, which is a second type of object (1006a) different from the first type, such as object 908a, which is a three-dimensional object (e.g., the second object is a three-dimensional object, such as a virtual three-dimensional representation of a car, building, clock, etc. in a three-dimensional environment). In some embodiments, when displaying a three-dimensional environment that includes the second object at its third position, wherein the second object has a third size in the three-dimensional environment and occupies a third amount of the field of view from the corresponding viewpoint, the electronic device receives (1006b) a second input corresponding to a request to move the second object away from the third position in the three-dimensional environment, such as... Figure 9A The input originates from the hand 903b pointing at the object 908a. For example, when the user's hand is greater than a threshold distance (e.g., 0.2, 0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm) from the third object, the user's index finger and thumb form a pinch gesture while the user's gaze is directed at the second object, and then the hand moves in the pinched hand shape; or when the user's hand is less than the threshold distance from the third object, the user's index finger and thumb pinch together, and then the hand moves in the pinched hand shape regardless of the user's gaze position. The hand movement may optionally be away from the user's viewpoint, which optionally corresponds to a request to move the third object away from the user's viewpoint in a three-dimensional environment, or the hand movement may optionally correspond to a request to move the third object toward the user's viewpoint. In some embodiments, the second input has one or more of the characteristics of the input described with reference to methods 800, 1200, 1400, and / or 1600.

[0214] In some implementations, in response to receiving a third input and based on determining that the second input corresponds to a request to move the second object away from the corresponding viewpoint (1006c), such as Figure 9A The input from the hand 903b pointing to the object 908a (e.g., a request to move the third object away from its position in the 3D environment from which the electronic device is displaying the 3D environment) is received. Based on this second input, the electronic device moves the second object away from the corresponding viewpoint from the third position (1006d) to a fourth position in the 3D environment, where the fourth position is farther from the corresponding viewpoint than the third position, without scaling the second object. This is such that when the second object is in the fourth position, it has a third size in the 3D environment and occupies a fourth amount smaller than the third amount in the field of view from the corresponding viewpoint. Figure 9B As shown in object 908a. For example, 3D objects are optionally not scaled based on their distance from the user's viewpoint (as 2D objects are optionally scaled based on their distance from the user's viewpoint compared to 2D objects). Therefore, if a 3D object moves further away from the user's viewpoint, the amount of field of view occupied by the 3D object optionally decreases, and if a 3D object moves closer to the user's viewpoint, the amount of field of view occupied by the 3D object optionally increases. Scales 2D objects but not non-3D objects treats 3D objects similarly to physical objects, which is familiar to the user and leads to the user's expected behavior, thereby improving user-device interaction and reducing errors in use.

[0215] In some implementations, the second object is displayed together with a control user interface for controlling one or more operations associated with the second object (1008a), such control user interfaces as... Figure 9A The object is 907a or 916a. For example, the second object is displayed along with a user interface element that is selectable and movable to allow the second object to move in a three-dimensional environment in a manner corresponding to that movement. For example, the user interface element is optionally a grab bar displayed below the second object, and it can be grabbed to move the second object in the three-dimensional environment. The control user interface optionally includes one or more of the following: a grab bar, a selectable option to stop displaying the second object in the three-dimensional environment, a selectable option to share the second object with another user, etc.

[0216] In some embodiments, when the second object is displayed in a third position, the control user interface is displayed in the third position and has a fourth size in the three-dimensional environment (1008b). In some embodiments, when the second object is displayed in a fourth position (e.g., in response to a second input for moving the second object), the control user interface is displayed in the fourth position and has a fifth size in the three-dimensional environment that is larger than the fourth size (1008c), as shown in objects 907a and 916a. For example, the control user interface moves with the second object according to the same second input. In some embodiments, even if the second object is not scaled in the three-dimensional environment based on the distance between the second object and the user's viewpoint, the control user interface displayed with the second object is scaled in the three-dimensional environment based on the distance between the control user interface and the user's viewpoint (e.g., to ensure continued interactivity of the control user interface elements for the user). In some embodiments, the control user interface elements are scaled in the same manner as the first object is scaled based on movement toward / away from the user's viewpoint. In some embodiments, the control user interface is scaled in a manner less or more than the first object is scaled based on movement toward / away from the user's viewpoint. In some implementations, the control user interface is scaled so that the amount of the user's field of view occupied by the control user interface does not change as the second object moves toward / away from the user's viewpoint. Scaled control user interfaces for 3D objects ensure that users can interact with control user interface elements regardless of the distance between the 3D object and the user's viewpoint, thereby improving user-device interaction and reducing errors in use.

[0217] In some implementations, when displaying a three-dimensional environment including a first object at a first location, the first object having a first size in the three-dimensional environment, wherein the corresponding viewpoint is a first viewpoint, the electronic device detects (1010a) a movement of the user's viewpoint from the first viewpoint to a second viewpoint, the movement changing the distance between the user's viewpoint and the first object, such as... Figure 9D The movement of the user's viewpoint 926. For example, the user moves within the physical environment of the electronic device and / or provides input to the electronic device to move the user's viewpoint from a first corresponding position in the three-dimensional environment to a second corresponding position, such that the electronic device displays the three-dimensional environment from the user's updated viewpoint. This movement of the viewpoint optionally causes the viewpoint to be closer to or further away from the first object compared to the distance from the first object when the viewpoint is the first viewpoint.

[0218] In some implementations, in response to detecting a viewpoint movement from a first viewpoint to a second viewpoint, the electronic device updates the display of the 3D environment (1010b) to reflect the viewpoint from the second viewpoint without scaling the size of a first object, such as a reference object, at the first location in the 3D environment. Figure 9DAs described in objects 906a, 908a, and 910a. For example, the first object maintains the same size in the 3D environment as when the viewpoint is the first viewpoint, but when the viewpoint is the second viewpoint, the amount of field of view occupied by the first object is optionally greater (if the viewpoint moves closer to the first object) or less (if the viewpoint moves further away from the first object) than the amount of field of view occupied by the first object when the viewpoint is the first viewpoint. Abandoning scaling the first object in response to changes in the distance between the first object and the user's corresponding viewpoint caused by the movement of the viewpoint (opposite to the movement of the object) ensures that changes in the 3D environment occur as expected (e.g., in response to user input) and reduce user disorientation, thereby improving user-device interaction and reducing errors in use.

[0219] In some implementations, the first object is an object of a first type (e.g., a content object, such as a two-dimensional user interface of an application on an electronic device; a three-dimensional representation of an object, such as a car; more generally, an object that is or corresponds to content, rather than an object that is or corresponds to a system (e.g., operating system) user interface of an electronic device), and the three-dimensional environment also includes a second object, which is an object of a second type different from the first type (1012a), such as... Figure 9C The object 916a in the image (e.g., a control user interface for the corresponding object as described above, such as a gripper bar for moving the corresponding object in a 3D environment). In some embodiments, when a 3D environment is displayed, which includes a second object at its third position (e.g., displaying a gripper bar for the corresponding object also at a third position in the 3D environment), wherein the second object has a third size in the 3D environment and the user's viewpoint is a first viewpoint, the electronic device detects (1012b) a movement of the viewpoint from the first viewpoint to the second viewpoint, which changes the distance between the user's viewpoint and the second object, such as... Figure 9D The movement of the user's viewpoint 926. For example, the user moves within the physical environment of the electronic device and / or provides input to the electronic device to move the user's viewpoint from a first corresponding position in the three-dimensional environment to a second corresponding position, causing the electronic device to display the three-dimensional environment from the user's updated viewpoint. This movement of the viewpoint optionally causes the viewpoint to be closer to or further away from the second object compared to the distance from the second object when the viewpoint is the first viewpoint.

[0220] In some implementations, in response to detecting movement of the corresponding viewpoint (1012c), the electronic device updates the display of the three-dimensional environment (1012d) from a second viewpoint, such as... Figure 9D As shown in the diagram. In some embodiments, the electronic device will scale (1012e) the size of the second object at the third location to a fourth size in the three-dimensional environment that differs from the third size, such as... Figure 9DThe scaling of object 916a in the document refers to scaling a second object in response to movement of the user's viewpoint based on an updated distance between the user's viewpoint and the second object. In some implementations, the size of the second object is reduced in the 3D environment if the user's viewpoint is closer to the second object, and increased if the user's viewpoint is farther away from the user's viewpoint. The amount of field of view occupied by the second object is optionally kept constant or increased or decreased in the manner described herein. Scaling certain types of objects in response to movement of the user's viewpoint ensures that the user can interact with the object regardless of the distance between the object and the user's viewpoint, even if the change in distance is due to movement of the viewpoint, thereby improving user-device interaction and reducing errors in use.

[0221] In some implementations, when displaying a three-dimensional environment including a first object at its first location, the electronic device detects (1014a) movement of the user's viewpoint from a first viewpoint to a second viewpoint within the three-dimensional environment, the movement altering the distance between the viewpoint and the first object, such as... Figure 9D The movement of the user's viewpoint 926. For example, the user moves within the physical environment of the electronic device and / or provides input to the electronic device to move the user's viewpoint in a three-dimensional environment to a second viewpoint, causing the electronic device to display the three-dimensional environment from the user's updated viewpoint. This movement of the viewpoint optionally causes the viewpoint to be closer to or further away from the first object compared to the distance from the first object when the viewpoint was at the first viewpoint.

[0222] In some implementations, in response to the detection of viewpoint movement, the electronic device updates the display of the 3D environment (1014b) to reflect the viewpoint from a second viewpoint without scaling the size of the first object at the first location in the 3D environment, such as... Figure 9D Objects 906a, 908a, and 910a are shown in the diagram. The first object is optionally not scaled in response to movement of the viewpoint, as previously described.

[0223] In some implementations, when a first object is displayed from a second viewpoint at a first position in a three-dimensional environment, the electronic device receives (1014c) a second input via one or more input devices corresponding to a request to move the first object away from the first position in the three-dimensional environment to a third position in the three-dimensional environment that is further away from the second corresponding position than the first position, such as... Figure 9DThe second input is a movement input pointing towards object 906a. For example, when the user's hand is greater than a threshold distance from the first object (e.g., 0.2, 0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm), the user's index finger and thumb form a pinch gesture when the user's gaze is directed towards the first object, and then the hand moves in the pinched hand shape; or when the user's hand is less than the threshold distance from the first object, the user's index finger and thumb pinch, and then the hand moves in the pinched hand shape regardless of the user's gaze position. The movement of the hand optionally moves away from the user's viewpoint, which optionally corresponds to a request to move the first object away from the user's viewpoint in a three-dimensional environment. In some embodiments, the second input has one or more of the characteristics of the inputs described with reference to methods 800, 1200, 1400, and / or 1600.

[0224] In some implementations, upon detecting the second input and before moving the first object away from the first position (e.g., in response to detecting a pinching motion of the user's index finger and thumb, such as when the tips of the thumb and index finger are detected to meet and touch, before detecting movement of the hand while maintaining the pinched hand shape), the electronic device scales (1014d) the size of the first object to a third size different from the first size, based on the distance between the first object and the second viewpoint at the start of detecting the second input. Figure 9E The object 906a is scaled before it is moved (e.g., when a pinching motion of the user's index finger and thumb is detected). If the viewpoint moves closer to the first object, the size of the first object is optionally reduced, and if the viewpoint moves further away from the first object, the size of the first object is optionally enlarged. The scaling amount of the first object (and / or the resulting amount of the field of view of the viewpoint occupied by the first object) is optionally as previously described with reference to the first object. Thus, in some embodiments, even if the first object is not scaled in response to movement of the user's viewpoint, it is scaled when it is detected that the initiation of a subsequent movement input will be based on the current distance between the first object and the user's viewpoint. Scaling the first object upon detection of movement input ensures that the first object has an appropriate size according to its current distance from the user's viewpoint, thereby improving user-device interaction.

[0225] In some embodiments, the three-dimensional environment also includes a second object at a third location within the three-dimensional environment (1016a). In some embodiments, in response to receiving a first input (1016b), based on determining that the first input corresponds to a request to move the first object to a fourth location in the three-dimensional environment (e.g., a location in the three-dimensional environment that does not include another object), the electronic device displays (1016c) the first object at the fourth location in the three-dimensional environment, where the first object has a third size in the three-dimensional environment. For example, the first object may be scaled based on the first distance, as previously described.

[0226] In some implementations, the determination of a first input is based on one or more criteria, including corresponding criteria that are met when the first input corresponds to a request to move a first object to a third location in a three-dimensional environment, such as... Figure 9A The input to object 914a is indicated by the middle pointer (e.g., the first object moves to the second object, where the second object is a valid delivery target for the first object. Valid and invalid delivery targets are described in more detail with reference to methods 1200 and / or 1400), the third position is a first distance from the corresponding viewpoint (e.g., the distance of the second object from the user's viewpoint is the same as the distance of the fourth position from the user's viewpoint), and the electronic device displays (1016d) the first object at the third position in the three-dimensional environment, where the first object has a fourth dimension in the three-dimensional environment that is different from the third dimension, such as... Figure 9B As shown in object 914a. In some embodiments, when the first object is moved to an object (e.g., a window) or within a threshold distance of that object such as 0.1, 0.2, 0.5, 1, 2, 3, 5, 10, 20, 30, or 50 cm (where the object is an effective delivery target for the first object), the electronic device scales the first object in a different manner than when the first object is not moved to an object (e.g., scaling the first object not based on the distance between the user's viewpoint and the first object). Therefore, even when the first object is moved to a third position, still at a first distance from the user's viewpoint, the first object has a different size and thus occupies a different amount of the user's field of view compared to when the first object is moved to a fourth position. Scaling the first object differently when it is moved to another object provides the user with visual feedback that the first object has been moved to another object, which is potentially the delivery target / container of the first object, thereby improving user-device interaction and reducing errors in use.

[0227] In some implementations, the fourth dimension of the first object is based on the dimension (1018) of the second object, such as Figure 9BAs shown in object 914a. For example, the first object has a size that fits within the second object. For example, if the second object is a user interface of a messaging application and the first object is a representation of a photograph, the first object is optionally scaled up or down to become an appropriate size for inclusion / display within the second object. In some embodiments, the first object is scaled such that it is a certain percentage (e.g., 1%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, or 70%) of the size of the second object. Scaled first object based on the size of the second object ensures that the first object has an appropriate size relative to the second object (e.g., not too large to obscure the second object, nor too small to be properly visible and / or interactive within the second object), thereby improving user-device interaction and reducing errors in use.

[0228] In some implementations, when the first object is in a third position in a three-dimensional environment and has a fourth dimension based on the size of the second object (e.g., when the first object is contained within the second object, such as a representation of a photo contained within a photo browsing and viewing user interface), the electronic device receives (1020a) a second input via one or more input devices corresponding to a request to move the first object away from the third position in the three-dimensional environment, such as relative to... Figure 9D The input of object 940a (e.g., removing the first object from the second object, such as the first object moving beyond a threshold distance from the second object, such as 0.1, 0.2, 0.5, 1, 2, 5, 10, 20, or 30 cm). For example, when the user's hand is greater than the threshold distance (e.g., 0.2, 0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm) and the user's gaze is directed at the first object, the user's index finger and thumb form a pinch gesture, and then the hand moves in the pinched hand shape; or when the user's hand is less than the threshold distance from the first object, the user's index finger and thumb form a pinch gesture, and then the hand moves in the pinched hand shape regardless of the user's gaze position. The movement of the hand optionally faces the user's viewpoint, which optionally corresponds to a request to move the first object toward the user's viewpoint (e.g., and / or away from the second object) in a three-dimensional environment. In some implementations, the second input has one or more of the characteristics of the input described in reference methods 800, 1200, 1400 and / or 1600.

[0229] In some implementations, in response to receiving a second input, the electronic device displays (1020b) the first object at a fifth size, wherein the fifth size is not based on the size of the second object, such as Figure 9EAs shown in object 940a. In some embodiments, when the first object is removed from the second object, the electronic device immediately scales the first object (e.g., before detecting movement of the first object away from the third location). In some embodiments, the electronic device scales the first object based on the distance between the first object and the user's viewpoint (e.g., at the moment the first object is removed from the second object), and is no longer based on or proportional to the size of the second object. In some embodiments, the fifth size is the same as the size displayed by the first object during the first input just before it arrives at / is added to the second object. Scales the first object to a size not based on the second object when it is removed from the second object ensures that the first object has an appropriate size according to its current distance from the user's viewpoint, thereby improving user-device interaction and reducing errors in use.

[0230] In some implementations, when the first input corresponds to moving the first object into a volume within a three-dimensional environment including a third location (such as... Figure 9B When a request is made to any location within a volume (930) of the second object, the corresponding criterion (1022) is satisfied. In some embodiments, the delivery area for the second object is a volume in the three-dimensional environment that covers a portion but not all of the second object, or covers at least some (including all) of the second object (e.g., whose boundaries are optionally not displayed in the three-dimensional environment). In some embodiments, the volume extends from the surface of the second object toward the user's viewpoint. In some embodiments, moving the first object anywhere within the volume causes the first object to scale based on the size of the second object (e.g., rather than based on the distance between the first object and the user's viewpoint). In some embodiments, detecting the end of the first input (e.g., detecting the user's hand release from a pinched hand shape) while the first object is within the volume causes the first object to be added to the second object, as described in reference methods 1200 and / or 1400. Providing a volume in which the first object is scaled based on the second object facilitates easier interaction between the first and second objects, thereby improving user-device interaction and reducing errors in use.

[0231] In some implementations, upon receiving the first input (e.g., and before the end of detecting the first input, as previously described), and based on determining that the first object has moved to a third position according to the first input and meets one or more criteria, the electronic device changes (1024) the appearance of the first object to indicate that the second object is a valid delivery target for the first object, such as a reference. Figure 9BAs described in object 914a. For example, changing the size of the first object, changing the color of the first object, changing the semi-transparency or brightness of the first object, displaying a badge with a "+" symbol overlaid on the upper right corner of the first object, and / or so on, to indicate that the second object is a valid delivery target for the first object. The change in the appearance of the first object may optionally be as described in methods 1200 and / or 1400 in the context of a valid delivery target. Changing the appearance of the first object to indicate that it has been moved to a valid delivery target provides the user with visual feedback that the first object will be added to the second object if the user terminates the movement input, thereby improving user-device interaction and reducing errors in use.

[0232] In some implementations, the one or more criteria include criteria that are met when the second object is a valid delivery target for the first object and not met when the second object is not a valid delivery target for the first object (1026a) (e.g., examples of valid and invalid delivery targets are described with reference to methods 1200 and / or 1400). In some implementations, in response to receiving the first input (1026b), based on the determination that the corresponding criteria are met but the first input does not meet the one or more criteria because the second object is not a valid delivery target for the first object (e.g., the first object has been moved to a position that would have allowed the first object to be added to the second object if the second object were a valid delivery target for the first object), the electronic device displays (1026c) the first object at a fourth position in the three-dimensional environment, wherein the first object has a third dimension in the three-dimensional environment, such as Figure 9C As shown in object 914a. For example, because the second object is not a valid target for the first object, the first object is not scaled based on the size of the second object, but rather based on the current distance between the user's viewpoint and the first object. Abandoning scaling the first object based on the second object provides the user with visual feedback that the second object is not a valid target for the first object, thereby improving user-device interaction and reducing errors in use.

[0233] In some implementations, in response to receiving a first input (1028a), based on determining that the first input meets one or more criteria (e.g., the first object has been moved to a delivery position for the second object, and the second object is a valid delivery target for the first object), the electronic device updates (1028b) the orientation of the first object relative to the corresponding viewpoint based on the orientation of the second object relative to the corresponding viewpoint, such as... Figure 9BObject 914a is shown relative to object 910a. As a supplement to or alternative to scaling the first object when it is moved to a valid delivery target, the electronic device updates / changes the pitch, yaw, and / or roll of the first object to align it with the orientation of the second object. For example, if the second object is a planar object or has a planar surface (e.g., a three-dimensional object with a planar surface), when the first object is moved to the second object, the electronic device changes the orientation of the first object such that the first object (or, if the first object is a three-dimensional object, a surface on the first object) is parallel to the second object (or the surface of the second object). Changing the orientation of the first object when it is moved to the second object ensures that the first object will be properly placed within the second object when it is delivered, thereby improving user-device interaction.

[0234] In some embodiments, the three-dimensional environment further includes a second object (1030a) at a third location within the three-dimensional environment. In some embodiments, upon receiving a first input (1030b), based on a determination that the first input corresponds to a request to move the first object through the third location and further away from the corresponding viewpoint (1030c) (e.g., an input corresponding to the first object moving through a second object, as described with reference to method 1200), the electronic device moves the first object away from the corresponding viewpoint from the first location (1030d) to the third location according to the first input, while simultaneously scaling the first object in the three-dimensional environment based on the distance between the corresponding viewpoint and the first object, such as from... Figures 9A to 9B The object 914a is moved and scaled until it reaches object 910a. For example, the first object moves freely backward away from the user's viewpoint based on a first input until it reaches the second object, as described in reference method 1200. Because the first object moves away from the user's viewpoint as it moves backward toward the second object, the electronic device optionally scales the first object based on the currently changing distance between the first object and the user's viewpoint, as previously described.

[0235] In some implementations, after the first object reaches the third position, the electronic device maintains (1030e) the display of the first object at the third position without scaling the first object, while continuing to receive the first input, such as... Figure 9BIf input from hand 903a pointing to object 914a corresponds to continued movement through object 910a while object 914a remains within volume 930, for example, similar to that described in reference method 1200, when the first object collides with and / or reaches the second object, the first object prevents movement through the second object, even if further input from the user's hand for moving the first object through the second object is detected. When the first object is held in the second / third position, because the distance between the user's viewpoint and the first object has not changed, the electronic device stops scaling the first object in the 3D environment based on further input for moving the first object through / through the second object. If input of a sufficient magnitude to pass through the second object is received (e.g., as described in reference method 1200), the electronic device optionally resumes scaling the first object based on the currently changing distance between the first object and the user's viewpoint. Abandoning scaling the first object when it is fixed to the second object provides the user with feedback that the first object is no longer moving in the 3D environment, thereby improving user-device interaction.

[0236] In some implementations, scaling the first object is based on determining that a second amount of the field of view occupied by the first object of the second size from the corresponding viewpoint is greater than a threshold amount of the field of view (1032a). (For example, the electronic device scales the first object based on the distance between the first object and the user's viewpoint whenever the amount of the first object occupying the user's field of view is greater than a threshold amount, such as 0.1%, 0.5%, 1%, 3%, 5%, 10%, 20%, 30%, or 50% of the field of view). In some implementations, when the first object is displayed at the corresponding size in a three-dimensional environment, where the first object occupies a first corresponding amount of the field of view from the corresponding viewpoint, the electronic device receives (1032b) a second input corresponding to a request to move the first object away from the corresponding viewpoint via one or more input devices. For example, when the user's hand is greater than a threshold distance from the first object (e.g., 0.2, 0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm), the user's index finger and thumb form a pinch gesture when the user's gaze is directed at the first object, and then the hand moves in the pinched hand shape. Alternatively, when the user's hand is less than the threshold distance from the first object, the user's index finger and thumb pinch, and then the hand moves in the pinched hand shape regardless of the user's gaze position. The movement of the hand optionally moves away from the user's viewpoint, which optionally corresponds to a request to move the first object away from the user's viewpoint in a three-dimensional environment. In some embodiments, the second input has one or more of the characteristics of the input described in reference methods 800, 1200, 1400, and / or 1600.

[0237] In some implementations, in response to receiving a second input (1032c), based on determining that a first corresponding amount of the field of view from the corresponding viewpoint is less than a threshold amount of the field of view (e.g., the first object has been moved a certain distance from the user's viewpoint in the 3D environment, at which distance the amount of the field of view occupied by the first object has reached and / or fallen below the threshold amount of the field of view), the electronic device moves the first object away from the corresponding viewpoint (1032d) according to the second input without scaling the size of the first object in the 3D environment, such as if device 101 stops moving object 906a from... Figure 9B Zoom to Figure 9C In some implementations, when the first object is sufficiently far from the user's viewpoint such that the amount of field of view occupied by the first object is less than a threshold amount, the electronic device ceases to scale the size of the first object in the 3D environment based on the distance between the first object and the user's viewpoint. In some implementations, the size of the first object remains constant in the 3D environment as the first object continues to move away from the user's viewpoint. In some implementations, if the first object is subsequently moved closer to the user's viewpoint such that the amount of field of view occupied by the first object reaches and / or exceeds the threshold amount, the electronic device resumes scaling the first object based on the distance between the first object and the user's viewpoint. Abandoning scaling the first object when its field of view is less than the user's threshold saves processing resources when interaction with the first object is ineffective, thereby reducing the power consumption of the electronic device.

[0238] In some implementations, the first input corresponds to a request to move a first object away from the corresponding viewpoint (1034a). In some implementations, in response to receiving a first portion of the first input and before moving the first object away from the corresponding viewpoint (1034b) (e.g., in response to detecting a user's hand performing a pinching gesture, i.e., the tip of the index finger approaches and touches the tip of the thumb, and before the hand in the pinching hand shape subsequently moves), the electronic device scales (1034c) the first object to have a third size different from the first size, based on the current distance between the first object and the corresponding viewpoint, such as if the size of object 906a is not based on the distance between the user's viewpoint and the first object, according to determining that a first size of the first object satisfies one or more criteria, including criteria satisfied when the first size does not correspond to the current distance between the first object and the corresponding viewpoint (e.g., if the current size of the first object is not based on the distance between the user's viewpoint and the first object when the first portion of the first input is detected). Figure 9AThe current distance between the object and the user's viewpoint is considered. Therefore, in some embodiments, in response to a first portion of the first input, the electronic device appropriately determines the size of the first object based on the current distance between the first object and the user's viewpoint. The third size is optionally larger or smaller than the first size, depending on the current distance between the first object and the user's viewpoint. Scaling the first object upon detecting a first portion of the movement input ensures that the first object has an appropriate size according to its current distance from the user's viewpoint, facilitating subsequent interaction with the first object and thus improving user-device interaction.

[0239] It should be understood that the specific order in which the operations in method 1000 are described is merely exemplary and not intended to indicate that the described order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein.

[0240] Figures 11A to 11E An example is shown of an electronic device, according to some implementation schemes, selectively preventing the movement of objects in a three-dimensional environment.

[0241] Figure 11A The electronic device 101 is shown via a display generation component (e.g., Figure 1 The display generation component 120) displays the three-dimensional environment 1102 from the viewpoint of the user 1126 shown in the top view (e.g., facing the rear wall of the physical environment where the device 101 is located). (Refer to the above) Figure 1 As depicted in Figure 6, the electronic device 101 optionally includes a display generation component (e.g., a touchscreen) and multiple image sensors (e.g., ...). Figure 3 Image sensor 314. The image sensor optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the electronic device 101 can use to capture one or more images of the user or a portion of the user (e.g., one or more of the user's hands) when the user interacts with the electronic device 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display including display generating components for displaying the user interface or a three-dimensional environment to the user, and sensors for detecting the physical environment and / or movement of the user's hands (e.g., external sensors facing outwards from the user) and / or sensors for detecting the user's gaze (e.g., internal sensors facing inwards towards the user's face). Device 101 optionally includes one or more buttons (e.g., physical buttons), which are optionally a power button 1140 and a volume control button 1141.

[0242] like Figure 11AAs shown, device 101 captures one or more images of the physical environment surrounding device 101 (e.g., operating environment 100) (including one or more objects in the physical environment surrounding device 101). In some embodiments, device 101 displays a representation of the physical environment in a three-dimensional environment 1102. For example, the three-dimensional environment 1102 includes a representation 1122a of a coffee table (corresponding to table 1122b in a top view), which optionally represents a physical coffee table in the physical environment, and the three-dimensional environment 1102 includes a representation 1124a of a sofa (corresponding to sofa 1124b in a top view), which optionally represents a physical sofa in the physical environment.

[0243] exist Figure 11A In this 3D environment 1102, virtual objects 1104a (corresponding to object 1104b in the top view), 1106a (corresponding to object 1106b in the top view), 1107a (corresponding to object 1107b in the top view), and 1109a (corresponding to object 1109b in the top view) are also included. Virtual objects 1104a and 1106a are optionally located at a distance relatively close to the viewpoint of user 1126, while virtual objects 1107a and 1109a are optionally located at a distance relatively far from the viewpoint of user 1126. Figure 11A In this embodiment, virtual object 1109a is furthest from the viewpoint of user 1126. In some implementations, virtual object 1107a is a valid target for virtual object 1104a, and virtual object 1109a is an invalid target for virtual object 1106a. For example, virtual object 1107a is a user interface (e.g., a messaging user interface) of an application configured to accept and / or display virtual object 1104a, which is optionally a two-dimensional photograph. Virtual object 1109a is optionally a user interface (e.g., a content browsing user interface) of an application that cannot accept and / or display virtual object 1106a, which is also optionally a two-dimensional photograph. In some implementations, virtual objects 1104a and 1106a are optionally one or more user interfaces of an application that includes content (e.g., a quick browse window displaying photographs), three-dimensional objects (e.g., a virtual clock, a virtual ball, a virtual car, etc.), or any other element displayed by device 101 that is not included in the physical environment of device 101.

[0244] In some implementations, the virtual object is displayed in the 3D environment 1102 with a corresponding orientation relative to the viewpoint of user 1126 (e.g., before receiving input for interaction with the virtual object in the 3D environment 1102, which will be described later). Figure 11AAs shown, virtual objects 1104a and 1106a have a first orientation (e.g., the forward surfaces of virtual objects 1104a and 1106a facing the viewpoint of user 1126 are tilted upwards / at a slight angle relative to the viewpoint of user 1126), virtual object 1107a has a second orientation different from the first orientation (e.g., the forward surface of virtual object 1107a facing the viewpoint of user 1126 is tilted to the left / at a slight angle relative to the viewpoint of user 1126, as shown in top view 1107b), and virtual object 1109a has a third orientation different from the first orientation (e.g., the forward surface of virtual object 1109a facing the viewpoint of user 1126 is tilted to the right / at a slight angle relative to the viewpoint of user 1126, as shown in top view 1109b). It should be understood that... Figure 11A The orientation of the objects in the diagram is merely exemplary, and other orientations are possible; for example, objects may optionally share the same orientation in the three-dimensional environment 1102.

[0245] In some implementations, the shadow of the virtual object is optionally displayed by device 101 on a valid projection target for the virtual object. For example, in Figure 11A In this embodiment, the shadow of virtual object 1104a is displayed as overlaying virtual object 1107a, which is the effective projection target for virtual object 1104a. In some embodiments, the relative size of the shadow of virtual object 1104a optionally changes in response to a change in the position of virtual object 1104a relative to virtual object 1107a; therefore, in some embodiments, the shadow of virtual object 1104a indicates the distance between objects 1104a and 1107a. For example, movement of virtual object 1104a closer to virtual object 1107a (e.g., further away from the viewpoint of user 1126) optionally decreases the size of the shadow of virtual object 1104a over virtual object 1107a, and movement of virtual object 1104a further away from virtual object 1107a (e.g., closer to the viewpoint of user 1126) optionally increases the size of the shadow of virtual object 1104a over virtual object 1107a. Figure 11A In this context, because virtual object 1109a is an invalid delivery target for virtual object 1106a, the shadow of virtual object 1106a is not displayed as covering virtual object 1109a.

[0246] In some embodiments, device 101 prevents an object from moving along a specific path or in a specific direction within the three-dimensional environment 1102; for example, device 101 prevents an object from moving along a path containing another object and / or in a direction passing through the other object within the three-dimensional environment 1102. In some embodiments, the first object is prevented from moving through the other object based on the determination that the other object is a valid delivery target for the first object. In some embodiments, the first object is not prevented from moving through the other object based on the determination that the other object is an invalid delivery target for the first object. Additional details regarding the aforementioned object movement are provided below with reference to method 1200.

[0247] exist Figure 11A In this embodiment, hand 1103a (e.g., in hand state A) is providing motion input pointing towards object 1104a, and hand 1105a (e.g., in hand state A) is providing motion input pointing towards object 1106a. Hand 1103a optionally provides input for moving object 1104a away from the viewpoint of user 1126 and in the direction of virtual object 1107a in the 3D environment 1102, and hand 1105a optionally provides input for moving object 1106a away from the viewpoint of user 1126 and in the direction of virtual object 1109a in the 3D environment 1102. In some embodiments, such motion input includes movement of the hand away from the user 1126's body when the user's hand is in a pinched hand shape (e.g., when the tips of the thumb and forefinger touch). For example, according to Figures 11A to 11B Device 101 optionally detects hand 1103a moving away from user 1126's body when in a pinched hand shape, and device 101 optionally detects hand 1105a moving away from user 1126's body when in a pinched hand shape. It should be understood that, although in Figures 11A to 11E Multiple hands and corresponding inputs are shown, but such hands and inputs do not need to be detected simultaneously by device 101; instead, in some embodiments, device 101 responds independently to such hands and / or inputs in response to the independent detection of the hands and / or inputs shown and described.

[0248] In response to Figure 11A Upon detecting motion input, device 101 moves objects 1104a and 1106a accordingly within the three-dimensional environment 1102, such as... Figure 11B As shown. In Figure 11AIn this context, hands 1104a and 1106a optionally have the same amount of movement in the same direction, as previously described. In response to a given amount of movement of hand 1103a away from the body of user 1126, device 101 has moved object 1104a away from the viewpoint of user 1126 and in the direction of virtual object 1107a, wherein the movement of virtual object 1104a has been stopped by virtual object 1107a in the 3D environment 1102 (e.g., when virtual object 1104a reaches and / or collides with virtual object 1107a), as... Figure 11B As shown in the top view. In response to the same given amount of movement of hand 1105a away from user 1126's body, device 101 has moved the viewpoint of object 1106a away from user 1126 by a distance greater than the distance covered by object 1104a, as shown in the top view. Figure 11B As shown in the top view. As discussed above, virtual object 1107a is optionally a valid delivery target for virtual object 1104a, and virtual object 1109a is optionally an invalid delivery target for virtual object 1106a. Therefore, as virtual object 1104a moves in the direction of virtual object 1107a by device 101 in response to a given amount of movement of hand 1103a, when virtual object 1104a reaches / contacts at least a portion of the surface of virtual object 1107a, virtual object 1104a is optionally blocked by device 101 by the movement of virtual object 1107a, as... Figure 11B As shown in the top view. On the other hand, as the virtual object 1106a moves in the direction of the virtual object 1109a in response to a given amount of movement by the device 101 in response to the hand 1105a, the movement of the virtual object 1106a is not hindered when the virtual object 1106a reaches / contacts the surface of the virtual object 1109a, although in Figure 11B In the middle, virtual object 1106a has not yet reached virtual object 1109a, such as Figure 11B As shown in the top view.

[0249] Furthermore, in some implementations, if another object is a valid delivery target for a certain object, then when the object approaches the other object or surface, device 101 automatically adjusts the orientation of the object to correspond to the other object or surface. For example, when virtual object 1104a moves to within a threshold distance (e.g., 0.1, 0.5, 1, 3, 6, 12, 24, 36, or 48 cm) of the surface of virtual object 1107a in response to a given amount of movement of hand 1103a, device 101 optionally adjusts the orientation of virtual object 1104a to correspond to and / or parallel to the surface of the approaching virtual object 1107a (e.g., as shown in the image). Figure 11BAs shown in the top view (in the diagram), virtual object 1107a is a valid delivery target for virtual object 1104a. In some embodiments, if another object is an invalid delivery target for a certain object, then when the object approaches the other object or surface, device 101 abandons automatically adjusting the orientation of the object to correspond to the other object or surface. For example, when virtual object 1106a moves within a threshold distance (e.g., 0.1, 0.5, 1, 3, 6, 12, 24, 36, or 48 cm) of the surface of virtual object 1109a in response to a given amount of movement of hand 1105a, device 101 abandons adjusting the orientation of virtual object 1106a to correspond to and / or parallel to the surface of the approaching virtual object 1109a (e.g., as shown in the top view), because virtual object 1107a is a valid delivery target for virtual object 1104a. Figure 11B (As shown in the top view), because virtual object 1109a is an invalid delivery target for virtual object 1106a.

[0250] Furthermore, in some implementations, when a corresponding object moves within a threshold distance of the surface of an object (e.g., physical or virtual), device 101 displays a badge on the corresponding object indicating whether the object is a valid or invalid delivery target. Figure 11B In this context, object 1107a is a valid delivery target for object 1104a; therefore, device 101 displays a badge 1125 overlaid on the upper right corner of object 1104a, indicating that object 1107a is a valid delivery target for object 1104a when virtual object 1104a moves within a threshold distance (e.g., 0.1, 0.5, 1, 3, 6, 12, 24, 36, or 48 cm) of the surface of virtual object 1107a. In some embodiments, badge 1125 optionally includes one or more symbols or characters (e.g., a "+" symbol indicating that virtual object 1107a is a valid delivery target for virtual object 1104a). In another example, if virtual object 1107a was previously an invalid delivery target for virtual object 1104a, badge 1125 would optionally include one or more symbols or characters (e.g., a "-" symbol or an "x" symbol) indicating that virtual object 1107a was an invalid delivery target for virtual object 1104a. In some implementations, when a corresponding object moves within a threshold distance of an object, if the object is a valid delivery target for that object, device 101 adjusts the size of the corresponding object to indicate that the object is a valid delivery target for that object. For example, in Figure 11BIn this context, when virtual object 1104a moves within a threshold distance of the surface of virtual object 1107a, virtual object 1104a is scaled down (or scaled up) in the three-dimensional environment 1102 in an angular (e.g., angular) dimension. The scaled size of object 1104a is optionally based on the size of object 1107a and / or the size of the area within object 1107a that can accommodate object 1104a (e.g., the larger object 1107a is, the larger the scaled size of object 1104a). Additional details regarding valid and invalid delivery targets, as well as the associated indications and other responses of device 101, are described in reference methods 1000, 1200, 1400, and / or 1600.

[0251] Furthermore, in some embodiments, device 101 controls the size of an object based on the distance of the object included in the 3D environment 1102 from the viewpoint of user 1126, to avoid the object consuming a large portion of the user 1126's field of view from its current viewpoint. Therefore, in some embodiments, objects are associated with appropriate or optimal sizes for their current distances from the user 1126's viewpoint, and device 101 automatically changes the size of the objects to conform to their appropriate or optimal sizes. However, in some embodiments, device 101 adjusts the size of the object only after detecting user input for moving the object. For example, in... Figure 11A In the three-dimensional environment 1102, objects 1104a and 1106a are displayed by device 101 at a first size. In response to the detection of inputs provided by hand 1103a for moving object 1104a in the three-dimensional environment 1102 and by hand 1105a for moving object 1106a in the three-dimensional environment 1102, device 101 optionally increases the size of objects 1104a and 1106a in the three-dimensional environment 1102, such as... Figure 11B The top view is shown in Figure 9. The increased dimensions of objects 1104a and 1106a optionally correspond to the current distances of objects 1104a and 1106a from the viewpoint of user 1126. Additional details regarding controlling the dimensions of these objects based on their distances from the user's viewpoint are described in a series of figures and method 1000 with reference to Figure 9.

[0252] In some implementations, depending on whether the second object is a valid delivery target for the first object, the device 101 applies different levels of resistance to the movement of the first object along the surface of the second object. For example, in Figure 11BIn one hand state, hand 1103b (e.g., in hand state B) is providing input for upward diagonal movement towards object 1104a, and hand 1105b (e.g., in hand state B) is providing input for upward diagonal movement towards object 1106a, while object 1104a is already in contact with object 1107a. In hand state B (e.g., when the hand is in a pinched hand shape (e.g., when the tips of the thumb and forefinger are touching)), hand 1103b optionally provides input to move object 1104a away from the viewpoint of user 1126 and diagonally into the surface of virtual object 1107a in the 3D environment 1102 (e.g., swipe to the right), and hand 1105b optionally provides input to move object 1106a away from the viewpoint of user 1126 and diagonally into the surface of virtual object 1109a in the 3D environment 1102 (e.g., swipe to the right).

[0253] In some implementations, in response to a given amount of hand movement, device 101 moves the first object by varying amounts in the three-dimensional environment 1102, depending on whether the first object is contacting the surface of a second object and whether the second object is a valid delivery target for the first object. For example, in Figure 11B In this context, the amount of movement of hands 1103b and 1105b is optionally the same. In response, such as... Figure 11C As shown, device 101 moved object 1106a diagonally in the three-dimensional environment 1102, by a greater amount than it would have moved object 1104a laterally and / or away from the viewpoint of user 1126 in the three-dimensional environment 1102. Specifically, in Figure 11C In response to diagonal movement of hand 1103b in three-dimensional environment 1102, which optionally includes a rightward component and a component away from user 1126's viewpoint, device 101 has blocked (e.g., disallowed) movement of object 1104a away from user 1126's viewpoint (e.g., according to the component of hand 1303b moving away from user 1126's viewpoint) because object 1104a is in contact with object 1107a and object 1107a is a valid delivery target for object 1104a, and the component of hand 1303b moving away from user 1126's viewpoint is insufficient to pass through object 1107a, as described later. Furthermore, in Figure 11C In this process, device 101 has moved object 1104a across the surface of object 1107a by a relatively small amount (e.g., less than the lateral movement of object 1106a) (e.g., less than the lateral movement of object 1106a) because object 1104a is in contact with object 1107a and object 1107a is an effective delivery target for object 1104a. Additionally or alternatively, in Figure 11CIn response to the movement of hand 1105b, device 101 diagonally moves virtual object 1106a in the three-dimensional environment 1102, so that virtual object 1106a is displayed by device 101 as being behind virtual objects 1109a and 1107a from the viewpoint of user 1126, as shown. Figure 11C As shown in the top view. Because virtual object 1109a is an invalid delivery target for virtual object 1106a, the movement of virtual object 1106a diagonally across virtual object 1109a is optionally not blocked by device 101, and the lateral movement of object 1106a and the movement of object 1106a away from the viewpoint of user 1126 (e.g., the rightward component of the movement of hand 1105b and the component away from the viewpoint of user 1126, respectively) are greater than the lateral movement of object 1104a and the movement of object 1104a away from the viewpoint of user 1126.

[0254] In some implementations, device 101 requires at least a threshold value of the motion of the corresponding object through the object to allow the corresponding object to pass through the object while the corresponding object is in contact with the surface of the object. For example, in Figure 11C In the present invention, hand 1103c (e.g., in hand state C) provides movement input pointing towards virtual object 1104a for moving virtual object 1104a across the surface of virtual object 1107a from the viewpoint of user 1126. In hand state C (e.g., when the hand is in a pinched hand shape (e.g., when the tips of the thumb and forefinger are touching)), hand 1103c optionally provides input for moving object 1104a away from the viewpoint of user 1126 and into (e.g., vertically into) the surface of virtual object 1107a in the three-dimensional environment 1102. In response to a first portion of the movement input moving virtual object 1104a into / through virtual object 1107a, device 101 optionally prevents movement of virtual object 1104a across virtual object 1107a. When hand 1103c applies a larger amount of motion in the second part of the motion input, causing virtual object 1104a to move through virtual object 1107a, device 101 optionally prevents the movement with an increased resistance level, the increased resistance level optionally being proportional to the increased amount of motion. In some embodiments, once the amount of motion pointing at virtual object 1104a reaches and / or exceeds a corresponding amount threshold (e.g., corresponding to a movement of 0.3, 0.5, 1, 2, 3, 5, 10, 20, 40, or 50 cm), device 101 moves virtual object 1104a through virtual object 1107a, as... Figure 11D As shown.

[0255] exist Figure 11DIn response to detecting that a movement input pointing to virtual object 1104a exceeds a corresponding threshold value, device 101 abandons preventing virtual object 1104a from moving through virtual object 1107a and allows virtual object 1104a to move through virtual object 1107a. Figure 11D In the process, device 101 moves virtual object 1104a to a position behind virtual object 1107a in the 3D environment 1102, such as... Figure 11D As shown in the top view. In some embodiments, after detecting that a movement input pointing to virtual object 1104a exceeds a corresponding magnitude threshold, device 101 provides a visual indication 1116 in the three-dimensional environment 1102 (e.g., at a location on the surface of object 1107a where object 1104a has been traversed), indicating that virtual object 1104a has moved from the viewpoint of user 1126 across the surface of virtual object 1107a. For example, in Figure 11D In this process, device 101 displays ripples 1116 on the surface of virtual object 1107a from the viewpoint of user 1126, indicating that virtual object 1104a has moved through virtual object 1107a according to the movement input.

[0256] In some implementations, device 101 provides a visual indication of the presence of virtual object 1104a behind virtual object 1107a in a three-dimensional environment 1102. For example, in Figure 11D In this embodiment, device 101 alters the appearance of virtual object 1104a and / or virtual object 1107a such that even if virtual object 1104a is behind virtual object 1107a in the 3D environment 1102, the corresponding position of virtual object 1104a can be identified from the viewpoint of user 1126. In some embodiments, the visual indication that virtual object 1104a is behind virtual object 1107a in the 3D environment 1102 is displayed as a faded or ghosted version of object 1104a passing through object 1107a (e.g., overlaid thereon), displayed as the outline of object 1104a passing through object 1107a (e.g., overlaid thereon), etc. In some embodiments, device 101 increases the transparency of virtual object 1107a to provide a visual indication of the presence of virtual object 1104a behind virtual object 1107a in the 3D environment 1102.

[0257] In some implementations, when the corresponding object is behind an object in the three-dimensional environment 1102, lateral movement of the corresponding object or movement of the corresponding object away from the viewpoint of the user 1126 is not prevented by the device 101. For example, in Figure 11EIf hand 1103d provides movement input pointing to virtual object 1104a for moving virtual object 1104a to a new position behind virtual object 1107a in 3D environment 1102 to the right, then device 101 will optionally move object 1104a to that new position without preventing the movement based on the movement input. Furthermore, in some embodiments, device 101 will update the display of visual indications of object 1104a in 3D environment 1102 (e.g., ghosting, outline, etc. of virtual object 1104a) to have different sizes and / or new portions of the surface extending through virtual object 1107a based on the updated distance of object 1104a from user 1126's viewpoint, the new portions corresponding to the new position of object 1104a behind virtual object 1107a as seen from user 1126's viewpoint.

[0258] In some implementations, movement of the virtual object 1104a from behind the virtual object 1107a to a corresponding position in front of the virtual object 1107a (e.g., through the virtual object 1107a) as seen from the user's 1126 viewpoint is not blocked by the device 101. Figure 11D In this context, hand 1103d (e.g., in hand state D) is providing movement input pointing to virtual object 1104a, which is used to move virtual object 1104a from behind virtual object 1107a to a corresponding position in front of virtual object 1107a from the viewpoint of user 1126 along a path passing through virtual object 1107a, as seen by virtual object 1126. Figure 11E As shown. In hand state D (e.g., when the hand is in a pinched hand shape (e.g., when the tips of the thumb and forefinger are touching)), hand 1103d optionally provides input for moving object 1104a closer to the viewpoint of user 1126 and into (e.g., vertically into) the rear surface of virtual object 1107a in three-dimensional environment 1102.

[0259] exist Figure 11E In response to detecting that virtual object 1104a has moved from behind virtual object 1107a to in front of virtual object 1107a, device 101 moves virtual object 1104a in the 3D environment 1102 from the viewpoint of user 1126 through virtual object 1107a to a corresponding position in front of virtual object 1107a, such as... Figure 11E As shown in the top view. Figure 11EIn this embodiment, when virtual object 1104a moves to the corresponding position in the three-dimensional environment 1102, the movement of virtual object 1104a through virtual object 1107a is not prevented by device 101. It should be understood that in some embodiments, subsequent movement of virtual object 1104a (e.g., in response to movement input provided by hand 1103e) in the three-dimensional environment 1102 toward virtual object 1107a (e.g., away from the viewpoint of user 1126) causes device 101 to prevent the movement of virtual object 1104a when virtual object 1104a sees / touches the surface of virtual object 1107a from the viewpoint of user 1126, as previously described.

[0260] In some implementations, the movement of the first virtual object from behind the second virtual object to in front of it is unimpeded, regardless of whether the second virtual object is a valid delivery target for the first virtual object. For example, in Figure 11E If, from the viewpoint of user 1126, virtual object 1106a (e.g., in response to movement input provided by hand 1105c) moves from behind virtual object 1109a (which is optionally an invalid delivery target for virtual object 1106a) to a corresponding position in front of virtual object 1109a, then device 101 will optionally abandon preventing virtual object 1106a from moving through virtual object 1109a to a corresponding position in front of virtual object 1109a in the three-dimensional environment 1102.

[0261] Figures 12A to 12G This is a flowchart illustrating a method 1200 for selectively preventing the movement of objects in a three-dimensional environment according to some embodiments. In some embodiments, method 1200 is implemented in a computer system (e.g., Figure 1 The computer system 101, such as a tablet computer, smartphone, wearable computer, or head-mounted device, performs the operation. This computer system includes a display generation component (e.g., Figure 1 , Figure 3 and Figure 4 The display generating component 120 (e.g., a head-up display, monitor, touchscreen, projector, etc.) and one or more cameras (e.g., cameras pointing downwards at the user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras) or cameras pointing forward from the user's head). In some embodiments, method 1200 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., ...). Figure 1 The control unit 110 in A executes instructions to manage the process. Some operations in method 1200 may be optionally combined, and / or the order of some operations may be optionally changed.

[0262] In some embodiments, method 1200 is performed at an electronic device (e.g., 101) communicating with a display generating component (e.g., 120) and one or more input devices (e.g., 314). For example, a mobile device (e.g., a tablet, smartphone, media player, or wearable device) or a computer. In some embodiments, the display generating component is a display integrated with the electronic device (optionally a touchscreen display), an external display such as a monitor, projector, television, or a hardware component (optionally integrated or external) for projecting a user interface or making the user interface visible to one or more users. In some embodiments, the one or more input devices include the ability to receive user input (e.g., capture user input, detect user input, etc.) and transmit information associated with that user input to the electronic device. Examples of input devices include touchscreens, mice (e.g., external), touchpads (optionally integrated or external), remote control devices (e.g., external), another mobile device (e.g., separate from the electronic device), handheld devices (e.g., external), controllers (e.g., external), cameras, depth sensors, eye-tracking devices, and / or motion sensors (e.g., hand-tracking devices, hand motion sensors), etc. In some embodiments, the electronic device communicates with a hand-tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touchscreens, touchpads)). In some embodiments, the hand-tracking device is a wearable device, such as a smart glove. In some embodiments, the hand-tracking device is a handheld input device, such as a remote control or stylus.

[0263] In some implementations, the electronic device displays (1202a) a three-dimensional environment (e.g., Figure 11A The three-dimensional environment 1102 includes a first object at a first location in the three-dimensional environment (e.g., Figure 11A Virtual objects 1104a and / or 1106a in the three-dimensional environment and a second object at a second position at a first distance from the first object in the three-dimensional environment (e.g., Figure 11AThe virtual objects 1107a and / or 1109a in the context. In some embodiments, the three-dimensional environment is generated, displayed, or otherwise made viewable electronically through a device (e.g., a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment). For example, the first object is a photograph (or a representation of a photograph) that can be projected onto a second object, the second object optionally being a container for accepting and / or displaying the photograph (e.g., the second object is a user interface of a messaging application including a text input field into which the photograph can be projected to be added to a messaging conversation displayed in the second object), and the second object is located at a first distance (e.g., 1, 2, 3, 5, 10, 12, 24, 26, 50, or 100 cm) from the first object (e.g., behind the first object from the user's viewpoint in the three-dimensional environment, and therefore farther from the user's viewpoint than the first object).

[0264] In some implementations, when displaying a three-dimensional environment comprising a first object at a first location and a second object at a second location, the electronic device receives (1202a) a first input via one or more input devices corresponding to a request to move the first object away from the first location in the three-dimensional environment by a second distance (e.g., to a third location in the three-dimensional environment), wherein the second distance is greater than the first distance, such as... Figure 11A In one embodiment, virtual object 1104a is moved by hand 1103a and / or virtual object 1106a is moved by hand 1105a (e.g., when the user's gaze is directed at the first object, the user's index finger and thumb form a pinch gesture, and then the hand moves towards a third position in the three-dimensional environment in the pinched hand shape, wherein the third position is optionally located at a second distance (e.g., 2, 3, 5, 10, 12, 24, 26, or 30 cm) from the first position, and wherein the second distance is optionally greater than the first distance. In some embodiments, during the first input, the user's hand is greater than a threshold distance (e.g., 0.2 cm). (0.5, 1, 2, 3, 5, 10, 12, 24, or 26 cm). In some embodiments, when the user's hand is less than a threshold distance from the first object, the first input is the user's index finger and thumb pinching together, and then the hand moving towards a third position in the three-dimensional environment in the pinched hand shape, regardless of the user's gaze position. In some embodiments, the first input corresponds to the movement of the first object away from the user's viewpoint in the three-dimensional environment. In some embodiments, the first input has one or more of the characteristics of the input described in reference methods 800, 1000, 1400, 1600, and / or 1800.

[0265] In some implementations, in response to receiving a first input (1202c), the system determines that the first input meets one or more criteria, wherein the first set of criteria includes a requirement that the first input corresponds to movement through a second position in the three-dimensional environment, such as movement of virtual object 1104a toward virtual object 1107a, etc. Figure 11A As shown (e.g., a hand movement corresponds to the first object moving far enough and / or moving through a second position in the three-dimensional environment. In some embodiments, if the movement of the first object does not pass through the second position in the three-dimensional environment because, for example, the direction of the movement is not toward the second position, then one or more of the first set of criteria are not met), the electronic device moves the first object away from the first position in the three-dimensional environment (1202d) a first distance (e.g., the movement of the virtual object 1104a away from the user 1126's field of view, such as...) according to the first input. Figure 11B (As shown). For example, a first object moves a first distance from a first position in a three-dimensional environment to a second object at a second position in the three-dimensional environment, and when the first object is moving toward a third position in the three-dimensional environment, the first object collides with the second object at the second position (or remains within a threshold distance of the second object, such as 0.1, 0.2, 0.5, 1, 2, 3, 5, 10, or 20 cm). In some embodiments, ...

Claims

1. A method comprising: At an electronic device in communication with a display generating component and one or more input devices: displaying, via the display generation component, a three-dimensional environment, the three-dimensional environment including a first object at a first location in the three-dimensional environment, wherein the first object has a first size in the three-dimensional environment and occupies a first amount of a field of view from a respective viewpoint; while displaying the three-dimensional environment including the first object at the first location in the three-dimensional environment, receiving, via the one or more input devices, a first input corresponding to a request to move the first object away from the first location in the three-dimensional environment; and In response to receiving the first input: Based on determining that the first input corresponds to a request to move the first object away from the corresponding viewpoint: moving the first object away from the corresponding viewpoint from the first position in the three-dimensional environment to a second position based on the first input, wherein the second position is farther from the corresponding viewpoint than the first position; as well as The first object is scaled so that when the first object is at the second position, the first object has a second size in the three-dimensional environment that is larger than the first size and occupies a second amount of the field of view from the corresponding viewpoint, wherein the second amount is smaller than the first amount.

2. The method according to claim 1, further comprising: Upon receiving the first input and in accordance with determining that the first input corresponds to the request to move the first object away from the corresponding viewpoint, the first object is continuously scaled to increasing sizes as the first object moves farther from the corresponding viewpoint.

3. The method of claim 1 , wherein the first object is an object of a first type, and the three-dimensional environment further includes a second object, the second object being an object of a second type different from the first type, the method further comprising: receiving, via the one or more input devices, a second input corresponding to a request to move the second object away from the third location in the three-dimensional environment while displaying the three-dimensional environment including the second object at a third location in the three-dimensional environment, wherein the second object has a third size in the three-dimensional environment and occupies a third amount of the field of view from the respective viewpoint; and In response to receiving the third input and in accordance with determining that the second input corresponds to a request to move the second object away from the corresponding viewpoint: The second object is moved from the third position in the three-dimensional environment to a fourth position away from the corresponding viewpoint based on the second input without scaling the second object, so that when the second object is at the fourth position, the second object has the third size in the three-dimensional environment and occupies a fourth amount of the field of view from the corresponding viewpoint that is less than the third amount, wherein the fourth position is farther from the corresponding viewpoint than the third position.

4. The method according to claim 3, wherein: The second object is displayed together with a control user interface for controlling one or more operations associated with the second object; When the second object is displayed at the third position, the control user interface is displayed at the third position and has a fourth size in the three-dimensional environment, and when the second object is displayed at the fourth position, the control user interface is displayed at the fourth position and has a fifth size greater than the fourth size in the three-dimensional environment.

5. The method according to claim 1, further comprising: while displaying the three-dimensional environment including the first object at the first position in the three-dimensional environment, the first object having the first size in the three-dimensional environment, detecting a movement of the user's viewpoint from a first viewpoint to a second viewpoint, the movement changing a distance between the viewpoint of the user and the first object, wherein the corresponding viewpoint is the first viewpoint; and In response to detecting the movement of the viewpoint from the first viewpoint to the second viewpoint, updating the display of the three-dimensional environment to be from the second viewpoint without scaling the size of the first object at the first location in the three-dimensional environment.

6. The method of claim 5, wherein the first object is an object of a first type, and the three-dimensional environment further includes a second object, the second object being an object of a second type different from the first type, the method further comprising: detecting a movement of the viewpoint from the first viewpoint to the second viewpoint, the movement changing a distance between the viewpoint of the user and the second object, when displaying the three-dimensional environment including the second object at a third position in the three-dimensional environment, wherein the second object has a third size in the three-dimensional environment and the viewpoint of the user is the first viewpoint; and In response to detecting the movement of the corresponding viewpoint: updating the display of the three-dimensional environment to be from the second viewpoint; and The size of the second object at the third position is scaled to a fourth size in the three-dimensional environment that is different from the third size.

7. The method according to claim 1, further comprising: while displaying the three-dimensional environment including the first object at the first position in the three-dimensional environment, detecting movement of the user's viewpoint from a first viewpoint to a second viewpoint in the three-dimensional environment, the movement changing a distance between the viewpoint and the first object; in response to detecting the movement of the viewpoint, updating a display of the three-dimensional environment to be from the second viewpoint without scaling a size of the first object at the first location in the three-dimensional environment; receiving, via the one or more input devices, a second input corresponding to a request to move the first object away from the first position in the three-dimensional environment to a third position in the three-dimensional environment that is further away from the first position than the second corresponding position, while the first object is displayed at the first position in the three-dimensional environment from the second viewpoint; as well as Upon detecting the second input and before moving the first object away from the first position, scaling the size of the first object to a third size different from the first size based on a distance between the first object and the second viewpoint when a start of the second input is detected.

8. The method of claim 1, wherein the three-dimensional environment further comprises a second object at a third location in the three-dimensional environment, the method further comprising: In response to receiving the first input: displaying the first object at a fourth position in the three-dimensional environment based on determining that the first input corresponds to a request to move the first object to a fourth position in the three-dimensional environment, wherein the first object has a third size in the three-dimensional environment and the fourth position is a first distance from the corresponding viewpoint; as well as Based on determining that the first input satisfies one or more criteria, displaying the first object at the third position in the three-dimensional environment, wherein the first object has a fourth size in the three-dimensional environment that is different from the third size, the one or more criteria including a corresponding criterion that is satisfied when the first input corresponds to a request to move the first object to the third position in the three-dimensional environment, and the third position is separated from the corresponding viewpoint by the first distance. The method of claim 8 , wherein the fourth size of the first object is based on a size of the second object.

10. The method according to claim 9, further comprising: receiving, via the one or more input devices, a second input corresponding to a request to move the first object away from the third location in the three-dimensional environment when the first object is at the third location in the three-dimensional environment and has the fourth size based on the size of the second object; as well as In response to receiving the second input, the first object is displayed at a fifth size, wherein the fifth size is not based on the size of the second object.

11. The method of any one of claims 8 to 10, wherein the corresponding criterion is met when the first input corresponds to a request to move the first object to any position within a volume of the three-dimensional environment that includes the third position.

12. The method according to any one of claims 8 to 11, further comprising: When the first input is received, and based on determining that the first object has been moved to the third position based on the first input and the one or more criteria are satisfied, the appearance of the first object is changed to indicate that the second object is a valid drop target for the first object.

13. The method of any one of claims 8 to 12, wherein the one or more criteria include criteria that are satisfied when the second object is a valid delivery target for the first object and are not satisfied when the second object is not a valid delivery target for the first object, the method further comprising: In response to receiving the first input: Based on determining that the corresponding criteria are met but the first input does not meet the one or more criteria because the second object is not a valid delivery target for the first object, displaying the first object at the fourth position in the three-dimensional environment, wherein the first object has the third size in the three-dimensional environment.

14. The method according to any one of claims 8 to 13, further comprising: In response to receiving the first input: Based on determining that the first input satisfies the one or more criteria, the orientation of the first object relative to the corresponding viewpoint is updated based on the orientation of the second object relative to the corresponding viewpoint.

15. The method according to any one of claims 1 to 14, wherein the three-dimensional environment further comprises a second object at a third position in the three-dimensional environment, the method further comprising: Upon receiving the first input: Based on determining that the first input corresponds to a request to move the first object through the third position and further from the corresponding viewpoint than the third position: moving the first object from the first position to the third position away from the corresponding viewpoint based on the first input, while scaling the first object in the three-dimensional environment based on a distance between the corresponding viewpoint and the first object; as well as After the first object reaches the third position, the first object is kept displayed at the third position without scaling the first object while continuing to receive the first input.

16. The method of any one of claims 1 to 15, wherein scaling the first object is based on determining that the second amount of the field of view from the corresponding viewpoint occupied by the first object of the second size is greater than a threshold amount of the field of view, the method further comprising: receiving, via the one or more input devices, a second input corresponding to a request to move the first object away from the respective viewpoint while the first object is displayed at the respective size in the three-dimensional environment, wherein the first object occupies a first respective amount of the field of view from the respective viewpoint; and In response to receiving the second input: Based on determining that the first corresponding amount of the field of view from the corresponding viewpoint is less than the threshold amount of the field of view, the first object is moved away from the corresponding viewpoint based on the second input without scaling the size of the first object in the three-dimensional environment.

17. The method according to any one of claims 1 to 16, wherein the first input corresponds to the request to move the first object away from the corresponding viewpoint, the method further comprising: In response to receiving a first portion of the first input and before moving the first object away from the corresponding viewpoint: Based on determining that the first size of the first object satisfies one or more criteria, the first object is scaled to have a third size different from the first size, the one or more criteria including a criterion that is satisfied when the first size does not correspond to a current distance between the first object and the corresponding viewpoint, and the third size is based on the current distance between the first object and the corresponding viewpoint.

18. An electronic device comprising: one or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 17.

19. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by one or more processors of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 17.

Images