DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR MAPS - Patent application

Abstract

In some embodiments, the electronic device simultaneously presents a navigation user interface element having a designated distinct physical location and a content element including content corresponding to the distinct physical location. In some embodiments, the electronic device presents navigation from a first physical location to a second physical location with reduced visual prominence in the content element in response to an input corresponding to a request to present content corresponding to a second physical location.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 164,296, filed March 22, 2021, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] It generally relates to a computer system having a display generating component and one or more input devices, including but not limited to an electronic device, that presents a graphical user interface through the display generating component, including a map. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays for computer systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include virtual objects, including digital images, videos, text, icons, and control elements such as buttons and other graphics.

[0004] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback to perform actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulation of virtual objects is complex and error-prone cause significant cognitive burden to users and detract from the experience of the virtual / augmented reality environment. In addition, the methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. Summary of the Invention

[0005] Thus, there is a need for a computer system having improved methods and interfaces for providing a user with a computer-generated experience that makes interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can optionally complement or replace conventional methods of providing an augmented reality experience to a user. Such methods and interfaces reduce the number, extent and / or type of inputs from a user by helping the user understand the connection between the inputs provided and the device response to the inputs, thereby creating a more efficient human-machine interface.

[0006] The above disadvantages and other problems associated with user interfaces of computer systems are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a watch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generation components, the output devices including one or more tactile output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a number of functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on a touch-sensitive surface, movements of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body as captured by cameras and other movement sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, creating spreadsheets, playing games, making phone calls, video conferencing, emailing, instant messaging, training support, digital photography, digital videography, web browsing, playing digital music, taking notes, and / or playing digital videos. Executable instructions to perform those functions are optionally contained in a transitory and / or non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] There is a need for electronic devices having improved methods and interfaces for navigating a user interface. Such methods and interfaces can complement or replace conventional methods for interacting with graphical user interfaces. Such methods and interfaces reduce the number, extent, and / or type of input from a user, producing a more efficient human-machine interface.

[0008] In some embodiments, the electronic device simultaneously presents content corresponding to a first physical location at a first location in a user interface and a navigational user interface element having an indication of the first physical location. In some embodiments, in response to an input corresponding to a request to display content corresponding to a second physical location, the electronic device displays navigation from the first physical location to the second physical location with reduced visual prominence.

[0009] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, especially in view of the drawings, the specification, and the claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and description, and not to define or limit the subject matter of the present invention.

[0010] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout: [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an operating environment for a computer system for providing an XR experience, according to some embodiments. [Diagram 2] FIG. 1 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience, according to some embodiments. [Diagram 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of an XR experience to a user, according to some embodiments. [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments. [Diagram 5] FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments. [Figure 6A] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments. [Figure 6B]1 illustrates an exemplary environment of an electronic device for providing an XR experience, according to some embodiments. [Figure 7A] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 7B] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 7C] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 7D] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 7E] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 7F] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 7G] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 7H] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 7I] 1 illustrates an example of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element including content that corresponds to the individual physical location, according to some embodiments. [Figure 8A] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8B] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8C] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8D] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8E] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8F]1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8G] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8H] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8I] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 8J] 1 is a flowchart illustrating a method for simultaneously presenting a navigation user interface element having a designated distinct physical location and a content element including content that corresponds to the distinct physical location, according to some embodiments. [Figure 9A] 1 is a flowchart illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9B] 1 is a flowchart illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9C]1 is a flowchart illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9D] 1 is a flowchart illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9E] 1 is a flowchart illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9F] 1 is a flowchart illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9G] 1 is a flowchart illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9H] 1 is a flowchart illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present disclosure relates to a user interface that provides a computer-generated reality (XR) experience to a user, according to some embodiments.

[0013] The systems, methods, and GUIs described herein provide improved ways for electronic devices to present content that corresponds to a physical location indicated within a navigation user interface element.

[0014] In some embodiments, the computer system simultaneously displays in the three-dimensional environment the specified first physical location and a navigation user interface element with the first content corresponding to the first physical location. In some embodiments, the navigation user interface element is displayed between the first content and a user's viewpoint in the three-dimensional environment. In some embodiments, in response to an input corresponding to a request to present the content corresponding to the second physical location, the electronic device stops displaying the first content and displays the second content at the location in the three-dimensional environment where the first content was displayed. Presenting the second content at the same location in the three-dimensional environment where the first content was displayed provides an efficient way of viewing the content corresponding to the physical location, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., without the user having to direct attention to different areas of the three-dimensional environment or provide input to continue displaying the content at the same location).

[0015] In some embodiments, the computer system simultaneously displays a navigation user interface element and a content element including content corresponding to a first physical location represented by the navigation user interface element. In some embodiments, the electronic device displays the first content with a first visual salience. In some embodiments, while detecting a user input corresponding to a request to display content corresponding to a second physical location, the electronic device displays, within the content element, navigation from the first physical location to the second physical location with reduced visual salience relative to the first visual salience. After displaying the navigation from the first physical location to the second physical location, the electronic device optionally displays second content having a visual salience greater than the reduced visual salience relative to the first visual salience. Reducing the visual prominence of the content element while detecting user input corresponding to a request to specify a second location provides an efficient way of indicating to the user that specifying the second location will update the content element, which further reduces power usage by allowing the user to use the electronic device more quickly and efficiently, improves the battery life of the electronic device, reduces usage errors that must be corrected by further user input, and reduces the need for the electronic device to fully render content corresponding to the first location and / or intermediate locations between the first and second locations (e.g., while the visual indication of the location corresponding to the content is being moved).

[0016] 1-6 illustrate an exemplary computer system for providing an XR experience to a user. FIGs. 7A-7I show examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. FIGs. 8-9 are flow diagrams of methods for providing navigation according to detection of a user's gaze, according to various embodiments. The user interfaces of FIGs. 7A-7I are used to illustrate the processes of FIGs. 8-9, respectively.

[0017] The processes described below enhance usability of the device and streamline the user-device interface (e.g., by helping the user provide appropriate inputs and reduce user errors when operating / interacting with the device) through a variety of techniques, including providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional controls that are displayed, performing an operation without requiring further user input when a set of conditions are met, improving privacy and / or security, and / or other techniques. These techniques also reduce power usage and improve the device's battery life by allowing the user to use the device more quickly and efficiently.

[0018] Furthermore, in methods described herein in which one or more steps are contingent on one or more conditions being met, it should be understood that the described method can be repeated in multiple iterations such that over the course of the iterations, all of the conditions on which the steps of the method are contingent are met in different iterations of the method. For example, if a method requires performing a first step if a condition is met and performing a second step if the condition is not met, one skilled in the art would understand that the steps recited in the claim are repeated in a particular order until the conditions are met and are no longer met. Thus, a method described with one or more steps that are dependent on one or more conditions being met can be rewritten as a method that is repeated until each condition recited in the method is met. However, this is not required for system or computer readable medium claims in which the system or computer readable medium includes instructions for performing a conditional action based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency is met without explicitly repeating the steps of the method until all conditions on which the steps of the method are contingent are met. Those skilled in the art will also understand that, as with methods having conditional steps, the system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0019] In some embodiments, as shown in Figure 1, an XR experience is provided to a user via an operating environment 100 that includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, tactile sensors, orientation sensors, proximity sensors, temperature sensors, location sensors, motion sensors, speed sensors, etc.), and optionally one or more peripheral devices 195 (e.g., home appliances, wearable devices, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensors 190, and the peripheral devices 195 are integrated with the display generation component 120 (eg, within a head-mounted or handheld device).

[0020] When describing an XR experience, various terms are used to individually refer to several related, but distinct, environments that a user senses and / or can interact with (e.g., using inputs detected by computer system 101 that cause the computer system generating the XR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the XR experience). Below is a subset of these terms:

[0021] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can sense and / or interact with the physical environment directly, such as through sight, touch, hearing, taste, and smell.

[0022] Augmented reality: In contrast, an extended reality (XR) environment refers to a fully or partially mimicked environment that people sense and / or interact with via electronic systems. In XR, a subset of a person's body movements or a representation thereof is tracked, and one or more properties of one or more virtual objects simulated within the XR environment are adjusted accordingly to behave with at least one law of physics. For example, an XR system may detect a person's head rotation and adjust the graphical content and sound field presented to the person accordingly, in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to a property(ies) of a virtual object(s) in the XR environment may be made in response to a representation of a body movement (e.g., a voice command). A person may sense and / or interact with an XR object using any one of these senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create an audio environment with 3D or spatial breadth that provides the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, a person may sense and / or interact with only audio objects.

[0023] Examples of XR include virtual reality and mixed reality.

[0024] Virtual Reality: A virtual reality (VR) environment refers to an emulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a number of virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0025] Mixed reality: A mixed reality (MR) environment refers to a mimetic environment designed to incorporate sensory input from or representations of a physical environment in addition to including computer-generated sensory input (e.g., virtual objects) as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtual continuum, a mixed reality environment is anywhere between but not including a complete physical environment at one end and a virtual reality environment at the other end. In some MR environments, the computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track location and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system may take into account movement so that a virtual tree appears stationary relative to the physical ground.

[0026] Examples of mixed reality include augmented reality and augmented virtuality.

[0027] Augmented reality: An augmented reality (AR) environment refers to a mimicked environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or semi-transparent display, whereby the person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment that are representations of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. The person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, a video of a physical environment shown on an opaque display is referred to as a "pass-through video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a mimicked environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) other than the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be distorted by graphically modifying (e.g., enlarging) a portion thereof, such that the modified portion is a modified version that represents the original captured image, but is non-photorealistic. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

[0028] Augmented Virtual: An augmented virtual (AV) environment refers to a mimicking environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0029] Perspective-locked virtual object: A virtual object is perspective-locked when the computer system displays the virtual object in the same location and / or position within the user's perspective, even as the user's perspective shifts (e.g., changes). In embodiments in which the computer system is a head-mounted device, the user's perspective is locked to the forward-facing orientation of the user's head (e.g., the user's perspective is at least a portion of the user's field of view when the user is looking straight ahead). Thus, the user's perspective remains fixed even as the user's line of sight moves without moving the user's head. In embodiments in which the computer system has a display generating component (e.g., a display screen) that can be repositioned relative to the user's head, the user's perspective is the augmented reality view being presented to the user on the display generating component of the computer system. For example, a perspective-locked virtual object that is displayed in the upper left corner of the user's perspective when the user's perspective is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's perspective when the user's perspective changes to a second orientation (e.g., the user's head is facing west). In other words, the location and / or position at which a viewpoint-locked virtual object is displayed in a user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments in which the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head such that the virtual object is also referred to as a "head-locked virtual object."

[0030] Environment-Locked Virtual Object: A virtual object is environment-locked (or "world-locked") when a computer system displays the virtual object at a location and / or position in a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) a location and / or object in a three-dimensional environment (e.g., a physical environment or a virtual environment). As the user's viewpoint shifts, the location and / or object in the environment relative to the user's viewpoint changes, resulting in the environment-locked virtual object being displayed at a different location and / or position in the user's viewpoint. For example, an environment-locked virtual object locked to a tree directly in front of the user will be displayed at the center of the user's viewpoint. If the user's viewpoint shifts to the right (e.g., the user's head is turned to the right) and the tree becomes more left in the user's viewpoint (e.g., the position of the tree in the user's viewpoint shifts), the environment-locked virtual object locked to the tree will be displayed more left in the user's viewpoint. In other words, the location and / or position at which the environment-locked virtual object is displayed in the user's viewpoint depends on the position and / or orientation of the location and / or object in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference frame (e.g., a coordinate system that is fixed to a fixed location and / or object in the physical environment) to determine a position at which to display an environment-locked virtual object in the user's viewpoint. The environment-locked virtual object can be locked to a stationary portion of the environment (e.g., a floor, wall, table, or other stationary object) or can be locked to a moving portion of the environment (e.g., a vehicle, an animal, a person, or a representation of a part of the user's body that moves independent of the user's viewpoint, such as the user's hand, wrist, arm, leg, etc.), such that the virtual object moves as the viewpoint or the part of the environment moves in order to maintain a fixed relationship between the virtual object and the part of the environment.

[0031] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits a delayed-following behavior that reduces or delays the movement of the environment-locked or viewpoint-locked virtual object relative to the movement of a reference point that the virtual object is following. In some embodiments, when exhibiting a delayed-following behavior, the computer system intentionally delays the movement of the virtual object when it detects movement of a reference point that the virtual object is following (e.g., a part of the environment, the viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 and 300 cm from the viewpoint). For example, when the reference point (e.g., a part of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but at a second speed that is slower than the first speed (e.g., until the reference point stops or slows down its movement, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits a delayed-following behavior, the device ignores small amounts of movement of the reference point (e.g., ignores movement of the reference point that is less than a threshold amount of movement, such as 0-5 degrees movement or 0-50 cm movement). For example, when the reference point (e.g., a viewpoint or portion of the environment to which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position with respect to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked) and when the reference point (e.g., a viewpoint or portion of the environment to which the virtual object is locked) moves by a second amount that is greater than the first amount, the distance between the reference point and the virtual object first increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position with respect to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked) and then decreases as the amount of movement of the reference point increases beyond a threshold (e.g., a “lag following” threshold) as the virtual object is moved by the computer system to maintain a fixed or substantially fixed position with respect to the reference point.In some embodiments, a virtual object maintaining a substantially fixed position relative to a reference point includes the virtual object being displayed within a threshold distance (e.g., 1, 2, 3, 5, 15, 20, 50 cm) of the reference point in one or more dimensions (e.g., above / below, left / right, and / or forward / backward relative to the position of the reference point).

[0032] Hardware: There are many different types of electronic systems that allow a person to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated viewing capabilities, windows with integrated viewing capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or translucent display rather than an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed to the person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination of these. In an embodiment, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology that projects a graphical image onto the person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as holograms or as physical surfaces.In some embodiments, the controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the display generation component 120 (e.g., an HMD, a display, a projector, a touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of one or more of the display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0033] In some embodiments, display generation component 120 is configured to provide an XR experience (e.g., at least a visual component of an XR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component 120.

[0034] According to some embodiments, the display generation component 120 provides an XR experience to the user while the user is virtually and / or physically present within the scene 105.

[0035] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head, on his / her hand, etc.). Thus, the display generation component 120 includes one or more XR displays provided to display the XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present the XR content, where the user holds a device with a display directed to the user's field of view and a camera directed to the scene 105. In some embodiments, the handheld device is optionally disposed in a housing worn on the user's head. In some embodiments, the handheld device is optionally disposed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is an XR chamber, housing, or room configured to present the XR content without the user wearing or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing an interaction with XR content that is triggered based on an interaction that occurs in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD where the interaction occurs in the space in front of the HMD and the XR content response is displayed via the HMD. Similarly, a user interface showing an interaction with CRG content that is triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD where the movement is caused by movement of the HMD relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0036] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will appreciate from this disclosure that various other features have not been shown for purposes of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0037] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communications interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communications buses 204 for interconnecting these and various other components.

[0038] In some embodiments, the one or more communications buses 204 include circuitry that interconnects and controls communications between the system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0039] The memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, the memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located remotely from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220, or the non-transitory computer-readable storage medium of the memory 220, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 230 and an XR experience module 240:

[0040] The operating system 230 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for respective groups of one or more users). To that end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0041] 1, and optionally from one or more of the input devices 125, the output devices 155, the sensors 190, and / or the peripheral devices 195. To that end, in various embodiments, the data acquisition unit 241 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0042] In some embodiments, the tracking unit 242 is configured to map the scene 105 and track the location / position of at least the display generation component 120 relative to the scene 105 of FIG. 1, and optionally the position of one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To that end, in various embodiments, the tracking unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the location / position of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the scene 105 of FIG. 1, relative to the display generation component 120, and / or relative to a coordinate system defined relative to the user's hand. The hand tracking unit 244 is described in more detail below with respect to FIG. 4. In some embodiments, eye tracking unit 243 is configured to track the position and movement of a user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)), or relative to XR content displayed via display generation component 120. Eye tracking unit 243 is described in more detail below with respect to FIG.

[0043] In some embodiments, the coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by the display generation component 120 and, optionally, by one or more of the output devices 155 and / or peripheral devices 195. To that end, in various embodiments, the coordination unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0044] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0045] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0046] Moreover, FIG. 2 is intended more to illustrate the functionality of various features that may be present in certain embodiments, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in FIG. 2 may be implemented within a single module, and various functions of a single functional block may be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, may vary from implementation to implementation, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0047] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, and by way of non-limiting example, in some embodiments the display generation component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communications interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communications buses 304 for interconnecting these and various other components.

[0048] In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communications between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0049] In some embodiments, the one or more XR displays 312 are configured to provide an XR experience to the user. In some embodiments, the one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emissive element display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, the one or more XR displays 312 correspond to a waveguide display, such as diffractive, reflective, polarized, holographic, etc. For example, the display generation component 120 (e.g., HMD) includes a single XR display. In another example, the display generation component 120 includes an XR display for each eye of the user. In some embodiments, the one or more XR displays 312 can present MR or VR content. In some embodiments, the one or more XR displays 312 can present MR or VR content.

[0050] In some embodiments, the one or more image sensors 314 are configured to capture image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to capture image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward (and may be referred to as a scene camera) to capture image data corresponding to a scene as the user would view if the display generating component 120 (e.g., an HMD) were not present. The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0051] The memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, the memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located remotely from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320, or the non-transitory computer-readable storage medium of the memory 320, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and an XR presentation module 340:

[0052] The operating system 330 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR presentation module 340 is configured to present XR content to a user via one or more XR displays 312. To that end, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0053] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0054] In some embodiments, the XR presentation unit 344 is configured to present XR content via one or more XR displays 312. To that end, in various embodiments, the XR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0055] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed) based on the media content data. To that end, in various embodiments, the XR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0056] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally to one or more of the input devices 125, the output devices 155, the sensors 190, and / or the peripheral devices 195. To that end, in various embodiments, the data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0057] Although the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 are shown as being present on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0058] Moreover, FIG. 3 is intended more to illustrate the functionality of various features that may be present in a particular implementation, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in FIG. 3 can be implemented within a single module, and various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0059] 4 is a schematic diagram of an example embodiment of a hand tracking device 140. In some embodiments, the hand tracking device 140 (FIG. 1) is controlled by a hand tracking unit 243 (FIG. 2) to track the location / position of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to the scene 105 of FIG. 1 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hand). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0060] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for distinguishing fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or images of the entire body, and can have either zoom capabilities or dedicated sensors with high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or serves as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0061] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data as well) to the controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives the display generation component 120 accordingly. For example, a user can interact with software running on the controller 110 by moving their hand 406 and changing the posture of their hand.

[0062] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spots of the pattern. This approach is advantageous in that it does not require the user to hold or wear any kind of beacon, sensor, or other marker. It gives the depth coordinate of a point in the scene relative to a predefined reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines a set of orthogonal x, y, z axes such that the depth coordinate of a point in the scene corresponds to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

[0063] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps including the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors to patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D locations of the user's wrist joints and fingertips.

[0064] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. The program can, for example, move and modify an image presented on the display generation component 120 or perform other functions in response to the pose and / or gesture information.

[0065] In some embodiments, the gesture includes an air gesture, which is a gesture that is detected without (or independent of) the user touching an input element that is part of a device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) and is based on detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) that is in the air, including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of a user's finger relative to another finger or part of the user's hand), and / or absolute movement of the user's body part (e.g., a tap gesture that includes movement of a hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture that includes a predetermined speed or amount of rotation of the user's body part).

[0066] In some embodiments, input gestures used in various examples and embodiments described herein include air gestures performed by movement of a user's finger(s) relative to other finger(s) or part(s) of the user's hand to interact with an XR environment (e.g., a virtual or mixed reality environment), according to some embodiments. In some embodiments, an air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and is based on detected movements of a part of the user's body, including movements of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movements of the user's body relative to another part of the user's body (e.g., movements of the user's hand relative to the user's shoulder, movements of the user's other hand relative to one of the user's hands, and / or movements of the user's fingers relative to another finger or part of the user's hand), and / or absolute movements of a part of the user's body (e.g., a tap gesture that includes movements of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture that includes rotation of a part of the user's body a predetermined speed or amount).

[0067] In some embodiments where the input gesture is an air gesture (e.g., in the absence of physical contact with an input device that provides a computer system with information about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., in the case of direct input, as described below). Thus, in implementations that include air gestures, the input gesture is detected attention (e.g., gaze) to a user interface element in combination (e.g., simultaneously) with the movement of the user's finger(s) and / or hand to perform pinch and / or tap input, as described in more detail below.

[0068] In some embodiments, an input gesture directed at a user interface object is performed directly or indirectly with reference to the user interface object. For example, user input is performed directly at a user interface object in response to performing an input gesture with a user's hand at a position corresponding to the position of the user interface object in the three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some embodiments, an input gesture is performed indirectly at a user interface object in response to a user performing an input gesture while the position of the user's hand is not at a position corresponding to the position of the user interface object in the three-dimensional environment while detecting the user's attention (e.g., gaze) to the user interface object. For example, in the case of a direct input gesture, a user can direct the user's input to a user interface object by initiating the gesture at or near a position corresponding to a displayed position of the user interface object (e.g., within a distance of 0.5 cm, 1 cm, 5 cm, or 0-5 cm, measured from an outer edge of an option or a central portion of an option). For indirect input gestures, a user can direct the user's input to a user interface object by paying attention to the user interface object (e.g., by gazing at the user interface object), and while paying attention to the option, the user initiates an input gesture (e.g., in any position detectable by the computer system) (e.g., in a position that does not correspond to the displayed position of the user interface object).

[0069] In some embodiments, input gestures (e.g., air gestures) used in various examples and embodiments described herein include pinch inputs and tap inputs for interacting with a virtual or mixed reality environment, according to some embodiments. For example, pinch inputs and tap inputs, described below, are performed as air gestures.

[0070] In some embodiments, the pinch input is part of an air gesture, including one or more of a pinch gesture, a long pinch gesture, a pinch-and-drag gesture, or a double pinch gesture. For example, the pinch gesture, which is an air gesture, includes moving two or more fingers of a hand to contact each other, i.e., optionally with a short interruption (e.g., within 0-1 seconds) after contacting each other. The long pinch gesture, which is an air gesture, includes moving two or more fingers of a hand to contact each other for at least a threshold amount of time (e.g., at least 1 second) before detecting an interruption in contacting each other. For example, the long pinch gesture includes a user holding a pinch gesture (e.g., when two or more fingers are in contact), and the long pinch gesture continues until an interruption in contact between the two or more fingers is detected. In some embodiments, the double pinch gesture, which is an air gesture, includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected directly (e.g., within a predefined period of time) in succession to each other. For example, a user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaking contact between two or more fingers), and performs a second pinch input within a predetermined period of time (e.g., within 1 second or within 2 seconds) after releasing the first pinch input.

[0071] In some embodiments, a pinch-and-drag gesture that is an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., followed by) a drag input that changes the position of the user's hand from a first position (e.g., a start position of the drag) to a second position (e.g., an end position of the drag). In some embodiments, the user maintains the pinch gesture while performing the drag input and releases the pinch gesture (e.g., spreading two or more fingers apart) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers together to touch each other and moves the same hand to a second position in the air in a drag gesture). In some embodiments, the pinch input is performed by a first hand of the user and the drag input is performed by a second hand of the user (e.g., the second hand of the user moves from a first position to a second position in the air while the user continues the pinch input with the first hand of the user). In some embodiments, an input gesture that is an air gesture includes an input (e.g., a pinch input and / or a tap input) performed using both of a user's hands. For example, an input gesture includes two (e.g., or more) pinch inputs performed in conjunction with one another (e.g., simultaneously or within a predefined period of time). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) performed using a first hand of a user and a second pinch input performed using the other hand (e.g., a second hand of the user's two hands) in conjunction with performing the pinch input using the first hand. In some embodiments, a movement between a user's hands (e.g., to increase and / or decrease the distance or relative orientation between the user's hands).

[0072] In some embodiments, a tap input performed as an air gesture (e.g., directed towards a user interface element) includes a movement of a user's finger(s) towards the user interface element, a movement of a user's hand towards the user interface element, optionally with the user's finger(s) extended towards the user interface element, a downward movement of a user's finger (e.g., mimicking a mouse click action or a tap on a touch screen), or other predefined movement of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture, moving the finger or hand away from the user's viewpoint and / or towards the object that is the target of the tap input followed by an end of the movement. In some embodiments, an end of the movement is detected based on a change in movement characteristics of the finger or hand performing the tap gesture (e.g., an end of the movement away from the user's viewpoint and / or towards the object that is the target of the tap input, a reversal of the finger or hand movement direction, and / or a reversal of the acceleration direction of the finger or hand movement).

[0073] In some embodiments, the user's attention is determined to be directed to a portion of the three-dimensional environment based on detection of a gaze directed to the portion of the three-dimensional environment (optionally without requiring other conditions). In some embodiments, the user's attention is determined to be directed to a portion of the three-dimensional environment based on detection of a gaze directed to the portion of the three-dimensional environment with one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., dwell time) while the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment for the device to determine that the user's attention is directed to the portion of the three-dimensional environment, and / or requiring the gaze to be directed to the portion of the three-dimensional environment, and if one of the additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until one or more additional conditions are met).

[0074] In some embodiments, detection of a ready configuration of a user or a portion of a user is detected by the computer system. Detection of the ready configuration of the hands is used by the computer system as an indication that the user is likely preparing to interact with the computer system using one or more air gesture inputs performed by the hands (e.g., a pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the ready state of a hand is determined based on whether the hand has a predefined hand geometry (e.g., a pre-pinch geometry with the thumb and one or more fingers extended and spaced apart, ready to perform a pinch or grab gesture, or a pre-tap with one or more fingers extended and the palm facing away from the user), whether the hand is in a predefined position relative to the user's viewpoint (e.g., under the user's head, on the user's waist, extended at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular manner (e.g., moved toward an area in front of the user, on the user's waist, under the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of a user interface is responsive to attentional (e.g., gaze) input.

[0075] In some embodiments, the software may be downloaded in electronic form to the controller 110, for example over a network, or may alternatively be provided on a tangible non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in a memory associated with the controller 110. Alternatively or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in FIG. 4 as a separate unit from the image sensor 440, by way of example, some or all of the processing functionality of the controller may be associated with the image sensor 404 by a suitable microprocessor and software, or by a dedicated circuit within the housing of the hand tracking device 402, or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or with any other suitable computerized device, such as a game console or media player. The sensing functions of the image sensor 404 may similarly be integrated into a computer or other computerized device that is controlled by the sensor output.

[0076] FIG. 4 further includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels with respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The brightness of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing gray levels as depth increases. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the sequence of depth maps.

[0077] 4 also illustrates, in accordance with some embodiments, a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406. In FIG. 4, the hand skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand that connects to the wrist, etc.) and optionally the wrist or arm connected to the hand are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0078] FIG. 5 illustrates an exemplary embodiment of the eye tracking device 130 (FIG. 1). In some embodiments, the eye tracking device 130 is controlled by the eye tracking unit 245 (FIG. 2) to track the position and movement of the user's gaze relative to the scene 105 or relative to the XR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, if the display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed in a wearable frame, the head-mounted device includes both a component for generating XR content for viewing by the user and a component for tracking the user's gaze relative to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, if the display generation component is a handheld device or an XR chamber, the eye tracking device 130 is, optionally, a device separate from the handheld device or the XR chamber. In some embodiments, the eye tracking device 130 is a head-mounted device or part of the head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a head-mounted or non-head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally used in combination with a head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally part of a non-head-mounted display generation component.

[0079] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or semi-transparent display that allows the user to view the physical environment directly and display virtual objects on the transparent or semi-transparent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, whereby an individual using the system can observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0080] As shown in FIG. 5, in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye tracking camera may be aimed at the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, may be aimed at a "hot" mirror disposed between the user's eye and a display panel that reflects the IR or NIR light from the eye to the eye tracking camera while allowing visual light to pass through. The eye tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a separate eye tracking camera and lighting source.

[0081] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for the particular operating environment 100, e.g., 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility prior to delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic calibration process or a manual calibration process. The user-specific calibration process may include estimation of the particular user's eye parameters, e.g., pupil position, central vision position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters have been determined for the eye tracking device 130, images captured by the eye tracking camera may be processed using a glint-assisted method to determine the user's current visual axis and viewpoint relative to the display.

[0082] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and a gaze tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be located between the user's eye(s) 592 and the display 510 (e.g., a left or right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) and may be pointed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown in the top of FIG. 5), or may be pointed at the user's eye(s) 592 to receive reflected IR or NIR light from the eye(s) 592 (e.g., as shown in the bottom of FIG. 5).

[0083] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses gaze tracking input 542 from the eye tracking camera 540 for various purposes, e.g., in processing the frames 562 for display. The controller 110 optionally estimates the user's viewpoint on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540, using a glint-assisted method or other suitable method. The viewpoint estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0084] Below, some possible use cases of the user's current gaze direction are described, which are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 can generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller can position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller can display certain virtual content in a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the XR experience and focus in the determined direction. The autofocus mechanism of the external camera can then focus on an object or surface in the environment that the user is currently looking at on the display 510. As another exemplary use case, the eyepiece 520 may be a focusable lens, and the eye tracking information is used by the controller to adjust the focus of the eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. The controller 110 can leverage the eye tracking information to orient and focus the eyepiece 520 so that a nearby object the user is looking at appears at the correct distance.

[0085] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR LED or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR light or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0086] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR ranges, and therefore does not introduce noise into the eye tracking system. Note that the locations and angles of the eye tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0087] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience.

[0088] FIG. 6A illustrates a glint-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is realized by a glint-assisted gaze tracking system (e.g., eye tracking device 130 as shown in FIG. 1 and FIG. 5). The glint-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the glint-assisted gaze tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted gaze tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0089] As shown in FIG. 6A, an eye-tracking camera may capture left and right images of a user's left and right eyes. The captured images are then input into an eye-tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye-tracking system may continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0090] At 610, if the tracking status is yes for the current captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eye.

[0091] At 640, proceeding from element 610, the current frame is analyzed to track pupils and glints based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no at element 660 and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0092] 6A is intended to serve as an example of eye-tracking technology that may be used in certain implementations. As will be recognized by those skilled in the art, other eye-tracking technologies, either currently existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with an XR experience according to various embodiments.

[0093] In some embodiments, the captured portion of the real-world environment 602 is used to provide the user with an XR experience, e.g., a mixed reality environment in which one or more virtual objects are overlaid on a representation of the real-world environment 602.

[0094] FIG. 6B illustrates an exemplary environment of the electronic device 101 for providing an XR experience, according to some embodiments. In FIG. 6B, the real-world environment 602 includes the electronic device 101, a user 608, and a real-world object (e.g., a table 604). As illustrated in FIG. 6B, the electronic device 101 is optionally mounted on a tripod or otherwise secured to the real-world environment 602 such that one or more hands of the user 608 are free (e.g., the user 608 is optionally not holding the device 101 with one or more hands). As described above, the device 101 optionally has one or more groups of sensors located on different sides of the device 101. For example, the device 101 optionally includes a sensor group 612-1 and a sensor group 612-2 located on the "back" and "front" sides of the device 101, respectively (e.g., capable of capturing information from each side of the device 101). As used herein, the front side of the device 101 is the side that faces the user 608 and the back side of the device 101 is the side that faces away from the user 608.

[0095] In some embodiments, sensor group 612-2 includes an eye tracking unit (e.g., eye tracking unit 245 described above with reference to FIG. 2) that includes one or more sensors for tracking the eyes and / or gaze of a user, and the eye tracking unit can "see" the user 608 and track the eye(s) of the user 608 in the manner described above. In some embodiments, the eye tracking unit of device 101 can capture the movement, orientation, and / or gaze of the eyes of user 608 and process the movement, orientation, and / or gaze as input.

[0096] In some embodiments, sensor group 612-1 includes a hand tracking unit (e.g., hand tracking unit 243 described above with reference to FIG. 2) that can track one or more hands of user 608 held on the “back” side of device 101, as shown in FIG. 6B. In some embodiments, a hand tracking unit is optionally included in sensor group 612-2 such that user 608 can additionally or alternatively hold one or more hands on the “front” side of device 101 while device 101 tracks the position of the one or more hands. As described above, the hand tracking unit of device 101 can capture movements, positions, and / or gestures of one or more hands of user 608 and process the movements, positions, and / or gestures as inputs.

[0097] In some embodiments, sensor group 612-1 optionally includes one or more sensors (e.g., image sensor 404 described above with reference to FIG. 4 ) configured to capture images of real-world environment 602, including table 604. As described above, device 101 can capture images of portions (e.g., part or all) of real-world environment 602 and present the captured portions of real-world environment 602 to the user via one or more display generation components of device 101 (e.g., a display of device 101 optionally located on a side of device 101 facing the user, opposite a side of device 101 facing the captured portions of real-world environment 602).

[0098] In some embodiments, the captured portion of the real-world environment 602 is used to provide the user with an XR experience, e.g., a mixed reality environment in which one or more virtual objects are overlaid on a representation of the real-world environment 602.

[0099] Thus, the description herein describes some embodiments of a three-dimensional environment (e.g., an XR environment) that includes representations of real-world objects and representations of virtual objects. For example, the three-dimensional environment optionally includes a representation of a table present in a physical environment that is captured and displayed in the three-dimensional environment (e.g., actively via a camera and display of the computer system, or passively via a transparent or semi-transparent display of the computer system). As described above, the three-dimensional environment is optionally a mixed reality system based on a physical environment where the three-dimensional environment is captured by one or more sensors of the computer system and displayed via a display generation component. As a mixed reality system, the computer system can optionally display portions and / or objects of the physical environment selectively such that individual portions and / or objects of the physical environment appear to exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system can optionally display virtual objects in the three-dimensional environment such that the virtual objects appear to exist in the real world (e.g., the physical environment) by placing the virtual objects at respective locations in the three-dimensional environment that have corresponding locations in the real world. For example, the computer system optionally displays the vase so that it appears as if the real vase were placed on a table in the physical environment, hi some embodiments, distinct locations in the three-dimensional environment have corresponding locations in the physical environment.Thus, when a computer system is described as displaying a virtual object at a distinct location relative to a physical object (e.g., at or near the location of a user's hand, or on or near a physical table, etc.), the computer system displays the virtual object at a particular location in the three-dimensional environment such that the virtual object appears to be at or near the physical object in the physical world (e.g., the virtual object is displayed at a location in the three-dimensional environment that corresponds to the location in the physical environment where the virtual object would be displayed if the virtual object were a real object at that particular location).

[0100] In some embodiments, real-world objects that exist in the physical environment (e.g., and / or that are viewable via a view generation component) that are displayed in the three-dimensional environment can interact with virtual objects that exist only in the three-dimensional environment. For example, the three-dimensional environment can include a table and a vase positioned on the table, where the table is a view (or representation) of the physical table in the physical environment and the vase is a virtual object.

[0101] Similarly, the user can optionally use one or more hands to interact with virtual objects in the three-dimensional environment as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the computer system optionally capture one or more of the user's hands and display a representation of the user's hands in the three-dimensional environment (e.g., in a manner similar to displaying real-world objects in the three-dimensional environment described above), or, in some embodiments, due to the transparency / semi-transparency of the user interface, or the projection of the user interface onto a transparent / semi-transparent surface, or the portion of the display generation component displaying the projection of the user interface to the user's eye or the field of view of the user's eye, the user's hands are visible through the display generation component by the ability to see the physical environment through the user interface. Thus, in some embodiments, the user's hands are displayed at separate locations in the three-dimensional environment and are processed as if they were objects in the three-dimensional environment that can interact with virtual objects in the three-dimensional environment as if they were actual physical objects in the physical environment. In some embodiments, the computer system can update the display of the representation of the user's hands in the three-dimensional environment in conjunction with the movement of the user's hands in the physical environment.

[0102] In some of the embodiments described below, for example, for purposes of determining whether a physical object is directly interacting with a virtual object (e.g., whether a hand is touching, grabbing, holding, etc., or within a threshold distance from a virtual object), the computer system can optionally determine an "effective" distance between a physical object in the physical world and a virtual object in the three-dimensional environment. For example, a hand directly interacting with a virtual object optionally includes one or more of a finger of a hand pressing a virtual button, a user's hand gripping a virtual vase, two fingers of a user's hand pinching / holding together an application's user interface, and any of the other types of interactions described herein. For example, when determining whether and / or how a user is interacting with a virtual object, the computer system optionally determines the distance between the user's hand and the virtual object. In some embodiments, the computer system determines the distance between the user's hand and the virtual object by determining the distance between the location of the hand in the three-dimensional environment and the location of the target virtual object in the three-dimensional environment. For example, one or more hands of a user are positioned at a particular position in the physical world, which the computer system optionally captures and displays at a particular corresponding position in the three-dimensional environment (e.g., the position in the three-dimensional environment at which the hand is displayed, if the hand is a virtual hand rather than a physical hand). The position of the hand in the three-dimensional environment is optionally compared to the position of a target virtual object in the three-dimensional environment to determine a distance between the user's one or more hands and the virtual object. In some embodiments, the computer system optionally determines the distance between the physical object and the virtual object by comparing positions in the physical world (e.g., as opposed to comparing positions in the three-dimensional environment).For example, when determining the distance between one or more hands of a user and a virtual object, the computer system optionally determines a corresponding location in the physical world of the virtual object (e.g., a position at which the virtual object is located in the physical world, if the virtual object is a physical object rather than a virtual object), and then determines the distance between the corresponding physical position and the one or more hands of the user. In some embodiments, the same technique is optionally used to determine the distance between any physical object and any virtual object. Thus, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, the computer system optionally performs any of the techniques described above to map the location of the physical object to the three-dimensional environment and / or to map the location of the virtual object to the physical environment.

[0103] In some embodiments, the same or similar techniques are used to determine where and what a user's gaze is directed at and / or where and what a physical stylus held by the user is directed at. For example, if a user's gaze is directed at a particular position in the physical environment, the computer system optionally determines a corresponding position in the three-dimensional environment (e.g., a virtual position of the gaze), and if a virtual object is located at that corresponding virtual position, the computer system optionally determines that the user's gaze is directed at that virtual object. Similarly, the computer system can optionally determine where the stylus is pointing in the physical environment based on the orientation of the physical stylus. In some embodiments, based on this determination, the computer system determines a corresponding virtual position in the three-dimensional environment that corresponds to the location in the physical environment where the stylus is pointing, and optionally determines that the stylus is pointing to the corresponding virtual position in the three-dimensional environment.

[0104] Similarly, the embodiments described herein may refer to the location of a user (e.g., a user of a computer system) and / or the location of the computer system within a three-dimensional environment. In some embodiments, a user of the computer system is holding, wearing, or otherwise located at or near the computer system. Thus, in some embodiments, the location of the computer system is used as a proxy for the location of the user. In some embodiments, the location of the computer system and / or the user within the physical environment corresponds to a distinct location within the three-dimensional environment. For example, if the user stands at a location facing a distinct portion of the physical environment displayed by the display generation component, the location of the computer system is the location within the physical environment (and its corresponding location within the three-dimensional environment) where the user would see objects within the physical environment in the same position, orientation, and / or size (e.g., absolutely and / or relative to each other) as the objects were displayed by the display generation component of the computer system within the three-dimensional environment. Similarly, if the virtual objects displayed in the three-dimensional environment were physical objects in the physical environment (e.g., the virtual objects are located in the same physical environment location and have the same physical environment size and orientation as in the three-dimensional environment), the location of the computer system and / or user is the position at which the user would see the virtual objects in the physical environment in the same position, orientation, and / or size (e.g., absolutely and / or relative to each other, and to real-world objects) as they were displayed by the display generation component of the computer system in the three-dimensional environment.

[0105] In the present disclosure, various input methods are described with respect to interaction with a computer system. If one example is provided using one input device or input method and another example is provided using another input device or input method, it should be understood that each example may be compatible with and optionally utilize the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. If one example is provided using one output device or output method and another example is provided using another output device or output method, it should be understood that each example may be compatible with and optionally utilize the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. If one example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with and optionally utilize the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples without exhaustively listing all features of the embodiments in the description of each exemplary embodiment. User Interface and Related Processes

[0106] Attention now turns to embodiments of user interfaces ("UI") and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, with a display generating component, one or more input devices, and (optionally) one or more cameras.

[0107] 7A-7I show examples of how an electronic device may simultaneously present navigational user interface elements having designated individual physical locations and content elements including content that corresponds to the individual physical locations, according to some embodiments.

[0108] FIG. 7A illustrates an electronic device 101 displaying a three-dimensional environment via a display generating component 120. It should be understood that in some embodiments, the electronic device 101 utilizes one or more of the techniques described with reference to FIGS. 7A-7I in a two-dimensional environment or user interface without departing from the scope of this disclosure. As described above with reference to FIGS. 1-6, the electronic device 101 optionally includes a display generating component 120 (e.g., a touch screen) and a number of image sensors 314. The image sensor optionally includes one or more of a visible light camera, an infrared camera, a depth sensor, or any other sensor that the electronic device 101 can use to capture one or more images of a user or a portion of a user while the user is interacting with the electronic device 101. In some embodiments, the display generating component 120 is a touch screen that can detect gestures and movements of the user's hands. In some embodiments, the user interfaces described below may also be implemented in a head-mounted display that includes a display generation component that displays the user interface to a user, and sensors that detect the physical environment and / or movement of the user's hands (e.g., external sensors facing outward from the user), and / or the user's line of sight (e.g., internal sensors facing inward toward the user's face).

[0109] 7A , the electronic device 101 presents a three-dimensional environment including navigation user interface elements 704a and content user interface elements 706. The three-dimensional environment further includes representations of real objects in the physical environment of the electronic device 101, such as a representation 702 of a real table, a representation 722' of a floor 722, and a representation 724' of a wall 724. The physical environment of the electronic device 101 further includes a doorway 726.

[0110] As described above, the three-dimensional environment presented by the electronic device 101 includes navigation user interface elements 704a that are displayed between the content user interface elements 706 and a user's viewpoint in the three-dimensional environment. In some embodiments, the user's viewpoint corresponds to a location within the three-dimensional environment at which the three-dimensional environment is presented by the electronic device 101. The navigation user interface elements 704a include representations 710a-c of physical objects (e.g., buildings) and a representation 714 of roads located within the physical area represented by the navigation user interface elements 704a. In some embodiments, the navigation user interface elements 704a further include representations of other physical objects, such as infrastructure, landmarks, terrain, vegetation, etc. In some embodiments, the navigation user interface elements 704a are (at least partial) three-dimensional maps of the physical area represented by the map.

[0111] The navigation user interface element 704a further includes a weather indication 712 (e.g., clouds). The weather indication 712 indicates weather conditions currently being experienced at a physical location corresponding to the location of the indication 712 in the navigation user interface element 704a. For example, if the physical cloud represented by the indication 712 moves, the electronic device 101 updates the position of the indication 712 according to the movement of the physical cloud. As another example, if the weather at the physical location corresponding to the location of the indication 712 in the navigation user interface element 704a becomes rain, the electronic device 101 updates the indication 712 (or more generally the navigation user interface element 704a) to include a representation of rain. As shown in FIG. 7A, the navigation user interface element 704a indicates the topography of the physical region represented by the navigation user interface element 704a (e.g., by displaying a cross-section of such topography (e.g., elevation changes, etc.) at the edges and / or interior regions of the navigation user interface element 704a).

[0112] The navigation user interface element 704a further includes an indication 716 of a particular location corresponding to the content 708a presented in the content user interface element 706, and a field of view indicator 718a indicating a field of view corresponding to the content 708a. For example, the content 708a is an image (or video) captured from a physical location corresponding to the location of the indication 716 in the navigation user interface element 704a having a boundary corresponding to the field of view indicator 718a. As shown in FIG. 7A, the electronic device 101 presents the navigation user interface element 704a such that the navigation user interface element appears to rest on a surface of a representation 702 of a real table in the physical environment of the electronic device 101. In some embodiments, the electronic device 101 displays the navigation user interface elements 704a at a location in the three-dimensional environment between the user's viewpoint and the content user interface elements 706 that are separate from any representation of an actual (e.g., horizontal) surface in the three-dimensional environment (e.g., floating in space in the three-dimensional environment and not connected to a surface in the three-dimensional environment), rather than displaying the navigation user interface elements 704a so that they appear to be resting on a representation of a surface in the physical environment of the electronic device 101.

[0113] As shown in Figure 7A, the electronic device 101 presents the electronic device 101 with selectable options 720a and 720b that, when selected, cause the electronic device 101 to adjust the view of the content in the content user interface element 706 by translating and / or rotating the location at which the content was taken (or the location of the content). In some embodiments, in response to an input for option 720a or 720b, the electronic device 101 updates the content in the content user interface element 706 and updates the position and / or orientation of the indication 716 and view indicator 718a in the navigation user interface element 704a, as described in more detail below with reference to Figures 7C-7D.

[0114] In some embodiments, the navigation user interface element 704a further includes one or more indications of a physical location(s) of one or more other electronic devices in communication with the electronic device 101. For example, the navigation user interface element 704a includes an indication of a second electronic device associated with a user other than the user of the electronic device 101 (e.g., a contact in an address book of the user of the electronic device). In some embodiments, in response to detecting a selection of an indication of the second electronic device, the electronic device 101 presents in the content user interface element 706 an image captured by the second electronic device. For example, the image is a live video feed from a camera in communication with the second electronic device.

[0115] In some embodiments, the navigation user interface element 704a includes a two-dimensional portion in addition to a three-dimensional portion similar to the navigation user interface element 704a shown in FIG. 7A. In some embodiments, the two-dimensional portion 704a of the navigation user interface element is displayed behind the three-dimensional portion of the navigation user interface element from the user's perspective. For example, the navigation user interface element 704a is folded and / or curved upward at the rear of the navigation user interface element 704a (e.g., the portion of the navigation user interface element 704a closer to the content user interface element 706), such that the two-dimensional portion of the navigation user interface element 704a is displayed at the location in the three-dimensional environment where the content user interface element 706 is displayed in FIG. 7A. In some embodiments, the physical location represented by the three-dimensional portion of the navigation user interface element 704a and the physical location represented by the two-dimensional portion of the navigation user interface element 704a are adjacent to one another in the real world. In some embodiments, in response to a user input to pan the navigation user interface element 704a, the electronic device pans the navigation user interface element 704a in accordance with the user input, including moving one or more representations from a three-dimensional portion of the navigation user interface element 704a to a two-dimensional portion of the navigation user interface element 704a and vice versa as appropriate.

[0116] As described above, the content user interface element 706 includes content 708a that corresponds to a physical location represented by an indication 716 in the navigation user interface element 704a. In some embodiments, the content 708a is an image captured at the location represented by the indication 716, or additionally / alternatively, is an image of the location indicated by the indication 716. In some embodiments, the edges of the content user interface element 706 are feathered relative to the remainder of the three-dimensional environment. In some embodiments, the boundaries of the content user interface element 706 are blurred or blended into the remainder of the three-dimensional environment surrounding the content user interface element 706.

[0117] In FIG. 7B, the electronic device 101 detects a user input corresponding to a request to update the content user interface element 706 to present content corresponding to a different physical location than the physical location specified in FIG. 7A. Detecting the input optionally includes simultaneously detecting a user's gaze 701d directed toward the navigation user interface element 704b (e.g., the indication 716 therein) and detecting a predefined gesture performed by the user's hand 728a. In some embodiments, the predefined gesture is a pinch gesture in which the user moves the thumb of the hand 728a to another finger of the hand. In some embodiments, detecting the pinch gesture causes the electronic device 101 to "pick up" the indication 716, such that the user can move the indication 716 according to the movement of the hand 728a while the user continues to touch his or her thumb to the other finger of the hand 728a. In some embodiments, in response to detecting that the user has removed their thumb from the other fingers of the hand 728a, the electronic device 101 “drops” the indication 716 at the location where the indication 716 was displayed when the electronic device 101 detected that the user had removed their thumb from the fingers.

[0118] While detecting the input shown in FIG. 7B, the electronic device 101 visually de-emphasizes the content user interface element 706 (e.g., including the content 708b within the content user interface element 706) relative to the amount of visual emphasis the content user interface element 706 was displayed with prior to detecting the input (e.g., in FIG. 7A). In some embodiments, visually de-emphasizing the content user interface element 706 includes blurring, fading, darkening, and / or increasing the translucency of the content user interface element 706. In some embodiments, the entire content user interface element 706 is de-emphasized. In some embodiments, only the edges of the content user interface element 706 are de-emphasized, and the portion of the content user interface element 706 that is at the center of the content user interface element 706 is not visually de-emphasized. In some embodiments, while the input is being detected, the electronic device 101 displays one or more images corresponding to intermediate locations of the indication 716 along with visual de-emphasis within the content user interface element 706.

[0119] In some embodiments, while the input of FIG. 7B is detected, the electronic device 101 modifies the appearance of representations (e.g., representation 710c, etc.) of physical objects (e.g., buildings, infrastructure, plants, other physical objects in the area represented by the navigation user interface elements 704b) in the navigation user interface elements 704b. Modifying the appearance of the representations of the physical objects in the navigation user interface elements 704b optionally includes one or more of fading the three-dimensional representations and / or reducing the height of the representations or flattening the representations. In some embodiments, the electronic device 101 blurs, fades, flattens, or stops displaying one or more representations at the location of the user's hand 728a (or within a threshold distance, such as 1, 2, 5, 10, 20 inches). For example, in Figure 7B, the electronic device 101 stops displaying the representations 710a and 710b shown in Figure 7A because they were displayed in the location of Figure 7A where hand 728a is located in Figure 7B. In some embodiments, the electronic device 101 pans the navigation user interface element 704b when the user provides input and stops displaying one or more representations of physical objects intersected by the boundaries of the navigation user interface element 704b (e.g., representations that do not "fit" within the boundaries of the navigation user interface element 704b).

[0120] In response to the input shown in Figure 7B, the electronic device 101 updates the three-dimensional environment to include content in the content user interface element 706 that corresponds to the updated location of the indication 716 in the navigation user interface element 704a, as shown in Figure 7C. In Figure 7C, the location of the indication 716 and the field of view indicator 713c are updated in the navigation user interface element 704a according to the input shown in Figure 7B, and the content user interface element 706 is updated to include content 708c that corresponds to the location of the indication 716 and the field of view indicator 718c in the navigation user interface element 704a. The location of the content user interface element 706 and the navigation user interface element 704b in the three-dimensional environment remains the same as shown in Figure 7A before detecting the input in Figure 7B, and after detecting the input in Figure 7B as shown in Figure 7C.

[0121] As shown in Figure 7C, the amount of visual de-emphasis of the content user interface element 706 is reduced relative to the amount of de-emphasis of the content user interface element 706 of Figure 7C while the input is detected. In some embodiments, as shown in Figure 7C, the amount of de-emphasis (or lack thereof) after receiving the input of Figure 7B is the same as the amount of de-emphasis of the content user interface 706 of Figure 7A before detecting the input of Figure 7B (e.g., no visual de-emphasis). In some embodiments, the amount of de-emphasis before and after receiving the input in Figure 7B is different.

[0122] The electronic device 101 also displays the representation in the navigation user interface element 704a in FIG. 7C with less de-emphasis than the amount of de-emphasis with which the representation in the navigation user interface element 704a was displayed in FIG. 7B while the input was detected. For example, if a representation (e.g., of a building or other physical structure or feature) was flattened, blurred, darkened, and / or faded in FIG. 7B, the representation is displayed higher, clearer, and / or brighter in FIG. 7C after the input in FIG. 7B is no longer detected. In some embodiments, the electronic device 101 displays the navigation user interface element 704a with the same degree of visual clarity / prominence in FIG. 7A before receiving the input in FIG. 7B as in FIG. 7C after receiving the input in FIG. 7B. In some embodiments, the degree of visual clarity / prominence of the navigation user interface element 704a in FIG. 7A and FIG. 7C is different.

[0123] In Figure 7C, the electronic device 101 detects an input directed at the selectable option 720b and updates a location corresponding to the content in the content user interface element 706. Detecting the input optionally includes simultaneously detecting a user's gaze 701a directed at the selectable option 720b and detecting that the user has made a pinch gesture with his or her hand 728b (e.g., moving the thumb to touch another finger of the hand and then moving the thumb away from the finger). In response to the input shown in Figure 7C, the electronic device 101 updates the location of the indication 716 and field of view indicator 718c in the navigation user interface element 704a, as shown in Figure 7D, and updates the content user interface element 706 to include content corresponding to the updated position / orientation of the indication 716 and field of view indicator 718c.

[0124] Figure 7D shows the three-dimensional environment updated in response to the input detected in Figure 7C. The electronic device 101 displays the indication 716 and the field of view indicator 718d at an updated position in the navigation user interface element 704a according to the input detected in Figure 7C. Because the input in Figure 7C to the option 720b to pan the content right in the content user interface element 706 was detected, the electronic device 101 moves the indication 716 and the field of view 718d in the navigation user interface element 704a to the right. The electronic device 101 also updates the content user interface element 706 to include content 708d that corresponds to the updated position of the indication 716 and the field of view indicator 718d.

[0125] In some embodiments, the electronic device 101 increases the size of the content user interface element 706 as shown in Figure 7E in response to detecting a user's gaze 701b directed at the content user interface element 706 for at least a predetermined time threshold (e.g., 0.5, 1, 2, 3, 5, 10 seconds, etc.) as shown in Figure 7D. Additionally or alternatively, in some embodiments, the electronic device 101 increases the size of the content user interface element 706 as shown in Figure 7E in response to a user 728c operating a physical button 703a on the electronic device 101 as shown in Figure 7D. In some embodiments, the physical buttons 703a and 703b on the electronic device 101 control the immersion of the virtual content in the three-dimensional environment. For example, while displaying a three-dimensional environment with a relatively high level of immersion, the electronic device increases the visual salience (e.g., size, brightness, clarity, opacity, etc.) of virtual objects in the three-dimensional environment (e.g., content user interface elements 706, navigation user interface elements 704a, content, other virtual objects, such as application user interface elements, etc.) and decreases the visual salience of representations of real objects in the three-dimensional environment (e.g., representation 724' of wall 724, representation 722' of floor 722, and representation 702 of table). As another example, while displaying the three-dimensional environment at a relatively low level of immersion, the electronic device decreases the visual salience (e.g., size, brightness, clarity, opacity, etc.) of virtual objects in the three-dimensional environment (e.g., content user interface elements 706, navigation user interface elements 704a, content, other virtual objects, such as application user interface elements, etc.) and increases the visual salience of representations of real objects in the three-dimensional environment (e.g., representation 724' of wall 724, representation 722' of floor 722, and representation 702 of table).

[0126] FIG. 7E illustrates a three-dimensional environment updated in response to one or more of the inputs shown in FIG. 7D. As shown in FIG. 7E, the size of the content user interface element 706 in FIG. 7E is larger than the size at which the content user interface element 706 was displayed in FIG. 7D prior to receiving the input(s) to enlarge the content user interface element 706. Updating the content user interface element 706 in FIG. 7E includes increasing the amount of content 708d corresponding to the location of the indicator 716 displayed in the content user interface element 706. The electronic device 101 also updates the field of view indicator 718e in the navigation user interface element 704a to reflect the updated width of the physical area shown in the content 708d. In some embodiments, rather than increasing the amount of content displayed (e.g., updating the field of view 718e), the electronic device 101 instead enlarges the content 708d shown in FIG. 7D and maintains the same field of view 718d of FIG. 7D in response to the input(s) of FIG. 7D.

[0127] In some embodiments, the electronic device 101 updates the content user interface element 706 according to the movement of the electronic device 101 and / or the user's head. For example, in response to detecting the movement of the electronic device 101 and / or the user's head, the electronic device 101 shifts the perspective at which the content 708d in the content user interface element 706 is displayed. In some embodiments, the electronic device shifts the content 708d using a parallax effect. In some embodiments, the electronic device moves and / or zooms the content user interface element 706 according to the movement of the electronic device 101. FIG. 7E shows a user 728d moving the electronic device 101 to the left. FIG. 7F shows an example manner in which the electronic device 101 updates the content user interface element 706 according to the movement of the electronic device 101 shown in FIG. 7E.

[0128] In FIG. 7F, the electronic device 101 displays the three-dimensional environment from the user's perspective updated according to the movement of the electronic device 101 in FIG. 7E. As the user 728d moves the electronic device 101 to the left in FIG. 7E, the electronic device 101 expands the content 708d in the content user interface element 706 to the left and updates (e.g., expands) the field of view indicator 718e according to the (e.g., larger) field of view of the content 708d. Expanding the content 708d includes presenting content including a real-world object to the left of the real-world object shown in the content user interface element 706 in FIG. 7E. In some embodiments, the electronic device 101 similarly expands the content user interface element 706 to the right, up, and / or down in response to detecting a right, up, and / or down movement of the electronic device 101 (and / or the user's head), respectively. In some embodiments, the electronic device 101 expands the width of the content user interface element 706 according to the movement of the electronic device 101. For example, in response to detecting movement of the electronic device 101 to the left, the electronic device 101 expands the content user interface element 706 (and therefore the content 708d displayed within the content user interface element 706) to the left in the three-dimensional environment.

[0129] As shown in FIG. 7F, the electronic device 101 stops displaying the portion of the navigation user interface element 704a presented on the portion of the representation 702 of the table that is no longer within the field of view of the display generation component 120 of the electronic device 101. In some embodiments, the electronic device 101 does not update the position of the navigation user interface element 704a in response to zooming in on the content user interface element 706 in response to movement of the electronic device 101 and / or the user's head. In some embodiments, if the entire navigation user interface element 704a is still within the field of view of the display generation component 120 during / after the movement of the electronic device 101 and / or the user's head is detected, the electronic device 101 continues to display the entire navigation user interface element 704a. For example, if the entire representation 702 of the table was still within the field of view of the display generation component 120 of the electronic device 101 in FIG. 7F, the electronic device 101 continues to present the entire navigation user interface element 704a in the same location in the three-dimensional environment where the navigation user interface element 704a was displayed prior to detecting the movement of the electronic device 101 and / or the user's head. In some embodiments, moving the electronic device 101 and / or the user's head updates the position of the user's viewpoint in the three-dimensional environment, and thus displaying the navigation user interface element 704a at a fixed location in the three-dimensional environment includes modifying the portion of the display generation component 120 that presents the map navigation user interface element 704a in some circumstances.

[0130] In some embodiments, pursuant to a determination that the user is looking at a representation of a real object in the three-dimensional environment (e.g., while moving the device 101), the electronic device 101 stops expanding the content user interface element 706 to prevent obscuring the representation of the real object the user is looking at. In FIG. 7F, the electronic device 101 detects a user's line of sight 701c directed at a representation 726' of a doorway in the physical environment of the electronic device 101 and, in response, only expands the content user interface element 706 to the boundary of the doorway and does not expand the content user interface element 706 to obscure the representation 726' of the doorway. In some embodiments, if the electronic device 101 did not detect a user's line of sight 701c directed at the representation 726' of the doorway (e.g., while moving the device 101), the electronic device 101 would have expanded the content user interface element 706 to obscure (at least a portion of) the representation 726' of the doorway.

[0131] In some embodiments, the electronic device 101 facilitates searching for physical locations corresponding to a search query provided by a user. For example, a user may search for a particular location (e.g., an address, a name of a building, a business, a landmark, and other locations) or a type of location (e.g., a type of business (e.g., coffee, gas station, grocery store), a landmark, and other locations). Figure 7G illustrates the electronic device 101 presenting the search results in a three-dimensional environment.

[0132] In FIG. 7G, the electronic device 101 presents a text entry field 730 (or more generally a search field) into which the user has entered the search query "city hall". The electronic device 101 also presents an indication 732a of the names of the search results, the distances of the locations of the search results from the current location of the electronic device, and images 732b of the search results. In addition, the electronic device 101 presents a (e.g., three-dimensional) rendering 708e of a physical object corresponding to the search results (e.g., a three-dimensional representation of the searched city hall building). In some embodiments, the rendering 708e of the physical object corresponding to the search results is presented in front of the image 732b of the physical object corresponding to the search results and / or at a position in the three-dimensional environment closer to the user's viewpoint than the image 732b is at the user's viewpoint. As shown in FIG. 7G, the electronic device 101 presents a navigation user interface element 704b between the user's viewpoint and the rendering 732b. The electronic device 101 presents a navigation user interface element between the user's viewpoint and the image 732b of the physical object corresponding to the search results.

[0133] In FIG. 7G, in response to detecting the search query (e.g., "city hall"), the electronic device 101 updates the navigation user interface element 704b to show the search result(s). In FIG. 7G, the electronic device 101 displays representations of physical locations / objects / features that do not match the search query (e.g., representation 710e) with reduced visual prominence (e.g., at least partially flattening the three-dimensional representations of the physical objects in the navigation user interface element 704b) while maintaining the visual prominence of representations 710d of objects that match the search query (e.g., buildings) and / or displays representations 710d in a color that is different from the color of representations of other objects (e.g., including representation 710e). In some embodiments, the electronic device 101 either updates the color of representations 710d of objects that match the search query or reduces the visual prominence of representations that do not match the search query (e.g., including representation 710e), but not both. In some embodiments, if more than one location matches the search query, the electronic device 101 displays the representations of the locations that match the search query with a different visual distinction (e.g., color, visual salience) than the visual distinction of the representations of the locations that do not match the search query. In some embodiments, the representation 710d of the physical object corresponding to the search query is a three-dimensional representation of the physical object. For example, the representation 710d is a three-dimensional rendering of the physical object presented from a different viewing angle than the viewing angle of the rendering 708e. In some embodiments, the electronic device 101 presents the rendering 708e and the representation 710d of the physical object in the navigation user interface element 704b from the same viewing angle.

[0134] In some embodiments, the electronic device 101 presents navigation directions from one physical location to another physical location in a three-dimensional environment. In some embodiments, in response to a request to present the navigation directions, the electronic device 101 presents a selectable option that, when selected, causes the electronic device 101 to present an animation of navigating the navigation directions. Figure 7H illustrates the electronic device 101 presenting an animation of navigating the navigation directions (e.g., in response to selection of a selectable option).

[0135] In FIG. 7H, the electronic device 101 presents an indication 734 of navigation direction including a start point and an end point, a route distance, and / or a route duration, a navigation user interface element 704c, and one or more images 708f (e.g., one or more still images, videos, animations) corresponding to the navigation route. The navigation user interface element 704c includes an indication 740 of the navigation route, an indication of an end point 738 of the route, and an indication 736 of a location corresponding to the image 708f currently presented in the content user interface element 706. In some embodiments, the navigation user interface element 704c is three-dimensional. In some embodiments, the navigation user interface element 704c is two-dimensional. In some embodiments, the navigation user interface element 704c is a bird's-eye (e.g., top-down) view. In some embodiments, the navigation user interface element is presented from a perspective view. The navigation user interface element 704c is presented between the user's viewpoint and the image 708f corresponding to the navigation route. Navigation user interface element 704c is presented between the user's viewpoint and the navigation direction indication 734. In some embodiments, the electronic device 101 presents an animation of the indication 736 navigating along the navigation path 740, and as the indication 736 moves along the path 740, the electronic device 101 presents an image 708f or a portion of a video or animation that corresponds to the current location represented by the indication 736. For example, image 708f is a first person video / view of navigating the navigation path that the electronic device 101 presents simultaneously with an animation of the indication 736 traversing the navigation path 740.

[0136] In some embodiments, the electronic device 101 selects the size of the navigation user interface elements based on the size of the electronic device 101's physical environment (e.g., one or more features thereof). For example, in FIGS. 7A-7E, the electronic device 101 displays the navigation user interface elements 704a to fit the area of ​​the representation 702 of the table top in the electronic device 101's physical environment (e.g., because the device 101 displays the navigation user interface elements 704a on the table top). In some embodiments, if the table top is smaller, the navigation user interface elements 704a are displayed at a smaller size, either by including a representation of a smaller physical area or by shrinking a representation of the same size physical area. In some embodiments, if the table top is larger, the navigation user interface elements 704a are displayed at a larger size, either by including a representation of a larger physical area or by enlarging a representation of the same size physical area. In some embodiments, when a navigation user interface element is presented on a different surface / feature within the three-dimensional environment or away from any surface / feature, the electronic device 101 sizes (e.g., and shapes) the navigation user interface element based on the surface / feature and / or area on which the navigation user interface element is displayed.

[0137] In FIG. 7I, the physical environment of the electronic device 101 is smaller than the physical environment of FIGS. 7A-7E. In particular, FIG. 7I shows the electronic device 101 being used in a cubicle 726. In FIG. 7I, the navigation user interface element 704d is displayed in a representation 726′ of the cubicle 726 at a smaller size than the size at which the navigation user interface element 704a was displayed in FIGS. 7A-7E because the cubicle 726 is smaller than the table corresponding to the virtual table 702. As shown in FIG. 7I, the electronic device 101 displays the navigation user interface element 704d between the user's viewpoint and the content user interface element 706 in the three-dimensional environment.

[0138] Additional or alternative details regarding the embodiments shown in FIGS. 7A-7I are provided below in the description of methods 800-900, which are described with reference to FIGS. 8-9 below.

[0139] 8A-8J are flow charts illustrating a method for simultaneously presenting navigation user interface elements having designated respective physical locations and content elements including content corresponding to the respective physical locations, according to some embodiments. In some embodiments, method 800 is performed on a computer system (e.g., computer system 101 of FIG. 1) that includes a display generation component (e.g., display generation component 120 of FIGS. 1, 3, and 4) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera pointing downward in a user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras) or a camera pointing forward from the user's head). In some embodiments, method 800 is governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 800 are optionally combined and / or the order of some operations is optionally changed.

[0140] In some embodiments, the method 800 is performed in an electronic device in communication with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer). In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touch screen display), an external display such as a monitor, projector, television, or a hardware component (optionally built-in or external) for projecting a user interface and making the user interface visible to one or more users, etc. In some embodiments, the one or more input devices include electronic devices or components capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information related to the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a track pad (optionally integrated or external), a touch pad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, an eye tracking device, and / or a motion sensor (e.g., hand tracking device, hand motion sensor), etc. In some embodiments, the electronic device is in communication with a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, track pad). In some embodiments, the hand tracking device is a wearable device such as a smart glove. In some embodiments, the hand tracking device is a handheld input device such as a remote control or a stylus.

[0141] In some embodiments, such as in FIG. 7A, the electronic device (e.g., 101), via the display generation component, displays (802a) a three-dimensional environment including a user interface including a first individual piece of content (e.g., 708a) corresponding to a first view of a first physical location, and the first individual piece of content (e.g., 708a) is displayed (802b) at the first individual location within the three-dimensional environment. In some embodiments, the first individual piece of content is an image taken from the first physical location and / or an image of the first physical location (e.g., a street level view image from the first physical location). In some embodiments, the first individual piece of content is a video recorded (e.g., live) at the first physical location.

[0142] In some embodiments, such as FIG. 7A, the electronic device (e.g., 101), via the display generation component, displays (802a) a three-dimensional environment including a user interface including a navigation user interface element (e.g., 704a) having a first location (e.g., 716) corresponding to a specified first physical location, and the navigation user interface element (e.g., 704a) is displayed in the three-dimensional environment between the first individual location where the first individual content (e.g., 708a) is displayed and a viewpoint of a user (e.g., a user of the electronic device) in the three-dimensional environment (802c). In some embodiments, the three-dimensional environment includes a representation of a virtual object including the user interface and a representation of a real object in the physical environment of the electronic device. The three-dimensional environment is presented, optionally, from a first-person perspective from a viewpoint of a user associated with the electronic device at an individual location (e.g., corresponding to a location of the electronic device) in the three-dimensional environment. In some embodiments, the navigation user interface element includes a three-dimensional topographical map of physical locations with a visual indication (e.g., a pin) at the first location corresponding to the first physical location. For example, the navigation user interface element includes a three-dimensional topographical map of a city including a three-dimensional representation of buildings, streets, and other landmarks with a flag, pin, or other visual indication displayed at a first location corresponding to a city address, landmark, or coordinate. In some embodiments, the navigation user interface element is oriented or floats along a horizontal plane in the three-dimensional environment, and the first individual content is oriented vertically. In some embodiments, the map navigation element is between a user's viewpoint and the first individual content in the three-dimensional environment. In some embodiments, the navigation user interface element and / or the first individual content are displayed in a three-dimensional environment (e.g., a computer-generated reality (XR) environment such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment) that is generated, displayed, or otherwise made viewable by the device.

[0143] In some embodiments, such as in FIG. 7B, while displaying the user interface and navigational user interface elements (e.g., 704b) via the display generation component, the electronic device detects (802d) a user input corresponding to a request to specify a second location, via one or more input devices, that corresponds to the second physical location. In some embodiments, the input is directed to the navigational user interface elements, such as an input to move an indication of the specified location from the first location to the second location on the three-dimensional terrain map. In some embodiments, the input is directed to the first individual piece of content, such as an input to change a viewpoint of the content from the first physical location to the second physical location or to rotate a view of the first individual piece of content.

[0144] In some embodiments, such as FIG. 7C, in response to detecting the user input, the electronic device updates the user interface to include second individual content (e.g., 708c) corresponding to a second view of the second physical location, where the second individual content (e.g., 708c) is displayed at the first individual location within the three-dimensional environment. In some embodiments, in response to detecting the user input, the electronic device further updates the navigation user interface elements to display a visual indication (e.g., a pin) at the second location and to cease displaying the visual indication (e.g., the pin) at the first location. In some embodiments, the second individual content is an image taken from the second physical location. In some embodiments, the second individual content is a video recorded (e.g., live) at the second physical location. In some embodiments, the electronic device maintains the location within the three-dimensional environment where the content and navigation user interface elements are displayed in response to the user input.

[0145] The above-described method of displaying a second individual piece of content at the same location within the three-dimensional environment where the first individual piece of content was displayed provides an efficient way of viewing content corresponding to a physical location, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., without the user having to direct their attention to a different area of ​​the three-dimensional environment or provide input to continue displaying the content at the second individual location within the three-dimensional environment).

[0146] In some embodiments, the view or style of the navigation user interface element changes depending on the level of zoom applied to the navigation user interface element. For example, when the zoom level is below a first threshold, the navigation user interface element is a globe. As another example, when the level of zoom is between the first threshold and a second threshold greater than the first threshold, the navigation user interface element is a topographical map of the area showing natural features (e.g., terrain, vegetation, etc.), and optionally not showing man-made features (e.g., buildings, infrastructure, etc.). As another example, when the level of zoom is greater than the second threshold, the navigation user interface element includes a topographical map including natural and man-made features, and the electronic device displays first-person content from the physical location indicated by the navigation user interface element. In some embodiments, the electronic device refrains from displaying first-person content at the first discrete location within the three-dimensional environment while the zoom level is below the second threshold.

[0147] In some embodiments, such as FIG. 7A, the three-dimensional environment includes a representation (804a) of a (e.g., horizontal) surface (e.g., 702) in the physical environment of the electronic device. In some embodiments, the representation of the surface is a representation (e.g., virtual or video pass-through) of the surface presented by the display generation component. In some embodiments, the representation of the surface is a view of the surface through a transparent portion of the display generation component (e.g., true or actual pass-through). For example, the surface is a table, shelf, counter, or floor in the environment of the electronic device. In some embodiments, such as FIG. 7A, the navigation user interface element (e.g., 704a) is displayed at a location (and orientation) in the three-dimensional environment that corresponds to the representation (e.g., horizontal) surface (e.g., 702). In some embodiments, the electronic device displays the navigation user interface element as if it were located (e.g., resting) on ​​the surface. For example, the electronic device displays the navigation user interface element to appear as if it were located on a table, shelf, counter, or floor in the physical environment of the electronic device between the first distinct location in the three-dimensional environment and the user's viewpoint.

[0148] The above-described method of displaying navigation user interface elements at locations corresponding to a representation of a surface provides an efficient way of integrating virtual objects with representations of current real-world objects in a mixed reality environment without consuming more space than necessary in the mixed reality environment, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to integrate virtual and real objects in the environment).

[0149] In some embodiments, the navigational user interface element is displayed (806a) at a location (and / or orientation) in the three-dimensional environment that does not correspond to a surface in the physical environment of the electronic device, such as, for example, displaying navigational user interface element 704a of FIG. 7A away from the representation 702 of the table. In some embodiments, the surface corresponds to a real object in the physical environment of the electronic device, such as a table, a shelf, a counter, or a floor. In some embodiments, while displaying the three-dimensional environment that includes a representation of the surface, the electronic device displays the navigational user interface element at a location away from the representation of the surface (e.g., floating in space, not attached to a physical or virtual object or surface). In some embodiments, the three-dimensional environment does not include a representation of the surface in the physical environment of the electronic device (e.g., because the three-dimensional environment is a virtual environment that does not include a representation of one or more real objects in the physical environment of the electronic device, because the surface is not within the field of view of the electronic device, because no such surface exists in the physical environment of the electronic device). In some embodiments, the navigation user interface elements include indications of physical locations, virtual tours, and landmarks that, when selected, cause the electronic device to present content associated with the selected indication (e.g., at a first discrete location within the navigation user interface element and / or within the three-dimensional environment).

[0150] The above-described methods of displaying navigation user interface elements at locations in a three-dimensional environment that do not correspond to surfaces in the physical environment of the electronic device provide flexibility in the placement of navigation user interface elements, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to display the navigation user interface elements).

[0151] In some embodiments, such as FIG. 7A , the first individual content (e.g., 708a) is surrounded by (e.g., overlaid on) a representation (e.g., 724′) of the physical environment of the electronic device (e.g., 101), and a boundary between the first individual content (e.g., 708a) and the representation (e.g., 724′) of the physical environment of the electronic device (e.g., 101) includes a gradual visual transition (e.g., feathering, gradual fading, blending) between the first individual content (e.g., 708a) and the representation (e.g., 724′) of the physical environment of the electronic device (808a). In some embodiments, the representation of the physical environment is a representation (e.g., virtual or video pass-through) of the physical environment presented by the display generation component. In some embodiments, the representation of the physical environment is a view of the physical environment through a transparent portion of the display generation component (e.g., true pass-through or actual pass-through). In some embodiments, the first individual content is surrounded by (e.g., overlaid on) a representation of the virtual environment, and a boundary between the first individual content and the representation of the virtual environment includes a gradual visual transition between the first individual content and the representation of the virtual environment. In some embodiments, the gradual visual transition is a feathering boundary, a blending boundary, and / or a blurring boundary between the first individual content and the representation of the physical (e.g., or virtual) environment. In some embodiments, the electronic device displays the first individual content between a user's viewpoint within the three-dimensional environment and a representation of the physical environment that includes a portion of the representation of the physical environment that surrounds the first individual content.

[0152] The above-described method of displaying the first individual content using a gradual visual transition between the first individual content and the representation of the physical environment maintains context for the user, which avoids erroneous inputs or the user having to switch back and forth between different user interfaces, which further reduces power usage and improves battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to view the representation of the physical environment and the first individual content).

[0153] In some embodiments, such as FIG. 7D , a first individual content (e.g., 708c) corresponding to a first view of a first physical location is displayed simultaneously with a representation (e.g., 724′) of the physical environment of the electronic device (e.g., 101) and occupies a first portion of a display area of ​​the display generation component (e.g., 120) (e.g., the display generation component displays the first individual content at a first size) (810a). In some embodiments, the representation of the physical environment is a representation (e.g., virtual or video pass-through) of the physical environment presented by the display generation component. In some embodiments, the representation of the physical environment is a view of the physical environment through a transparent portion of the display generation component (e.g., true pass-through or actual pass-through). In some embodiments, the first individual content obscures, blocks, and / or does not obscure and / or block portions of the physical environment (e.g., representation) that are displayed in front of and appear to surround the first individual content. In some embodiments, such as in FIG. 7D , while simultaneously displaying, via the display generation component (e.g., 120), a first individual piece of content (e.g., 708c) corresponding to a first view of a first physical location occupying a first portion of the display area with the representation of the physical environment (e.g., 724′), the electronic device (e.g., 101) detects (810b) an individual input corresponding to a request to obscure the display of the representation of the physical environment (e.g., 724′) via one or more input devices (e.g., a physical button (e.g., pressing) or dial (e.g., rotating) of the electronic device, or a different input device in communication with the electronic device, selection of one or more user interface elements displayed via the display generation component). In some embodiments, the request to obscure the display of the representation of the physical environment corresponds to a request to reduce a number of representations of real objects in the physical environment of the electronic device presented by the electronic device.In some embodiments, the request to obscure the display of the representation of the physical environment corresponds to a request to cease displaying the representation of the physical environment (e.g., instead display the user interface, the first individual content, and the navigation user interface elements in the virtual environment). In some embodiments, the request to obscure the display of the representation of the physical environment corresponds to a request to increase a portion of a display area of ​​the display generation component occupied by virtual objects and / or content (e.g., the first individual content). In some embodiments, the request to obscure the display of the representation of the physical environment corresponds to a request to dim the physical environment (e.g., the representation of the physical environment). In some embodiments, such as FIG. 7E, in response to detecting the paired individual input, the electronic device (e.g., 101) obscures (810c) the display of the representation of the physical environment (e.g., 724′), including updating (e.g., 708d) the first individual content (e.g., 708d) to correspond to a second view (810d) that is larger than the first view of the first physical location (e.g., increasing the amount of the first individual content displayed by the display generation component). 7E, in response to detecting the discrete input, the electronic device (e.g., 101) obscures (810c) a display of the representation (e.g., 724') of the physical environment, including updating (e.g., increasing the size at which the first discrete content is displayed by the display generating component) the first discrete content to occupy (810e) a second portion of a display area of ​​the display generating component (e.g., 120) that is larger than the first portion. In some embodiments, the display area of ​​the display generating component occupied by the representation of the physical environment and the view of the physical environment is reduced in response to detecting the discrete input.

[0154] The above-described method of obscuring the display of a representation of the physical environment in response to individual inputs provides an efficient way of presenting a larger view of a first physical location, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to view the first physical location).

[0155] In some embodiments, such as FIG. 7E, while displaying the first individual content (e.g., 708d) corresponding to the first view of the first physical location, the electronic device (e.g., 101) detects (812a) via one or more input devices (e.g., accelerometer(s), gyroscope(s)) a movement of a user's viewpoint of the electronic device (e.g., electronic device movement) (e.g., electronic device moves less than 1, 2, 3, 5, 10, 30, 50 centimeters, etc. and / or rotates less than 0.1, 0.5, 1, 2, 3, 4, 5, etc.) in the three-dimensional environment. In some embodiments, the electronic device detects the movement of the electronic device. For example, the electronic device is a wearable device (e.g., head-mounted device, smart watch), and a movement of a predefined part of the user (e.g., user's head, user's wrist) causes the electronic device to move. In some embodiments, as in FIG. 7F, in response to detecting a movement of a viewpoint of a user of the electronic device (e.g., 101) in the three-dimensional environment, in accordance with a determination that the user's gaze (e.g., 701c) was directed toward the first individual piece of content (e.g., 708d) when the movement of the user's viewpoint was detected, the electronic device updates (812b) a display of the first view of the first physical location corresponding to the first individual piece of content with a simulated parallax effect in accordance with the movement of the user's viewpoint. In some embodiments, updating the display of the first view of the first physical location with a parallax effect includes shifting a location of an object(s) in the foreground of the first view of the first physical location by a distance(s) that is greater than a distance(s) by which an object(s) in the background of the first view of the first physical location is shifted. In some embodiments, updating the display of the first view at the first physical location using the parallax effect causes one or more portions of the first individual content that were previously not visible (e.g., because they were blocked by other portions of the first individual content) to become visible.

[0156] The above-described method of updating the first view of the first physical location using the parallax effect provides an intuitive way of updating the first content as the user's viewpoint moves, which accommodates the viewpoint movement and enables the display of additional content that was not displayed before the movement, allowing the user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to update the first individual content as the user's viewpoint moves), further reducing power usage and improving battery life of the electronic device.

[0157] In some embodiments, such as in FIG. 7E, while displaying the first individual content (e.g., 708d) corresponding to the first view of the first physical location, the electronic device (e.g., 101) detects (814a) a movement of the electronic device (e.g., 101) via one or more input devices (e.g., 1, 2, 3, 5, 10, 30, 50 centimeters, etc., and / or rotate at least 0.1, 0.5, 1, 2, 3, 4, 5, etc.) that exceeds a predetermined threshold. In some embodiments, the electronic device is a wearable device (e.g., a head-mounted device, a smart watch), and movement of a predefined part of the user (e.g., the user's head, the user's wrist) causes the electronic device to move. Thus, in some embodiments, while displaying a first individual content corresponding to a first view of a first physical location, the electronic device detects movement of a predetermined part of the user (e.g., head, hand, arm, wrist) (e.g., meeting one or more criteria such as a speed or distance of movement exceeding a threshold (e.g., 1, 2, 3, 5, 10, 30, 50 centimeters, etc.)). In some embodiments, as in FIG. 7F, in response to detecting a movement of the electronic device (e.g., 101) (e.g., in response to detecting a movement of the user or a predefined portion of the electronic device), in accordance with a determination that the movement of the electronic device (e.g., 101) (e.g., in response to detecting a movement of the user or a predefined portion of the electronic device) satisfies one or more criteria (e.g., exceeds a speed threshold, exceeds a distance threshold (e.g., 1, 2, 3, 5, 10, 30, 50 centimeters, etc.), exceeds a rotation threshold (e.g., 0.1, 0.5, 1, 2, 3, 4, 5, etc.)), the electronic device (e.g., 101) displays (814b) additional content (e.g., 708d) (e.g., an extension of the first individual content, additional information about the first physical location, etc.) corresponding to the first physical location, via the display generation component (e.g., 120). In some embodiments, the additional information is displayed at a location corresponding to the movement of the electronic device (e.g., in response to detecting a movement to the left), the electronic device displays the additional information to the left of the first individual content.In some embodiments, the electronic device magnifies a view of the first individual piece of content in a direction in which the electronic device is moved, increasing the field of view of the first individual piece of content, and updating the indication of the field of view accordingly.

[0158] The above-described method of displaying additional information in response to detecting movement of an electronic device provides an efficient way of presenting additional information to a user in response to intuitive input, allowing the user to use the electronic device more quickly and efficiently, thereby further reducing power usage and improving battery life of the electronic device.

[0159] In some embodiments, such as FIG. 7C, while displaying a first individual piece of content (e.g., 708c) corresponding to a first view of a first physical location, the electronic device (e.g., 101) detects (816a) an input via one or more input devices directed to a selectable user interface element (e.g., 720b) ​​displayed in a user interface. In some embodiments, the user interface element is a user interface element for shifting a physical location corresponding to a viewpoint of the individual piece of content. In some embodiments, the user interface element is displayed adjacent to or overlaid on the first individual piece of content. In some embodiments, the user interface element is one of a plurality of arrows. In some embodiments, the arrow is a translation arrow (e.g., for laterally moving a viewpoint of the first individual piece of content) or a rotation arrow (e.g., for rotationally moving a viewpoint of the first individual piece of content). In some embodiments, the electronic device detects the selection of a user interface element by detecting, via the eye tracking device, that the user's gaze is directed at the user interface element while detecting, via the hand tracking device, a predefined pose of the user's hand (e.g., a pinch pose in which the thumb of the hand is within a threshold distance (e.g., 0.1, 0.2, 0.3, 0.5, 1, 2 centimeters, etc.) of another finger of the hand). In some embodiments, in response to detecting an input directed at the selectable user interface element (e.g., 720b) ​​of FIG. 7C, the electronic device displays (816b) a third individual content (e.g., 708d) corresponding to a individual view of the individual physical location, where the third individual content (e.g., 708d) is displayed at a first individual location in the three-dimensional environment, as in FIG. 7D. For example, in response to detecting a selection of an option to shift the viewpoint of the individual content to the left, the electronic device displays a third individual content corresponding to a physical location to the left of the first physical location.As another example, in response to detecting a selection of an option to rotate the viewpoint of the individual content to the left, the electronic device displays a fourth individual content corresponding to the first physical location with the view rotated to the left from the view of the first individual content. In some embodiments, the electronic device maintains the size (e.g., width) of the field of view of the first individual content, but the location of the field of view is updated in response to an input directed at the selectable user interface element.

[0160] The above-described method of displaying a third individual content at the same location within the three-dimensional environment where the first individual content was displayed provides an efficient way of viewing content corresponding to a physical location, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., without the user having to direct their attention to a different area of ​​the three-dimensional environment or provide input to continue displaying content at the third individual location within the three-dimensional environment).

[0161] In some embodiments, such as FIG. 7E, while displaying the first individual content (e.g., 708d) corresponding to the first view of the first physical location, the electronic device (e.g., 101) detects a movement of the electronic device (e.g., 101) relative to the three-dimensional environment (818a). In some embodiments, the electronic device is a wearable device (e.g., a head-mounted device, a smart watch), and a movement of a predefined part of the user (e.g., the user's head, the user's wrist) causes the electronic device to move. In some embodiments, while displaying the first individual content corresponding to the first view of the first physical location, the electronic device detects a movement of a predefined part of the user (e.g., the head, the hand, the arm, the wrist). In some embodiments, such as FIG. 7F, in accordance with a determination that the movement of the electronic device (e.g., 101) relative to the three-dimensional environment satisfies one or more criteria, the electronic device (e.g., 101) displays, via the display generation component, additional content (e.g., an augmentation of the first individual content, additional information about the first physical location, etc.) corresponding to the first physical location (818b). In some embodiments, the one or more criteria are met when movement of the electronic device (e.g., or a predefined portion of the user) positions the electronic device such that the electronic device is not directed toward a distinct real object (e.g., a window, a doorway, a display screen, a person, an animal, an electronic device, an appliance, a vehicle, a building, etc.) in the electronic device's physical environment to which the user may wish to direct their attention, and / or is not directed toward a distinct virtual object (e.g., content, a user interface, etc.) in the three-dimensional environment to which the user wishes to direct their attention. In some embodiments, following a determination that the movement of the electronic device relative to the three-dimensional environment does not satisfy the one or more criteria, the electronic device refrains from displaying additional content corresponding to the first physical location (e.g., while maintaining the display of the first distinct content) (818c), such as by refraining from updating the content user interface element 706 of FIG. 7F. For example, in response to a movement of the electronic device that directs the electronic device toward an empty wall, the electronic device displays the additional content overlaid on a portion of the empty wall.As another example, in response to a movement of the electronic device pointing the electronic device toward an individual real object (e.g., a window, a doorway, a display screen, a person, an animal, an electronic device, an appliance, a vehicle, a building, etc.) to which the user may wish to attend, the electronic device refrains from displaying additional content overlaid on the individual object. In some embodiments, the electronic device expands the view of the first content to obscure parts of the three-dimensional environment other than the individual real object to which the user may wish to attend, and stops when the first individual content reaches a distance of the individual real object (e.g., within a threshold (e.g., 1, 5, 10, 20, 30, 40, 50, 100 centimeters, etc.)). As another example, in response to a movement of the electronic device pointing the electronic device toward a virtual object (e.g., content, a user interface) to which the user may wish to attend, the electronic device refrains from displaying additional content overlaid on the virtual object. Thus, in some embodiments, the user can see the individual real object or the individual virtual object without the individual real object or the individual virtual object being obscured by the additional content. In some embodiments, when the electronic device magnifies the view of the first individual piece of content, the electronic device magnifies the field of view of the first individual piece of content and updates an indication of the field of view accordingly.

[0162] The above-described method of selectively displaying additional content in response to movement of the electronic device relative to the three-dimensional environment provides an efficient way of preventing content from obscuring portions of the three-dimensional environment, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to view objects in the three-dimensional environment other than the additional content, such as an input to stop displaying the additional content in order to view other objects).

[0163] In some embodiments, such as in FIG. 7F, the one or more criteria include a criterion that is met based on a change in orientation of the electronic device (e.g., 101) (e.g., relative to one or more respective objects in the three-dimensional environment) and / or a criterion that is met based on whether the user's gaze (e.g., 701c) is directed toward one or more respective objects in the three-dimensional environment (820a). In some embodiments, the electronic device refrains from displaying the additional content pursuant to a determination that the electronic device is directed toward an individual real object (e.g., a window, a doorway, a display screen, a person, an animal, an electronic device, an appliance, a vehicle, a building, etc.) in the physical environment of the electronic device to which the user wishes to direct their attention and / or an individual virtual object (e.g., content, a user interface, etc.) in the three-dimensional environment to which the user may wish to direct their attention. In some embodiments, the electronic device refrains from displaying the additional content pursuant to a determination that the user is looking at an individual real object (e.g., a window, a doorway, a display screen, a person, an animal, an electronic device, an appliance, a vehicle, a building, etc.) in the electronic device's physical environment to which the user may wish to direct their attention and / or toward an individual virtual object (e.g., content, a user interface, etc.) in the three-dimensional environment to which the user may wish to direct their attention (e.g., as determined by an eye tracking device in communication with the electronic device). In some embodiments, pursuant to a determination that the electronic device is not pointed at an individual real object (e.g., a window, a doorway, a display screen, a person, an animal, an electronic device, an appliance, a vehicle, a building, etc.) in the electronic device's physical environment to which the user may wish to direct their attention and / or toward an individual virtual object (e.g., content, a user interface, etc.) in the three-dimensional environment to which the user may wish to direct their attention, the electronic device displays the additional content pursuant to movement of the electronic device.In some embodiments, pursuant to a determination that the user's gaze is not directed to an individual real object (e.g., a window, a doorway, a display screen, a person, an animal, an electronic device, an appliance, a vehicle, a building, etc.) in the electronic device's physical environment to which the user may wish to direct their attention and / or an individual virtual object (e.g., content, a user interface, etc.) in the three-dimensional environment to which the user may wish to direct their attention, the electronic device displays the additional content in accordance with the movement of the electronic device. Thus, in some embodiments, the electronic device obscures the individual real object or the individual virtual object with the additional content. In some embodiments, the electronic device obscures the individual real or virtual object with the additional content if the user's gaze is not directed to the individual real or virtual object while the movement of the electronic device is detected. In some embodiments, the electronic device does not display the additional content at a location in the three-dimensional environment obscuring the individual real or virtual object if the user's gaze is directed to the real or virtual object while the movement of the electronic device is detected.

[0164] The above-described method of selectively displaying additional content in response to changes in the orientation of the electronic device relative to the three-dimensional environment and / or the user's line of sight provides an efficient way of allowing a user to maintain visibility of content in the three-dimensional environment other than the additional content, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., reducing the number of inputs required to view objects in the three-dimensional environment other than the additional content, such as an input to stop displaying the additional content in order to view other objects).

[0165] In some embodiments, such as FIG. 7D , while displaying a first individual piece of content (e.g., 708d) occupying a first portion of a display area of ​​the display generation component (e.g., 120) (e.g., a first individual portion of a view of a first physical location), the electronic device (e.g., 101) detects (822a) via the eye tracking device that the user's gaze (e.g., 701b) is directed toward the first individual piece of content (e.g., 708d) for an amount of time that exceeds a predetermined time threshold (e.g., 0.05, 0.1, 0.2, 0.3, 0.5, 1, 2, 5, 10 seconds, etc.). In some embodiments, such as FIG. 7E, in response to detecting a user's gaze (e.g., 701b) directed toward the first individual content (e.g., 708d) for an amount of time that exceeds a predetermined time threshold (e.g., 0.05, 0.1, 0.2, 0.3, 0.5, 1, 2, 5, 10 seconds, etc.), the electronic device updates (822b) the first individual content (e.g., 708d) to occupy a second portion of the display area of ​​the display generating component (e.g., 120) that is larger than the first portion (e.g., a second individual portion of the view of the first physical location that is larger (e.g., includes more content, objects, etc.) than the first individual portion of the view of the first physical location). In some embodiments, in response to detecting a user's gaze directed toward the first individual content for an amount of time that exceeds a predetermined time threshold (e.g., 0.05, 0.1, 0.2, 0.3, 0.5, 1, 2 seconds, etc.), the electronic device increases the size of the first individual content. In some embodiments, in response to increasing the size of the first individual piece of content, the electronic device increases the field of view of the first individual piece of content (e.g., without updating the orientation of the field of view) and updates an indication of the field of view (e.g., displayed in a navigation user interface element) accordingly. In some embodiments, increasing the display area of ​​the first individual piece of content does not increase the field of view of the first individual piece of content; rather, in some embodiments, the first individual piece of content is enlarged while the field of view (e.g., size and orientation) remains the same.

[0166] The above-described method of increasing the size of a first individual content in response to detecting a user's gaze directed at the first individual content for a time period exceeding a predetermined threshold provides an efficient method of increasing the view of a first physical location by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs to pan the view of the first physical location to view the first physical location), further reducing power usage and improving battery life of the electronic device.

[0167] In some embodiments, while displaying the first individual content, the electronic device displays within the navigation user interface element an indication of a field of view of a first physical location corresponding to the first individual content (824a). In some embodiments, the indication of field of view is displayed in proximity or association with (e.g., adjacent to or embedded in) a visual indication of the first location included in the navigation user interface element. In some embodiments, the indication of field of view indicates a boundary of the first physical location displayed within the first individual content. In some embodiments, while the first individual content occupies a first portion of a display area of ​​the display generation component, the indication of field of view indicates (824b) the first field of view (e.g., a boundary of the first physical location included in the first individual content while the first individual content occupies a first portion of a display area). In some embodiments, while the first individual content occupies a second portion of the display area of ​​the display generation component, the indication of the field of view indicates a second field of view that is larger than the first field of view (e.g., a boundary of a first physical location included in the first individual content while the first individual content occupies the second portion of the display area) (824c). In some embodiments, the second portion of the display area is larger than the first portion of the display area and / or the second field of view is larger than the first field of view.

[0168] The above-described method of displaying an indication of field of view according to the display area occupied by the first individual content provides an efficient way of indicating the portion of the first physical location represented by the first individual content, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0169] In some embodiments, such as FIG. 7E, the first individual content (e.g., 708d) corresponds to a first field of view (e.g., 718e) having a first orientation (e.g., relative to a first physical location) (826a). In some embodiments, such as FIG. 7E, while displaying the first individual content (e.g., 708d) and displaying an indication of the first field of view (e.g., 718e) within a navigation user interface element (e.g., 704a), the electronic device (e.g., 101) detects an input via one or more input devices corresponding to a request to display a third individual content corresponding to the second field of view having a second orientation (826b). In some embodiments, the third individual content and the first individual content are images captured from the same physical location but with different orientations (e.g., facing different directions while at the same location). In some embodiments, the third individual content and the first individual content are images captured from different physical locations (e.g., different orientations). In some embodiments, the widths of the first field of view and the second field of view are the same. In some embodiments, the widths of the first field of view and the second field of view are different. In some embodiments, the input corresponding to the request to display the third individual content is a request to rotate the viewpoint from the viewpoint of the first individual content to the viewpoint of the third individual content. In some embodiments, the input corresponds to a request to expand the first individual content in one or more (e.g., lateral) directions. In some embodiments, the input corresponds to a request to display the third individual content corresponding to a third physical location (e.g., lateral movement relative to the first location). In some embodiments, such as FIG. 7F, in response to detecting (826c) an input corresponding to a request to display (826c) a third individual content (e.g., 708d) corresponding to the second field of view (e.g., 718e) having a second orientation (e.g., corresponding to the first physical location or corresponding to a third physical location), the electronic device (e.g., 101) displays (826d) the third individual content (e.g., 708d) (e.g., at the first individual location in the three-dimensional environment) via the display generation component (e.g., 120).In some embodiments, such as FIG. 7F, in response to detecting an input corresponding to a request to display a third individual piece of content (e.g., 708d) corresponding to a second field of view (e.g., 718) having a second orientation (826c), the electronic device (e.g., 101) updates (826e) an indication of the first field of view in a navigation user interface element (e.g., 704a) to an indication of the second field of view (e.g., 718e). In some embodiments, the indication of the field of view is displayed in proximity to or in association with (e.g., adjacent to or embedded in) a visual indication of a physical location corresponding to the third individual piece of content included in the navigation user interface element. In some embodiments, the indication of the second field of view indicates a boundary of a physical location displayed within the third individual piece of content.

[0170] The above-described method of displaying an indication of the field of view in accordance with the third individual content provides an efficient way of indicating the portion of the individual physical location represented by the third individual content, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0171] In some embodiments, such as FIG. 7A , in accordance with a determination that the physical environment of the electronic device (e.g., 101) satisfies one or more criteria, including criteria that are satisfied based on an amount of unobstructed space (e.g., 0.5, 1, 2, 3, 4, 5 meters, etc. in one or more directions) around the electronic device (e.g., 101) in the physical environment of the electronic device (e.g., 101), the electronic device (e.g., 101) displays (828a) a navigation user interface element (e.g., 704a) at a first size within the three-dimensional environment. In some embodiments, the electronic device displays the navigation user interface element at a location within the three-dimensional environment that does not overlay or intersect with a location of a physical object within the three-dimensional environment. In some embodiments, the size at which the navigation user interface element is displayed depends on the amount of unobstructed space around the electronic device in the physical environment of the electronic device. In some embodiments, such as in FIG. 7I, following a determination that the physical environment of the electronic device (e.g., 101) does not satisfy one or more criteria, the electronic device (e.g., 101) displays (828b) the navigational user interface element (e.g., 704d) in the three-dimensional environment at a second size that is different (e.g., smaller) than the first size. For example, when the electronic device displays the navigational user interface element in a room with free space above a dining room table (e.g., approximately 2 meters of unobstructed space in multiple directions around the electronic device), the electronic device displays the navigational user interface element at a size larger than the size at which the electronic device displays the navigational user interface element in an office cubicle (e.g., approximately 0.5 to 1 meter of unobstructed space in multiple directions around the electronic device).

[0172] The above-described method of displaying navigation user interface elements at a size that corresponds to the amount of unobstructed space around the electronic device in the electronic device's physical environment provides an efficient way of presenting representations of objects in the user's physical environment simultaneously with the navigation user interface elements, which further reduces power usage and improves battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to switch between the display of navigation user interface elements and representations of objects in the electronic device's physical environment).

[0173] In some embodiments, a navigation element (e.g., 704a in FIG. 7A ) includes a first discrete portion (e.g., including three-dimensional representations of physical objects (e.g., buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, vegetation, natural features) at a physical location indicated by the navigation user interface element) that is displayed in three dimensions in the three dimensional environment, and a second discrete portion (e.g., including two-dimensional representations of physical objects (e.g., buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, vegetation, natural features) at a physical location indicated by the navigation user interface element) that is displayed in two dimensions in the three dimensional environment, where the first discrete portion is closer to a user's viewpoint than the second discrete portion (e.g., within a threshold distance (e.g., 0.5, 1, 2, 3, 4, 5 meters, etc.) (830a). In some embodiments, a navigation element (e.g., 704a in FIG. 7A ) includes a first discrete portion (e.g., including three-dimensional representations of physical objects (e.g., buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, vegetation, natural features) at a physical location indicated by the navigation user interface element) that is displayed in two dimensions in the three dimensional environment, where the first discrete portion is closer to a user's viewpoint than the second discrete portion (e.g., within a threshold distance (e.g., 0.5, 1, 2, 3, 4, 5 meters, etc.) (830a). In some embodiments, a navigation element (e.g., 704a in FIG. 7A ) includes a navigation element (e.g., including a first discrete portion and a second discrete portion and a third discrete portion and a fourth discrete portion and a fourth discrete portion and a fourth discrete portion and a fourth discrete portion and a fourth discrete portion and a fourth discrete portion and a fourth discrete portion and a fourth discrete portion and a fourth In response to an input corresponding to a request to move the first discrete portion of the navigation user interface elements (to within a user threshold distance defining a boundary between the first and second respective portions of the navigation user interface elements), the electronic device displays a representation of the physical object in a three-dimensional manner within the second discrete portion of the navigation user interface elements. In some embodiments, in response to an input corresponding to a request to move the first discrete portion of the navigation user interface elements (to within a user threshold distance defining a boundary between the first and second respective portions of the navigation user interface elements), the electronic device displays a representation of the physical object in a two-dimensional manner within the first discrete portion of the navigation user interface elements. In some embodiments, the first discrete portion of the navigation user interface elements is oriented along a horizontal plane (e.g., a real or virtual surface, a virtual plane) within the three-dimensional environment, and the second discrete portion of the navigation user interface elements is displayed vertically behind the first discrete portion of the navigation user interface elements (e.g., curved upward from the horizontal plane).In some embodiments, the first distinct portion and the second distinct portion of the navigation user interface element are joined by a curved portion of the navigation user interface element.

[0174] The above-described method of displaying a first distinct portion of a navigation user interface element in three dimensions and a second distinct portion of the navigation user interface element in two dimensions provides an efficient way of increasing the area of ​​navigation user interface elements that a user can view simultaneously, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0175] In some embodiments, such as FIG. 7A, while displaying (832a) the first individual content (e.g., 708a), the navigation user interface elements (e.g., 704a) correspond to a first physical region that includes the first physical location (832b). In some embodiments, the navigation user interface elements include representations of other physical locations in the first physical region, including representations of buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, vegetation, natural features, and the like. In some embodiments, such as FIG. 7A, while displaying (832a) the first individual content (e.g., 708a), the navigation user interface elements (e.g., 704a) include an indication of the terrain of the first physical region (832c). In some embodiments, the navigation user interface elements are displayed in three dimensions and include a three-dimensional rendering of the terrain (e.g., of the ground) of the first physical region. For example, if the first physical region includes hills, the navigation user interface elements include a three-dimensional rendering of the hills. In some embodiments, the edges of a navigation user interface element represent a cross-section of the terrain of the physical location that corresponds to the edge of the navigation user interface element.

[0176] The above-described method of displaying an indication of the terrain of a first physical area within a navigation user interface element provides an efficient way of representing not only the objects at the first physical location, but also the terrain of the first physical location, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0177] In some embodiments, such as FIG. 7A , the navigation user interface element (e.g., 704a) includes individual content (e.g., 710a, 710b) displayed in three dimensions (e.g., three-dimensional representations of physical objects (e.g., buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, plants, natural features) in the physical area corresponding to the navigation user interface element (834a). In some embodiments, such as FIG. 7B , the electronic device (e.g., 101) detects movement of a predetermined part (e.g., 728a) of the user (e.g., hand, arm, head, etc.) via one or more input devices (e.g., one or more cameras, distance sensors, hand tracking devices, eye tracking devices) while displaying the navigation user interface element (e.g., 704b) including the individual content displayed in three dimensions. In some embodiments, such as FIG. 7B , in response to detecting (834c) movement of a predefined portion (e.g., 728a) of the user, in accordance with a determination that the predefined portion (e.g., 728a) of the user is at a location corresponding to the individual content displayed in three dimensions within the navigation user interface element (e.g., 704b) (e.g., the location of the predefined portion of the user within the three dimensional environment intersects or is within a threshold distance (e.g., 0.1, 0.2, 0.3, 0.5, 1, 2, 3, 4, 5, 10 centimeters, etc.) of the location of the individual content within the three dimensional environment), the electronic device (e.g., 101) updates (834d) the navigation user interface element (e.g., 704b) to visually de-emphasize (e.g., no longer include) the individual content displayed in three dimensions (e.g., stop displaying or de-emphasize the display (e.g., make it more transparent, less opaque) or display the individual content in two dimensions), as in FIG. 7B . In some embodiments, the electronic device stops displaying three-dimensional content at a location that corresponds to the user's default portion and maintains three-dimensional display of other content of the navigation user interface elements at locations that do not correspond to the location of the user's default portion.In some embodiments, the electronic device presents a representation of a predefined portion of the user. In some embodiments, the representation of the predefined portion of the user is a representation of the predefined portion of the user presented by the display generation component (e.g., a virtual or video pass-through). In some embodiments, the representation of the predefined portion of the user is a view of the predefined portion of the user through a transparent portion of the display generation component (e.g., a true or actual pass-through).

[0178] The above-described method of ceasing three-dimensional display of individual content in response to detection of movement of a predefined portion of the user to a location corresponding to the individual content provides an efficient way of maintaining visibility of a representation of a predefined portion of the user and allows the user to use the electronic device more quickly and efficiently, thereby reducing power usage and improving battery life of the electronic device.

[0179] In some embodiments, the electronic device displays (836a) an indication of the second electronic device (e.g., associated with a user different from the user of the electronic device) in a navigation user interface element displayed at a separate location in the navigation user interface element that corresponds to a separate physical location of the second electronic device, such as displaying an indication of the second electronic device in navigation user interface element 704a of FIG. 7A. In some embodiments, the electronic device receives (e.g., from a server, from the second electronic device) an indication of the location of the second electronic device. For example, the user of the electronic device and the second user of the second electronic device are connected via a service that presents the location of the second electronic device to the electronic device and presents the location of the electronic device to the second electronic device. In some embodiments, while displaying the indication of the second electronic device in the navigation user interface element, the electronic device detects (836b) a selection of the indication of the second electronic device via one or more input devices in a manner similar to detecting selection of option 720b of FIG. 7C. In some embodiments, detecting the selection of the indication includes detecting, via the eye tracking device, that the user's gaze is directed toward the indication of the second electronic device, and detecting, via the hand tracking device, that the user performs a predefined gesture (e.g., a pinch gesture in which the user moves the thumb within a threshold distance (e.g., 0.05, 0.1, 0.5, 1, 2 centimeters, etc.) of another finger of the hand). In some embodiments, detecting the selection of the indication includes detecting, via the hand tracking device, that the user performs a predefined gesture (e.g., a pinch gesture in which the user moves the thumb within a threshold distance (e.g., 0.05, 0.1, 0.5, 1, 2 centimeters, etc.) of another finger of the hand) while a predefined portion of the user (e.g., the hand) is within a threshold distance (e.g., 1, 2, 3, 5, 10, 20 centimeters, etc.) of a location corresponding to the indication of the second electronic device.In some embodiments, in response to detecting a selection of the indication of the second electronic device, the electronic device displays (836c) via the display generation component (e.g., at a first discrete location within the three-dimensional environment) content captured by the second electronic device at the discrete physical location of the second electronic device in a manner similar to how the electronic device presents content 708a corresponding to indication 716 of FIG. 7A. For example, the content is live video captured by the second electronic device (e.g., one or more cameras and / or microphones in communication therewith). In some embodiments, the content is one or more images and / or video content associated with the physical location of the second electronic device (e.g., captured or not captured by the second electronic device).

[0180] The above-described method of presenting content captured by the second electronic device in response to detecting a selection of an indication of the second electronic device displayed at a location within a navigation user interface element that corresponds to the location of the second electronic device provides an efficient way of viewing content corresponding to the location of the second electronic device, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0181] In some embodiments, such as FIG. 7G, the first individual content includes a two-dimensional representation (e.g., 732b) of an object (e.g., a building, a landmark, a road, infrastructure, a geographic feature, a landform, a body of water, a plant, a natural feature) located at (or visible from) the first physical location (838a). In some embodiments, such as FIG. 7G, the electronic device (e.g., 101) simultaneously displays (838b) the first individual content including the two-dimensional representation (e.g., 732b) of the object and a three-dimensional representation (e.g., 708e) of the object (e.g., displayed before the first individual content). In some embodiments, while displaying the two-dimensional and three-dimensional representations of the object, the electronic device maintains display of a navigation user interface element. In some embodiments, the three-dimensional representation of the object is displayed in response to detecting a selection of a user interface element displayed at the first individual location of the three-dimensional environment (e.g., within the first individual content, proximate to the first individual content, overlaid on the first individual content, etc.). In some embodiments, the three-dimensional representation of the object is displayed in response to detecting a selection of the two-dimensional representation of the object. In some embodiments, a three-dimensional representation of the object is displayed between the user's viewpoint and the two-dimensional representation of the object.

[0182] The above-described method of simultaneously displaying two-dimensional and three-dimensional representations of an object provides an efficient way of seeing more information about an object (e.g., different views of the object), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0183] 9A-9H are flow charts illustrating a method for presenting navigation from a first physical location to a second physical location with reduced visual prominence in content elements in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. In some embodiments, the method 900 is performed on a computer system (e.g., computer system 101 of FIG. 1 ) that includes a display generating component (e.g., display generating component 120 of FIGS. 1 , 3, and 4 ) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera that is placed on a user's non-hand and points downward (e.g., color sensors, infrared sensors, and other depth-sensing cameras) or a camera that points forward from the user's head). In some embodiments, the method 900 is governed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0184] In some embodiments, the method 900 is performed in an electronic device (e.g., 101) in communication with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer). In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touch screen display), an external display such as a monitor, projector, television, or a hardware component (optionally built-in or external) for projecting a user interface and making the user interface visible to one or more users, etc. In some embodiments, the one or more input devices include electronic devices or components capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information related to the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a track pad (optionally integrated or external), a touch pad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, an eye tracking device, and / or a motion sensor (e.g., hand tracking device, hand motion sensor), etc. In some embodiments, the electronic device is in communication with a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, track pad). In some embodiments, the hand tracking device is a wearable device such as a smart glove. In some embodiments, the hand tracking device is a handheld input device such as a remote control or a stylus.

[0185] In some embodiments, such as FIG. 7A, the electronic device (e.g., 101), via the display generation component (e.g., 120), displays (902a) a user interface including a navigational user interface element (e.g., 704a) having a first location (902b) (e.g., 716) corresponding to a first physical location. In some embodiments, the user interface includes a three-dimensional environment including representations of virtual objects including the user interface and representations of real objects in the physical environment of the electronic device. The three-dimensional environment is optionally presented from a first-person perspective from the perspective of a user associated with the electronic device at a distinct location (e.g., corresponding to a location of the electronic device) in the three-dimensional environment. In some embodiments, the navigational user interface element includes a three-dimensional topographical map of physical locations having a visual indication (e.g., a pin) at the first location corresponding to the first physical location, as described with reference to method 800. For example, the navigation user interface element includes a three-dimensional topographical map of a city including a three-dimensional representation of buildings, streets, and other landmarks with a flag, pin, or other visual indication displayed at a first location corresponding to a city address, landmark, or coordinate. In some embodiments, the electronic device displays the navigation user interface element having a first location corresponding to a specified first physical location.

[0186] In some embodiments, such as FIG. 7A, the electronic device (e.g., 101), via the display generation component (e.g., 120), displays (902a) a user interface including a content element (e.g., 706) including a first (e.g., 708a) individual piece of content corresponding to a first view of a first physical location (e.g., 716), where the content element (e.g., 706) is displayed (902c) with a first visual prominence. In some embodiments, the first individual piece of content is an image taken from the first physical location and / or an image of the first physical location (e.g., a street level view image from the first physical location). In some embodiments, the first individual piece of content is a video recorded (e.g., live) at the first physical location. In some embodiments, the first individual piece of content has one or more characteristics of the first individual piece of content described with reference to method 800. In some embodiments, the degree of visual salience is an amount of blur applied to the content element and / or the first individual content included in the content element, and displaying the content element with the first degree of visual salience is displaying the content element and / or the first individual content included in the content element with no (or relatively low) blur. In some embodiments, the degree of visual salience is an amount of transparency (e.g., relatively low transparency), contrast (e.g., relatively high contrast), color (e.g., relatively high or bright color), size (e.g., relatively large size), and / or position (e.g., relatively prominent / centered position) within the user interface. In some embodiments, the navigation user interface element and / or the content element and / or the first individual content are displayed in a three-dimensional environment (e.g., a computer-generated reality (XR) environment such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment) generated, displayed, or otherwise made viewable by the device.

[0187] In some embodiments, such as FIG. 7B, while displaying the content element (e.g., 706) having the first visual salience via the display generation component (e.g., 120), the electronic device (e.g., 101) detects (902d) a user input (or a series of multiple user inputs) corresponding to a request to specify a second location corresponding to the second physical location via one or more input devices. In some embodiments, the input is directed to a navigation user interface element, such as an input to move an indication of the specified location from the first location to the second location on a terrain map. In some embodiments, the input is directed to the first individual piece of content, such as an input to change a viewpoint of the content from the first physical location to the second physical location or to rotate a view of the first individual piece of content. In some embodiments, the user input has one or more characteristics of a user input corresponding to a request to specify a second location corresponding to the second physical location described with reference to method 800.

[0188] In some embodiments, such as FIG. 7B , in response to (e.g., while) detecting the user input, the electronic device (e.g., 101) displays (902e) within the content element (e.g., 706) navigation from a first location to a second location, which includes displaying representations of one or more locations between the first location and the second location with reduced visual prominence relative to the first visual prominence. In some embodiments, the degree of visual prominence is one or more of a degree of translucency (e.g., relatively high translucency), contrast (e.g., relatively low contrast), color (e.g., relatively low or dark color), size (e.g., relatively small size), and / or position (e.g., relatively subtle / peripheral position) within the user interface. In some embodiments, while the user input is provided, the electronic device displays the content element and / or content included in the content element with a blurred (e.g., relatively high blur) appearance. In some embodiments, the input is a user moving a visual indication of a discrete location (e.g., a pin) corresponding to a discrete physical location from one location in the navigation user interface element to a different location in the navigation user interface element. For example, the user picks up the visual indication from a first location and places it in a second location in the navigation user interface element, and the electronic device blurs the content element while the user is moving the visual indication. In some embodiments, while detecting the user input (or a sequence of multiple user inputs), the content element includes a first discrete content having reduced visual salience (e.g., a relatively high amount of blur). In some embodiments, while detecting the user input (or a sequence of multiple user inputs), the content element updates to include discrete content corresponding to a current position of the visual indication of the discrete location in the map navigation element that is displayed with reduced visual salience (e.g., a relatively high amount of blur).

[0189] In some embodiments, such as FIG. 7C, the electronic device (e.g., 101), after displaying navigation from the first location to the second location (and in response to detecting user input), displays (902f) a second individual content (e.g., 708c) (e.g., such as that described with reference to method 800) corresponding to a second view of the second physical location (e.g., 716) in the content element (e.g., 706) with visual salience greater than the reduced visual salience relative to the first visual salience. In some embodiments, after the user has provided one or more inputs to designate the second location, the electronic device updates the content element to include the second individual content and updates the navigation user interface to include a visual indication (e.g., a pin) of the designation of the second physical location. In some embodiments, the second individual content is an image taken from the second physical location. In some embodiments, the first individual content is a video recorded (e.g., live) at the second physical location. In some embodiments, the electronic device maintains a location within the three-dimensional environment where content and navigational user interface elements are displayed in response to a user input. In some embodiments, in response to detecting the input and / or after the input is detected, the electronic device displays the second distinct content (and / or other characteristics of visual salience corresponding to the first visual salience, as described above) without blurring. In some embodiments, the visual salience greater than the reduced visual salience is the same as the first visual salience at which the user interface is displayed prior to detecting the input. In some embodiments, the visual salience greater than the reduced visual salience is more visually salient than the first visual salience at which the user interface is displayed prior to detecting the input. In some embodiments, the visual salience greater than the reduced visual salience is less visually salient than the first visual salience at which the user interface is displayed prior to detecting the input.

[0190] The above-described method of updating visual characteristics of a content element while detecting user input corresponding to a request to specify a second location provides an efficient way of indicating to a user that specifying the second location will update the content element, which further reduces power usage by allowing the user to use the electronic device more quickly and efficiently, improves the battery life of the electronic device, reduces usage errors that must be corrected by further user input, and reduces the need for the electronic device to fully render content corresponding to the first location and / or intermediate locations between the first and second locations (e.g., while a visual indication of the location corresponding to the content is moved).

[0191] 7B, displaying the content element (e.g., 706) with reduced visual salience includes displaying the entire content element (e.g., 706) with reduced visual salience (904a) (e.g., blurring, fading, darkening the entire area of ​​the content element, etc.). In some embodiments, displaying the entire content element with reduced visual salience includes displaying the entire individual content included in the content element with reduced visual salience.

[0192] The above-described method of displaying the entire content element with a reduced degree of visual salience provides an efficient way of indicating to a user that specifying a second location will update the content element, which further reduces power usage by allowing the user to use the electronic device more quickly and efficiently, improves the battery life of the electronic device, reduces usage errors that must be corrected by further user input, and reduces the need for the electronic device to fully render content corresponding to the first location and / or intermediate locations between the first and second locations (e.g., while a visual indication of the location corresponding to the content is being moved).

[0193] In some embodiments, displaying the content element with reduced visual salience includes displaying a first region of the content element that (e.g., at least partially) surrounds a second region of the content element (e.g., an edge of the content element) with reduced visual salience (e.g., blurred, faded, darkened, etc.) and displaying the second region of the content element with the first visual salience (906a), such as displaying a central portion of the content user interface element 704b of FIG. 7B without reduced visual salience (e.g., not blurred, faded, darkened). In some embodiments, displaying the first region of the content element with reduced visual salience includes displaying a portion of the individual content included in the first region of the content element with reduced visual salience. In some embodiments, the electronic device content element with edges is blurred, faded, and / or darkened.

[0194] The above-described method of displaying a first region of a content element with a reduced degree of visual salience provides an efficient way of indicating to a user that specifying a second location will update the content element, which further reduces power usage by allowing the user to use the electronic device more quickly and efficiently, improves the battery life of the electronic device, reduces usage errors that must be corrected by further user input, and reduces the need for the electronic device to fully render content corresponding to the first location and / or intermediate locations between the first and second locations (e.g., while a visual indication of the location corresponding to the content is being moved).

[0195] In some embodiments, such as FIG. 7A , the electronic device (e.g., 101) displays (908a) at least a portion of a navigation user interface element (e.g., 704a) in a second visual salience (e.g., in three dimensions, full color) while displaying the content element (e.g., 706) in the first visual salience and before detecting user input corresponding to a request to specify a second location corresponding to the second physical location. In some embodiments, the navigation user interface element is a three-dimensional map of an area that includes the first location. For example, the navigation user interface element includes three-dimensional representations of buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, vegetation, natural features, etc. in the area. In some embodiments, the electronic device displays all (or at least a portion) of the representations in the navigation user interface element in three dimensions, full color, semi-transparently, etc. while displaying (at least a portion of) the navigation user interface element in the second visual salience. In some embodiments, such as FIG. 7B, while detecting the user input and while displaying the content element (e.g., 706) with reduced visual prominence relative to the first visual prominence, the electronic device (e.g., 101) displays (908b) at least a portion of the navigation user interface element (e.g., 704b) with reduced visual prominence relative to the second visual prominence (e.g., two-dimensional, faded / darkened color, highly translucent). In some embodiments, the electronic device updates the navigation user interface element to include one or more two-dimensional and / or darkened, faded, and / or translucent representations of objects in the area represented by the navigation user interface element while detecting the user input. In some embodiments, the electronic device displays the entire navigation user interface element with reduced visual prominence while detecting the user input.In some embodiments, the electronic device displays at least a portion of a navigation user interface element (e.g., an edge, a portion at a location corresponding to a predefined part of the user (e.g., a hand, an arm, a head, a finger)) with reduced visual prominence while detecting user input.

[0196] The above-described method of displaying at least a portion of a navigation user interface element with reduced visual prominence while detecting user input reduces the need for the electronic device to fully render the navigation user interface element while the input is detected, which reduces power usage and improves battery life of the electronic device by reducing the number and / or complexity of operations being performed by the electronic device.

[0197] In some embodiments, such as in FIG. 7C , the electronic device (e.g., 101), after displaying navigation from a first location to a second location, displays navigation user interface elements (e.g., 704a) at greater visual salience than the reduced visual salience for a second visual salience (e.g., at a second visual salience, at a different visual salience than the second visual salience) while displaying content elements (e.g., 706) at greater visual salience than the reduced visual salience for the first visual salience (910a). In some embodiments, after displaying navigation from the first location to the second location, the electronic device displays navigation user interface elements at reduced visual salience for the second visual salience while displaying navigation user interface elements that have one or more representations of three-dimensional physical objects (e.g., buildings, landmarks, roads, infrastructure, geographic features, landforms, bodies of water, plants, natural features) and / or have colors that are more saturated, less translucent, and less blurred relative to the colors of the representations. In some embodiments, while displaying navigation from the first location to the second location, the electronic device displays navigation user interface elements with reduced visual prominence, and after displaying navigation from the first location to the second location, the electronic device increases the visual prominence of the navigation user interface elements.

[0198] The above-described method of increasing the visual prominence with which navigation user interface elements are displayed after displaying navigation from a first location to a second location provides an efficient way of automatically improving the visibility of navigation user interface elements, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to view a navigation user interface element after displaying navigation from a first location to a second location).

[0199] In some embodiments, such as FIG. 7B, detecting a user input corresponding to a request to designate a second location includes detecting (912a) via an eye tracking device that a gaze (e.g., 701d) of a user of an electronic device (e.g., 101) is directed toward a navigation user interface element (e.g., 704b). In some embodiments, the electronic device detects that the user's gaze is directed toward a visual indication of the designation of the first physical location (e.g., displayed at a first location within the navigation user interface element). In some embodiments, the electronic device detects that the user's gaze is directed toward a second location of the navigation user interface element that corresponds to the second physical location. In some embodiments, such as FIG. 7B , detecting user input corresponding to a request to designate a second location includes detecting (912a) via one or more input devices (e.g., hand tracking devices) that a gesture of a predefined part (e.g., 728a) of the user of the electronic device (e.g., 101) (e.g., finger(s), hand, arm, head, etc.) satisfies one or more criteria. In some embodiments, the predefined gesture corresponds to a pinch gesture in which the user moves the thumb of their hand to touch another finger of the hand. In some embodiments, the electronic device detects that the user is looking at a visual indication of the designation at the first location while making a gesture that satisfies one or more criteria for “picking up” the designation, and initiates a process of moving the designation (e.g., following the movement of the predefined part of the user). In some embodiments, the electronic device detects that the user is looking at the second location while making a gesture that satisfies one or more criteria for moving the designation to the second location.

[0200] The above-described method of navigating from a first location to a second location in response to detection of a user's gaze and gestures performed by the user provides an intuitive way of navigating a user interface (e.g., without manipulating a tactile input device), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0201] In some embodiments, such as FIG. 7B , while displaying navigation from a first location to a second location in a content element (e.g., 706), the electronic device (e.g., 101) displays (914a) a corresponding animation of navigation from the first location to the second location in a navigation user interface element (e.g., 704b). In some embodiments, the corresponding animation of navigation from the first location to the second location in the navigation user interface element includes panning and / or zooming the navigation user interface element. In some embodiments, the corresponding animation of navigation from the first location to the second location in the navigation user interface element includes displaying an animation of moving a visual indication of the specified location (e.g., a flag, a pin, etc.) from the first location to the second location in the navigation user interface element.

[0202] The above-described method of displaying an animation of navigation from a first location to a second location in a navigation user interface element provides an efficient way of indicating to a user that specifying a second location will update the user interface, which further reduces power usage by allowing a user to use the electronic device more quickly and efficiently, improving the battery life of the electronic device, and reducing usage errors that must be corrected by further user input.

[0203] In some embodiments, the electronic device displays (916a) a selectable option within the user interface that is selectable to display a preview of a navigation path from a first distinct physical location to a second distinct physical location (e.g., such as the navigation path preview of FIG. 7H). In some embodiments, the selectable option is displayed in response to a request to present a navigation path from the first distinct physical location to the second distinct physical location. In some embodiments, the selectable option is displayed simultaneously with an indication of the navigation path from the first distinct physical location to the second distinct physical location. In some embodiments, such as FIG. 7H, in response to detecting (916b) a selection of the selectable option, the electronic device (e.g., 101) displays (916c) an animation of a view (e.g., 708f) corresponding to navigating the navigation path within the content element (e.g., 706). In some embodiments, the animation of a view corresponding to navigating the navigation path is video content and / or a series of images from the perspective of someone navigating the navigation path. In some embodiments, such as in FIG. 7H, in response to detecting 916b a selection of the selectable option, the electronic device (e.g., 101) displays 916d a corresponding indication (e.g., 736) within a navigation user interface element (e.g., 704c) for navigating a navigation path (e.g., 740). In some embodiments, the corresponding indication displayed within the navigation user interface element includes a visual indication of the navigation path (e.g., a navigation path overlaid on the navigation user interface element) and an indication of a respective physical location that corresponds to the portion of the animation displayed within the content element.In some embodiments, as the animation in the content element progresses, indications of the individual physical locations corresponding to the portions of the animation displayed in the content element move such that the indications in the navigation user interface element move in a manner synchronized with the animation displayed in the content element. In some embodiments, the visual indication of the navigation path is a representation of the navigation path that is displayed with visual characteristics (e.g., color, thickness, etc.) that differ from the visual characteristics of other roads included in the navigation user interface element that are not along the navigation path.

[0204] The above-described method of displaying animations on content elements and displaying corresponding indications on navigation user interface elements provides an efficient way of presenting animations simultaneously with an indication of a particular physical location corresponding to a currently displayed portion of the animation, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to determine the physical location corresponding to a portion of the animation).

[0205] In some embodiments, such as while detecting the input shown in FIG. 7B , the navigation user interface element (e.g., 704b) includes (918a) one or more (e.g., three-dimensional, two-dimensional) representations (e.g., 710c) of respective objects (e.g., buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, plants, natural features) located at respective physical locations corresponding to the navigation user interface element (e.g., 704b). In some embodiments, the navigation user interface element includes three-dimensional representations of physical objects located at respective physical regions corresponding to regions displayed in the navigation user interface element. In some embodiments, while displaying (918b) the navigation user interface element (e.g., 704b in FIG. 7B ), pursuant to a determination that a boundary of the navigation user interface element (e.g., 704b) coincides with a respective one of the one or more representations (e.g., 710c) of the respective objects, the electronic device (e.g., 101) withholds (918c) the respective one of the one or more representations of the respective objects in the navigation user interface element. In some embodiments, while displaying the navigation user interface element, in accordance with a determination that a respective one of the one or more representations of the respective object is within a boundary of the navigation user interface element, the electronic device maintains a display of the respective one of the one or more representations of the respective object within the navigation user interface element. In some embodiments, the electronic device displays a representation of the object that is entirely within a boundary of the navigation user interface element.In some embodiments, in response to panning the navigation user interface element, the electronic device updates the navigation user interface element to stop displaying a representation of an object currently displayed at a location corresponding to a boundary of the navigation user interface element (e.g., a boundary between the navigation user interface element and a representation of the device's physical environment displayed surrounding the navigation user interface element) due to panning of the navigation user interface element. In some embodiments, in response to panning the navigation user interface element, the electronic device updates the navigation user interface element to begin displaying a representation of an object fully displayed at a location within the boundary of the navigation user interface element due to panning of the navigation user interface element. In some embodiments, the electronic device withholds displaying a representation of a first type of object (e.g., a building, a plant) located at a location corresponding to the boundary of the navigation user interface element and displays a portion of a representation of a second type of object (e.g., a road, a geographic feature, a body of water) located at a location corresponding to the boundary of the navigation user interface element.

[0206] The above-described method of falsifying the display of representations of objects that coincide with the boundaries of navigation user interface elements reduces the amount of content displayed by the electronic device via the display generation component, which reduces power usage (e.g., by rendering fewer representations) and improves battery life of the electronic device.

[0207] In some embodiments, such as FIG. 7A, while displaying the navigation user interface element (e.g., 704a), the navigation user interface element (e.g., 704a) includes a first representation (e.g., 710a) of a first (e.g., physical) object located at a first physical location and a second representation (e.g., 710b) of a second (e.g., physical) object located at a second physical location, the first representation (e.g., 710a) of the first object and the second representation (e.g., 710b) of the second object are displayed in three dimensions in the navigation user interface element (e.g., 704a), and the electronic device (e.g., 101) detects (920a), via one or more input devices, an input corresponding to a request to perform a search corresponding to the information represented by the navigation user interface element (e.g., 704a) based on a respective search query. In some embodiments, the navigation user interface element includes a three-dimensional representation of physical objects (e.g., buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, plants, natural features) located within the respective physical area. In some embodiments, the distinct search queries correspond to distinct physical locations, distinct landmarks, distinct addresses, etc. In some embodiments, in response to detecting 920b an input corresponding to a request to search a navigation user interface element based on the distinct search query, and in accordance with a determination that the distinct search query corresponds to a first object and not a second object (e.g., the first object matches the query and the second object does not match the query), the electronic device (e.g., 101) displays 920c a first representation (e.g., 710d) of the first object in three dimensions within the navigation user interface element (e.g., 704b) and refrains from displaying 920c at least a portion (e.g., or all) of the second representation of the second object in three dimensions within the navigation user interface element (e.g., 704b), as in FIG.In some embodiments, the second representation of the second object (and representations of other objects not responsive to the respective search query) is displayed in two dimensions. In some embodiments, the electronic device stops displaying the second representation of the second object (and representations of other objects not responsive to the respective search query). In some embodiments, the electronic device reduces a height of the second representation of the second object (and representations of other objects not responsive to the respective search query) and / or reduces the extent to which the second representation of the second object (and representations of other objects not responsive to the respective search query) is displayed in three dimensions. In some embodiments, in response to detecting 920b an input corresponding to a request to search a navigation user interface element (e.g., 704b) based on the respective search query, the electronic device (e.g., 101) displays 920d a second representation (e.g., 710d) of the second object in three dimensions within the navigation user interface element (e.g., 704b) and refrains from displaying 920d at least a portion (or all) of the first representation of the first object in three dimensions within the navigation user interface element (e.g., 704b) in accordance with a determination that the respective search query corresponds to the second object and not the first object (e.g., the second object matches the query and the first object does not match the query), as in FIG. 7G. In some embodiments, the first representation of the first object (and representations of other objects not corresponding to the respective search query) are displayed in two dimensions. In some embodiments, the electronic device ceases displaying the first representation of the first object (and representations of other objects not corresponding to the respective search query).In some embodiments, prior to detecting an input corresponding to a request to search the navigation user interface elements based on the respective search query, the representations of the objects in the navigation user interface elements are displayed in two dimensions, and in response to the search query, the electronic device updates the navigation user interface elements to display representations of the objects that match the search query in three dimensions (e.g., maintains a display of representations of the objects that do not match the search query in two dimensions). In some embodiments, prior to detecting an input corresponding to a request to search the navigation user interface elements based on the respective search query, the representations of the objects in the navigation user interface elements are displayed in three dimensions, and in response to the search query, the electronic device maintains a display of representations of the objects that match the search query in three dimensions (e.g., updates the navigation user interface elements to display representations of the objects that do not match the search query in two dimensions).

[0208] The above-described method of displaying objects that correspond to a search query in three dimensions and suppressing the display of three-dimensional representations of objects that do not correspond to the search query in navigation user interface elements provides an efficient way of highlighting representations of objects that match individual search queries, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to determine which objects correspond to the search query and which do not).

[0209] In some embodiments, such as FIG. 7A , while displaying a navigation user interface element (e.g., 704a), the navigation user interface element (e.g., 704a) includes a first representation (e.g., 710a) of a first (e.g., physical) object located at a first physical location and a second representation (e.g., 710b) of a second (e.g., physical) object located at a second physical location, where the first representation (e.g., 710a) of the first object and the second representation (e.g., 710b) of the second object are displayed with visual characteristics (e.g., color, color scheme, translucency, brightness, saturation) having a first value, and the electronic device (e.g., 101) detects (922a), via one or more input devices, an input corresponding to a request to perform a search corresponding to information represented by the navigation user interface element based on a respective search query. In some embodiments, the navigation user interface elements include representations of physical objects (e.g., buildings, landmarks, roads, infrastructure, geographic features, terrain, bodies of water, plants, natural features) located within the distinct physical region that are displayed in a first color or colors. For example, the representations are displayed in a color that corresponds to the type of object that the representation corresponds to (e.g., buildings in a first color, roads in a second color, plants in a third color). As another example, the representations are displayed in full color that corresponds to the color of the object that the representation corresponds to. In some embodiments, the distinct search queries correspond to distinct physical locations, distinct landmarks, distinct addresses, etc.In some embodiments, in response to detecting an input corresponding to a request to search for a navigation user interface element based on the respective search query (922b), in accordance with a determination that the respective search query corresponds to the first object and not to the second object (e.g., the first object matches the query and the second object does not match the query), the electronic device (e.g., 101) displays (922c) in the navigation user interface element (e.g., 704b) at least a portion (or all) of a first representation (e.g., 710d) of the first object having a visual characteristic having a second value (e.g., a value the same as or different from the first value) and displays (922c) in the navigation user interface element (e.g., 704b) at least a portion (or all) of a second representation (e.g., 710e) of the second object having a visual characteristic having a third value different from the second value (e.g., a value the same as or different from the first value), as in FIG. 7G. In some embodiments, the second representation of the second object (and representations of other objects not corresponding to the particular search query) is displayed in a different color, a different color scheme or setting, etc., than the first representation of the first object corresponding to the particular search query.In some embodiments, in response to detecting an input corresponding to a request to search a navigation user interface element based on the respective search query (922b), in accordance with a determination that the respective search query corresponds to a second object rather than the first object (e.g., the second object matches the query and the first object does not match the query), the electronic device (e.g., 101) displays (922d) in the navigation user interface element (e.g., 704b) at least a portion (or all) of a second representation (e.g., 710d) of the second object having a visual characteristic having a second value (e.g., the same or different from the first value) and displays (922d) in the navigation user interface element (e.g., 704b) at least a portion (or all) of the first representation (e.g., 710e) of the first object having a visual characteristic having a third value (e.g., the same or different from the first value). In some embodiments, a first representation of a first object (and representations of other objects not corresponding to a respective search query) is displayed in a different color, a different color scheme, setting, etc. than a second representation of a second object corresponding to a respective search query. In some embodiments, the electronic device maintains the color, color scheme, setting, etc. of the representation of the object corresponding to the search query, and modifies the color, color scheme, setting, etc. of the representation of the object not corresponding to the search query. For example, the electronic device reduces the saturation or contrast of the representation not corresponding to the search query, or displays the representation not corresponding to the search query in a predetermined color. In some embodiments, the electronic device maintains the color, color scheme, setting, etc. of the representation of the object not corresponding to the search query, and modifies the color, color scheme, setting, etc. of the representation of the object corresponding to the search query. For example, the electronic device highlights, brightens, or uses a predetermined color for the representation of the object corresponding to the search query.In some embodiments, the electronic device displays representations of objects that correspond to the search query in a first color or in a first color scheme or setting that differs from how the representations of the objects were displayed prior to detecting the search request, and displays representations of objects that do not correspond to the search query in a second color or in a second color scheme or setting that differs from how the representations of the objects were displayed prior to detecting the search request.

[0210] The above-described method of displaying objects that correspond to a search query and objects that do not correspond to a search query in a navigation user interface element using visual characteristics having different values ​​provides an efficient way of distinguishing representations of objects that match individual search queries, which allows a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to determine which objects correspond to a search query and which do not), thereby further reducing power usage and improving battery life of the electronic device.

[0211] In some embodiments, while displaying a navigation user interface element (e.g., navigation user interface element 704a of FIG. 7A), the electronic device (e.g., 101) detects (924a), via one or more input devices, an input corresponding to a request to perform a search corresponding to the information represented by the navigation user interface element (e.g., 704a) based on a respective search query. In some embodiments, the respective search query corresponds to a respective physical location, a respective landmark, a respective address, or the like. In some embodiments, such as in FIG. 7G, in response to detecting (924b) an input corresponding to a request to search the navigation user interface element (e.g., 704b) based on the respective search query, the electronic device (e.g., 101) displays (924c), via the display generation component (e.g., 120), a three-dimensional representation (e.g., 708e) of a physical object (e.g., and / or location) corresponding to the search query (e.g., within a content element). In some embodiments, the three-dimensional representation is actual scale with respect to the physical object to which the representation corresponds. In some embodiments, the electronic device displays an indication of the location of the physical object within the navigation user interface element simultaneously with the three-dimensional representation of the physical object. In some embodiments, the electronic device displays information about the physical object simultaneously with a three-dimensional representation of the physical object. In some embodiments, the electronic device displays the three-dimensional representation of the physical object in response to detecting a selection of a representation of the object in a content element or in a navigation user interface element. In some embodiments, the three-dimensional representation is displayed according to one or more steps of method 800.

[0212] The above-described method of displaying a three-dimensional representation of a physical object corresponding to a search query provides an efficient method of identifying an object corresponding to a search query, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0213] In some embodiments, such as FIG. 7A, a navigation user interface element (e.g., 704a) includes (926a) a discrete piece of content (e.g., 710a, 710b) (e.g., displayed in three dimensions). In some embodiments, the navigation user interface element includes a representation of a physical object (e.g., a building, a landmark, a road, infrastructure, a geographic feature, a landform, a body of water, a plant, a natural feature) in a physical region. In some embodiments, such as FIG. 7B, while displaying a navigation user interface element (e.g., 704b) including the displayed discrete piece of content, the electronic device detects (926b) a movement of a predefined portion (e.g., 728a) of a user (e.g., a finger, a hand, an arm, a head, etc.) via one or more input devices. In some embodiments, the electronic device presents a representation of the predefined portion of the user in a three-dimensional environment including the user interface. In some embodiments, the representation is a photorealistic representation (e.g., a video or a virtual pass-through) of the predefined portion of the user displayed via a display generating component of the electronic device. In some embodiments, the representation is a view of a predefined portion of the user visible through a transparent portion of the display generating component (e.g., actual or true pass-through). In some embodiments, such as FIG. 7B, in response to detecting (926c) movement of the predefined portion of the user (e.g., 728a), the electronic device (e.g., 101) visually de-emphasizes (e.g., removes, flattens, blurs, increases translucency) (926d) the predefined portion of the user (e.g., 728a) in a location corresponding to the individual content in the navigation user interface element (e.g., 704b) (e.g., the location of the predefined portion of the user in the three-dimensional environment intersects or is within a threshold distance (e.g., 0.1, 0.2, 0.3, 0.5, 1, 2, 3, 4, 5, 10 centimeters, etc.) of the location of the individual content in the three-dimensional environment).For example, the electronic device displays the distinct content in two dimensions pursuant to a determination that a predefined portion of the user is at a location corresponding to the distinct content in the content navigation user interface element, while maintaining a three-dimensional display of other content in the navigation user interface element. In some embodiments, the electronic device visually de-emphasizes the distinct content in response to detecting a predefined portion of the user at a location corresponding to the distinct content according to one or more steps of method 800.

[0214] The above-described method of visually de-emphasizing individual content pursuant to a determination that a user's pre-defined portion is at a location corresponding to the individual content within a navigation user interface element provides an efficient method of maintaining visibility of a representation of the user's pre-defined portion, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by providing the user with visual feedback of the user's pre-defined portion while providing input to the electronic device using the user's pre-defined portion).

[0215] In some embodiments, such as FIG. 7A , the navigation element (e.g., 704a) includes an indication (e.g., 712) of weather conditions (e.g., precipitation, clouds, sunshine, wind, fog, haze) at the respective physical location represented by the navigation user interface element (e.g., 704a) (e.g., the weather condition is occurring, has occurred, or is predicted to occur) and includes an indication (928a) of a three-dimensional location of the weather conditions at the respective physical location (e.g., 712). In some embodiments, the indication is a pictorial representation of the weather conditions displayed at a location within the navigation user interface element. For example, if the respective physical location is experiencing rain, the electronic device displays a representation of rain clouds and rain at a three-dimensional location within the navigation user interface element that corresponds to the respective three-dimensional physical location experiencing the rain. As another example, the electronic device displays a three-dimensional representation of a cloud at a three-dimensional location that corresponds to the three-dimensional physical location of the cloud. For example, the representation of high altitude clouds is an image of clouds displayed at a position corresponding to a relatively high altitude above the physical location corresponding to the navigation user interface element, and the representation of low altitude clouds is an image of clouds displayed at a position corresponding to a relatively low altitude above the physical location corresponding to the navigation user interface element. In some embodiments, the indication of a particular weather condition at a particular three-dimensional location within the navigation user interface element is an indication of the weather condition and an indication of the three-dimensional location of the weather condition at the physical location. For example, the indication of clouds at a first location within the navigation user interface element is an indication of overcast weather and an indication of the physical location experiencing the overcast weather, and the indication of smog at a second location within the navigation user interface element is an indication of smog and an indication of the physical location experiencing the smog.The above-described method of displaying an indication of weather conditions on a navigation user interface element provides an efficient way of presenting weather conditions at one or more physical locations represented by the navigation user interface element, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to view weather conditions at multiple locations simultaneously).

[0216] In some embodiments, aspects and / or operations of methods 800 and 900 may be interchanged, substituted, and / or added between these methods, the details of which will not be repeated here for the sake of brevity.

[0217] The above has been described with reference to specific embodiments for purposes of explanation. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been chosen and described in order to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suited to the particular uses contemplated.

[0218] As mentioned above, one aspect of the present technology is to collect and use data available from various sources to improve a user's XR experience. This disclosure contemplates that in some cases, this collected data may include personal information data that uniquely identifies a particular person or that can be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records regarding the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0219] This disclosure recognizes that the use of such personal information data in the present technology may be for the benefit of the user. For example, the personal information data may be used to enhance the user's XR experience. Additionally, other uses of personal information data that benefit the user are also contemplated by this disclosure. For example, health and fitness data may be used to provide insight into the user's overall wellness, or may be used as proactive feedback to individuals using the technology in pursuit of wellness goals.

[0220] This disclosure contemplates that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will adhere to robust privacy policies and / or privacy practices. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for keeping personal information data confidential and secure. Such policies should be easily accessible by users and should be updated as data collection and / or use changes. Personal information from users should be collected for the entity's lawful and legitimate use and should not be shared or sold except for those lawful uses. Furthermore, such collection / sharing should be carried out after informing and obtaining consent from the user. Moreover, such entities should consider taking all necessary measures to protect and secure access to such personal information data and ensure that others who have access to the personal information data adhere to their privacy policies and procedures. Furthermore, such entities may subject themselves to third-party assessments to attest to their adherence to widely accepted privacy policies and practices. Moreover, policies and practices should be adapted to the specific types of personal information data being collected and / or accessed, and should conform to applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, collection of or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). Meanwhile, health data in other countries may be subject to other regulations and policies and should be addressed accordingly. Therefore, different privacy practices should be maintained with respect to different types of personal data in each country.

[0221] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which a user selectively blocks use of or access to personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case of an XR experience, the present technology may be configured to allow a user to select to "opt-in" or "opt-out" of participating in the collection of personal information data during registration for the service or at any time thereafter. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notice regarding access or use of personal information. For example, the user may be informed upon download of an app that will access the user's personal information data, and then again immediately prior to the personal information data being accessed by the app.

[0222] Moreover, it is the intent of this disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintended or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting it when it is no longer needed. Additionally, where applicable in certain health-related applications, anonymization of data can be used to protect user privacy. De-identification can be facilitated by removing certain identifiers (e.g., date of birth, etc.) where appropriate, controlling the amount or specificity of data stored (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0223] Thus, while this disclosure broadly encompasses the use of personal information data to implement one or more of the various disclosed embodiments, this disclosure also contemplates that the various embodiments may be implemented without requiring access to such personal information data. That is, the various embodiments of the technology are not rendered inoperable by the absence of all or a portion of such personal information data. For example, an XR experience may be generated by inferring preferences based on non-personal information data, such as content requested by a device associated with a user, or a minimal amount of personal information, other non-personal information available to the service, or publicly available information.

Claims

1. 1. A method comprising: An electronic device in communication with a display generation component and one or more input devices, comprising: a user interface via the display generation component, a first individual piece of content corresponding to a first view of a first physical location, the first individual piece of content being displayed at the first individual location within the three-dimensional environment, displayed simultaneously with the representation of the physical environment of the electronic device, and occupying a first portion of a display area of ​​the display generation component; displaying the three-dimensional environment including a user interface including a navigation user interface element having a first location designated therein that corresponds to the first physical location, the navigation user interface element being displayed within the three-dimensional environment between the first individual location at which the first individual content is displayed and a user's viewpoint within the three-dimensional environment; while displaying the user interface and the navigation user interface elements via the display generation component, detecting user input directed to the navigation user interface elements via the one or more input devices corresponding to a request to specify a second location corresponding to a second physical location; updating the user interface to include a second distinct content displayed at the first distinct location within the three dimensional environment corresponding to a second view of the second physical location in response to detecting the user input; detecting, via the one or more input devices, a discrete input corresponding to a request to obscure the display of the representation of the physical environment while displaying, via the display generation component, the first discrete content corresponding to the first view of the first physical location occupying the first portion of the display area simultaneously with the representation of the physical environment; and obscuring a display of the representation of the physical environment in response to detecting the discrete input, the obscuring comprising: corresponding to a third view of the first physical location that is larger than the first view; occupying a second portion of the display area of ​​the display generation component that is larger than the first portion; updating the first individual piece of content such that

2. the three-dimensional environment includes a representation of a surface in a physical environment of the electronic device; The method of claim 1 , wherein the navigational user interface elements are displayed at locations within the three-dimensional environment that correspond to the representation of the surface.

3. The method of claim 1 , wherein the navigational user interface elements are displayed at locations in the three-dimensional environment that do not correspond to surfaces in the physical environment of the electronic device.

4. 2. The method of claim 1 , wherein the first individual content is surrounded by a representation of a physical environment of the electronic device, and a boundary between the first individual content and the representation of the physical environment of the electronic device includes a gradual visual transition between the first individual content and the representation of the physical environment of the electronic device.

5. detecting, via the one or more input devices, a movement of the viewpoint of the user of the electronic device in the three-dimensional environment while displaying the first individual piece of content corresponding to the first view of the first physical location; 2. The method of claim 1, further comprising: in response to detecting a movement of the viewpoint of the user of the electronic device in the three-dimensional environment, and in accordance with a determination that the user's gaze was directed at the first individual piece of content when the movement of the viewpoint of the user was detected, updating a display of the first view of the first physical location corresponding to the first individual piece of content with a simulated parallax effect in accordance with the movement of the viewpoint of the user.

6. detecting a movement of the electronic device via the one or more input devices while displaying the first individual piece of content corresponding to the first view of the first physical location; 13. The method of claim 1, further comprising: in response to detecting the movement of the electronic device, displaying, via the display generation component, additional content corresponding to the first physical location in accordance with a determination that the movement of the electronic device satisfies one or more criteria.

7. detecting, while displaying the first individual piece of content corresponding to the first view of the first physical location, an input directed via the one or more input devices to a selectable user interface element displayed within the user interface; 2. The method of claim 1 , further comprising: in response to detecting the input directed at the selectable user interface element, displaying a third distinct content corresponding to a distinct view of a distinct physical location, the third distinct content being displayed at the first distinct location within the three-dimensional environment.

8. detecting a movement of the electronic device relative to the three-dimensional environment while displaying the first individual piece of content corresponding to the first view of the first physical location; displaying, via the display generation component, additional content corresponding to the first physical location in accordance with a determination that the movement of the electronic device relative to the three-dimensional environment satisfies one or more criteria; and 10. The method of claim 1, further comprising: withholding display of the additional content corresponding to the first physical location in accordance with a determination that the movement of the electronic device relative to the three-dimensional environment does not satisfy the one or more criteria.

9. The method of claim 8 , wherein the one or more criteria include a criterion that is met based on a change in orientation of the electronic device and / or a criterion that is met based on whether the user's line of sight is directed towards one or more respective objects in the three-dimensional environment.

10. The first individual piece of content corresponds to a first field of view having a first orientation, and the method further comprises: while displaying the first individual piece of content and displaying an indication of the first field of view in the navigation user interface element, detecting an input via the one or more input devices corresponding to a request to display a third individual piece of content corresponding to a second field of view having a second orientation; in response to detecting the input corresponding to the request to display the third distinct content corresponding to the second field of view having the second orientation; displaying the third separate content via the display generation component; and The method of claim 1 , further comprising: updating, in the navigation user interface element, the indication of the first field of view to be an indication of the second field of view.

11. displaying the navigation user interface element at a first size within the three-dimensional environment in accordance with a determination that the physical environment of the electronic device satisfies one or more criteria, including a criterion that is satisfied based on an amount of unobstructed space around the electronic device in the physical environment of the electronic device; 2. The method of claim 1, further comprising: displaying the navigation user interface element in the three-dimensional environment at a second size different from the first size in accordance with a determination that the physical environment of the electronic device does not satisfy the one or more criteria.

12. 2. The method of claim 1 , wherein the navigation user interface element includes a first distinct portion that is displayed in three dimensions in the three dimensional environment and a second distinct portion that is displayed in two dimensions in the three dimensional environment, the first distinct portion being closer to the user's viewpoint than the second distinct portion.

13. During display of the first individual content, the navigation user interface element corresponds to a first physical area that includes the first physical location; The method of claim 1 , wherein the navigation user interface element includes an indication of a topography of the first physical region.

14. The navigation user interface elements include individual pieces of content that are displayed in three dimensions, and the method further comprises: detecting a movement of a predetermined portion of the user via the one or more input devices while displaying the navigation user interface element including the respective content displayed in three dimensions; in response to detecting the movement of the predetermined portion of the user, 2. The method of claim 1, further comprising: updating the navigation user interface element to visually de-emphasize the individual content displayed in three dimensions in accordance with a determination that the predetermined portion of the user is at a location corresponding to the individual content displayed in three dimensions within the navigation user interface element.

15. displaying within the navigation user interface element an indication of the second electronic device displayed at a distinct location within the navigation user interface element that corresponds to a distinct physical location of the second electronic device; detecting a selection of the indication of the second electronic device via the one or more input devices while displaying the indication of the second electronic device on the navigation user interface element; 2. The method of claim 1, further comprising: in response to detecting the selection of the indication of the second electronic device, displaying, via the display generation component, content captured by the second electronic device at the respective physical location of the second electronic device.

16. The first individual piece of content includes a two-dimensional representation of an object located at the first physical location, the method comprising: The method of claim 1 , further comprising simultaneously displaying the first separate content including the two-dimensional representation of the object and the three-dimensional representation of the object.

17. 1. An electronic device comprising: one or more processors; Memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 16.

18. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method of any one of claims 1 to 16.

Images