Devices, methods, and graphical user interfaces for maps

Abstract

This relates to devices, methods, and graphical user interfaces for maps. [Solution] In some embodiments, an electronic device simultaneously presents a navigation user interface element having a specified individual physical location and a content element containing content corresponding to the individual physical location. In some embodiments, in response to an input corresponding to a request to present content corresponding to a second physical location, the electronic device presents navigation from the first physical location to the second physical location with reduced visual prominence within the content element.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 164,296, filed on March 22, 2021, the contents of which are hereby incorporated by reference in their entirety for all purposes.

[0002] This generally relates to computer systems having one or more input devices that present a graphical user interface, including but not limited to an electronic device that presents a graphical user interface via a display generation component and a display generation component that includes a map.

Background Art

[0003] The development of computer systems for augmented reality has advanced significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices such as cameras, controllers, joysticks, touch - sensitive surfaces, and touch - screen displays for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Exemplary virtual elements include virtual objects that contain digital images, videos, text, icons, and control elements such as buttons and other graphics.

[0004] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and restrictive. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve desired results in augmented reality environments, and systems where manipulating virtual objects is complex and error-prone impair the user's cognitive burden and detract from the virtual / augmented reality experience. In addition, these methods are unnecessarily time-consuming, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. [Overview of the project]

[0005] Therefore, there is a need for computer systems with improved methods and interfaces to provide users with computer-generated experiences that make interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can optionally complement or replace conventional methods of providing users with augmented reality experiences. Such methods and interfaces reduce the number, extent, and / or types of user input by helping the user understand the connection between the inputs provided and the device response to those inputs, thereby generating a more efficient human-machine interface.

[0006] The drawbacks and other problems associated with the user interface of a computer system described above are mitigated or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, tablet computer, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a wristwatch or head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a “touchscreen” or “touchscreen display”). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, the computer system has one or more output devices in addition to a display generation component, the output devices include one or more tactile output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in memory for performing multiple functions. In some embodiments, the user interacts with the GUI (and / or computer system) through stylus and / or finger touch and gestures on a touch-sensitive surface, the movement of the user's eyes and hands in space relative to the user's body as captured by a camera and other motion sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through interaction optionally include image editing, drawing, presentation, word processing, spreadsheet creation, gameplay, making phone calls, video conferencing, sending emails, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note-taking, and / or digital video playback. The executable instructions for performing those functions optionally reside in a primary computer-readable storage medium and / or a non-primary computer-readable storage medium, or in other computer program products configured to be executed by one or more processors.

[0007] There is a need for electronic devices having improved methods and interfaces for navigating user interfaces. Such methods and interfaces can complement or replace conventional methods for interacting with graphical user interfaces. Such methods and interfaces reduce the number, extent, and / or types of user input, resulting in a more efficient human-machine interface.

[0008] In some embodiments, the electronic device simultaneously presents content corresponding to a first physical location in a first location within the user interface, and a navigation user interface element having an indication of the first physical location. In some embodiments, in response to an input corresponding to a request to display content corresponding to a second physical location, the electronic device displays navigation from the first physical location to the second physical location with reduced visual prominence.

[0009] It should be noted that the various embodiments described herein can be combined with any other embodiments described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, in particular, in light of the drawings, specification and claims. Furthermore, it should be noted that the language used herein has been selected solely for readability and explanatory purposes and not to define or limit the subject matter of the invention.

[0010] To better understand the various embodiments described, the following “Modes for Carrying Out the Invention” should be referenced in conjunction with the following drawings, and similar reference numbers throughout the following drawings refer to the corresponding parts. [Brief explanation of the drawing]

[0011] [Figure 1] This block diagram shows the operating environment of a computer system for providing an XR experience, according to several embodiments. [Figure 2] A block diagram showing a controller for a computer system configured to manage and adjust the user's XR experience, according to several embodiments. [Figure 3] This block diagram shows a display generation component of a computer system configured to provide a user with visual components of an XR experience, according to several embodiments. [Figure 4] This is a block diagram showing a hand tracking unit for a computer system configured to capture user gesture input, according to several embodiments. [Figure 5] This is a block diagram showing an eye-tracking unit for a computer system configured to capture user eye-gaze input, according to several embodiments. [Figure 6A] This is a flowchart showing a glint-assisted eye-tracking pipeline according to several embodiments. [Figure 6B]This document illustrates exemplary environments of electronic devices for providing XR experiences, based on several embodiments. [Figure 7A] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 7B] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 7C] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 7D] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 7E] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 7F] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 7G] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 7H] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 7I] Examples of how an electronic device may simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location, according to several embodiments. [Figure 8A] This flowchart shows a method, according to several embodiments, for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element containing content corresponding to that individual physical location. [Figure 8B] This flowchart shows a method, according to several embodiments, for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element containing content corresponding to that individual physical location. [Figure 8C] This flowchart shows a method, according to several embodiments, for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element containing content corresponding to that individual physical location. [Figure 8D] This flowchart shows a method, according to several embodiments, for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element containing content corresponding to that individual physical location. [Figure 8E] This flowchart shows a method, according to several embodiments, for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element containing content corresponding to that individual physical location. [Figure 8F]A flowchart showing a method for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element including content corresponding to the individual physical location, according to some embodiments. [Figure 8G] A flowchart showing a method for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element including content corresponding to the individual physical location, according to some embodiments. [Figure 8H] A flowchart showing a method for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element including content corresponding to the individual physical location, according to some embodiments. [Figure 8I] A flowchart showing a method for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element including content corresponding to the individual physical location, according to some embodiments. [Figure 8J] A flowchart showing a method for simultaneously presenting a navigation user interface element having a specified individual physical location and a content element including content corresponding to the individual physical location, according to some embodiments. [Figure 9A] A flowchart showing a method for presenting navigation from a first physical location to a second physical location with reduced visual salience within a content element in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9B] A flowchart showing a method for presenting navigation from a first physical location to a second physical location with reduced visual salience within a content element in response to an input corresponding to a request to present content corresponding to the second physical location, according to some embodiments. [Figure 9C]This flowchart shows a method for presenting navigation from a first physical location to a second physical location, with reduced visual prominence within the content elements, in response to an input corresponding to a request to present content corresponding to a second physical location, according to several embodiments. [Figure 9D] This flowchart shows a method for presenting navigation from a first physical location to a second physical location, with reduced visual prominence within the content elements, in response to an input corresponding to a request to present content corresponding to a second physical location, according to several embodiments. [Figure 9E] This flowchart shows a method for presenting navigation from a first physical location to a second physical location, with reduced visual prominence within the content elements, in response to an input corresponding to a request to present content corresponding to a second physical location, according to several embodiments. [Figure 9F] This flowchart shows a method for presenting navigation from a first physical location to a second physical location, with reduced visual prominence within the content elements, in response to an input corresponding to a request to present content corresponding to a second physical location, according to several embodiments. [Figure 9G] This flowchart shows a method for presenting navigation from a first physical location to a second physical location, with reduced visual prominence within the content elements, in response to an input corresponding to a request to present content corresponding to a second physical location, according to several embodiments. [Figure 9H] This flowchart shows a method for presenting navigation from a first physical location to a second physical location, with reduced visual prominence within the content elements, in response to an input corresponding to a request to present content corresponding to a second physical location, according to several embodiments. [Modes for carrying out the invention]

[0012] This disclosure relates to user interfaces that provide users with computer-generated reality (XR) experiences, in several embodiments.

[0013] The systems, methods, and GUIs described herein provide improved methods for electronic devices to present content corresponding to physical locations indicated within navigation user interface elements.

[0014] In some embodiments, a computer system simultaneously displays a navigation user interface element in a three-dimensional environment, accompanied by a specified first physical location and first content corresponding to the first physical location. In some embodiments, the navigation user interface element is displayed between the first content and the user's viewpoint in the three-dimensional environment. In some embodiments, in response to input corresponding to a request to present content corresponding to a second physical location, the electronic device stops displaying the first content and displays the second content at the location in the three-dimensional environment where the first content was displayed. Presenting the second content at the same location in the three-dimensional environment where the first content was displayed provides an efficient way to view content corresponding to physical locations, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, without the user having to direct their attention to different areas of the three-dimensional environment or provide input to continue displaying content at the same location).

[0015] In some embodiments, a computer system simultaneously displays a navigation user interface element and a content element containing content corresponding to a first physical location represented by the navigation user interface element. In some embodiments, an electronic device displays the first content with a first visual splendor. In some embodiments, while detecting user input corresponding to a request to display content corresponding to a second physical location, the electronic device displays navigation from the first physical location to the second physical location within the content element with a visual splendor reduced to the first visual splendor. After displaying the navigation from the first physical location to the second physical location, the electronic device optionally displays the second content with a visual splendor greater than the reduced visual splendor relative to the first visual splendor. Reducing the visual prominence of a content element while detecting user input corresponding to a request to specify a second location provides an efficient way to indicate to the user that specifying a second location will update the content element, which further reduces power consumption, improves the battery life of the electronic device, reduces usage errors that must be corrected by further user input, and reduces the need for the electronic device to fully render the content corresponding to the first location and / or intermediate locations between the first and second locations (e.g., while the visual indication of the location corresponding to the content is moving).

[0016] Figures 1-6 illustrate exemplary computer systems for providing an XR experience to a user. Figures 7A-7I show examples of how an electronic device provides user interface navigation according to the detection of the user's gaze, according to several embodiments. Figures 8-9 are flowcharts of how navigation is provided according to the detection of the user's gaze, according to various embodiments. The user interfaces in Figures 7A-7I are used to illustrate the processes in Figures 8-9, respectively.

[0017] The processes described below enhance the usability of the device and streamline the user-device interface through various technologies, including providing users with improved visual feedback, reducing the number of inputs required to perform operations, offering additional control options without cluttering the user interface with additional controls displayed, performing operations without requiring further user input when a set of conditions is met, improving privacy and / or security, and / or other technologies. These technologies also reduce power consumption and improve the device's battery life by enabling users to use the device more quickly and efficiently.

[0018] Furthermore, in any method described herein that is conditional on one or more conditions being met in one or more steps, it should be understood that the method described can be repeated in multiple iterations such that all the conditions that the steps of the method are conditional on are met in different iterations of the method. For example, if a method requires that a first step be performed if a condition is met, and a second step be performed if the condition is not met, a person skilled in the art will understand that the steps described in the claim are repeated in a specific order until the conditions are met and then not met. Thus, a method described in one or more steps that depends on one or more conditions being met can be rewritten as a method that is repeated until each of the conditions described in the method is met. However, this is not required for a claim of a system or computer-readable medium that includes instructions for performing a conditional operation based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency has been met without explicitly repeating the steps of the method until all the conditions that the steps of the method are conditional on are met. Those skilled in the art will also understand that, as with a method having conditional steps, a system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0019] In some embodiments, as shown in Figure 1, the XR experience is provided to the user via an operating environment 100 which includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, a velocity sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a home appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, output device 155, sensor 190, and peripheral device 195 are integrated with the display generation component 120 (for example, in a head-mounted device or handheld device).

[0020] When describing an XR experience, various terms are used to refer individually to several related but distinct environments that the user perceives and / or interacts with (for example, using inputs detected by the computer system 101, which causes the computer system generating the XR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101 that generates the XR experience). The following is a subset of these terms.

[0021] Physical Environment: The physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic systems. Examples of physical environments, such as a physical park, include physical objects such as physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through their senses of sight, touch, hearing, taste, and smell.

[0022] Augmented Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people perceive and / or interact with through an electronic system. In XR, a subset of a person's bodily movements or their representations are tracked, and accordingly, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. For example, an XR system may detect a person's head rotation and, accordingly, adjust the graphic content and sound field presented to the person in a similar manner to how such views and sounds would change in a physical environment. In some circumstances (e.g., for reasons of accessibility), adjustments to the properties(s) of virtual objects(s) in the XR environment may be made in response to representations of bodily movements (e.g., voice commands). A person may perceive and / or interact with XR objects using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person may perceive and / or interact with audio objects that create a 3D or spatially expansive audio environment, providing the perception of a point source in 3D space. In another example, audio objects may enable audio transparency, selectively incorporating ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, a person may perceive and / or interact with audio objects only.

[0023] Examples of XR include virtual reality and mixed reality.

[0024] Virtual reality: A virtual reality (VR) environment refers to a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can perceive and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can perceive and / or interact with virtual objects in a VR environment through a simulation of their presence within the computer-generated environment and / or through a simulation of a subset of their physical movement within the computer-generated environment.

[0025] Mixed Reality: A mixed reality (MR) environment is a simulated environment designed to incorporate sensory input or its representation from a physical environment, in addition to including computer-generated sensory input (e.g., virtual objects), in contrast to a virtual reality (VR) environment designed to rely entirely on computer-generated sensory input. On a virtual continuum, a mixed reality environment is any location between, but not encompassing, the complete physical environment at one end and the virtual reality environment at the other. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Also, some electronic systems for presenting an MR environment may track location and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical articles or their representations from the physical environment). For example, the system may take movement into account so that a virtual tree appears stationary relative to the physical ground.

[0026] Examples of mixed reality include augmented reality and augmented virtual reality.

[0027] Augmented Reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on or onto a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display that allows a person to directly view the physical environment. The system may also be configured to present virtual objects on the transparent or translucent display, thereby allowing a person to use the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture an image or video of the physical environment, which is a representation of the physical environment. The system composites the image or video with the virtual objects and presents the composite on the opaque display. A person uses this system to perceive the virtual objects superimposed on the physical environment by indirectly viewing the physical environment through the image or video of the physical environment. As used herein, a video of the physical environment shown on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example, as holograms, into or onto the physical environment, so that a person can use the system to perceive the virtual objects superimposed on the physical environment. An augmented reality environment also refers to an imitation environment in which the representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may transform one or more sensor images to plane a selected perspective (e.g., viewpoint) different from the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion of it, so that the modified portion is a non-photorealistic altered version of the original captured image. As yet another example, the representation of the physical environment may be transformed by graphically removing or obscuring a portion of it.

[0028] Augmented Virtual: An Augmented Virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. These sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park might have virtual trees and virtual buildings, while people with faces are realistically reproduced from images of real people. Another example is that a virtual object might adopt the shape or color of a physical article captured by one or more imaging sensors. A further example is that a virtual object might adopt shadows that correspond to the position of the sun in the physical environment.

[0029] Viewpoint-locked virtual objects: A virtual object is viewpoint-locked when the computer system displays the virtual object in the same location and / or position within the user's view, even if the user's view shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's view is locked in the forward direction of the user's head (e.g., the user's view is at least a portion of the user's field of vision when the user is looking straight ahead). Thus, the user's view remains fixed even if the user's gaze moves, without moving the user's head. In embodiments where the computer system has a display generation component (e.g., a display screen) that can be repositioned relative to the user's head, the user's view is the augmented reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper-left corner of the user's view when the user's view is in a first orientation (e.g., the user's head is facing north) will continue to be displayed in the upper-left corner of the user's view even if the user's view changes to a second orientation (e.g., the user's head is facing west). In other words, the location and / or position in which a viewpoint-locked virtual object is displayed from the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, so that the virtual object is also referred to as a "head-locked virtual object."

[0030] Environment-Locked Virtual Objects: A virtual object is environment-locked (or "world-locked") when a computer system displays it at a location and / or position in the user's viewpoint that is based on (e.g., selected by reference to and / or fixed to) a location and / or object in a three-dimensional environment (e.g., a physical or virtual environment). As the user's viewpoint shifts, the location and / or object in the environment relative to the user's viewpoint changes, and as a result, the environment-locked virtual object will appear at a different location and / or position in the user's viewpoint. For example, an environment-locked virtual object locked to a tree directly in front of the user will appear at the center of the user's viewpoint. If the user's viewpoint shifts to the right (e.g., the user's head is turned to the right) and the tree becomes left-leaning in the user's viewpoint (e.g., the tree's position in the user's viewpoint shifts), the environment-locked virtual object locked to the tree will appear left-leaning in the user's viewpoint. In other words, the location and / or position in which an environment-locked virtual object is displayed in the user's viewpoint depends on the location and / or object's position and / or orientation in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference frame (e.g., a fixed location in the physical environment and / or a coordinate system fixed to an object) to determine the position in which the environment-locked virtual object is displayed from the user's viewpoint. The environment-locked virtual object can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object) or to a moving part of the environment (e.g., a vehicle, animal, person, or a representation of a part of the user's body that moves independently of the user's viewpoint, such as the user's hands, wrists, arms, or feet), so that the virtual object moves as the viewpoint or the part of the environment moves in order to maintain a fixed relationship between the virtual object and the part of the environment.

[0031] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits delayed tracking behavior, reducing or delaying its movement in response to the movement of a reference point that the virtual object is following. In some embodiments, when exhibiting delayed tracking behavior, the computer system detects movement of the reference point that the virtual object is following (e.g., a part of the environment, a viewpoint, or a point fixed to the viewpoint, such as a point between 5 and 300 cm from the viewpoint) and intentionally delays the movement of the virtual object. For example, when the reference point (e.g., a part of the environment or the viewpoint) moves at a first velocity, the virtual object is moved by the device so as to remain locked to the reference point, but at a second velocity slower than the first velocity (e.g., the virtual object begins to catch up to the reference point until the reference point stops or slows down). In some embodiments, when a virtual object exhibits delayed tracking behavior, the device ignores small movements of the reference point (e.g., ignoring movements of the reference point that are below a threshold movement amount, such as a movement of 0 to 5 degrees or a movement of 0 to 50 cm). For example, when the reference point (e.g., the part of the environment or viewpoint from which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a different viewpoint or part of the environment from which the virtual object is locked), and when the reference point (e.g., the part of the environment or viewpoint from which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object first increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a different viewpoint or part of the environment from which the virtual object is locked), and then decreases as the amount of movement of the reference point increases beyond a threshold (e.g., a "delayed tracking" threshold) as the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point.In some embodiments, a virtual object that maintains a substantially fixed position with respect to a reference point includes the virtual object being displayed within a threshold distance (e.g., 1, 2, 3, 5, 15, 20, 50 cm) of the reference point in one or more dimensions (e.g., above / below, left / right, and / or forward / behind the position of the reference point).

[0032] Hardware: There are many different types of electronic systems that enable a person to perceive and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned over a person's eyes (similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to receive an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or videos of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology to project a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as holograms or as physical surfaces.In some embodiments, the controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, the controller 110 includes a preferred combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to Figure 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server located outside the scene 105 (e.g., a cloud server, a central server, etc.). In some embodiments, the controller 110 is coupled to a display generation component 120 (e.g., an HMD, display, projector, touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within a housing (e.g., a physical housing) of one or more of the display generation components 120 (e.g., a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0033] In some embodiments, the display generation component 120 is configured to provide the user with an XR experience (e.g., at least the visual components of an XR experience). In some embodiments, the display generation component 120 includes a preferred combination of software, firmware, and / or hardware. The display generation component 120 is described in more detail below with reference to Figure 3. In some embodiments, the functions of the controller 110 are provided by and / or combined with the display generation component 120.

[0034] According to some embodiments, the display generation component 120 provides the user with an XR experience while the user is virtually and / or physically present in the scene 105.

[0035] In some embodiments, the display generation component is mounted on a part of the user's body (e.g., their head or hand). Thus, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds the device, which has a display directed towards the user's field of view and a camera directed towards scene 105. In some embodiments, the handheld device is optionally placed in a housing mounted on the user's head. In some embodiments, the handheld device is optionally placed on a support in front of the user (e.g., a tripod). In some embodiments, the display generation component 120 is an XR chamber, housing, or room configured to present XR content when the user is not wearing or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interaction with XR content triggered based on interaction occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD where the interaction occurs in the space in front of the HMD and the XR content response is displayed through the HMD. Similarly, a user interface showing interaction with CRG content triggered based on the movement of a handheld or tripod-mounted device relative to the physical environment (e.g., Scene 105 or a part of the user's body (e.g., the user's eyes, head, or hands)) may be implemented similarly to an HMD where the movement is triggered by the movement of the HMD relative to the physical environment (e.g., Scene 105 or a part of the user's body (e.g., the user's eyes, head, or hands)).

[0036] While relevant features of the operating environment 100 are shown in Figure 1, those skilled in the art will understand from this disclosure that various other features for the sake of simplification are not shown so as not to obscure more suitable embodiments of the exemplary embodiments disclosed herein.

[0037] Figure 2 is a block diagram of an example of the controller 110 according to several embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more suitable embodiments of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), central processing unit (CPU), processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global Mobile Communication System (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZiGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0038] In some embodiments, one or more communication buses 204 include circuits that interconnect system components and control communication between system components. In some embodiments, one or more I / O devices 206 include at least one of the following: a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0039] Memory 220 includes high-speed random-access memory such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-temporary computer-readable storage medium. In some embodiments, memory 220, or the non-temporary computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 230 and XR experience module 240.

[0040] The operating system 230 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for each group of one or more users). For this purpose, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0041] In some embodiments, the data acquisition unit 241 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the display generation component 120 in Figure 1, and optionally from one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data acquisition unit 241 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0042] In some embodiments, the tracking unit 242 is configured to map scene 105 and track the location / position of at least the display generation component 120 relative to scene 105 in Figure 1, and optionally the location of one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the tracking unit 242 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the location / position of one or more parts of the user's hand relative to scene 105 in Figure 1, relative to the display generation component 120, and / or relative to a coordinate system defined for the user's hand, and / or the movement of one or more parts of the user's hand. The hand tracking unit 244 is described in more detail below with reference to Figure 4. In some embodiments, the eye-tracking unit 243 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to the scene 105 (e.g., the physical environment and / or the user (e.g., the user's hands)) or to XR content displayed via the display generation component 120. The eye-tracking unit 243 is described in more detail below with reference to Figure 5.

[0043] In some embodiments, the adjustment unit 246 is configured to manage and adjust the XR experience presented to the user by the display generation component 120 and optionally by one or more of the output devices 155 and / or peripheral devices 195. For this purpose, in various embodiments, the adjustment unit 246 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0044] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral devices 195. For this purpose, in various embodiments, the data transmission unit 248 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0045] While the data acquisition unit 241, tracking unit 242 (including, for example, an eye-tracking unit 243 and a hand-tracking unit 244), adjustment unit 246, and data transmission unit 248 are shown as residing on a single device (e.g., a controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, tracking unit 242 (including, for example, an eye-tracking unit 243 and a hand-tracking unit 244), adjustment unit 246, and data transmission unit 248 may be located in separate computing devices.

[0046] Furthermore, Figure 2 is intended to illustrate the functionality of various features that may be present in a particular embodiment, in contrast to the structural schematics of the embodiments described herein. As will be recognized by those skilled in the art, the separately shown items can be combined, and some items can be separated. For example, several functional modules shown separately in Figure 2 can be implemented within a single module, and the various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of certain functions and how functions are assigned between them, will vary depending on the implementation and, in some embodiments, will partially depend on a particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0047] Figure 3 is a block diagram of an example of a display generation component 120 according to several embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more suitable embodiments of the embodiments disclosed herein. For that purpose, in some non-limiting examples, the display generation component 120 (e.g., HMD) may include one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, Bluetooth, ZiGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional in-facing and / or out-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0048] In some embodiments, one or more communication buses 304 include circuits that interconnect system components and control communication between system components. In some embodiments, one or more I / O devices and sensors 306 include at least one of the following: an inertial measuring unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.).

[0049] In some embodiments, one or more XR displays 312 are configured to provide the user with an XR experience. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more XR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holographic. For example, a display generation component 120 (e.g., HMD) includes a single XR display. In another embodiment, the display generation component 120 includes an XR display for each of the user's eyes. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0050] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hands and optionally a portion of the user's arms (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene that the user would view if a display generation component 120 (e.g., an HMD) were not present (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., complementary metal-oxide-semiconductor (CMOS) image sensors or charge-coupled device (CCD) image sensors), one or more infrared (IR) cameras, one or more event-based cameras, and / or similar.

[0051] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-temporary computer-readable storage medium. In some embodiments, memory 320, or the non-temporary computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 330 and XR presentation module 340.

[0052] The operating system 330 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR presentation module 340 is configured to present XR content to the user via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0053] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 in Figure 1. For this purpose, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0054] In some embodiments, the XR presentation unit 344 is configured to present XR content via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0055] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment on which computer-generated objects can be placed) based on media content data. For this purpose, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0056] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110 and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral devices 195. For this purpose, in various embodiments, the data transmission unit 348 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0057] Although the data acquisition unit 342, XR presentation unit 344, XR map generation unit 346, and data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 in Figure 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, XR presentation unit 344, XR map generation unit 346, and data transmission unit 348 may be located in separate computing devices.

[0058] Furthermore, Figure 3 is intended to illustrate the functionality of various features that may be present in a particular implementation, in contrast to the structural schematics of the embodiments described herein. As will be recognized by those skilled in the art, the separately shown items can be combined, and some items can be separated. For example, several functional modules shown separately in Figure 3 can be realized within a single module, and the various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of certain functions and how functions are assigned between them, will vary depending on the implementation and, in some embodiments, will partially depend on a particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0059] Figure 4 is a schematic diagram of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 (Figure 1) is controlled by the hand tracking unit 243 (Figure 2) to track the location / position of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the scene 105 of Figure 1 (e.g., relative to a part of the physical environment surrounding the user, relative to the display generation component 120, or relative to a part of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined for the user's hand). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0060] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures a hand image with sufficient resolution to allow for the distinction of fingers and their respective positions. The image sensor 404 can typically capture images of other parts of the user's body, or images of the entire body, and may have either a zoom function or a dedicated sensor with high magnification to capture an image of the hand at a desired resolution. In some embodiments, the image sensor 404 also captures a 2D color video image of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor or a portion thereof is used to define an interaction space in which hand movements captured by the image sensor are processed as input to the controller 110.

[0061] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to the controller 110, thereby extracting high-level information from the map data. This high-level information is typically provided to an application running on the controller via an application programming interface (API), which drives the display generation component 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving their hand 406 and changing the orientation of their hand.

[0062] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spot in the pattern. This approach is advantageous in that the user does not need to hold or wear any kind of beacon, sensor, or other marker. This gives the depth coordinates of points in the scene relative to a given reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines a set of orthogonal x, y, and z axes such that the depth coordinates of points in the scene correspond to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods such as stereoscopic imaging or time-of-flight measurement based on one or more cameras or other types of sensors.

[0063] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves their hand (e.g., the entire hand or one or more fingers). Software running on the processor in the image sensor 404 and / or controller 110 processes the 3D map data to extract patch descriptors of the hand within these depth maps. Based on a previous learning process, the software matches these descriptors against patch descriptors stored in the database 408 to estimate the hand pose in each frame. The pose typically includes the 3D location of the user's wrist and fingertips.

[0064] The software can also analyze the trajectory of the hand and / or fingers across multiple frames in a sequence to identify gestures. The posture estimation function described herein may be interleaved with the motion tracking function, so that patch-based posture estimation is performed only once every two (or more) frames, while tracking is used to detect changes in posture that occur over the remaining frames. Posture, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify the image presented on the display generation component 120, or perform other functions, depending on the posture and / or gesture information.

[0065] In some embodiments, the gesture includes an air gesture. An air gesture is a gesture detected by the user without (or independently of) touching an input element that is part of a device (e.g., a computer system 101, one or more input devices 125, and / or a hand tracking device 140), and is based on detected movement of a part of the user's body in the air (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs), including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture including movement of the hand in a predetermined posture by a predetermined amount and / or speed, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body).

[0066] In some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures, as in some embodiments, performed by moving one or more of the user's fingers relative to other fingers or parts of the user's hand for interacting with an XR environment (e.g., a virtual or mixed reality environment). In some embodiments, an air gesture is a gesture detected without the user touching an input element that is part of the device (or independently of an input element that is part of the device), and is based on detected movement of a part of the user's body, including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of the user's other hand relative to one hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture involving movement of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture involving rotation of a part of the user's body by a predetermined speed or amount).

[0067] In some embodiments where the input gesture is an air gesture (i.e., without physical contact with an input device that provides the computer system with information about which user interface element is the target of user input, such as contact with a user interface element displayed on a touchscreen or contact with a mouse or trackpad to move a cursor over a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of user input (e.g., in the case of direct input, as described below). Thus, in implementations involving air gestures, the input gesture is the detected attention (e.g., gaze) to the user interface element in combination (e.g., simultaneously) with the movement of the user's fingers (one or more) and / or hand to perform pinch and / or tap input, as described in more detail below.

[0068] In some embodiments, input gestures directed towards a user interface object are performed directly or indirectly by reference to the user interface object. For example, user input is performed directly towards the user interface object in response to the user performing an input gesture with their hand at a position corresponding to the user interface object's position in a three-dimensional environment (e.g., determined based on the user's current viewpoint). In some embodiments, the input gesture is performed indirectly towards the user interface object according to the user performing the input gesture while the user's hand position is not at a position corresponding to the user interface object's position in a three-dimensional environment, while detecting the user's attention (e.g., gaze) to the user interface object. For example, in the case of a direct input gesture, the user can direct their input towards the user interface object by initiating the gesture at or near a position corresponding to the user interface object's display position (e.g., within a distance of 0.5 cm, 1 cm, 5 cm, or 0-5 cm from the optional outer edge or optional central portion). In the case of indirect input gestures, the user can direct their input towards the user interface object by paying attention to the user interface object (for example, by gazing at the user interface object), and while paying attention to the options, the user initiates the input gesture (for example, at any position detectable by the computer system) (for example, at a position that does not correspond to the display position of the user interface object).

[0069] In some embodiments, the input gestures (e.g., air gestures) used in the various examples and embodiments described herein include pinch and tap inputs for interacting with virtual or mixed reality environments, as in some embodiments. For example, the pinch and tap inputs described later are performed as air gestures.

[0070] In some embodiments, a pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch-and-drag gesture, or a double pinch gesture. For example, a pinch gesture that is an air gesture involves moving two or more fingers of a hand to touch each other, i.e., including an optional interruption (e.g., within 0 to 1 second) immediately after the touch. A long pinch gesture that is an air gesture involves moving two or more fingers of a hand to touch each other for at least a threshold time amount (e.g., at least 1 second) before detecting an interruption of contact between them. For example, a long pinch gesture includes the user holding a pinch gesture (e.g., if two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture that is an air gesture includes two (e.g., or more) pinch inputs (e.g., performed with the same hand) that are detected directly and consecutively (e.g., within a predetermined period of time) to each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaks contact between two or more fingers), and then performs a second pinch input within a predetermined period (e.g., within 1 second or 2 seconds) after releasing the first pinch input.

[0071] In some embodiments, an air gesture, a pinch-and-drag gesture, includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in relation to (e.g., after) a drag input that changes the user's hand position from a first position (e.g., a drag initiation position) to a second position (e.g., a resistance termination position). In some embodiments, the user maintains the pinch gesture while performing the drag input and releases the pinch gesture (e.g., spreading two or more fingers) to terminate the drag gesture (e.g., at the second position). In some embodiments, the pinch input and drag input are performed by the same hand (e.g., the user pinches two or more fingers together and touches them to each other, and then moves the same hand to a second position in the air with a drag gesture). In some embodiments, the pinch input is performed by the user's first hand and the drag input is performed by the user's second hand (e.g., the user's second hand moves from the first position to the second position in the air while the user continues the pinch input with the user's first hand). In some embodiments, an input gesture that is an air gesture includes an input (e.g., a pinch input and / or a tap input) performed using both of the user's hands. For example, an input gesture includes two (e.g., or more) pinch inputs performed in relation to each other (e.g., simultaneously or within a predetermined period of time). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) performed using the user's first hand, and a second pinch input performed using the other hand (e.g., a second hand of the user's hands) in relation to performing a pinch input using the first hand. In some embodiments, movement between the user's hands (e.g., to increase and / or decrease the distance or relative orientation between the user's hands).

[0072] In some embodiments, a tap input performed as an air gesture (e.g., directed towards a user interface element) includes the movement of one or more of the user's fingers toward the user interface element, the movement of the user's hand toward the user interface element with the user's fingers (one or more) optionally extended toward the user interface element, a downward movement of the user's fingers (e.g., mimicking a mouse click or a tap on a touchscreen), or other default movements of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on the movement characteristics of the finger or hand that performs the tap gesture movement away from the user's viewpoint and / or toward the object that is the target of the tap input, followed by the end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand that performs the tap gesture (e.g., away from the user's viewpoint and / or the end of the movement toward the object that is the target of the tap input, a reversal of the direction of the finger or hand movement, and / or a reversal of the direction of acceleration of the finger or hand movement).

[0073] In some embodiments, the user's attention is determined to be directed towards a part of the three-dimensional environment based on the detection of a gaze directed towards that part of the three-dimensional environment (optionally, without requiring any other conditions). In some embodiments, for the device to determine that the user's attention is directed towards a part of the three-dimensional environment, the device determines that the user's attention is directed towards a part of the three-dimensional environment based on the detection of a gaze directed towards a part of the three-dimensional environment, with one or more additional conditions such as the gaze being directed towards the part of the three-dimensional environment for at least a threshold duration (e.g., dwell time) while the user's viewpoint is within a distance threshold from the part of the three-dimensional environment, and / or the gaze being directed towards a part of the three-dimensional environment. If one of the additional conditions is not met, the device determines that the user's attention is not directed towards the part of the three-dimensional environment to which the gaze is directed (e.g., until one or more additional conditions are met).

[0074] In some embodiments, the detection of a ready state configuration of the user or a part of the user is detected by the computer system. The detection of a ready state configuration of the hand is used by the computer system as an indication that the user is likely to be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the ready state of a hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape where the thumb and one or more fingers are extended and spaced apart, ready to perform a pinch or grab gesture, or a pre-tap shape where one or more fingers are extended and the palm is facing away from the user), whether the hand is in a predetermined position relative to the user's line of sight (e.g., below the user's head, above the user's waist, or extended at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular way (e.g., moved towards the area in front of the user above the user's waist, below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of the user interface is responsive to attention (e.g., gaze) input.

[0075] In some embodiments, the software may be downloaded electronically to the controller 110, for example, over a network, or instead, it may be provided on a tangible non-temporary medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively or additionally, some or all of the computer's described functions may be implemented in dedicated hardware such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). The controller 110 is shown in Figure 4, for example, as a separate unit from the image sensor 440, but some or all of the controller's processing functions may be associated with the image sensor 404 by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (for example, in a television set, handheld device, or head-mounted device), or by any other suitable computerized device such as a game console or media player. The sensing function of the image sensor 404 can also be integrated into a computer or other computerized device controlled by the sensor output.

[0076] Figure 4 further includes schematic diagrams of depth maps 410 captured by image sensor 404 according to several embodiments. The depth map includes a matrix of pixels, each having a depth value, as described above. Pixels 412 corresponding to the hand 406 are segmented in this map from the background and the wrist. The brightness of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from image sensor 404, with the gradation becoming richer as the depth increases. Controller 110 processes these depth values ​​to identify and segment image components (i.e., adjacent pixel groups) that have the characteristics of a human hand. These characteristics may include, for example, the overall size, shape, and frame-to-frame movement of the depth map sequence.

[0077] Figure 4 also schematically shows the hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to several embodiments. In Figure 4, the hand skeleton 414 is superimposed on the hand background 416, which has been segmented from the original depth map. In some embodiments, the hand (e.g., finger joints, fingertips, center of the palm, end of the hand connected to the wrist), and optionally major feature points on the wrist or arm connected to the hand, are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these major feature points across multiple image frames are used by the controller 110 to determine, according to several embodiments, a hand gesture performed by the hand or the current state of the hand.

[0078] Figure 5 shows an exemplary embodiment of the eye-tracking device 130 (Figure 1). In some embodiments, the eye-tracking device 130 is controlled by an eye-tracking unit 245 (Figure 2) to track the position and movement of the user's gaze toward the scene 105 or toward the XR content displayed via the display generation component 120. In some embodiments, the eye-tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, if the display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device positioned in a wearable frame, the head-mounted device includes both a component for generating XR content for user viewing and a component for tracking the user's gaze toward the XR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, if the display generation component is a handheld device or an XR chamber, the eye-tracking device 130 is optionally a separate device from the handheld device or XR chamber. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used with a display generation component that is mounted on the head or a display generation component that is not mounted on the head. In some embodiments, the eye-tracking device 130 is optionally used in combination with a head-mounted display generation component, rather than being a head-mounted device. In some embodiments, the eye-tracking device 130 is optionally part of a non-head-mounted display generation component, rather than being a head-mounted device.

[0079] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames containing left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include, or be coupled to, one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display on which the user can directly view the physical environment and display virtual objects on a transparent or translucent display. In some embodiments, the display generation component projects virtual objects onto the physical environment. The virtual objects are projected, for example, onto a physical surface or as holograms, so that the individual can use the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0080] As shown in Figure 5, in some embodiments, the eye-tracking device 130 includes at least one eye-tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be directed toward the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, it may be directed toward a "hot" mirror positioned between the user's eye and a display panel that reflects IR or NIR light from the eye to the eye-tracking camera while allowing visual light to pass through. The eye-tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60 to 120 frames per second (fps)), analyzes the images to generate eye-tracking information, and communicates the eye-tracking information to the controller 110. In some embodiments, both of the user's eyes are tracked separately by their respective eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked by a separate eye-tracking camera and light source.

[0081] In some embodiments, the eye-tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye-tracking device for a specific operating environment 100, e.g., the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at the factory or another facility before delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automated calibration process or a manual calibration process. The user-specific calibration process may include estimating the eye parameters of a particular user, e.g., pupil position, central visual position, optical axis, visual axis, interpupillary distance. According to some embodiments, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, the images captured by the eye-tracking camera can be processed using a glint-assisted method to determine the user's current visual axis and viewpoint relative to the display.

[0082] As shown in Figure 5, the eye-tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye-tracking system which includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eyes(s) 592. The eye-tracking camera 540 is positioned between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display or projector of a handheld device) and may be directed towards a mirror 550 that transmits visible light while reflecting IR or NIR light from the eye(s) 592 (e.g., as shown at the top of Figure 5), or may be directed towards the user's eye(s) 592 to receive reflected IR or NIR light from the eye(s) 592 (e.g., as shown at the bottom of Figure 5).

[0083] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames of left and right display panels) and provides the frames 562 to the display 510. For various purposes, for example, when processing the frames 562 for display, the controller 110 uses gaze tracking input 542 from the eye-tracking camera 540. The controller 110 optionally uses a glint-assisted method or other appropriate method to estimate the user's viewpoint on the display 510 based on the gaze tracking input 542 obtained from the eye-tracking camera 540. The viewpoint estimated from the gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0084] The following describes, but is not intended to be limiting, several possible use cases of the user's current gaze direction. As an exemplary use case, the controller 110 may render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content at a higher resolution in the central visual region determined from the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content within the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content within the view based at least partially on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 may capture the physical environment of the XR experience and orient an external camera to focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface in the environment that the user is currently viewing on the display 510. In another exemplary use case, the eyepiece 520 may be a focusing lens, and the controller uses eye-tracking information to adjust the focus of the eyepiece 520 so that the virtual object currently being viewed by the user has appropriate binocular coordination to match the convergence of the user's eye 592. The controller 110 can use the eye-tracking information to orient and adjust the focus of the eyepiece 520 so that the nearby object being viewed by the user appears at the correct distance.

[0085] In some embodiments, the eye-tracking device is part of a head-mounted device mounted on a wearable housing, which includes a display (e.g., display 510), two eyepieces (e.g., one or more eyepieces 520), an eye-tracking camera (e.g., one or more eye-tracking cameras 540), and a light source (e.g., a light source 530 (e.g., an IR LED or NIR LED)). The light source emits light (e.g., IR light or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in Figure 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0086] In some embodiments, the display 510 emits light within the visible light range and does not emit light within the IR or NIR range, thus not introducing noise into the eye-tracking system. Note that the location and angle of the eye-tracking camera(s) 540 are given as examples and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is positioned on each side of the user's face. In some embodiments, two or more NIR cameras 540 can be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0087] Embodiments of eye-tracking systems, such as those shown in Figure 5, can be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide users with computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experiences.

[0088] Figure 6A shows a glint-assisted eye-tracking pipeline in several embodiments. In some embodiments, the eye-tracking pipeline is implemented by a glint-assisted eye-tracking system (e.g., an eye-tracking device 130 as shown in Figures 1 and 5). The glint-assisted eye-tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the glint-assisted eye-tracking system tracks the pupil contour and glint in the current frame by using prior information from previous frames when analyzing the current frame. When not in tracking state, the glint-assisted eye-tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues in tracking state for the next frame.

[0089] As shown in Figure 6A, the eye-tracking camera can capture left and right images of the user's left and right eyes. The captured images are then fed into the eye-tracking pipeline for processing, which begins at 610. As indicated by the arrow returning to element 600, the eye-tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each set of captured images may be fed into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0090] At 610, if the tracking status is yes for the currently captured image, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as shown in 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. If they are not successfully detected, the method returns to element 610 and processes the next image of the user's eyes.

[0091] At 640, if the process proceeds from element 610, the current frame is analyzed and the pupil and glint are tracked, based in part on prior information from the previous frame. At 640, if the process proceeds from element 630, the tracking state is initialized based on the detected pupil and glint in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results may be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are unreliable, the tracking state is set to no at element 660, and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze.

[0092] Figure 6A is intended to serve as an example of an eye-tracking technology that may be used in a particular implementation. As will be recognized by those skilled in the art, other eye-tracking technologies that currently exist or may be developed in the future may be used in computer system 101 to provide users with XR experiences in various embodiments, either in place of or in combination with the Glint-assisted eye-tracking technology described herein.

[0093] In some embodiments, the captured portion of the real-world environment 602 is used to provide the user with an XR experience, for example, a mixed reality environment in which one or more virtual objects are superimposed on a representation of the real-world environment 602.

[0094] Figure 6B shows an exemplary environment for an electronic device 101 to provide an XR experience, according to several embodiments. In Figure 6B, the real-world environment 602 includes the electronic device 101, a user 608, and real-world objects (e.g., a table 604). As shown in Figure 6B, the electronic device 101 is optionally mounted on a tripod or otherwise fixed to the real-world environment 602 such that one or more of the user 608's hands are free (e.g., the user 608 is not optionally holding the device 101 with one or more hands). As described above, the device 101 optionally has one or more groups of sensors located on different sides of the device 101. For example, the device 101 optionally includes sensor groups 612-1 and 612-2 located on the "rear" and "front" sides of the device 101, respectively (e.g., information can be captured from each side of the device 101). As used herein, the front side of device 101 is the side facing user 608, and the rear side of device 101 is the side facing away from user 608.

[0095] In some embodiments, the sensor group 612-2 includes an eye-tracking unit (e.g., the eye-tracking unit 245 described above with reference to Figure 2) which includes one or more sensors for tracking the user's eyes and / or gaze, and the eye-tracking unit can "look" at user 608 and track user 608's eyes (one or more) in the manner described above. In some embodiments, the eye-tracking unit of device 101 can capture the movement, orientation, and / or gaze of user 608's eyes and process the movement, orientation, and / or gaze as input.

[0096] In some embodiments, the sensor group 612-1 includes a hand tracking unit (e.g., the hand tracking unit 243 described above with reference to Figure 2) that can track one or more hands of user 608 held on the "rear" side of device 101, as shown in Figure 6B. In some embodiments, a hand tracking unit is optionally included in sensor group 612-2 so that user 608 can additionally or alternatively hold one or more hands on the "front" side of device 101 while device 101 tracks the position of one or more hands. As described above, the hand tracking unit of device 101 can capture the movement, position, and / or gestures of one or more hands of user 608 and process the movement, position, and / or gestures as input.

[0097] In some embodiments, the sensor group 612-1 optionally includes one or more sensors (e.g., the image sensor 404 described above with reference to Figure 4) configured to capture images of the real-world environment 602, including the table 604. As described above, the device 101 can capture images of a portion (e.g., part or all) of the real-world environment 602 and present the captured portion of the real-world environment 602 to the user via one or more display generation components of the device 101 (e.g., the display of the device 101 optionally located on the user-facing side of the device 101, opposite to the side of the device 101 facing the captured portion of the real-world environment 602).

[0098] In some embodiments, the captured portion of the real-world environment 602 is used to provide the user with an XR experience, for example, a mixed reality environment in which one or more virtual objects are superimposed on a representation of the real-world environment 602.

[0099] Accordingly, this description describes several embodiments of three-dimensional environments (e.g., XR environments) that include representations of real-world objects and virtual objects. For example, a three-dimensional environment optionally includes a representation of a table existing in a physical environment, which is captured and displayed within the three-dimensional environment (e.g., actively via a computer system's camera and display, or passively via a computer system's transparent or translucent display). As described above, a three-dimensional environment optionally is a mixed reality system based on a physical environment, in which the three-dimensional environment is captured by one or more sensors of a computer system and displayed via a display generation component. As a mixed reality system, the computer system may optionally selectively display parts and / or objects of the physical environment so that individual parts and / or objects of the physical environment appear to exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system may optionally display virtual objects in a three-dimensional environment so that the virtual objects appear to exist in the real world (e.g., a physical environment) by placing virtual objects in each location within the three-dimensional environment that have corresponding locations in the real world. For example, a computer system may optionally display a vase in such a way that it appears as if a real vase were placed on a table in a physical environment. In some embodiments, individual locations in a three-dimensional environment have corresponding locations in the physical environment.Therefore, when a computer system is described as displaying virtual objects in separate locations relative to physical objects (for example, at or near the location of the user's hand, or on or near a physical table), the computer system displays the virtual objects in specific locations within a three-dimensional environment so that they appear to be at or near physical objects in the physical world (for example, if the virtual object were a real object at that specific location, the virtual object would be displayed in the location within the three-dimensional environment that corresponds to the location within the physical environment where the virtual object is displayed).

[0100] In some embodiments, real-world objects existing in a physical environment displayed within a three-dimensional environment (e.g., real-world objects that can be seen via and / or display generation components) can interact with virtual objects that exist only within the three-dimensional environment. For example, the three-dimensional environment may include a table and a vase placed on the table, where the table is a view (or representation) of a physical table in the physical environment, and the vase is a virtual object.

[0101] Similarly, just as virtual objects are real objects in a physical environment, the user can optionally interact with virtual objects in a three-dimensional environment using one or more hands. For example, as described above, one or more sensors in the computer system can optionally capture one or more of the user's hands and display a representation of the user's hands in a three-dimensional environment (in a similar manner to, for example, displaying real-world objects in a three-dimensional environment as described above), or, in some embodiments, the user's hands can be seen through the display-generating component, due to the ability to see the physical environment through the user interface, due to the transparency / transparency of part of the display-generating component displaying the user interface, or the projection of the user interface onto a transparent / translucent surface, or the projection of the user interface onto the user's eyes or field of view of the user's eyes. Thus, in some embodiments, the user's hands are displayed at separate locations in the three-dimensional environment and are treated as if they were objects in a three-dimensional environment that can interact with virtual objects in the three-dimensional environment as if they were actual physical objects in the physical environment. In some embodiments, the computer system can update the display of the representation of the user's hands in the three-dimensional environment in conjunction with the movement of the user's hands in the physical environment.

[0102] In some of the embodiments described below, for example, to determine whether a physical object is directly interacting with a virtual object (e.g., whether a hand is touching, grasping, or holding a virtual object, or whether it is within a threshold distance from the virtual object), the computer system may optionally determine the "effective" distance between the physical object in the physical world and the virtual object in the three-dimensional environment. For example, a hand directly interacting with a virtual object may optionally include one or more of the fingers of a hand pressing a virtual button, a user's hand grasping a virtual vase, two fingers of a user's hand pinching / holding an application's user interface together, and other types of interactions described herein. For example, when determining whether a user is interacting with a virtual object and / or how a user is interacting with a virtual object, the computer system may optionally determine the distance between the user's hand and the virtual object. In some embodiments, the computer system determines the distance between the user's hand and the virtual object by determining the distance between the location of the hand in the three-dimensional environment and the location of the virtual object of interest in the three-dimensional environment. For example, one or more of the user's hands are located in a specific position in the physical world, which the computer system optionally captures and displays at a specific corresponding position in a three-dimensional environment (e.g., the position in the three-dimensional environment where the hands are displayed, if the hands are virtual hands rather than physical hands). The position of the hands in the three-dimensional environment is optionally compared to the position of a target virtual object in the three-dimensional environment to determine the distance between the one or more of the user's hands and the virtual object. In some embodiments, the computer system optionally determines the distance between the physical object and the virtual object by comparing the position in the physical world (as opposed to comparing the position in the three-dimensional environment).For example, when determining the distance between one or more of the user's hands and a virtual object, the computer system optionally determines the corresponding location of the virtual object in the physical world (e.g., the position in the physical world where the virtual object is located if it is a physical object rather than a virtual object), and then determines the distance between the corresponding physical position and one or more of the user's hands. In some embodiments, the same technique is optionally used to determine the distance between any physical object and any virtual object. Thus, when determining whether a physical object is in contact with a virtual object, or whether a physical object is within a threshold distance of a virtual object, as described herein, the computer system optionally performs one of the techniques described above to map the location of the physical object to a three-dimensional environment and / or to map the location of the virtual object to a physical environment.

[0103] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed, and / or where and what the physical stylus held by the user is directed. For example, if the user's gaze is directed to a particular position in the physical environment, the computer system optionally determines the corresponding position in the three-dimensional environment (e.g., the virtual position of the gaze), and if a virtual object is located at that corresponding virtual position, the computer system optionally determines that the user's gaze is directed to that virtual object. Similarly, the computer system optionally determines, based on the orientation of the physical stylus, where in the physical environment the stylus is pointing. In some embodiments, based on this determination, the computer system optionally determines the corresponding virtual position in the three-dimensional environment corresponding to the location in the physical environment that the stylus is pointing to, and optionally determines that the stylus is pointing to the corresponding virtual position in the three-dimensional environment.

[0104] Similarly, embodiments described herein may refer to the location of a user (e.g., a user of a computer system) and / or the location of a computer system in a three-dimensional environment. In some embodiments, the user of a computer system is holding, wearing, or otherwise positioned near the computer system. Thus, in some embodiments, the location of the computer system is used as a proxy for the user's location. In some embodiments, the location of the computer system and / or the user in the physical environment corresponds to individual locations in the three-dimensional environment. For example, if a user stands at a location facing an individual part of the physical environment displayed by a display-generating component, the location of the computer system is the location in the physical environment (and its corresponding location in the three-dimensional environment) where the user will see objects in the physical environment in the same position, orientation, and / or size (e.g., absolutely and / or relative to each other) as the objects are displayed by the display-generating component of the computer system in the three-dimensional environment. Similarly, if a virtual object displayed in a three-dimensional environment is a physical object in a physical environment (for example, the virtual object is located in the same physical environment location as it is in the three-dimensional environment, and has the same size and orientation as it does in the three-dimensional environment), then the computer system and / or user's location is the position from which the user views the virtual object in the physical environment in the same position, orientation, and / or size (for example, absolutely, and / or relative to each other, and in relation to real-world objects) as it was displayed by the computer system's display generation component in the three-dimensional environment.

[0105] This disclosure describes various input methods for interaction with computer systems. Where one example is provided using one input device or method, and another example is provided using a different input device or method, each example may be compatible with the input device or method described in the other example, and their use should be considered optional. Similarly, various output methods for interaction with computer systems are described. Where one example is provided using one output device or method, and another example is provided using a different output device or method, each example may be compatible with the output device or method described in the other example, and their use should be considered optional. Similarly, various methods for interaction with virtual or mixed reality environments via computer systems are described. Where one example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, each example may be compatible with the method described in the other example, and their use should be considered optional. Therefore, this disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment. User interface and related processes

[0106] Here, we focus on embodiments of a user interface ("UI") and related processes that may be performed in a computer system such as a portable multifunction device or head-mounted device, which includes a display generation component, one or more input devices, and (optionally) one or more cameras.

[0107] Figures 7A to 7I illustrate examples of how an electronic device, in several embodiments, can simultaneously present a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location.

[0108] Figure 7A shows an electronic device 101 that displays a three-dimensional environment via a display generation component 120. In some embodiments, it should be understood that the electronic device 101 utilizes one or more techniques described with reference to Figures 7A to 7I in a two-dimensional environment or user interface without departing from the scope of this disclosure. As described above with reference to Figures 1 to 6, the electronic device 101 optionally includes a display generation component 120 (e.g., a touchscreen) and a plurality of image sensors 314. The image sensors optionally include one or more of a visible light camera, an infrared camera, a depth sensor, or any other sensor, and the electronic device 101 can be used to capture one or more images of the user or a part of the user while the user is interacting with the electronic device 101. In some embodiments, the display generation component 120 is a touchscreen that can detect the user's hand gestures and movements. In some embodiments, the user interface described below can also be implemented in a head-mounted display that includes a display generation component that displays the user interface to the user, and sensors that detect the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward from the user) and / or the user's line of sight (e.g., internal sensors facing inward towards the user's face).

[0109] In Figure 7A, the electronic device 101 presents a three-dimensional environment including a navigation user interface element 704a and a content user interface element 706. The three-dimensional environment further includes representations of real objects within the physical environment of the electronic device 101, such as a representation of a real table 702, a representation of a floor 722 722', and a representation of a wall 724 724'. The physical environment of the electronic device 101 further includes an entrance / exit 726.

[0110] As described above, the three-dimensional environment presented by the electronic device 101 includes a navigation user interface element 704a displayed between the content user interface element 706 and the user's viewpoint in the three-dimensional environment. In some embodiments, the user's viewpoint corresponds to a location within the three-dimensional environment presented by the electronic device 101. The navigation user interface element 704a includes representations 710a-c of physical objects (e.g., buildings) and representations 714 of roads located within the physical area represented by the navigation user interface element 704a. In some embodiments, the navigation user interface element 704a further includes representations of other physical objects such as infrastructure, landmarks, terrain, and vegetation. In some embodiments, the navigation user interface element 704a is a (at least partial) three-dimensional map of the physical area represented by the map.

[0111] The navigation user interface element 704a further includes a weather indication 712 (e.g., clouds). The weather indication 712 shows the weather conditions currently experienced at the physical location corresponding to the location of the indication 712 within the navigation user interface element 704a. For example, if the physical clouds represented by the indication 712 move, the electronic device 101 updates the position of the indication 712 according to the movement of the physical clouds. As another example, if the weather at the physical location corresponding to the location of the indication 712 within the navigation user interface element 704a becomes rainy, the electronic device 101 updates the indication 712 (or more generally the navigation user interface element 704a) to include a representation of rain. As shown in Figure 7A, the navigation user interface element 704a shows the terrain of the physical area represented by the navigation user interface element 704a (for example, by displaying a cross-section of such terrain (e.g., elevation changes) in the edge and / or internal area of ​​the navigation user interface element 704a).

[0112] The navigation user interface element 704a further includes an indication 716 of a specific location corresponding to content 708a presented within the content user interface element 706, and a field of view indicator 718a that shows the field of view corresponding to content 708a. For example, content 708a is an image (or video) captured from a physical location corresponding to the location of the indication 716 within the navigation user interface element 704a, having a boundary corresponding to the field of view indicator 718a. As shown in Figure 7A, the electronic device 101 presents the navigation user interface element 704a in such a way that the navigation user interface element appears to be placed on the surface of a representation 702 of a real table in the physical environment of the electronic device 101. In some embodiments, the electronic device 101 displays the navigation user interface element 704a in a location within a three-dimensional environment between the user's viewpoint and a content user interface element 706 that is detached from both the user's viewpoint and actual (e.g., horizontal) surface representations within the three-dimensional environment (e.g., floating in space within the three-dimensional environment and not connected to surfaces within the three-dimensional environment), rather than displaying the navigation user interface element 704a in such a way that it appears to rest on a surface representation within the physical environment of the electronic device 101.

[0113] As shown in Figure 7A, the electronic device 101, when selected, presents the electronic device 101 with selectable options 720a and 720b to adjust the field of view of the content within the content user interface element 706 by translating and / or rotating the location (or location) where the content was captured. In some embodiments, in response to input for option 720a or 720b, the electronic device 101 updates the content within the content user interface element 706 and updates the position and / or orientation of the indication 716 and field of view indicator 718a within the navigation user interface element 704a, as will be described in more detail below with reference to Figures 7C-7D.

[0114] In some embodiments, the navigation user interface element 704a further includes one or more indications of the physical location(s) of one or more other electronic devices communicating with the electronic device 101. For example, the navigation user interface element 704a includes an indication of a second electronic device associated with a user other than the user of the electronic device 101 (e.g., a contact in the electronic device user's address book). In some embodiments, upon detecting a selection of an indication of a second electronic device, the electronic device 101 presents an image captured by the second electronic device within the content user interface element 706. For example, the image is a live video feed from a camera communicating with the second electronic device.

[0115] In some embodiments, the navigation user interface element 704a includes a two-dimensional portion in addition to a three-dimensional portion similar to the navigation user interface element 704a shown in Figure 7A. In some embodiments, the two-dimensional portion 704a of the navigation user interface element is displayed behind the three-dimensional portion of the navigation user interface element from the user's perspective. For example, the navigation user interface element 704a is folded upward and / or curved at its rear (e.g., the portion of the navigation user interface element 704a closer to the content user interface element 706), thereby displaying the two-dimensional portion of the navigation user interface element 704a at a location in the three-dimensional environment where the content user interface element 706 is displayed in Figure 7A. In some embodiments, the physical location represented by the three-dimensional portion of the navigation user interface element 704a and the physical location represented by the two-dimensional portion of the navigation user interface element 704a are adjacent to each other in the real world. In some embodiments, in response to user input for panning the navigation user interface element 704a, the electronic device pans the navigation user interface element 704a in accordance with the user input, including, as appropriate, moving one or more representations from the three-dimensional portion of the navigation user interface element 704a to the two-dimensional portion of the navigation user interface element 704a and vice versa.

[0116] As described above, the content user interface element 706 includes content 708a corresponding to a physical location represented by an indication 716 within the navigation user interface element 704a. In some embodiments, the content 708a is an image captured at the location represented by the indication 716, or additionally / alternatively, an image of the location indicated by the indication 716. In some embodiments, the edges of the content user interface element 706 are feathered against the rest of the three-dimensional environment. In some embodiments, the boundaries of the content user interface element are blurred or blended with the rest of the three-dimensional environment surrounding the content user interface element 706.

[0117] In Figure 7B, the electronic device 101 detects user input corresponding to a request to update the content user interface element 706 to present content corresponding to a physical location different from the physical location specified in Figure 7A. Detecting input optionally includes simultaneously detecting the user's gaze 701d directed towards a navigation user interface element 704b (e.g., an indication 716 within it) and detecting a predetermined gesture performed by the user's hand 728a. In some embodiments, the predetermined gesture is a pinch gesture in which the user moves the thumb of hand 728a to another finger of hand. In some embodiments, detecting a pinch gesture causes the electronic device 101 to "pick up" the indication 716, and as a result, the user can move the indication 716 in accordance with the movement of hand 728a while the user continues to touch the other finger of hand 728a with their thumb. In some embodiments, in response to detecting that the user has released their thumb from the other finger of hand 728a, the electronic device 101 "drops" the indication 716 to the location where the indication 716 was displayed when the electronic device 101 detected that the user had released their thumb from the finger.

[0118] While detecting the input shown in Figure 7B, the electronic device 101 visually de-emphasizes the content user interface element 706 (e.g., including the content 708b within the content user interface element 706) with respect to the amount of visual emphasis that the content user interface element 706 was displayed before the input was detected (e.g., in Figure 7A). In some embodiments, visually de-emphasizing the content user interface element 706 includes blurring, fading, darkening, and / or increasing the translucency of the content user interface element 706. In some embodiments, the entire content user interface element 706 is de-emphasized. In some embodiments, only the edges of the content user interface element 706 are de-emphasized, and a portion of the content user interface element 706 at its center is not visually de-emphasized. In some embodiments, while the input is detected, the electronic device 101 displays one or more images corresponding to intermediate locations of the indication 716, along with the visual de-emphasis within the content user interface element 706.

[0119] In some embodiments, while the input in Figure 7B is detected, the electronic device 101 modifies the appearance of the representation (e.g., representation 710c, etc.) of a physical object within the navigation user interface element 704b (e.g., a building, infrastructure, plants, or other physical object within the area represented by the navigation user interface element 704b). Modifying the appearance of the representation of a physical object within the navigation user interface element 704b optionally includes one or more of fading the three-dimensional representation and / or reducing the height of the representation or flattening the representation. In some embodiments, the electronic device 101 blurs, fades, flattens, or stops the display of one or more representations at the location of the user's hand 728a (or within a threshold distance such as 1, 2, 5, 10, 20 inches, etc.). For example, in Figure 7B, the electronic device 101 stops displaying the representations 710a and 710b shown in Figure 7A because these representations 710a and 710b were displayed at the location in Figure 7A where hand 728a is located in Figure 7B. In some embodiments, the electronic device 101 pans the navigation user interface element 704b when the user provides input and stops displaying one or more representations of physical objects intersected by the boundary of the navigation user interface element 704b (e.g., representations that do not "fit" within the boundary of the navigation user interface element 704b).

[0120] In response to the input shown in Figure 7B, the electronic device 101 updates the three-dimensional environment to include content within the content user interface element 706 that corresponds to the updated location of the indication 716 in the navigation user interface element 704a, as shown in Figure 7C. In Figure 7C, the location of the indication 716 and the field of view indicator 713c are updated within the navigation user interface element 704a according to the input shown in Figure 7B, and the content user interface element 706 is updated to include content 708c that corresponds to the location of the indication 716 and the field of view indicator 718c within the navigation user interface element 704a. The locations of the content user interface element 706 and the navigation user interface element 704b in the three-dimensional environment remain the same as shown in Figure 7A before the input is detected in Figure 7B and after the input is detected in Figure 7B, as shown in Figure 7C.

[0121] As shown in Figure 7C, the amount of visual de-emphasis of the content user interface element 706 is reduced relative to the amount of de-emphasis of the content user interface element 706 in Figure 7C while input is being detected. In some embodiments, as shown in Figure 7C, the amount of de-emphasis (or lack thereof) after receiving input in Figure 7B is the same as the amount of de-emphasis of the content user interface 706 in Figure 7A before detecting input in Figure 7B (e.g., no visual de-emphasis). In some embodiments, the amount of de-emphasis before and after receiving input in Figure 7B is different.

[0122] The electronic device 101 also displays the representation in the navigation user interface element 704a in Figure 7C with less de-emphasis than the amount of de-emphasis when the representation in the navigation user interface element 704a was displayed in Figure 7B while input was being detected. For example, if a representation (e.g., of a building or other physical structure or feature) is flattened, blurred, darkened, and / or faded in Figure 7B, the representation will be displayed higher, more clearly, and / or brighter in Figure 7C after the input in Figure 7B is no longer detected. In some embodiments, the electronic device 101 displays the navigation user interface element 704a with the same degree of visual clarity / striking as in Figure 7A before receiving the input in Figure 7B as in Figure 7C after receiving the input in Figure 7B. In some embodiments, the degree of visual clarity / striking of the navigation user interface element 704a in Figure 7A and Figure 7C is different.

[0123] In Figure 7C, the electronic device 101 detects input directed towards the selectable option 720b and updates the location corresponding to the content within the content user interface element 706. Detecting input optionally includes simultaneously detecting the user's gaze 701a directed towards the selectable option 720b and detecting that the user has made a pinch gesture with their hand 728b (e.g., moving the thumb to touch another finger on the hand, and then lifting the thumb away from the finger). In response to the input shown in Figure 7C, the electronic device 101 updates the locations of the indication 716 and field of view indicator 718c within the navigation user interface element 704a, as shown in Figure 7D, and updates the content user interface element 706 to include content corresponding to the updated positions / orientations of the indication 716 and field of view indicator 718c.

[0124] Figure 7D shows the three-dimensional environment updated in response to the input detected in Figure 7C. Electronic device 101 displays the indication 716 and field of view indicator 718d in the updated position within the navigation user interface element 704a according to the input detected in Figure 7C. Since input in Figure 7C has been detected to option 720b for panning the content in the content user interface element 706 to the right, electronic device 101 moves the indication 716 and field of view 718d in the navigation user interface element 704a to the right. Electronic device 101 also updates the content user interface element 706 to include content 708d corresponding to the updated position of the indication 716 and field of view indicator 718d.

[0125] In some embodiments, the electronic device 101 increases the size of the content user interface element 706, as shown in Figure 7E, in response to detecting the user's gaze 701b directed towards the content user interface element 706 for at least a predetermined time threshold (e.g., 0.5, 1, 2, 3, 5, 10 seconds, etc.), as shown in Figure 7D. Additionally or alternatively, in some embodiments, the electronic device 101 increases the size of the content user interface element 706, as shown in Figure 7E, in response to the operation of a physical button 703a on the electronic device 101 by a user 728c, as shown in Figure 7D. In some embodiments, the physical buttons 703a and 703b on the electronic device 101 control the level of immersion in the virtual content of the three-dimensional environment. For example, while displaying a three-dimensional environment with a relatively high level of immersion, the electronic device increases the visual prominence (e.g., size, brightness, clarity, opacity, etc.) of virtual objects in the three-dimensional environment (e.g., content user interface element 706, navigation user interface element 704a, content, application user interface elements, and other virtual objects) and decreases the visual prominence of representations of real objects in the three-dimensional environment (e.g., representation 724' of wall 724, representation 722' of floor 722, and representation 702 of table). As another example, while displaying a three-dimensional environment at a relatively low level of immersion, the electronic device reduces the visual prominence (e.g., size, brightness, clarity, opacity, etc.) of virtual objects in the three-dimensional environment (e.g., content user interface element 706, navigation user interface element 704a, content, application user interface elements, and other virtual objects) and increases the visual prominence of representations of real objects in the three-dimensional environment (e.g., representation 724' of wall 724, representation 722' of floor 722, and representation 702 of table).

[0126] Figure 7E shows a three-dimensional environment updated in response to one or more of the inputs shown in Figure 7D. As shown in Figure 7E, the size of the content user interface element 706 in Figure 7E is larger than the size of the content user interface element 706 displayed in Figure 7D before receiving input(s) to enlarge the content user interface element 706. Updating the content user interface element 706 in Figure 7E involves increasing the amount of content 708d corresponding to the location of the indicator 716 displayed on the content user interface element 706. The electronic device 101 also updates the field of view indicator 718e within the navigation user interface element 704a to reflect the updated width of the physical area shown within the content 708d. In some embodiments, instead of increasing the amount of content displayed (e.g., updating the field of view 718e), the electronic device 101 instead enlarges the content 708d shown in Figure 7D and maintains the same field of view 718d in Figure 7D in response to input(s) in Figure 7D.

[0127] In some embodiments, the electronic device 101 updates the content user interface element 706 in accordance with the movement of the electronic device 101 and / or the user's head. For example, in response to detecting the movement of the electronic device 101 and / or the user's head, the electronic device 101 shifts the viewpoint from which the content 708d in the content user interface element 706 is displayed. In some embodiments, the electronic device shifts the content 708d using a parallax effect. In some embodiments, the electronic device moves and / or zooms the content user interface element 706 in accordance with the movement of the electronic device 101. Figure 7E shows user 728d moving the electronic device 101 to the left. Figure 7F shows an exemplary way in which the electronic device 101 updates the content user interface element 706 in accordance with the movement of the electronic device 101 shown in Figure 7E.

[0128] In Figure 7F, the electronic device 101 displays a three-dimensional environment from the user's viewpoint, updated in accordance with the movement of the electronic device 101 in Figure 7E. As user 728d moves the electronic device 101 to the left in Figure 7E, the electronic device 101 expands the content 708d within the content user interface element 706 to the left and updates the field of view indicator 718e (e.g., expands) in accordance with the (e.g., larger) field of view of the content 708d. Expanding the content 708d includes presenting the content, which includes real-world objects, to the left of the real-world objects shown in the content user interface element 706 in Figure 7E. In some embodiments, the electronic device 101 also expands the content user interface element 706 to the right, up, and / or down in response to detecting movement of the electronic device 101 (and / or the user's head) to the right, up, and / or down, respectively. In some embodiments, the electronic device 101 expands the width of the content user interface element 706 in accordance with the movement of the electronic device 101. For example, in response to detecting a leftward movement of the electronic device 101, the electronic device 101 expands the content user interface element 706 (and therefore the content 708d displayed within the content user interface element 706) to the left in the three-dimensional environment.

[0129] As shown in Figure 7F, the electronic device 101 stops displaying a portion of the navigation user interface element 704a that is presented on a portion of the table representation 702 that is no longer within the field of view of the electronic device 101's display generation component 120. In some embodiments, the electronic device 101 does not update the position of the navigation user interface element 704a in response to the zooming of the content user interface element 706 in response to the movement of the electronic device 101 and / or the user's head. In some embodiments, if the entire navigation user interface element 704a is still within the field of view of the display generation component 120 during / after the detection of movement of the electronic device 101 and / or the user's head, the electronic device 101 continues to display the entire navigation user interface element 704a. For example, if the entire table representation 702 is still within the field of view of the electronic device 101's display generation component 120 in Figure 7F, the electronic device 101 continues to present the entire navigation user interface element 704a at the same location in the three-dimensional environment where the navigation user interface element 704a was displayed before the detection of movement of the electronic device 101 and / or the user's head. In some embodiments, moving the electronic device 101 and / or the user's head updates the user's viewpoint position in the three-dimensional environment, and therefore, displaying the navigation user interface element 704a at a specific location in the three-dimensional environment includes changing the portion of the display generation component 120 that presents the map navigation user interface element 704a in some situations.

[0130] In some embodiments, upon determining that the user is viewing a representation of a real object in a three-dimensional environment (for example, while the device 101 is moving), the electronic device 101 stops expanding the content user interface element 706 to prevent obscuring the representation of the real object that the user is viewing. In Figure 7F, the electronic device 101 detects the user's line of sight 701c directed towards the representation of an entrance / exit 726' in the physical environment of the electronic device 101, and accordingly expands the content user interface element 706 only to the boundary of the entrance / exit, and does not expand the content user interface element 706 to obscure the representation of the entrance / exit 726'. In some embodiments, if the electronic device 101 does not detect the user's line of sight 701c directed towards the representation of an entrance / exit 726' (for example, while the device 101 is moving), the electronic device 101 expands the content user interface element 706 to obscure (at least a portion of) the representation of the entrance / exit 726'.

[0131] In some embodiments, the electronic device 101 facilitates the retrieval of physical locations corresponding to search queries provided by the user. For example, the user can search for specific locations (e.g., addresses, building names, companies, landmarks, and other locations) or types of locations (e.g., types of companies (e.g., coffee shops, gas stations, grocery stores), landmarks, and other locations). Figure 7G shows the electronic device 101 presenting search results in a three-dimensional environment.

[0132] In Figure 7G, the electronic device 101 presents a text input field 730 (or more generally, a search field) into which the user has entered the search query "City Hall Building". The electronic device 101 also presents an indication 732a of the name of the search result, the distance of the search result's location from the electronic device's current location, and an image 732b of the search result. In addition, the electronic device 101 presents a (e.g., three-dimensional) rendering 708e of the physical object corresponding to the search result (e.g., a three-dimensional representation of the searched City Hall Building). In some embodiments, the rendering 708e of the physical object corresponding to the search result is presented before the image 732b of the physical object corresponding to the search result, and / or in a position in the three-dimensional environment closer to the user's viewpoint than the image 732b is at the user's viewpoint. As shown in Figure 7G, the electronic device 101 presents a navigation user interface element 704b between the user's viewpoint and the rendering 732b. The electronic device 101 presents a navigation user interface element between the user's viewpoint and the image 732b of the physical object corresponding to the search result.

[0133] In Figure 7G, upon detecting a search query (e.g., "City Hall"), the electronic device 101 updates the navigation user interface element 704b to display the search results (one or more). In Figure 7G, the electronic device 101 displays representations of physical locations / objects / features that do not match the search query (e.g., representation 710e) with reduced visual saturation (e.g., at least partially flattening the three-dimensional representation of the physical object in the navigation user interface element 704b), while maintaining the visual saturation of representations 710d of objects that do match the search query (e.g., buildings), and / or displaying representation 710d in a different color than the representations of other objects (e.g., representation 710e). In some embodiments, the electronic device 101 either updates the color of representation 710d of objects that match the search query, or reduces the visual saturation of representations that do not match the search query (e.g., representation 710e), but not both. In some embodiments, if two or more locations match the search query, the electronic device 101 displays multiple representations of the locations that match the search query with different visual distinctions (e.g., color, visual striability) than the visual distinctions used for representations of locations that do not match the search query. In some embodiments, the representation 710d of a physical object corresponding to the search query is a three-dimensional representation of the physical object. For example, representation 710d is a three-dimensional rendering of the physical object presented from a different viewing angle than the viewing angle of rendering 708e. In some embodiments, the electronic device 101 presents the rendering 708e and representation 710d of the physical object to the navigation user interface element 704b from the same viewing angle.

[0134] In some embodiments, the electronic device 101 presents a navigation direction from one physical location to another in a three-dimensional environment. In some embodiments, upon request for a navigation direction to be presented, the electronic device 101 presents selectable options, which, if selected, cause the electronic device 101 to present an animation navigating the navigation direction. Figure 7H shows the electronic device 101 presenting an animation navigating the navigation direction (for example, depending on the selection of one of the selectable options).

[0135] In Figure 7H, the electronic device 101 presents a navigation direction indication 734 that includes a start and end point, route distance and / or route duration, a navigation user interface element 704c, and one or more images 708f (e.g., one or more still images, videos, animations) corresponding to the navigation route. The navigation user interface element 704c includes a navigation route indication 740, an indication of the route's end point 738, and a location indication 736 corresponding to the image 708f currently presented in the content user interface element 706. In some embodiments, the navigation user interface element 704c is three-dimensional. In some embodiments, the navigation user interface element 704c is two-dimensional. In some embodiments, the navigation user interface element 704c is a bird's-eye view (e.g., top-down view). In some embodiments, the navigation user interface element is presented from an oblique view. The navigation user interface element 704c is presented between the user's viewpoint and the image 708f corresponding to the navigation route. Navigation user interface element 704c is presented between the user's viewpoint and the navigation direction indication 734. In some embodiments, the electronic device 101 presents an animation of an indication 736 navigating along a navigation path 740, and as the indication 736 moves along the path 740, the electronic device 101 presents an image 708f or a portion of a video or animation corresponding to the current location represented by the indication 736. For example, image 708f is a first-person video / view navigating the navigation path, presented by the electronic device 101 simultaneously with the animation of the indication 736 traversing the navigation path 740.

[0136] In some embodiments, the electronic device 101 selects the size of the navigation user interface element based on the size of the physical environment of the electronic device 101 (e.g., one or more of its features). For example, in Figures 7A to 7E, the electronic device 101 displays the navigation user interface element 704a in a way that fits the area of ​​the representation 702 on the table surface in the physical environment of the electronic device 101 (e.g., so that the device 101 displays the navigation user interface element 704a on the table surface). In some embodiments, if the table surface is smaller, the navigation user interface element 704a is displayed in a smaller size by including a representation of a smaller physical area or by reducing the representation of a physical area of ​​the same size. In some embodiments, if the table surface is larger, the navigation user interface element 704a is displayed in a larger size by including a representation of a larger physical area or by enlarging the representation of a physical area of ​​the same size. In some embodiments, if a navigation user interface element is presented on a different surface / feature in a three-dimensional environment, or away from any surface / feature, the electronic device 101 sizes (e.g., and shapes) the navigation user interface element based on the surface / feature and / or area on which the navigation user interface element is displayed.

[0137] In Figure 7I, the physical environment of the electronic device 101 is smaller than the physical environments in Figures 7A to 7E. In particular, Figure 7I shows the electronic device 101 used within the cubicle 726. In Figure 7I, the navigation user interface element 704d is displayed within the representation 726' of the cubicle 726 at a smaller size than the navigation user interface element 704a is displayed in Figures 7A to 7E. This is because the cubicle 726 is smaller than the table corresponding to the virtual table 702. As shown in Figure 7I, the electronic device 101 displays the navigation user interface element 704d between the user's viewpoint and the content user interface element 706 in the three-dimensional environment.

[0138] Additional or alternative details relating to the embodiments shown in Figures 7A to 7I are provided below in the description of methods 800 to 900, which are described with reference to Figures 8 to 9 below.

[0139] Figures 8A to 8J are flowcharts illustrating methods, according to several embodiments, for simultaneously presenting a navigation user interface element having a designated individual physical location and a content element containing content corresponding to that individual physical location. In some embodiments, Method 800 is executed on a computer system (e.g., computer system 101 in Figure 1) which includes a display generation component (e.g., display generation component 120 in Figures 1, 3, and 4) (e.g., a head-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., a camera facing downward in the user's hand (e.g., a color sensor, an infrared sensor, and other depth-sensing cameras) or a camera facing forward from the user's head). In some embodiments, Method 800 is stored on a non-temporary computer-readable storage medium and controlled by instructions executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 in Figure 1A). Some operations of Method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0140] In some embodiments, Method 800 is performed in an electronic device that communicates with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer). In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touchscreen display), an external display such as a monitor, projector, or television, or a hardware component (optionally built-in or external) for projecting a user interface and making the user interface visible to one or more users. In some embodiments, the one or more input devices include an electronic device or component that can receive user input (e.g., capture user input, detect user input, etc.) and transmit information related to the user input to the electronic device. Examples of input devices include touchscreens, mice (e.g., external), trackpads (optionally integrated or external), touchpads (optionally integrated or external), remote control devices (e.g., external), another mobile device (e.g., separate from the electronic device), handheld devices (e.g., external), controllers (e.g., external), cameras, depth sensors, eye-tracking devices, and / or motion sensors (e.g., hand-tracking devices, hand motion sensors). In some embodiments, the electronic device communicates with the hand-tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touchscreen, trackpad)). In some embodiments, the hand-tracking device is a wearable device such as a smart glove. In some embodiments, the hand-tracking device is a handheld input device such as a remote control or stylus.

[0141] In some embodiments, such as those shown in Figure 7A, an electronic device (e.g., 101) displays a three-dimensional environment including a user interface (802a) via a display generation component, which includes a first individual content (e.g., 708a) corresponding to a first view of a first physical location, and the first individual content (e.g., 708a) is displayed at the first individual location within the three-dimensional environment (802b). In some embodiments, the first individual content is an image taken from the first physical location and / or an image of the first physical location (e.g., a street-level view image from the first physical location). In some embodiments, the first individual content is (e.g., live) video recorded at the first physical location.

[0142] In some embodiments, such as those shown in Figure 7A, an electronic device (e.g., 101) displays a three-dimensional environment (802a) via a display generation component, which includes a user interface including a navigation user interface element (e.g., 704a) having a first location (e.g., 716) corresponding to a specified first physical location, and the navigation user interface element (e.g., 704a) is displayed between a first individual location in the three-dimensional environment where a first individual content (e.g., 708a) is displayed and the viewpoint of a user (e.g., a user of the electronic device) in the three-dimensional environment (802c). In some embodiments, the three-dimensional environment includes a representation of virtual objects including the user interface and a representation of real objects in the physical environment of the electronic device. The three-dimensional environment is optionally presented from a first-person viewpoint from the viewpoint of a user associated with the electronic device at an individual location in the three-dimensional environment (e.g., corresponding to a location of the electronic device). In some embodiments, the navigation user interface element includes a three-dimensional terrain map of the physical location having a visual indication (e.g., a pin) at the first location corresponding to the first physical location. For example, a navigation user interface element includes a three-dimensional topographic map of a city, including a three-dimensional representation of buildings, streets, and other landmarks having flags, pins, or other visual indications displayed at a first location corresponding to a city address, landmark, or coordinates. In some embodiments, the navigation user interface element is oriented along a horizontal plane in the three-dimensional environment or floats along a horizontal plane, and the first individual content is oriented vertically. In some embodiments, the map navigation element is located between the user's viewpoint and the first individual content in the three-dimensional environment. In some embodiments, the navigation user interface element and / or the first individual content are displayed in a three-dimensional environment (e.g., a computer-generated reality (XR) environment such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment) that is generated, displayed, or otherwise made visible by the device.

[0143] In some embodiments, such as Figure 7B, while displaying user interface and navigation user interface elements (e.g., 704b) via a display generation component, the electronic device detects user input via one or more input devices that corresponds to a request to specify a second location corresponding to a second physical location (802d). In some embodiments, the input is directed to a navigation user interface element, such as an input that moves the indication of the specified location from the first location to the second location on a three-dimensional terrain map. In some embodiments, the input is directed to a first individual content, such as an input that changes the viewpoint of the content from the first physical location to the second physical location, or rotates the field of view of the first individual content.

[0144] In some embodiments, such as Figure 7C, in response to detecting user input, the electronic device updates the user interface to include a second separate piece of content (e.g., 708c) corresponding to a second view of a second physical location (802e), which is then displayed at the first separate location in a three-dimensional environment. In some embodiments, in response to detecting user input, the electronic device further updates the navigation user interface elements to display a visual indication (e.g., a pin) at the second location and to stop displaying a visual indication (e.g., a pin) at the first location. In some embodiments, the second separate piece of content is an image taken from the second physical location. In some embodiments, the second separate piece of content is (e.g., live) video recorded at the second physical location. In some embodiments, the electronic device maintains the locations in a three-dimensional environment where the content and navigation user interface elements are displayed in response to user input.

[0145] The method described above, which displays a second individual piece of content at the same location within a three-dimensional environment where a first individual piece of content is displayed, provides an efficient way to view content corresponding to a physical location, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, without the user having to direct their attention to different areas of the three-dimensional environment or without having to provide input to keep the content displayed at the second individual location within the three-dimensional environment).

[0146] In some embodiments, the view or style of a navigation user interface element changes depending on the zoom level applied to the navigation user interface element. For example, when the zoom level is below a first threshold, the navigation user interface element is the Earth. As another example, when the zoom level is between the first threshold and a second threshold greater than the first threshold, the navigation user interface element is a topographic map of an area showing natural features (e.g., terrain, vegetation, etc.) and optionally does not show artificial features (e.g., buildings, infrastructure, etc.). As yet another example, when the zoom level is greater than the second threshold, the navigation user interface element includes a topographic map containing both natural and artificial features, and the electronic device displays first-person content from the physical location indicated by the navigation user interface element. In some embodiments, the electronic device refrains from displaying first-person content at a first individual location in a three-dimensional environment while the zoom level is below the second threshold.

[0147] In some embodiments, such as Figure 7A, the three-dimensional environment includes a representation of a surface (e.g., horizontal) in the physical environment of the electronic device (e.g., 702) (804a). In some embodiments, the surface representation is a representation of a surface presented by a display generation component (e.g., virtual or video passthrough). In some embodiments, the surface representation is a view of the surface through a transparent portion of the display generation component (e.g., true or actual passthrough). For example, the surface is a table, shelf, counter, or floor in the environment of the electronic device. In some embodiments, such as Figure 7A, a navigation user interface element (e.g., 704a) is displayed in a location (and orientation) in the three-dimensional environment corresponding to a representation of a surface (e.g., horizontal) (e.g., 702) (804b). In some embodiments, the electronic device displays the navigation user interface element as if it were placed on a surface (e.g., resting on one). For example, the electronic device displays the navigation user interface element as if it were placed on a table, shelf, counter, or floor in the physical environment of the electronic device between a first distinct location in the three-dimensional environment and the user's viewpoint.

[0148] The above method of displaying navigation user interface elements at locations corresponding to surface representations provides an efficient way to integrate virtual objects with representations of real-world objects in a mixed-reality environment without consuming unnecessary space in the mixed-reality environment. This further reduces power consumption and improves the battery life of electronic devices by allowing users to use electronic devices more quickly and efficiently (for example, by reducing the number of inputs required to integrate virtual and real-world objects in the environment).

[0149] In some embodiments, navigation user interface elements are displayed in locations (and / or orientations) within a three-dimensional environment that do not correspond to surfaces within the physical environment of the electronic device (806a), such as displaying the navigation user interface element 704a in Figure 7A away from the table representation 702. In some embodiments, surfaces correspond to real objects within the physical environment of the electronic device, such as tables, shelves, counters, or floors. In some embodiments, while displaying a three-dimensional environment that includes a surface representation, the electronic device displays the navigation user interface elements in locations away from the surface representation (e.g., floating in space, not attached to a physical or virtual object or surface). In some embodiments, the three-dimensional environment does not include a surface representation within the physical environment of the electronic device (e.g., because such surfaces do not exist within the physical environment of the electronic device, the surfaces are not within the electronic device's field of view, and the three-dimensional environment is a virtual environment that does not include a representation of one or more real objects within the physical environment of the electronic device). In some embodiments, the navigation user interface elements include indications of physical locations, virtual tours, and landmarks, which, when selected, cause the electronic device to present the content associated with the selected indication (for example, within the navigation user interface element and / or at a first separate location in the three-dimensional environment).

[0150] The above method of displaying navigation user interface elements at locations in a three-dimensional environment that do not correspond to surfaces in the physical environment of an electronic device provides flexibility in the placement of navigation user interface elements, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to display navigation user interface elements).

[0151] In some embodiments, such as Figure 7A, a first individual content (e.g., 708a) is surrounded (e.g., overlaid on) by a representation (e.g., 724') of the physical environment of an electronic device (e.g., 101), and the boundary between the first individual content (e.g., 708a) and the representation (e.g., 724') of the physical environment of the electronic device (e.g., 101) includes a gradual visual transition (e.g., feathering, gradual fade, blending) between the first individual content (e.g., 708a) and the representation (e.g., 724') of the physical environment of the electronic device (e.g., 724') (808a). In some embodiments, the representation of the physical environment is a representation of the physical environment presented by a display-generating component (e.g., virtual or video passthrough). In some embodiments, the representation of the physical environment is a view of the physical environment through a transparent portion of the display-generating component (e.g., true passthrough or actual passthrough). In some embodiments, the first individual content is surrounded (e.g., overlaid) by a representation of a virtual environment, and the boundary between the first individual content and the representation of the virtual environment includes a gradual visual transition between the first individual content and the representation of the virtual environment. In some embodiments, the gradual visual transition is a feathering boundary, a blended boundary, and / or a blurred boundary between the first individual content and the representation of a physical (e.g., or virtual) environment. In some embodiments, an electronic device displays the first individual content between the user's viewpoint in a three-dimensional environment and a representation of the physical environment that includes a portion of the representation of the physical environment surrounding the first individual content.

[0152] The above method of displaying the first individual content using a gradual visual transition between the first individual content and the representation of the physical environment maintains context for the user, which avoids incorrect input or the need for the user to switch back and forth between different user interfaces, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to view the representation of the physical environment and the first individual content).

[0153] In some embodiments, such as Figure 7D, a first individual content (e.g., 708c) corresponding to a first view of a first physical location is displayed simultaneously with a representation (e.g., 724') of the physical environment of an electronic device (e.g., 101) and occupies a first portion of the display area of ​​a display generating component (e.g., 120) (810a) (e.g., the display generating component displays the first individual content at a first size). In some embodiments, the representation of the physical environment is a representation of the physical environment presented by the display generating component (e.g., virtual or video passthrough). In some embodiments, the representation of the physical environment is a view of the physical environment through a transparent portion of the display generating component (e.g., true passthrough or actual passthrough). In some embodiments, the first individual content does not obscure, block, and / or appear to be displayed in front of a portion of the physical environment (e.g., its representation) that surrounds the first individual content. In some embodiments, such as those shown in Figure 7D, while simultaneously displaying a first individual piece of content (e.g., 708c) corresponding to a first view of a first physical location occupying a first portion of a display area via a display generation component (e.g., 120) alongside a representation of the physical environment (e.g., 724'), the electronic device (e.g., 101) detects individual inputs (810b) via one or more input devices (e.g., physical buttons (e.g., pressed) or dials (e.g., rotated) of the electronic device, or different input devices communicating with the electronic device, or selection of one or more user interface elements displayed via the display generation component) that correspond to a request to obscure the display of the representation of the physical environment (e.g., 724'). In some embodiments, the request to obscure the display of the representation of the physical environment corresponds to a request to reduce the number of representations of real objects in the physical environment of the electronic device presented by the electronic device.In some embodiments, a request to obscure the display of a representation of the physical environment corresponds to a request to stop displaying the representation of the physical environment (e.g., instead displaying a user interface, a first individual content, and navigation user interface elements in the virtual environment). In some embodiments, a request to obscure the display of a representation of the physical environment corresponds to a request to increase the portion of the display area of ​​a display-generating component occupied by virtual objects and / or content (e.g., a first individual content). In some embodiments, a request to obscure the display of a representation of the physical environment corresponds to a request to darken the physical environment (e.g., a representation of the physical environment). In some embodiments, such as Figure 7E, in response to detecting a pair of individual inputs, an electronic device (e.g., 101) obscures the display of a representation of the physical environment (e.g., 724') (810c), including updating the first individual content (e.g., 708d) to correspond to a second view (810d) that is larger than the first view of the first physical location (e.g., increasing the amount of the first individual content displayed by the display-generating component). In some embodiments, such as Figure 7E, in response to detecting an individual input, an electronic device (e.g., 101) obscures the display of a representation of the physical environment (e.g., 724'), including updating the first individual content so that it occupies a second portion larger than the first portion of the display area of ​​a display generating component (e.g., 120) (810e) (e.g., increasing the size of the first individual content displayed by the display generating component). In some embodiments, the display area of ​​the display generating component occupied by the representation of the physical environment and the view of the physical environment is reduced in response to detecting an individual input.

[0154] The above method of obscuring the display of representations of the physical environment in response to individual inputs provides an efficient way to present a larger view of the first physical location, which further reduces power consumption and improves the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to view the first physical location).

[0155] In some embodiments, such as Figure 7E, while displaying first individual content (e.g., 708d) corresponding to a first view of a first physical location, the electronic device (e.g., 101) detects, via one or more input devices (e.g., accelerometers(s) or gyroscopes(s)) the movement of the user's viewpoint of the electronic device (e.g., 101) in a three-dimensional environment (e.g., movement of the electronic device) ((e.g., the electronic device moves less than 1, 2, 3, 5, 10, 30, 50 centimeters, etc., and / or rotates less than 0.1, 0.5, 1, 2, 3, 4, 5 centimeters, etc.) less than a predetermined threshold) (812a). In some embodiments, the electronic device detects the movement of the electronic device. For example, the electronic device is a wearable device (e.g., a head-mounted device, a smartwatch), and the movement of a predetermined part of the user (e.g., the user's head, the user's wrist) causes the movement of the electronic device. In some embodiments, as shown in Figure 7F, in response to detecting a shift in the user's viewpoint of an electronic device (e.g., 101) in a three-dimensional environment, and in accordance with the determination that the user's gaze (e.g., 701c) was directed towards a first individual content (e.g., 708d) when the shift in the user's viewpoint was detected, the electronic device updates the display of a first view of a first physical location corresponding to the first individual content with a simulated parallax effect (812b). In some embodiments, updating the display of a first view of a first physical location with a parallax effect includes shifting the location of objects(s) in the foreground of the first view of the first physical location by a distance(s) greater than the distance(s) by which objects(s) in the background of the first view of the first physical location are shifted. In some embodiments, updating the display of a first view of a first physical location using parallax effects makes visible one or more portions of the first individual content that were previously invisible (for example, because they were blocked by other portions of the first individual content).

[0156] The above method of updating the first view of a first physical location using parallax effects provides an intuitive way to update the first content in accordance with the user's viewpoint movement, which further reduces power consumption and improves the battery life of the electronic device by enabling the display of additional content that was not displayed before the movement and allowing the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to update the first individual content when the user's viewpoint moves).

[0157] In some embodiments, such as Figure 7E, while displaying first individual content (e.g., 708d) corresponding to a first view of a first physical location, an electronic device (e.g., 101) detects, via one or more input devices, the movement of the electronic device (e.g., 101) ((e.g., 1, 2, 3, 5, 10, 30, 50 centimeters, etc., and / or rotations of at least 0.1, 0.5, 1, 2, 3, 4, 5, etc.)) exceeding a predetermined threshold (814a). In some embodiments, the electronic device is a wearable device (e.g., a head-mounted device, a smartwatch), and movement of a predetermined part of the user (e.g., the user's head, the user's wrist) causes movement of the electronic device. Therefore, in some embodiments, while displaying a first individual content corresponding to a first view of a first physical location, the electronic device detects movement of a predetermined part of the user (e.g., head, hands, arms, wrists) that meets one or more criteria, such as speed or distance of movement exceeding a threshold (e.g., 1, 2, 3, 5, 10, 30, 50 centimeters, etc.). In some embodiments, as shown in Figure 7F, in response to the detection of movement of an electronic device (e.g., 101) (e.g., or the detection of movement of a user-defined part), the electronic device (e.g., 101) displays additional content corresponding to the first physical location (e.g., 708d) (e.g., an extension of the first individual content, additional information about the first physical location, etc.) via a display generation component (e.g., 120) (814b) according to the determination that the movement of the electronic device (e.g., 101) (e.g., or the movement of a user-defined part) satisfies one or more criteria (e.g., exceeding a speed threshold, exceeding a distance threshold (e.g., 1, 2, 3, 5, 10, 30, 50 centimeters, etc.), or exceeding a rotation threshold (e.g., 0.1, 0.5, 1, 2, 3, 4, 5, etc.). In some embodiments, the additional information is displayed at the location corresponding to the movement of the electronic device (e.g., or the movement of a user-defined part). For example, in response to detection of movement to the left, the electronic device displays additional information to the left of the first individual content.In some embodiments, the electronic device expands the view of the first individual content in the direction the electronic device is moving, increases the field of view of the first individual content, and updates the field of view indication accordingly.

[0158] The above method of displaying additional information in response to the detection of movement of an electronic device provides an efficient way to present additional information to the user in response to intuitive input, thereby further reducing power consumption and improving the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0159] In some embodiments, such as Figure 7C, while displaying a first individual content (e.g., 708c) corresponding to a first view of a first physical location, an electronic device (e.g., 101) detects input (816a) directed to a selectable user interface element (e.g., 720b) ​​displayed in the user interface via one or more input devices. In some embodiments, the user interface element is a user interface element for shifting the physical location corresponding to the viewpoint of the individual content. In some embodiments, the user interface element is displayed adjacent to or overlaid on the first individual content. In some embodiments, the user interface element is one of several arrows. In some embodiments, the arrow is a move arrow (e.g., for moving the viewpoint of the first individual content laterally) or a rotate arrow (e.g., for moving the viewpoint of the first individual content rotatably). In some embodiments, the electronic device detects the selection of a user interface element by detecting, via an eye-tracking device, that the user's gaze is directed towards the user interface element, while also detecting, via a hand-tracking device, a default pose of the user's hand (e.g., a pinch pose where the thumb is within a threshold distance (e.g., 0.1, 0.2, 0.3, 0.5, 1, 2 centimeters, etc.) of another finger of the hand). In some embodiments, upon detecting input directed towards a selectable user interface element (e.g., 720b) ​​in Figure 7C, the electronic device displays a third individual content (e.g., 708d) corresponding to an individual view of an individual physical location (816b), which is then displayed at a first individual location in a three-dimensional environment, as shown in Figure 7D. For example, upon detecting the selection of an option to shift the viewpoint of the individual content to the left, the electronic device displays a third individual content corresponding to a physical location to the left of the first physical location.As another example, upon detecting the selection of an option to rotate the viewpoint of individual content to the left, the electronic device displays a fourth individual content corresponding to a first physical location, with the view rotated to the left from the view of the first individual content. In some embodiments, the electronic device maintains the size (e.g., width) of the field of view of the first individual content, but updates the location of the field of view, in response to input directed to a selectable user interface element.

[0160] The above method of displaying a third individual piece of content at the same location in a three-dimensional environment where a first individual piece of content is displayed provides an efficient way to view content corresponding to a physical location, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, without the user having to direct their attention to different areas of the three-dimensional environment or without having to provide input to keep displaying content at a third individual location in the three-dimensional environment).

[0161] In some embodiments, such as Figure 7E, while displaying first individual content (e.g., 708d) corresponding to a first view of a first physical location, the electronic device (e.g., 101) detects the movement of the electronic device (e.g., 101) relative to the three-dimensional environment (818a). In some embodiments, the electronic device is a wearable device (e.g., a head-mounted device, a smartwatch), and movement of a default part of the user (e.g., the user's head, the user's wrists) triggers movement of the electronic device. In some embodiments, while displaying first individual content corresponding to a first view of a first physical location, the electronic device detects movement of a default part of the user (e.g., head, hands, arms, wrists). In some embodiments, such as Figure 7F, according to a determination that the movement of the electronic device (e.g., 101) relative to the three-dimensional environment satisfies one or more criteria, the electronic device (e.g., 101) displays additional content corresponding to the first physical location (e.g., an extension of the first individual content, additional information about the first physical location, etc.) via a display generation component (818b). In some embodiments, one or more criteria are met when the movement of an electronic device (e.g., or a user-defined portion) positions the electronic device so that it is not directed towards individual real objects in the physical environment of the electronic device that the user may want to direct their attention to (e.g., windows, doorways, display screens, people, animals, electronic devices, fixtures, vehicles, buildings, etc.) and / or towards individual virtual objects in a three-dimensional environment that the user may want to direct their attention to (e.g., content, user interfaces, etc.). In some embodiments, according to the determination that the movement of the electronic device relative to the three-dimensional environment does not meet one or more criteria, the electronic device refrains from displaying additional content corresponding to the first physical location (e.g., while maintaining the display of the first individual content), such as refraining from updating the content user interface element 706 in Figure 7F (818c). For example, in response to the movement of the electronic device to face an empty wall, the electronic device displays additional content overlaid on a portion of the empty wall.As another example, in response to the movement of an electronic device to point it towards a specific real-world object that the user may want to direct their attention to (e.g., a window, doorway, display screen, person, animal, electronic device, fixture, vehicle, building, etc.), the electronic device refrains from displaying additional content overlaid on the specific object. In some embodiments, the electronic device expands the view of the first content to obscure portions of the three-dimensional environment other than the specific real-world object that the user may want to direct their attention to, and stops when the first specific content reaches a distance (e.g., within a threshold of 1, 5, 10, 20, 30, 40, 50, 100 centimeters, etc.) from the specific real-world object. As yet another example, in response to the movement of an electronic device to point it towards a virtual object that the user may want to direct their attention to (e.g., content, user interface), the electronic device refrains from displaying additional content overlaid on the virtual object. Thus, in some embodiments, the user can see the specific real-world object or the specific virtual object without it being obstructed by the additional content. In some embodiments, when an electronic device expands the view of a first individual piece of content, the electronic device expands the field of view of the first individual piece of content and updates the field of view indication accordingly.

[0162] The above method of selectively displaying additional content in response to the movement of an electronic device in a three-dimensional environment provides an efficient way to prevent the content from obscuring parts of the three-dimensional environment, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to view objects in the three-dimensional environment other than the additional content, such as inputs that stop displaying the additional content in order to view other objects).

[0163] In some embodiments, such as those shown in Figure 7F, one or more criteria are met based on a change in the orientation of the electronic device (e.g., 101) (e.g., with respect to one or more objects in the three-dimensional environment), and / or based on whether the user's line of sight (e.g., 701c) is directed towards one or more objects in the three-dimensional environment (820a). In some embodiments, the electronic device refrains from displaying additional content in accordance with the determination that the electronic device is directed towards individual real objects in the physical environment of the electronic device that the user wants to direct their attention to (e.g., windows, doorways, display screens, people, animals, electronic devices, fixtures, vehicles, buildings, etc.) and / or individual virtual objects in the three-dimensional environment that the user may want to direct their attention to (e.g., content, user interfaces, etc.). In some embodiments, the electronic device refrains from displaying additional content in accordance with the movement of the electronic device, based on the determination that the user is looking at a specific real object in the physical environment of the electronic device that the user may want to direct their attention to (e.g., a window, doorway, display screen, person, animal, electronic device, fixture, vehicle, building, etc.) and / or towards a specific virtual object in a three-dimensional environment that the user may want to direct their attention to (e.g., content, user interface, etc.) (as determined, for example, by an eye-tracking device communicating with the electronic device). In some embodiments, the electronic device displays additional content in accordance with the movement of the electronic device, based on the determination that the electronic device is not directed towards a specific real object in the physical environment of the electronic device that the user may want to direct their attention to (e.g., a window, doorway, display screen, person, animal, electronic device, fixture, vehicle, building, etc.) and / or towards a specific virtual object in a three-dimensional environment that the user may want to direct their attention to (e.g., content, user interface, etc.).In some embodiments, the electronic device displays additional content as it moves, based on the determination that the user's gaze is not directed towards individual real objects in the physical environment of the electronic device that the user may wish to direct their attention to (e.g., windows, doorways, display screens, people, animals, electronic devices, fixtures, vehicles, buildings, etc.) and / or individual virtual objects in a three-dimensional environment that the user may wish to direct their attention to (e.g., content, user interfaces, etc.). Thus, in some embodiments, the electronic device obscures individual real or virtual objects with additional content. In some embodiments, the electronic device obscures individual real or virtual objects with additional content if the user's gaze is not directed towards an individual real or virtual object while the movement of the electronic device is detected. In some embodiments, if the user's gaze is directed towards a real or virtual object while the movement of the electronic device is detected, the electronic device does not display additional content at locations in the three-dimensional environment that obscure individual real or virtual objects.

[0164] The above method of selectively displaying additional content in response to changes in the orientation of the electronic device to a three-dimensional environment and / or the user's line of sight provides an efficient method that enables the user to maintain visibility of content in the three-dimensional environment other than the additional content, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to view objects in the three-dimensional environment other than the additional content, such as inputs to stop displaying the additional content in order to view other objects).

[0165] In some embodiments, such as Figure 7D, an electronic device (e.g., 101) displays a first individual piece of content (e.g., 708d) occupying a first portion of the display area of ​​a display generating component (e.g., 120) (e.g., a first individual piece of the view of a first physical location), and via an eye-tracking device, detects that the user's gaze (e.g., 701b) has been directed towards the first individual piece of content (e.g., 708d) for a period of time exceeding a predetermined time threshold (e.g., 0.05, 0.1, 0.2, 0.3, 0.5, 1, 2, 5, 10 seconds, etc.) (822a). In some embodiments, such as Figure 7E, the electronic device updates the first individual content (e.g., 708d) to occupy a second portion of the display area of ​​the display generation component (e.g., 120) that is larger than the first portion (e.g., the second individual portion of the view of the first physical location is larger than the first individual portion of the view of the first physical location (e.g., including more content, objects, etc.) (822b) in response to detecting that the user's gaze (e.g., 701b) has been directed towards the first individual content (e.g., 708d) for a period of time exceeding a predetermined time threshold (e.g., 0.05, 0.1, 0.2, 0.3, 0.5, 1, 2 seconds, etc.). In some embodiments, in response to enlarging the size of the first individual content, the electronic device increases the field of view of the first individual content (without updating the field of view orientation, for example) and updates the field of view indication (displayed on navigation user interface elements, for example). In some embodiments, increasing the display area of ​​the first individual content does not enlarge the field of view of the first individual content; rather, in some embodiments, the first individual content is enlarged while the field of view (size and orientation, etc.) remains the same.

[0166] The above method of enlarging the size of a first individual piece of content in response to detecting a user's gaze directed at the first individual piece of content for a period of time exceeding a predetermined threshold provides an efficient way to increase the view of a first physical location by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to pan the view of the first physical location in order to see the first physical location), further reducing power consumption and improving the battery life of the electronic device.

[0167] In some embodiments, while displaying a first individual content, the electronic device displays a field of view indication of a first physical location corresponding to the first individual content within a navigation user interface element (824a). In some embodiments, the field of view indication is displayed adjacent to or related to (e.g., adjacent to or embedded in) a visual indication of the first location contained within the navigation user interface element. In some embodiments, the field of view indication indicates the boundary of the first physical location displayed within the first individual content. In some embodiments, while the first individual content occupies a first portion of the display area of ​​the display generating component, the field of view indication indicates the first field of view (e.g., the boundary of the first physical location contained within the first individual content while the first individual content occupies a first portion of the display area) (824b). In some embodiments, while the first individual content occupies a second portion of the display area of ​​the display generation component, the field of view indication indicates a second field of view that is larger than the first field of view (e.g., the boundary of a first physical location contained in the first individual content while the first individual content occupies a second portion of the display area) (824c). In some embodiments, the second portion of the display area is larger than the first portion of the display area, and / or the second field of view is larger than the first field of view.

[0168] The above method of displaying a field of view indication according to the display area occupied by the first individual content provides an efficient method of indicating the portion of the first physical location represented by the first individual content, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0169] In some embodiments, such as Figure 7E, a first individual content (e.g., 708d) corresponds to a first field of view (e.g., 718e) having a first orientation (e.g., with respect to a first physical location) (826a). In some embodiments, such as Figure 7E, while displaying the first individual content (e.g., 708d) and displaying an indication of the first field of view (e.g., 718e) within a navigation user interface element (e.g., 704a), an electronic device (e.g., 101) detects an input via one or more input devices that corresponds to a request to display a third individual content corresponding to a second field of view having a second orientation (826b). In some embodiments, the third individual content and the first individual content are images captured from the same physical location but with different orientations (e.g., facing different directions while in the same location). In some embodiments, the third individual content and the first individual content are images captured from different physical locations (e.g., with different orientations). In some embodiments, the widths of the first and second fields of view are the same. In some embodiments, the widths of the first and second fields of view are different. In some embodiments, the input corresponding to a request to display a third individual content is a request to rotate the viewpoint from the viewpoint of the first individual content to the viewpoint of the third individual content. In some embodiments, the input corresponds to a request to zoom in on the first individual content in one or more directions (e.g., lateral). In some embodiments, the input corresponds to a request to display a third individual content corresponding to a third physical location (e.g., lateral movement relative to the first location). In some embodiments, such as Figure 7F, an electronic device (e.g., 101) detects an input corresponding to a request to display a third individual content (e.g., 708d) corresponding to a second field of view (e.g., 718e) having a second orientation (e.g., 718e) (e.g., corresponding to a first physical location or a third physical location) (826c), and displays the third individual content (e.g., 708d) (e.g., at the first individual location in a three-dimensional environment) (826d) via a display generation component (e.g., 120).In some embodiments, such as Figure 7F, upon detecting input corresponding to a request to display a third separate content (e.g., 708d) corresponding to a second field of view (e.g., 718) having a second orientation (826c), the electronic device (e.g., 101) updates the indication of the first field of view in a navigation user interface element (e.g., 704a) to become an indication of the second field of view (e.g., 718e) (826e). In some embodiments, the field of view indication is displayed adjacent to or in relation to (e.g., adjacent to or embedded in) a visual indication of a physical location corresponding to the third separate content contained within the navigation user interface element. In some embodiments, the indication of the second field of view indicates the boundary of the physical location displayed within the third separate content.

[0170] The above method of displaying field indications according to a third individual content provides an efficient way of indicating the portion of an individual physical location represented by the third individual content, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0171] In some embodiments, such as Figure 7A, the electronic device (e.g., 101) displays a navigation user interface element (e.g., 704a) in a three-dimensional environment at a first size (828a) according to a determination that the physical environment of the electronic device (e.g., 101) satisfies one or more criteria, including criteria that are satisfied based on the amount of unobstructed space around the electronic device (e.g., 0.5, 1, 2, 3, 4, 5 meters in one or more directions). In some embodiments, the electronic device displays the navigation user interface element at a location in the three-dimensional environment that does not overlay or intersect with the location of physical objects in the three-dimensional environment. In some embodiments, the size at which the navigation user interface element is displayed depends on the amount of unobstructed space around the electronic device in the physical environment of the electronic device. In some embodiments, such as those shown in Figure 7I, the electronic device (e.g., 101) displays the navigation user interface element (e.g., 704d) in a three-dimensional environment at a second size that is different from (e.g., smaller than) the first size, according to the determination that the physical environment of the electronic device (e.g., 101) does not meet one or more criteria (828b). For example, when the electronic device displays the navigation user interface element in a room with empty space above a dining room table (e.g., about 2 meters of unobstructed space in multiple directions around the electronic device), it displays the navigation user interface element at a larger size than when the electronic device displays it in an office cubicle (e.g., about 0.5 to 1 meter of unobstructed space in multiple directions around the electronic device).

[0172] The above method of displaying navigation user interface elements in a size corresponding to the amount of unobstructed space around the electronic device in the electronic device's physical environment provides an efficient way to present representations of objects in the user's physical environment simultaneously with the navigation user interface elements. This further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to switch between the display of navigation user interface elements and representations of objects in the electronic device's physical environment).

[0173] In some embodiments, a navigation element (e.g., 704a in Figure 7A) includes a first separate part displayed in three dimensions in a three-dimensional environment (e.g., including a three-dimensional representation of physical objects in a physical location indicated by the navigation user interface element (e.g., buildings, landmarks, roads, infrastructure, geographical features, terrain, bodies of water, plants, natural features)) and a second separate part displayed in two dimensions in a three-dimensional environment (e.g., including a two-dimensional representation of physical objects in a physical location indicated by the navigation user interface element (e.g., buildings, landmarks, roads, infrastructure, geographical features, terrain, bodies of water, plants, natural features)), wherein the first separate part is closer to the user's viewpoint than the second separate part (e.g., at a threshold distance (e.g., 0.5, 1, 2, 3, 4, 5 meters, etc.)) (830a). In some embodiments, the second separate part of the navigation user interface element (the representation of physical objects contained within it) is placed closer to the user (e.g., between each of the first parts of the navigation user interface element and the second In response to an input corresponding to a request to move (within a user threshold distance defining the boundary between each of the two parts), the electronic device displays a three-dimensional representation of the physical object within the second individual part of the navigation user interface element. In some embodiments, in response to an input corresponding to a request to move the first individual part of the navigation user interface element (the representation of the physical object contained within it) further away from the user (e.g., outside the user threshold distance defining the boundary between each of the first and second individual parts of the navigation user interface element), the electronic device displays a two-dimensional representation of the physical object within the first individual part of the navigation user interface element. In some embodiments, the first individual part of the navigation user interface element is oriented along a horizontal plane in a three-dimensional environment (e.g., a real or virtual surface, a virtual plane), and the second individual part of the navigation user interface element is displayed vertically behind the first individual part of the navigation user interface element (e.g., curved upward from the horizontal plane).In some embodiments, a first separate portion and a second separate portion of a navigation user interface element are joined by a curved portion of the navigation user interface element.

[0174] The above method of displaying a first individual part of a navigation user interface element in three dimensions and a second individual part of a navigation user interface element in two dimensions provides an efficient way to increase the area of ​​the navigation user interface element that the user can view simultaneously, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0175] In some embodiments, such as Figure 7A, while displaying a first individual content (e.g., 708a) (832a), the navigation user interface element (e.g., 704a) corresponds to a first physical area including a first physical location (832b). In some embodiments, the navigation user interface element includes representations of other physical locations within the first physical area, including representations of buildings, landmarks, roads, infrastructure, geographical features, topography, bodies of water, plants, natural features, etc. In some embodiments, such as Figure 7A, while displaying a first individual content (e.g., 708a) (832a), the navigation user interface element (e.g., 704a) includes an indication of the topography of the first physical area (832c). In some embodiments, the navigation user interface element is displayed in three dimensions and includes a three-dimensional rendering of the topography (e.g., ground) of the first physical area. For example, if the first physical area includes hills, the navigation user interface element includes a three-dimensional rendering of the hills. In some embodiments, the edges of the navigation user interface elements indicate a cross-section of the terrain of the physical location corresponding to the edges of the navigation user interface elements.

[0176] The above method of displaying terrain indications of a first physical area within a navigation user interface element provides an efficient way to represent not only objects in the first physical location but also the terrain of the first physical location, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0177] In some embodiments, such as Figure 7A, a navigation user interface element (e.g., 704a) includes individual content displayed in three dimensions (e.g., 710a, 710b) (e.g., a three-dimensional representation of physical objects in a physical area corresponding to the navigation user interface element (e.g., buildings, landmarks, roads, infrastructure, geographical features, terrain, bodies of water, plants, natural features)) (834a). In some embodiments, such as Figure 7B, an electronic device (e.g., 101) detects the movement of a default part of the user (e.g., 728a) (e.g., hands, arms, head, etc.) via one or more input devices (e.g., one or more cameras, distance sensors, hand tracking devices, eye tracking devices) (834b) while displaying a navigation user interface element (e.g., 704b) containing individual content displayed in three dimensions. In some embodiments, such as Figure 7B, upon detecting movement of a user's default part (e.g., 728a) (834c), the electronic device (e.g., 101) updates the navigation user interface element (e.g., 704b) (834d) to visually de-emphasize the individual content displayed in three dimensions, as shown in Figure 7B (e.g., stop displaying the individual content, or de-emphasize the display (e.g., make it more transparent, less opaque), or display the individual content in two dimensions). This is done in response to the detection of movement of a user's default part (e.g., 728a) within the navigation user interface element (e.g., 704b) (e.g., the location of the user's default part in the three-dimensional environment intersects with or is within a threshold distance (e.g., 0.1, 0.2, 0.3, 0.5, 1, 2, 3, 4, 5, 10 centimeters, etc.) of the location of the individual content in the three-dimensional environment. In some embodiments, the electronic device stops the three-dimensional display of content located in a location corresponding to the user's default portion, while maintaining the three-dimensional display of other content of navigation user interface elements in locations not corresponding to the user's default portion.In some embodiments, the electronic device presents a representation of the user's default portion. In some embodiments, the representation of the user's default portion is a representation of the user's default portion presented by a display generation component (e.g., a virtual or video passthrough). In some embodiments, the representation of the user's default portion is a view of the user's default portion through the transparency of the display generation component (e.g., a true or actual passthrough).

[0178] The method described above, which stops the three-dimensional display of individual content in response to detection of the user's default movement to a location corresponding to the individual content, provides an efficient way to maintain the visibility of the user's default representation, thereby reducing power consumption and improving the battery life of the electronic device, enabling the user to use the electronic device more quickly and efficiently.

[0179] In some embodiments, the electronic device displays an indication of the second electronic device (e.g., associated with a user different from the user of the electronic device) displayed in a navigation user interface element corresponding to a specific physical location of the second electronic device, such as displaying an indication of the second electronic device in the navigation user interface element 704a in Figure 7A (836a). In some embodiments, the electronic device receives an indication of the location of the second electronic device (e.g., from the second electronic device, from a server). For example, the user of the electronic device and the second user of the second electronic device are connected via a service that presents the location of the second electronic device to the electronic device and presents the location of the electronic device to the second electronic device. In some embodiments, while displaying an indication of the second electronic device in the navigation user interface element, the electronic device detects a selection of the indication of the second electronic device via one or more input devices in a manner similar to detecting the selection of option 720b in Figure 7C (836b). In some embodiments, detecting indication selection includes detecting, via an eye-tracking device, that the user's gaze is directed towards an indication on a second electronic device, and detecting, via a hand-tracking device, that the user performs a predetermined gesture (e.g., a pinch gesture in which the user moves their thumb within a threshold distance (e.g., 0.05, 0.1, 0.5, 1, 2 centimeters, etc.) of another finger on their hand). In some embodiments, detecting indication selection includes detecting, via a hand-tracking device, that the user performs a predetermined gesture (e.g., a pinch gesture in which the user moves their thumb within a threshold distance (e.g., 0.05, 0.1, 0.5, 1, 2 centimeters, etc.) of another finger on their hand while a predetermined part of the user (e.g., their hand) is within a threshold distance (e.g., 1, 2, 3, 5, 10, 20 centimeters, etc.) of a location corresponding to an indication on a second electronic device.In some embodiments, upon detecting the selection of an indication for the second electronic device, the electronic device displays, via a display generation component (e.g., in a first distinct location in a three-dimensional environment), content captured by the second electronic device at a distinct physical location of the second electronic device in a manner similar to how the electronic device presents content 708a corresponding to indication 716 in Figure 7A (836c). For example, the content is live video captured by the second electronic device (e.g., one or more cameras and / or microphones communicating with it). In some embodiments, the content is one or more image and / or video content associated with the physical location of the second electronic device (e.g., captured or not captured by the second electronic device).

[0180] The above method of presenting content captured by a second electronic device in response to detecting the selection of an indication for the second electronic device displayed in a navigation user interface element corresponding to the location of the second electronic device provides an efficient way to view content corresponding to the location of the second electronic device, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0181] In some embodiments, such as Figure 7G, the first individual content includes a two-dimensional representation (e.g., 732b) of an object (e.g., a building, landmark, road, infrastructure, geographical feature, terrain, body of water, plant, natural feature) located at (or visible from) a first physical location (838a). In some embodiments, such as Figure 7G, an electronic device (e.g., 101) simultaneously displays the first individual content, which includes both a two-dimensional representation (e.g., 732b) and a three-dimensional representation (e.g., 708e) of the object (e.g., displayed before the first individual content) (838b). In some embodiments, while displaying the two-dimensional and three-dimensional representations of an object, the electronic device maintains the display of navigation user interface elements. In some embodiments, the three-dimensional representation of an object is displayed in response to the detection of a selection of a user interface element displayed at a first individual location in a three-dimensional environment (e.g., within the first individual content, adjacent to the first individual content, overlaid on the first individual content, etc.). In some embodiments, the three-dimensional representation of an object is displayed in response to the detection of a selection of the two-dimensional representation of the object. In some embodiments, the three-dimensional representation of an object is displayed between the user's viewpoint and the two-dimensional representation of the object.

[0182] The method described above, which simultaneously displays two-dimensional and three-dimensional representations of an object, provides an efficient way to view more information about the object (e.g., different views of the object), which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0183] Figures 9A–9H are flowcharts illustrating methods for presenting navigation from a first physical location to a second physical location, with reduced visual prominence within the content elements, in response to inputs corresponding to a request to present content corresponding to a second physical location, according to several embodiments. In some embodiments, Method 900 is executed on a computer system (e.g., computer system 101 in Figure 1) which includes a display generating component (e.g., display generating component 120 in Figures 1, 3, and 4) (e.g., a head-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., a camera placed on the user's non-hand and facing downward (e.g., a color sensor, an infrared sensor, and other depth-sensing cameras) or a camera facing forward from the user's head). In some embodiments, Method 900 is stored on a non-temporary computer-readable storage medium and controlled by instructions executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 in Figure 1A). Some operations of method 900 are combined in an optional manner, and / or the order of some operations is changed in an optional manner.

[0184] In some embodiments, Method 900 is performed in an electronic device (e.g., 101) that communicates with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer). In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touchscreen display), an external display such as a monitor, projector, or television, or a hardware component (optionally built-in or external) for projecting a user interface and making the user interface visible to one or more users. In some embodiments, one or more input devices include an electronic device or component that can receive user input (e.g., capture user input, detect user input, etc.) and transmit information related to the user input to the electronic device. Examples of input devices include touchscreens, mice (e.g., external), trackpads (optionally integrated or external), touchpads (optionally integrated or external), remote control devices (e.g., external), another mobile device (e.g., separate from the electronic device), handheld devices (e.g., external), controllers (e.g., external), cameras, depth sensors, eye-tracking devices, and / or motion sensors (e.g., hand-tracking devices, hand motion sensors). In some embodiments, the electronic device communicates with the hand-tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touchscreen, trackpad)). In some embodiments, the hand-tracking device is a wearable device such as a smart glove. In some embodiments, the hand-tracking device is a handheld input device such as a remote control or stylus.

[0185] In some embodiments, such as Figure 7A, an electronic device (e.g., 101) displays a user interface (e.g., 704a) via a display generation component (e.g., 120) that includes a navigation user interface element (e.g., 704a) having a first location (902b) (e.g., 716) corresponding to a first physical location (902a). In some embodiments, the user interface includes a three-dimensional environment that includes a representation of a virtual object containing the user interface and a representation of real objects in the physical environment of the electronic device. The three-dimensional environment is optionally presented from a first-person perspective from the viewpoint of a user associated with the electronic device at individual locations within the three-dimensional environment (e.g., locations of the electronic device). In some embodiments, the navigation user interface element includes a three-dimensional terrain map of the physical location having a visual indication (e.g., a pin) at the first location corresponding to the first physical location, as described with reference to Method 800. For example, a navigation user interface element includes a three-dimensional topographic map of a city, which includes a three-dimensional representation of buildings, streets, and other landmarks having flags, pins, or other visual indications displayed at a first location corresponding to a city address, landmark, or coordinates. In some embodiments, an electronic device displays a navigation user interface element having a first location corresponding to a specified first physical location.

[0186] In some embodiments, such as Figure 7A, an electronic device (e.g., 101) displays a user interface via a display generation component (e.g., 120) that includes a content element (e.g., 706) containing a first (e.g., 708a individual content) corresponding to a first view of a first physical location (e.g., 716) (902a), the content element (e.g., 706) being displayed with first visual prominence (902c). In some embodiments, the first individual content is an image taken from the first physical location and / or an image of the first physical location (e.g., a street-level view image from the first physical location). In some embodiments, the first individual content is (e.g., live) video recorded at the first physical location. In some embodiments, the first individual content has one or more characteristics of the first individual content described with reference to Method 800. In some embodiments, the degree of visual splendor is the amount of blur applied to the content element and / or the first individual content contained within the content element, and displaying the content element at the first degree of visual splendor means displaying the content element and / or the first individual content contained within the content element with no (or relatively low) blur. In some embodiments, the degree of visual splendor is the amount of transparency (e.g., relatively low transparency), contrast (e.g., relatively high contrast), color (e.g., relatively high or bright color), size (e.g., relatively large size), and / or position (e.g., relatively prominent / central position) within the user interface. In some embodiments, the navigation user interface element and / or the content element and / or the first individual content are displayed in a three-dimensional environment (e.g., a computer-generated reality (XR) environment such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment) that is generated, displayed, or otherwise made visible by the device.

[0187] In some embodiments, such as those shown in Figure 7B, while displaying a content element (e.g., 706) having a first visual splendor via a display generation component (e.g., 120), an electronic device (e.g., 101) detects user input (or a set of user inputs) via one or more input devices that correspond to a request to specify a second location corresponding to a second physical location (902d). In some embodiments, the input is directed to a navigation user interface element, such as an input that moves the indication of the specified location on a terrain map from the first location to the second location. In some embodiments, the input is directed to a first individual content, such as an input that changes the viewpoint of the content from the first physical location to the second physical location, or rotates the field of view of the first individual content. In some embodiments, the user input has one or more characteristics of a user input that corresponds to a request to specify a second location corresponding to a second physical location, as described with reference to Method 800.

[0188] In some embodiments, such as Figure 7B, in response to detecting user input (e.g., while detecting it), the electronic device (e.g., 101) displays within a content element (e.g., 706) a navigation from a first location to a second location (902e), which includes displaying representations of one or more locations between the first and second locations with reduced visual stellarity relative to the first. In some embodiments, the degree of visual stellarity is one or more of the following in the user interface: translucency (e.g., relatively high translucency), contrast (e.g., relatively low contrast), color (e.g., relatively low or dark color), size (e.g., relatively small size), and / or position (e.g., relatively subtle / peripheral position). In some embodiments, while user input is provided, the electronic device displays the content element and / or the content contained within the content element with a blurred appearance (e.g., relatively high blur). In some embodiments, the input is that a user moves a visual indication (e.g., a pin) of a specific location corresponding to a specific physical location from one location within a navigation user interface element to a different location within the navigation user interface element. For example, the user picks up the visual indication from a first location and places it at a second location within the navigation user interface element, and while the user moves the visual indication, the electronic device blurs the content element. In some embodiments, while detecting user input (or a sequence of multiple user inputs), the content element includes a first specific content with reduced visual splendor (e.g., a relatively high amount of blur). In some embodiments, while detecting user input (or a sequence of multiple user inputs), the content element updates to include specific content corresponding to the current position of the visual indication of a specific location within a map navigation element, displayed with reduced visual splendor (e.g., a relatively high amount of blur).

[0189] In some embodiments, such as Figure 7C, the electronic device (e.g., 101) displays a second separate content (e.g., 708c) (e.g., as described with reference to Method 800) corresponding to a second view of the second physical location (e.g., 716) in a content element (e.g., 706) with a visual stellarity greater than that of the first visual stellarity (902f), after displaying navigation from a first location to a second location (and in response to detecting user input). In some embodiments, after the user has finished providing one or more inputs to specify the second location, the electronic device updates the content element to include the second separate content and updates the navigation user interface to include a visual indication (e.g., a pin) of the specification of the second physical location. In some embodiments, the second separate content is an image taken from the second physical location. In some embodiments, the first separate content is (e.g., live) video recorded at the second physical location. In some embodiments, the electronic device maintains the location of content and navigation user interface elements in a three-dimensional environment where they are displayed in response to user input. In some embodiments, in response to and / or after the detection of input, the electronic device displays a second, separate, unblurred content (and / or other characteristics of visual splendor corresponding to the first visual splendor, as described above). In some embodiments, a visual splendor greater than the reduced visual splendor is the same as the first visual splendor at which the user interface is displayed before input is detected. In some embodiments, a visual splendor greater than the reduced visual splendor is visually more prominent than the first visual splendor at which the user interface is displayed before input is detected. In some embodiments, a visual splendor greater than the reduced visual splendor is visually less prominent than the first visual splendor at which the user interface is displayed before input is detected.

[0190] The method described above, which updates the visual properties of a content element while detecting user input corresponding to a request to specify a second location, provides an efficient way to show the user that specifying a second location will update the content element, which further reduces power consumption by enabling the user to use the electronic device more quickly and efficiently, improves the battery life of the electronic device, reduces usage errors that must be corrected by further user input, and reduces the need for the electronic device to fully render the content corresponding to the first location and / or intermediate locations between the first and second locations (for example, while the visual indication of the location corresponding to the content is moving).

[0191] In some embodiments, such as Figure 7B, displaying a content element (e.g., 706) with reduced visual spiciness includes displaying the entire content element (e.g., 706) with reduced visual spiciness (e.g., blurring, fading, or darkening the entire area of ​​the content element) (904a). In some embodiments, displaying the entire content element with reduced visual spiciness includes displaying the entire individual content contained within the content element with reduced visual spiciness.

[0192] The above method of displaying the entire content element with reduced visual splendor provides an efficient way to indicate to the user that specifying a second location will update the content element, which further reduces power consumption, improves the battery life of the electronic device, reduces usage errors that must be corrected by further user input, and reduces the need for the electronic device to fully render the content corresponding to the first location and / or intermediate locations between the first and second locations (for example, while the visual indication of the location corresponding to the content is moving).

[0193] In some embodiments, displaying a content element with reduced visual spiciness includes displaying a first region of the content element that (e.g., at least partially) surrounds a second region of the content element (e.g., the edges of the content element) having reduced visual spiciness (e.g., blur, fade, darken), and displaying a second region of the content element having first visual spiciness, such as displaying the central portion of the content user interface element 704b in Figure 7B without having reduced visual spiciness (e.g., without blurring, fading, or darkening) (906a). In some embodiments, displaying a first region of a content element with reduced visual spiciness includes displaying a portion of individual content contained within the first region of the content element having reduced visual spiciness. In some embodiments, an electronic device content element with edges is blurred, faded, and / or darkened.

[0194] The above method of displaying a first area of ​​a content element with reduced visual splendor provides an efficient way to indicate to the user that specifying a second location will update the content element, which further reduces power consumption, improves the battery life of the electronic device, reduces usage errors that must be corrected by further user input, and reduces the need for the electronic device to fully render the content corresponding to the first location and / or intermediate locations between the first and second locations (e.g., while the visual indication of the location corresponding to the content is being moved).

[0195] In some embodiments, such as Figure 7A, an electronic device (e.g., 101) displays at least a portion of a navigation user interface element (e.g., 704a) in a second visual splendor (e.g., in three dimensions and full color) (908a) while displaying a content element (e.g., 706) in a first visual splendor and before detecting user input corresponding to a request to specify a second location corresponding to a second physical location. In some embodiments, the navigation user interface element is a three-dimensional map of a region containing the first location. For example, the navigation user interface element includes three-dimensional representations of buildings, landmarks, roads, infrastructure, geographical features, terrain, bodies of water, plants, natural features, etc., within the region. In some embodiments, while displaying at least a portion of the navigation user interface element in a second visual splendor, the electronic device displays all (or at least a portion) of the representations within the navigation user interface element in three dimensions, in full color, semi-transparent, etc. In some embodiments, such as Figure 7B, while detecting user input and displaying content elements (e.g., 706) with reduced visual saturation relative to a first visual saturation, the electronic device (e.g., 101) displays at least a portion of the navigation user interface elements (e.g., 704b) with reduced visual saturation relative to a second visual saturation (e.g., two-dimensional, faded / darkened colors, and high translucency) (908b). In some embodiments, while detecting user input, the electronic device updates the navigation user interface elements to include one or more two-dimensional representations of objects within the area represented by the navigation user interface elements, as well as / or darkened representations, faded representations, and / or translucent representations. In some embodiments, while detecting user input, the electronic device displays the entire navigation user interface element with reduced visual saturation.In some embodiments, the electronic device displays at least a portion of the navigation user interface elements (e.g., edges, portions located in locations corresponding to default parts of the user (e.g., hands, arms, head, fingers)) with reduced visual prominence while detecting user input.

[0196] The above method of displaying at least a portion of the navigation user interface elements with reduced visual prominence while detecting user input reduces the need for the electronic device to fully render the navigation user interface elements while input is being detected, which reduces power consumption and improves the battery life of the electronic device by reducing the number and / or complexity of operations performed by the electronic device.

[0197] In some embodiments, such as Figure 7C, the electronic device (e.g., 101) displays navigation from a first location to a second location, and then displays a navigation user interface element (e.g., 704a) (910a) with a visual saturation greater than the reduced visual saturation for a second visual saturation (e.g., at the second visual saturation, with a visual saturation different from the second visual saturation) while displaying a content element (e.g., 706) with a visual saturation greater than the reduced visual saturation for a first visual saturation. In some embodiments, after displaying navigation from a first location to a second location, the electronic device displays a navigation user interface element with a reduced visual saturation for a second visual saturation while displaying a navigation user interface element that has one or more representations of three-dimensional physical objects (e.g., buildings, landmarks, roads, infrastructure, geographical features, terrain, bodies of water, plants, natural features) and / or has colors that are more saturated, less translucent, and less blurred than the colors of the representations. In some embodiments, while displaying navigation from a first location to a second location, the electronic device displays navigation user interface elements with reduced visual prominence, and after displaying navigation from the first location to the second location, the electronic device increases the visual prominence of the navigation user interface elements.

[0198] The above method of increasing the visual prominence of the navigation user interface element after displaying navigation from the first location to the second location provides an efficient method of automatically improving the visibility of the navigation user interface element, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to see the navigation user interface element after displaying navigation from the first location to the second location).

[0199] In some embodiments, such as Figure 7B, detecting user input corresponding to a request to specify a second location includes detecting, via an eye-tracking device, that the user's gaze (e.g., 701d) on an electronic device (e.g., 101) is directed towards a navigation user interface element (e.g., 704b) (912b) (912a). In some embodiments, the electronic device detects that the user's gaze is directed towards a visual indication of specifying a first physical location (e.g., displayed at a first location within the navigation user interface element). In some embodiments, the electronic device detects that the user's gaze is directed towards a second location on the navigation user interface element corresponding to a second physical location. In some embodiments, such as Figure 7B, detecting user input corresponding to a request to specify a second location includes detecting (912c) that a gesture of a default part of the user (e.g., 728a) (e.g., fingers, hand, arm, head, etc.) of an electronic device (e.g., 101) satisfies one or more criteria via one or more input devices (e.g., hand tracking devices). In some embodiments, the default gesture corresponds to a pinch gesture in which the user moves their thumb to touch another finger of their hand. In some embodiments, the electronic device detects that the user is looking at a visual indication of a designation at a first location while making a gesture that satisfies one or more criteria for “picking up” the designation, and initiates a process to move the designation (e.g., according to the movement of the default part of the user). In some embodiments, the electronic device detects that the user is looking at a second location while making a gesture that satisfies one or more criteria for moving the designation to a second location.

[0200] The above method of navigating from a first location to a second location in response to the detection of the user's gaze and gestures performed by the user provides an intuitive way to navigate the user interface (e.g., without operating a tactile input device), which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0201] In some embodiments, such as those shown in Figure 7B, while a content element (e.g., 706) displays navigation from a first location to a second location, an electronic device (e.g., 101) displays a corresponding animation of navigation from the first location to the second location in a navigation user interface element (e.g., 704b) (914a). In some embodiments, the corresponding animation of navigation from the first location to the second location in the navigation user interface element includes panning and / or zooming the navigation user interface element. In some embodiments, the corresponding animation of navigation from the first location to the second location within the navigation user interface element includes displaying an animation of moving a visual indication of a specified location (e.g., a flag, pin, etc.) from the first location to the second location within the navigation user interface element.

[0202] The above method of displaying an animation of navigation from a first location to a second location in a navigation user interface element provides an efficient way to show the user that specifying the second location will update the user interface, which further reduces power consumption, improves the battery life of the electronic device, and reduces usage errors that must be corrected by further user input, by enabling the user to use the electronic device more quickly and efficiently.

[0203] In some embodiments, the electronic device displays selectable options within the user interface to display a preview of a navigation path from a first individual physical location to a second individual physical location (e.g., the navigation path preview in Figure 7H) (916a). In some embodiments, the selectable options are displayed in response to a request to present a navigation path from a first individual physical location to a second individual physical location. In some embodiments, the selectable options are displayed simultaneously with the indication of the navigation path from a first individual physical location to a second individual physical location. In some embodiments, such as Figure 7H, in response to detecting a selection of the selectable options (916b), the electronic device (e.g., 101) displays an animation of a view corresponding to navigating the navigation path (e.g., 708f) within a content element (e.g., 706) (916c). In some embodiments, the animation of a view corresponding to navigating the navigation path is video content and / or a series of images from the perspective of someone navigating the navigation path. In some embodiments, such as Figure 7H, upon detecting the selection of a selectable option (916b), the electronic device (e.g., 101) displays a corresponding indication (e.g., 736) within a navigation user interface element (e.g., 704c) for navigating a navigation path (e.g., 740) (916d). In some embodiments, the corresponding indication displayed within the navigation user interface element includes a visual indication of the navigation path (e.g., a navigation path overlaid on the navigation user interface element) and indications of separate physical locations corresponding to the animated portion displayed within the content element.In some embodiments, as the animation within a content element progresses, the indication of the individual physical location corresponding to the portion of the animation displayed within the content element moves in a manner that synchronizes with the animation displayed within the content element. In some embodiments, the visual indication of the navigation path is a representation of the navigation path displayed with different visual characteristics (e.g., color, thickness, etc.) than the visual characteristics of other roads included in the navigation user interface element that are not along the navigation path.

[0204] The above method of displaying animations on content elements and corresponding indicators on navigation user interface elements provides an efficient way to present animations simultaneously with indicators of individual physical locations corresponding to the currently displayed portion of the animation. This further reduces power consumption and improves the battery life of electronic devices by enabling users to use electronic devices more quickly and efficiently (for example, by reducing the number of inputs required to determine the physical location corresponding to the portion of the animation).

[0205] In some embodiments, while detecting input as shown in Figure 7B, the navigation user interface element (e.g., 704b) includes one or more (e.g., three-dimensional, two-dimensional) representations (e.g., 710c) of each object (e.g., building, landmark, road, infrastructure, geographical feature, terrain, body of water, plant, natural feature) located in each physical location corresponding to the navigation user interface element (e.g., 704b) (918a). In some embodiments, the navigation user interface element includes a three-dimensional representation of a physical object located in a separate physical region corresponding to the area displayed in the navigation user interface element. In some embodiments, while displaying the navigation user interface element (e.g., 704b in Figure 7B) (918b), the electronic device (e.g., 101) refrains from displaying one of the one or more representations of each object in the navigation user interface element (918c) upon determination that the boundary of the navigation user interface element (e.g., 704b) coincides with one of the individual representations (e.g., 710c) of each object. In some embodiments, while displaying the navigation user interface element, the electronic device maintains a display of one individual of one or more representations of each object within the navigation user interface element, according to the determination that one individual of one or more representations of each object is within the boundaries of the navigation user interface element. In some embodiments, the electronic device displays a representation of an object that is entirely within the boundaries of the navigation user interface element.In some embodiments, in response to panning of a navigation user interface element, the electronic device updates the navigation user interface element to stop displaying representations of objects currently displayed at a location corresponding to the boundary of the navigation user interface element (e.g., the boundary between the navigation user interface element and the representation of the physical environment of the device displayed surrounding the navigation user interface element) as a result of the panning of the navigation user interface element. In some embodiments, in response to panning of a navigation user interface element, the electronic device updates the navigation user interface element to start displaying representations of objects that are fully displayed at a location within the boundary of the navigation user interface element as a result of the panning of the navigation user interface element. In some embodiments, the electronic device stops displaying representations of a first type of object (e.g., buildings, plants) located at a location corresponding to the boundary of the navigation user interface element, and displays a portion of representations of a second type of object (e.g., roads, geographical features, bodies of water) located at a location corresponding to the boundary of the navigation user interface element.

[0206] The above method of falsifying the display of object representations that coincide with the boundaries of navigation user interface elements reduces the amount of content displayed by electronic devices via display generation components, which reduces power consumption (for example, by rendering fewer representations) and improves the battery life of electronic devices.

[0207] In some embodiments, such as those shown in Figure 7A, while displaying a navigation user interface element (e.g., 704a), the navigation user interface element (e.g., 704a) includes a first representation (e.g., 710a) of a first (e.g., physical) object located at a first physical location and a second representation (e.g., 710b) of a second (e.g., physical) object located at a second physical location, the first representation (e.g., 710a) of the first object and the second representation (e.g., 710b) of the second object are displayed in three dimensions in the navigation user interface element (e.g., 704a), and an electronic device (e.g., 101) detects input (920a) via one or more input devices that corresponds to a request to perform a search corresponding to the information represented by the navigation user interface element (e.g., 704a) based on a specific search query. In some embodiments, the navigation user interface element includes a three-dimensional representation of a physical object (e.g., a building, landmark, road, infrastructure, geographical feature, topography, body of water, plant, natural feature) located within a specific physical area. In some embodiments, individual search queries correspond to individual physical locations, individual landmarks, individual addresses, etc. In some embodiments, upon detecting input corresponding to a request to search for a navigation user interface element based on an individual search query (920b), and in accordance with the determination that the individual search query corresponds to a first object and not to a second object (e.g., the first object matches the query and the second object does not), the electronic device (e.g., 101) displays a first representation (e.g., 710d) of the first object in three dimensions within the navigation user interface element (e.g., 704b) (920c), as shown in Figure 7G, and refrains from displaying at least some (e.g., all) of the second representation of the second object in three dimensions within the navigation user interface element (e.g., 704b) (920c).In some embodiments, the second representation of the second object (and representations of other objects not corresponding to specific search queries) is displayed in two dimensions. In some embodiments, the electronic device stops displaying the second representation of the second object (and representations of other objects not corresponding to specific search queries). In some embodiments, the electronic device reduces the height of the second representation of the second object (and representations of other objects not corresponding to specific search queries) and / or reduces the degree to which the second representation of the second object (and representations of other objects not corresponding to specific search queries) is displayed in three dimensions. In some embodiments, the electronic device (e.g., 101) detects input corresponding to a request to search for a navigation user interface element (e.g., 704b) based on an individual search query (920b), and determines that the individual search query corresponds to a second object and not to a first object (e.g., the second object matches the query and the first object does not), it displays a second representation of the second object (e.g., 710d) in three dimensions within the navigation user interface element (e.g., 704b) as shown in Figure 7G (920d), and refrains from displaying at least some (or all) of the first representation of the first object in three dimensions within the navigation user interface element (e.g., 704b) (920d). In some embodiments, the first representation of the first object (and representations of other objects that do not correspond to an individual search query) are displayed in two dimensions. In some embodiments, the electronic device stops displaying the first representation of the first object (and representations of other objects that do not correspond to an individual search query).In some embodiments, before detecting input corresponding to a request to search for navigation user interface elements based on individual search queries, the representation of objects within the navigation user interface elements is displayed in two dimensions, and in response to the search query, the electronic device updates the navigation user interface elements to display representations of objects that match the search query in three dimensions (for example, maintaining a two-dimensional display for representations of objects that do not match the search query).

[0208] The method described above, which displays objects corresponding to a search query in three dimensions and stops displaying the three-dimensional representation of objects that do not correspond to the search query in the navigation user interface elements, provides an efficient way to highlight the representation of objects that match individual search queries. This further reduces power consumption and improves the battery life of electronic devices by enabling users to use electronic devices more quickly and efficiently (for example, by reducing the number of inputs required to determine which objects correspond to a search query and which do not).

[0209] In some embodiments, such as those shown in Figure 7A, while displaying a navigation user interface element (e.g., 704a), the navigation user interface element (e.g., 704a) includes a first representation (e.g., 710a) of a first (e.g., physical) object located at a first physical location and a second representation (e.g., 710b) of a second (e.g., physical) object located at a second physical location, wherein the first representation (e.g., 710a) of the first object and the second representation (e.g., 710b) of the second object are displayed with visual properties (e.g., color, color scheme, translucency, brightness, saturation) having first values, and an electronic device (e.g., 101) detects input (922a) via one or more input devices that corresponds to a request to perform a search corresponding to the information represented by the navigation user interface element based on a separate search query. In some embodiments, the navigation user interface element includes representations of physical objects (e.g., buildings, landmarks, roads, infrastructure, geographical features, topography, bodies of water, plants, natural features) located within individual physical areas, which are displayed in a first color(s). For example, the representation is displayed in a color corresponding to the type of object to which the representation belongs (e.g., buildings in the first color, roads in the second color, plants in the third color). In another example, the representation is displayed in full color corresponding to the color of the object to which the representation belongs. In some embodiments, individual search queries correspond to individual physical locations, individual landmarks, individual addresses, etc.In some embodiments, upon detecting input corresponding to a request to search for a navigation user interface element based on an individual search query (922b), the electronic device (e.g., 101) displays in the navigation user interface element (e.g., 704b) at least part (or all) of a first representation (e.g., 710d) of a first object having visual characteristics having a second value (e.g., the same as or different from the first value), as shown in Figure 7G (922c), and displays in the navigation user interface element (e.g., 704b) at least part (or all) of a second representation (e.g., 710e) of a second object having visual characteristics having a third value different from the second value (e.g., the same as or different from the first value) (922c). In some embodiments, the second representation of the second object (and the representations of other objects that do not correspond to individual search queries) are displayed in a different color, color scheme, or setting than the first representation of the first object that corresponds to individual search queries.In some embodiments, upon detecting input corresponding to a request to search for a navigation user interface element based on an individual search query (922b), and in accordance with the determination that the individual search query corresponds to a second object rather than a first object (e.g., the second object matches the query and the first object does not), the electronic device (e.g., 101) displays in the navigation user interface element (e.g., 704b) at least part (or all) of a second representation (e.g., 710d) of a second object having visual characteristics having a second value (e.g., the same as or different from the first value) (922d), and displays in the navigation user interface element (e.g., 704b) at least part (or all) of a first representation (e.g., 710e) of a first object having visual characteristics having a third value (e.g., the same as or different from the first value) (922d). In some embodiments, the first representation of a first object (and representations of other objects that do not correspond to individual search queries) is displayed with a different color, color scheme, or settings than the second representation of a second object that corresponds to individual search queries. In some embodiments, the electronic device maintains the color, color scheme, or settings of the representation of the object that corresponds to the search query and modifies the color, color scheme, or settings of the representation of the object that does not correspond to the search query. For example, the electronic device reduces the saturation or contrast of the representation that does not correspond to the search query, or displays the representation that does not correspond to the search query in a predetermined color. In some embodiments, the electronic device maintains the color, color scheme, or settings of the representation of the object that does not correspond to the search query and modifies the color, color scheme, or settings of the representation of the object that does correspond to the search query. For example, the electronic device emphasizes, brightens, or uses a predetermined color for the representation of the object that corresponds to the search query.In some embodiments, the electronic device displays representations of objects corresponding to a search query in a first color, or in a first color scheme or setting different from how the object representation was displayed before the search request was detected, and displays representations of objects not corresponding to a search query in a second color, or in a second color scheme or setting different from how the object representation was displayed before the search request was detected.

[0210] The method described above, which displays objects corresponding to a search query and objects that do not correspond to a search query in a navigation user interface element using visual characteristics with different values, provides an efficient way to distinguish the representation of objects that match individual search queries. This further reduces power consumption and improves the battery life of electronic devices by enabling users to use electronic devices more quickly and efficiently (for example, by reducing the number of inputs required to determine which objects correspond to a search query and which do not).

[0211] In some embodiments, while displaying a navigation user interface element (e.g., navigation user interface element 704a in Figure 7A), an electronic device (e.g., 101) detects input via one or more input devices that corresponds to a request to perform a search corresponding to information represented by the navigation user interface element (e.g., 704a) based on a specific search query (924a). In some embodiments, the specific search query corresponds to a specific physical location, a specific landmark, a specific address, etc. In some embodiments, such as Figure 7G, upon detecting input that corresponds to a request to search the navigation user interface element (e.g., 704b) based on a specific search query (924b), the electronic device (e.g., 101) displays a three-dimensional representation (e.g., 708e) of a physical object (e.g., and / or location) corresponding to the search query (e.g., within a content element) via a display generation component (e.g., 120) (924c). In some embodiments, the three-dimensional representation is the actual scale of the physical object to which the representation corresponds. In some embodiments, the electronic device displays an indication of the location of the physical object within the navigation user interface element simultaneously with the three-dimensional representation of the physical object. In some embodiments, the electronic device displays a three-dimensional representation of the physical object and simultaneously displays information about the physical object. In some embodiments, the electronic device displays a three-dimensional representation of the physical object in response to detecting a selection of an object representation within a content element or a navigation user interface element. In some embodiments, the three-dimensional representation is displayed according to one or more steps of Method 800.

[0212] The above method of displaying a three-dimensional representation of a physical object corresponding to a search query provides an efficient way to identify the object corresponding to the search query, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0213] In some embodiments, such as Figure 7A, a navigation user interface element (e.g., 704a) includes individual content (e.g., 710a, 710b) (e.g., displayed in three dimensions) (926a). In some embodiments, the navigation user interface element includes a representation of physical objects within a physical area (e.g., buildings, landmarks, roads, infrastructure, geographical features, terrain, bodies of water, plants, natural features). In some embodiments, such as Figure 7B, while displaying a navigation user interface element (e.g., 704b) containing individual content to be displayed, the electronic device detects the movement of a default part of the user (e.g., 728a) (e.g., fingers, hands, arms, head, etc.) via one or more input devices (926b). In some embodiments, the electronic device presents a representation of the default part of the user in a three-dimensional environment including the user interface. In some embodiments, the representation is a photorealistic representation (e.g., video or virtual passthrough) of the default part of the user displayed via a display generation component of the electronic device. In some embodiments, the representation is a view of the user's default portion visible through the transparency of the display generating component (e.g., actual or true passthrough). In some embodiments, such as Figure 7B, in response to detecting movement of the user's default portion (e.g., 728a) (926c), the electronic device (e.g., 101) visually de-emphasizes the individual content within the navigation user interface element (e.g., 704b) (e.g., 926d) according to the determination that the user's default portion (e.g., 728a) is in a location corresponding to individual content within a navigation user interface element (e.g., 704b) (e.g., the location of the user's default portion in the three-dimensional environment intersects with or is within a threshold distance (e.g., 0.1, 0.2, 0.3, 0.5, 1, 2, 3, 4, 5, 10 centimeters, etc.) of the location of the individual content in the three-dimensional environment (e.g., 0.1, 0.2, 0.3, 0.5, 1, 2, 3, 4, 5, 10 centimeters, etc.)).For example, an electronic device displays individual content in two dimensions, while maintaining a three-dimensional display of other content within the navigation user interface element, based on a determination that the user's default portion is located in a location corresponding to that individual content within the content navigation user interface element. In some embodiments, the electronic device visually de-emphasizes individual content in response to detecting the user's default portion at a location corresponding to that individual content, according to one or more steps of Method 800.

[0214] The above method of visually de-emphasizing individual content based on the determination that the user default portion is located in a location corresponding to individual content within a navigation user interface element provides an efficient way to maintain the visibility of the representation of the user default portion, which further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by providing the user with visual feedback of the user default portion while providing input to the electronic device using the user default portion).

[0215] In some embodiments, such as Figure 7A, a navigation element (e.g., 704a) includes an indication (e.g., 712) of a weather condition (e.g., precipitation, clouds, sunshine, wind, fog, haze) at an individual physical location represented by a navigation user interface element (e.g., 704a) (e.g., where the weather condition is occurring, has occurred, or is predicted to occur), and includes an indication (e.g., 712) of the three-dimensional location of the weather condition at the individual physical location (928a). In some embodiments, the indication is a visual representation of the weather condition displayed at a location within the navigation user interface element. For example, if an individual physical location is experiencing rain, the electronic device displays a representation of rain clouds and rain at the three-dimensional location within the navigation user interface element corresponding to the individual three-dimensional physical location experiencing rain. As another example, the electronic device displays a three-dimensional representation of clouds at the three-dimensional location corresponding to the three-dimensional physical location of clouds. For example, a representation of high-altitude clouds is an image of clouds displayed at a position corresponding to a relatively high altitude above a physical location corresponding to a navigation user interface element, and a representation of low-altitude clouds is an image of clouds displayed at a position corresponding to a relatively low altitude above a physical location corresponding to a navigation user interface element. In some embodiments, an indication of individual weather conditions at individual three-dimensional locations within a navigation user interface element is an indication of weather conditions and an indication of the three-dimensional location of the weather conditions at the physical location. For example, a cloud indication at a first location within a navigation user interface element is an indication of cloudy skies and an indication of a physical location experiencing cloudy skies, and a smog indication at a second location within a navigation user interface element is an indication of smog and an indication of a physical location experiencing smog.The above method of displaying weather condition indications on navigation user interface elements provides an efficient way to present weather conditions for one or more physical locations represented by the navigation user interface elements, which further reduces power consumption and improves the battery life of electronic devices by enabling users to use electronic devices more quickly and efficiently (for example, by reducing the number of inputs required to view weather conditions for multiple locations simultaneously).

[0216] In some embodiments, aspects and / or operations of methods 800 and 900 may be interchangeable, substituted, and / or added to among these methods. For brevity, those details will not be repeated here.

[0217] The above is written with reference to specific embodiments for illustrative purposes. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the invention and its practical applications, thereby enabling other persons skilled in the art to best use the invention and the various described embodiments with various modifications suitable for specific applications that may be conceived.

[0218] As described above, one aspect of this technology involves collecting and using data available from various sources to improve the user's XR experience. This disclosure considers that in some cases, such collected data may include personal information data that uniquely identifies a particular person, or personal information data that can be used to contact a particular person or locate them. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records relating to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0219] This disclosure acknowledges that such use of personal data in the technology may be for the benefit of the user. For example, personal data can be used to enhance the user's XR experience. Furthermore, other uses of personal data that benefit the user are also intended by this disclosure. For example, health and fitness data can be used to provide insights into the user's overall wellness, or as positive feedback to individuals using the technology to pursue wellness goals.

[0220] This disclosure assumes that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal data will adhere to a robust privacy policy and / or privacy practice. Specifically, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for the strict confidentiality of personal data. Such policies should be readily accessible to users and should be updated as data collection and / or use changes. Personal data from users should be collected for the lawful and legitimate use of the entity and should not be shared or sold for any other purpose. Furthermore, such collection / sharing should be carried out only after informing and obtaining the user's consent. In addition, such entities should consider taking all necessary steps to protect and secure access to such personal data and to ensure that others with access to personal data faithfully adhere to those privacy policies and procedures. Furthermore, such entities may undergo third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be adapted to the specific types of personal data collected and / or accessed, and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection or access to certain health data may be subject to federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA). On the other hand, health data in other countries may be subject to other regulations and policies and should be addressed accordingly. Therefore, different privacy practices should be maintained in each country with respect to different types of personal data.

[0221] Notwithstanding the foregoing, the Disclosure also envisions embodiments that allow users to selectively prevent the use of or access to personal data. Specifically, the Disclosure intends that hardware and / or software elements may be provided to prevent or prevent access to such personal data. For example, in the case of an XR experience, the Technology may be configured to allow a user to choose to “opt in” or “opt out” of participating in the collection of personal data during or at any time thereafter when registering for the service. In addition to providing “opt-in” and “opt-out” options, the Disclosure intends to provide notices regarding access to or use of personal data. For example, the user may be notified when downloading an app that will access their personal data, and then again immediately before the app accesses their personal data.

[0222] Furthermore, the intent of this disclosure is that personal data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data when it is no longer needed. Additionally, where applicable in certain health-related applications, data anonymization can be used to protect user privacy. De-identification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of data stored (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or by other means.

[0223] Therefore, while this disclosure broadly covers the use of personal data to implement one or more of the disclosed embodiments, it is conceivable that these embodiments can also be implemented without requiring access to such personal data. That is, the various embodiments of the technology are not rendered inoperable by the absence of all or part of such personal data. For example, XR experiences can be generated by inferring preferences based on non-personal data such as content requested by a device associated with the user, or a minimum amount of personal information, other non-personal information available for the service, or publicly available information.

Claims

1. It is a method, In an electronic device that communicates with a display generation component and one or more input devices, The user interface is provided via the display generation component, A first individual content corresponding to a first view of a first physical location, which is displayed at the first individual location in a three-dimensional environment, Displaying a three-dimensional environment including a user interface, which includes a navigation user interface element that specifies a first location corresponding to the first physical location, and which is displayed in the three-dimensional environment between the first individual location where the first individual content is displayed and the user's viewpoint in the three-dimensional environment; While the user interface and the navigation user interface elements are being displayed via the display generation component, user input corresponding to a request specifying a second location corresponding to a second physical location is detected via one or more input devices. A method comprising: updating the user interface to include a second individual content corresponding to a second view of the second physical location in response to the detection of user input, wherein the second individual content is displayed at the first individual location in the three-dimensional environment.

2. The three-dimensional environment includes a representation of the surface in the physical environment of the electronic device, The method according to claim 1, wherein the navigation user interface element is displayed at a location in the three-dimensional environment corresponding to the representation of the surface.

3. The method according to claim 1 or 2, wherein the navigation user interface element is displayed at a location in the three-dimensional environment that does not correspond to a surface in the physical environment of the electronic device.

4. The method according to any one of claims 1 to 3, wherein the first individual content is surrounded by a representation of the physical environment of the electronic device, and the boundary between the first individual content and the representation of the physical environment of the electronic device includes a gradual visual transition between the first individual content and the representation of the physical environment of the electronic device.

5. The first individual content corresponding to the first view of the first physical location is displayed simultaneously with the representation of the physical environment of the electronic device, occupies a first portion of the display area of ​​the display generation component, and the method While displaying the first individual content corresponding to the first view of the first physical location that occupies the first portion of the display area simultaneously with the representation of the physical environment via the display generation component, detecting individual inputs via one or more input devices that correspond to requests to obscure the display of the representation of the physical environment, In response to detecting the individual inputs, the further includes obscuring the display of the physical environment, and obscuring is Corresponding to a second view of the first physical location that is larger than the first view, The method according to any one of claims 1 to 4, comprising updating the first individual content so as to occupy a second portion of the display area of ​​the display generation component that is larger than the first portion.

6. While displaying the first individual content corresponding to the first view of the first physical location, the movement of the user's viewpoint of the electronic device in the three-dimensional environment is detected via one or more input devices. The method according to any one of claims 1 to 5, further comprising: detecting the movement of the user's viewpoint of the electronic device in the three-dimensional environment, and, in accordance with the determination that the user's gaze was directed towards the first individual content when the movement of the user's viewpoint was detected, updating the display of the first view of the first physical location corresponding to the first individual content with a simulated parallax effect in accordance with the movement of the user's viewpoint.

7. While displaying the first individual content corresponding to the first view of the first physical location, the movement of the electronic device is detected via one or more input devices. The method according to any one of claims 1 to 6, further comprising: detecting the movement of the electronic device, and, in accordance with a determination that the movement of the electronic device satisfies one or more criteria, displaying additional content corresponding to the first physical location via the display generation component.

8. While displaying the first individual content corresponding to the first view of the first physical location, the system detects input directed to selectable user interface elements displayed within the user interface via one or more input devices, The method according to any one of claims 1 to 7, further comprising: detecting the input directed to the selectable user interface element, displaying a third individual content corresponding to an individual view of an individual physical location, wherein the third individual content is displayed at the first individual location in the three-dimensional environment.

9. While displaying the first individual content corresponding to the first view of the first physical location, the movement of the electronic device relative to the three-dimensional environment is detected. In accordance with the determination that the movement of the electronic device in relation to the three-dimensional environment satisfies one or more criteria, additional content corresponding to the first physical location is displayed via the display generation component, The method according to any one of claims 1 to 8, further comprising: refraining from displaying the additional content corresponding to the first physical location in accordance with a determination that the movement of the electronic device in relation to the three-dimensional environment does not satisfy one or more of the criteria.

10. The method according to claim 9, wherein the one or more criteria include criteria that are satisfied based on a change in the orientation of the electronic device, and / or criteria that are satisfied based on whether the user's line of sight is directed towards one or more objects in the three-dimensional environment.

11. While displaying the first individual content that occupies a first portion of the display area of ​​the display generation component, the eye-tracking device detects that the user's gaze has been directed towards the first individual content for a period of time exceeding a predetermined time threshold, The method according to any one of claims 1 to 10, further comprising detecting that the user's gaze has been directed to the first individual content for a period of time exceeding a predetermined time threshold, updating the first individual content to occupy a second portion of the display area of ​​the display generation component that is larger than the first portion.

12. While the first individual content is being displayed, the navigation user interface element further includes displaying an indication of the field of view of the first physical location corresponding to the first individual content, While the first individual content occupies the first portion of the display area of ​​the display generation component, the indication in the field of view indicates the first field of view. The method according to claim 11, wherein while the first individual content occupies the second portion of the display area of ​​the display generation component, the indication of the field of view indicates a second field of view that is larger than the first field of view.

13. The first individual content corresponds to a first field of view having a first orientation, and the method, While displaying the first individual content and displaying an indication of the first field of view in the navigation user interface element, the system detects input via one or more input devices that corresponds to a request to display a third individual content corresponding to a second field of view having a second orientation. In response to the detection of the input corresponding to the request for displaying the third individual content corresponding to the second field of view having the second orientation, The third individual content is displayed via the display generation component, The method according to any one of claims 1 to 12, further comprising updating the indication of the first field of view in the navigation user interface element to become an indication of the second field of view.

14. Displaying the navigation user interface elements in the three-dimensional environment at a first size, according to a determination that the physical environment of the electronic device satisfies one or more criteria, including criteria that are satisfied based on the amount of unobstructed space around the electronic device within the physical environment of the electronic device, The method according to any one of claims 1 to 13, further comprising displaying the navigation user interface element in the three-dimensional environment at a second size different from the first size, in accordance with the determination that the physical environment of the electronic device does not meet one or more of the criteria.

15. The method according to any one of claims 1 to 14, wherein the navigation element includes a first separate portion that is displayed in three dimensions in the three-dimensional environment and a second separate portion that is displayed in two dimensions in the three-dimensional environment, the first separate portion being closer to the user's viewpoint than the second separate portion.

16. While displaying the first individual content, The navigation user interface element corresponds to a first physical area including the first physical location, The method according to any one of claims 1 to 15, wherein the navigation user interface element includes an indication of the terrain of the first physical area.

17. The navigation user interface element includes individual content displayed in three dimensions, and the method is While displaying the navigation user interface element, which includes the individual content displayed in three dimensions, the movement of a default portion of the user is detected via one or more input devices. In response to the detection of the movement of the aforementioned default portion of the user, The method according to any one of claims 1 to 16, further comprising updating the navigation user interface element to visually de-emphasize the individual content displayed in three dimensions, in accordance with the determination that the default portion of the user is located in a location corresponding to the individual content displayed in three dimensions within the navigation user interface element.

18. Within the navigation user interface element, an indication of the second electronic device displayed at a specific location within the navigation user interface element corresponding to the specific physical location of the second electronic device is displayed. While the indication of the second electronic device is displayed on the navigation user interface element, the selection of the indication of the second electronic device is detected via one or more input devices. The method according to any one of claims 1 to 17, further comprising: detecting the selection of the indication of the second electronic device, and displaying the content captured by the second electronic device at the individual physical locations of the second electronic device via the display generation component.

19. The first individual content includes a two-dimensional representation of an object located at the first physical location, and the method is The method according to any one of claims 1 to 18, further comprising simultaneously displaying the first separate content, which includes the two-dimensional representation of the object and the three-dimensional representation of the object.

20. It is an electronic device, One or more processors, Memory and An electronic device comprising one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, The user interface is generated via a display generation component, A first individual content corresponding to a first view of a first physical location, which is displayed at the first individual location in a three-dimensional environment, A navigation user interface element that includes a navigation user interface element that is displayed in the three-dimensional environment between the first individual location where the first individual content is displayed and the user's viewpoint in the three-dimensional environment, wherein the three-dimensional environment is displayed, and the navigation user interface element is specified for a first location corresponding to the first physical location. While the user interface and the navigation user interface elements are being displayed via the display generation component, user input corresponding to a request to specify a second location corresponding to a second physical location is detected via one or more input devices. An electronic device including a command to update the user interface to include a second individual content corresponding to a second view of the second physical location in response to detection of user input, wherein the second individual content is displayed at the first individual location in the three-dimensional environment.

21. A non-temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device... The user interface is generated via a display generation component, A first individual content corresponding to a first view of a first physical location, which is displayed at the first individual location in a three-dimensional environment, Displaying a three-dimensional environment including a user interface, which includes a navigation user interface element that specifies a first location corresponding to the first physical location, and which is displayed in the three-dimensional environment between the first individual location where the first individual content is displayed and the user's viewpoint in the three-dimensional environment; While the user interface and the navigation user interface elements are being displayed via the display generation component, user input corresponding to a request specifying a second location corresponding to a second physical location is detected via one or more input devices. A non-temporary computer-readable storage medium for executing instructions, which includes updating the user interface to include a second separate content corresponding to a second view of the second physical location in response to detection of user input, wherein the second separate content is displayed in the first separate location within the three-dimensional environment.

22. It is an electronic device, One or more processors, Memory and The user interface is generated via a display generation component, A first individual content corresponding to a first view of a first physical location, which is displayed at the first individual location in a three-dimensional environment, A means for displaying a three-dimensional environment including a user interface, the navigation user interface element being displayed in the three-dimensional environment between the first individual location where the first individual content is displayed and the user's viewpoint in the three-dimensional environment, the navigation user interface element being specified for a first location corresponding to the first physical location, the navigation user interface element being displayed in the three-dimensional environment. Means for detecting user input corresponding to a request to specify a second location corresponding to a second physical location via one or more input devices while the user interface and the navigation user interface elements are being displayed via the display generation component, An electronic device comprising means for updating the user interface to include a second individual content corresponding to a second view of the second physical location in response to detection of user input, wherein the second individual content is displayed at the first individual location in the three-dimensional environment.

23. An information processing device for use in an electronic device, wherein the information processing device is The user interface is generated via the display generation component, A first individual content corresponding to a first view of a first physical location, which is displayed at the first individual location in a three-dimensional environment, A means for displaying a three-dimensional environment including a user interface, the navigation user interface element being displayed in the three-dimensional environment between the first individual location where the first individual content is displayed and the user's viewpoint in the three-dimensional environment, the navigation user interface element being specified for a first location corresponding to the first physical location, the navigation user interface element being displayed in the three-dimensional environment. Means for detecting user input corresponding to a request to specify a second location corresponding to a second physical location via one or more input devices while the user interface and the navigation user interface elements are being displayed via the display generation component, An information processing device comprising means for updating the user interface to include a second individual content corresponding to a second view of the second physical location in response to detection of user input, wherein the second individual content is displayed at the first individual location in the three-dimensional environment.

24. It is an electronic device, One or more processors, Memory and An electronic device comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 1 to 19.

25. A non-temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device causes the electronic device to perform the method according to any one of claims 1 to 19.

26. It is an electronic device, One or more processors, Memory and An electronic device comprising means for carrying out the method described in any one of claims 1 to 19.

27. An information processing device for use in an electronic device, wherein the information processing device is An information processing apparatus comprising means for performing the method described in any one of claims 1 to 19.

28. It is a method, In an electronic device that communicates with a display generation component and one or more input devices, The user interface is provided via the display generation component, A navigation user interface element having a first location corresponding to a first physical location, Displaying a user interface that includes a content element containing first individual content corresponding to a first view of the first physical location, and a content element displayed with first visual prominence; While the content element having the first visual prominence is being displayed via the display generation component, user input corresponding to a request specifying a second location corresponding to a second physical location is detected via one or more input devices. In response to detecting the user input, the content element displays a navigation from the first location to the second location, which includes displaying representations of one or more locations between the first location and the second location with reduced visual saturation relative to the first visual saturation. A method comprising displaying navigation from the first location to the second location, and then displaying a second separate piece of content within the content element corresponding to a second view of the second physical location with a visual spiciness greater than the reduced visual spiciness relative to the first visual spiciness.

29. The method according to claim 28, wherein displaying the content element having reduced visual splendor includes displaying the entire content element having reduced visual splendor.

30. The method according to claim 28, wherein displaying the content element having reduced visual saturation includes displaying a first region of the content element surrounding a second region of the content element having reduced visual saturation, and displaying the second region of the content element having the first visual saturation.

31. Displaying at least a portion of the navigation user interface elements with a second visual splendor while the content elements are being displayed with the first visual splendor, and before detecting the user input corresponding to the request for specifying the second location corresponding to the second physical location, The method according to any one of claims 28 to 30, further comprising displaying at least a portion of the navigation user interface elements with reduced visual saturation relative to the second visual saturation while detecting the user input and while displaying the content elements with reduced visual saturation relative to the first visual saturation.

32. The method according to claim 31, further comprising displaying the navigation user interface elements at a visual saturation greater than the reduced visual saturation relative to the second visual saturation, after displaying the navigation from the first location to the second location and while displaying the content elements at a visual saturation greater than the reduced visual saturation relative to the first visual saturation.

33. Detecting the user input corresponding to the request for specifying the second location, The eye-tracking device detects that the user's gaze is directed towards the navigation user interface element of the electronic device, The method according to any one of claims 28 to 32, comprising detecting, via one or more input devices, that a gesture of the user's default portion of the electronic device satisfies one or more criteria.

34. The method according to any one of claims 28 to 33, further comprising displaying a corresponding animation of the navigation from the first location to the second location within the navigation user interface element while the navigation from the first location to the second location is displayed within the content element.

35. The user interface displays selectable options for displaying a preview of the navigation route from a first individual physical location to a second individual physical location. In response to the detection of the selection of the aforementioned selectable option, In the aforementioned content element, display an animation of a view corresponding to navigating the navigation path, The method according to any one of claims 28 to 34, further comprising displaying a corresponding indication for navigating the navigation path in the navigation user interface element.

36. The navigation user interface element includes one or more representations of each object located at each physical location corresponding to the navigation user interface element, and the method While the aforementioned navigation user interface element is displayed, The method according to any one of claims 28 to 35, further comprising suspending the display of one of the one or more representations of each object in the navigation user interface element, in accordance with the determination that the boundary of the navigation user interface element coincides with one of the one or more representations of each object.

37. While the navigation user interface element is displayed, the navigation user interface element includes a first representation of a first object located at a first physical location and a second representation of a second object located at a second physical location, wherein the first representation of the first object and the second representation of the second object are displayed in three dimensions within the navigation user interface element, and the system detects inputs corresponding to requests to perform searches based on individual search queries via one or more input devices, wherein the system performs searches corresponding to the information represented by the navigation user interface element. In response to detecting the input corresponding to the request to search for the navigation user interface elements based on individual search queries, In accordance with the determination that the individual search query corresponds to the first object and not to the second object, the first representation of the first object is not displayed in three dimensions within the navigation user interface element, and at least a portion of the second representation of the second object is not displayed in three dimensions within the navigation user interface element. The method according to any one of claims 28 to 36, further comprising: determining that the individual search query corresponds to the second object and not to the first object, displaying the second representation of the second object in three dimensions within the navigation user interface element, and refraining from displaying at least a portion of the first representation of the first object in three dimensions within the navigation user interface element.

38. While the navigation user interface element is displayed, the navigation user interface element includes a first representation of a first object located at a first physical location and a second representation of a second object located at a second physical location, wherein the first representation of the first object and the second representation of the second object are displayed with visual characteristics having a first value, and the system detects inputs corresponding to requests to perform a search corresponding to information represented by the navigation user interface element based on individual search queries via one or more input devices. In response to detecting the input corresponding to the request to search for the navigation user interface elements based on individual search queries, In accordance with the determination that the individual search query corresponds to the first object and not to the second object, the navigation user interface element displays at least a portion of the first representation of the first object having the visual characteristics having the second value, and displays at least a portion of the second representation of the second object having the visual characteristics having a third value different from the second value. The method according to any one of claims 28 to 37, further comprising: displaying within the navigation user interface element at least a portion of the second representation of the second object having the visual characteristics having the second value, and displaying within the navigation user interface element at least a portion of the first representation of the first object having the visual characteristics having the third value, in accordance with the determination that the individual search query corresponds to the second object rather than the first object.

39. While the navigation user interface element is displayed, input corresponding to a request to perform a search corresponding to the information represented by the navigation user interface element based on an individual search query is detected via one or more input devices. In response to detecting the input corresponding to the request to search for the navigation user interface elements based on the individual search queries, The method according to any one of claims 28 to 38, further comprising displaying a three-dimensional representation of a physical object corresponding to the search query via the display generation component.

40. The navigation user interface element includes individual content, and the method, While the navigation user interface element, including the displayed individual content, is being shown, the movement of the user's default portion is being detected via one or more input devices. In response to the detection of the movement of the aforementioned default portion of the user, The method according to any one of claims 28 to 39, further comprising visually de-emphasizing the individual content within the navigation user interface element in accordance with the determination that the default portion of the user is located in a location corresponding to the individual content within the navigation user interface element.

41. The method according to any one of claims 28 to 40, wherein the navigation element includes an indication of weather conditions at individual physical locations represented by the navigation user interface element, and includes an indication of the three-dimensional location of the weather conditions at the individual physical locations.

42. It is an electronic device, One or more processors, Memory and An electronic device comprising one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, The user interface is generated via a display generation component, A navigation user interface element having a first location corresponding to a first physical location, Displaying a user interface that includes a content element containing first individual content corresponding to a first view of the first physical location, and a content element displayed with first visual prominence; While the content element having the first visual prominence is being displayed via the display generation component, user input corresponding to a request specifying a second location corresponding to a second physical location is detected via one or more input devices. In response to detecting the user input, the content element displays navigation from the first location to the second location, including displaying representations of one or more locations between the first location and the second location with reduced visual saturation relative to the first visual saturation. An electronic device including a command to display, after displaying navigation from the first location to the second location, a second separate piece of content corresponding to a second view of the second physical location within the content element, with visual spiciness greater than the reduced visual spiciness relative to the first visual spiciness.

43. A non-temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device... The user interface is generated via a display generation component, A navigation user interface element having a first location corresponding to a first physical location, Displaying a user interface that includes a content element containing first individual content corresponding to a first view of the first physical location, and a content element displayed with first visual prominence; While the content element having the first visual prominence is being displayed via the display generation component, user input corresponding to a request specifying a second location corresponding to a second physical location is detected via one or more input devices. In response to detecting the user input, the content element displays a navigation from the first location to the second location, which includes displaying representations of one or more locations between the first location and the second location with reduced visual saturation relative to the first visual saturation. A non-temporary computer-readable storage medium that causes a method to be performed which includes, after displaying navigation from the first location to the second location, displays a second separate piece of content within the content element corresponding to a second view of the second physical location with a visual stellarity greater than the reduced visual stellarity relative to the first visual stellarity.

44. It is an electronic device, One or more processors, Memory and The user interface is generated via a display generation component, A navigation user interface element having a first location corresponding to a first physical location, A means for displaying a user interface, including a content element containing first individual content corresponding to a first view of the first physical location, and a content element displayed with first visual prominence; Means for detecting user input corresponding to a request specifying a second location corresponding to a second physical location via one or more input devices while the content element having the first visual prominence is being displayed via the display generation component, In response to detecting the user input, the content element displays a navigation from the first location to the second location, showing representations of one or more locations between the first location and the second location with reduced visual saturation relative to the first visual saturation. An electronic device comprising means for displaying navigation from the first location to the second location, and then displaying a second separate piece of content within the content element corresponding to a second view of the second physical location with a visual spiciness greater than the reduced visual spiciness relative to the first visual spiciness.

45. An information processing device for use in an electronic device, wherein the information processing device is The user interface is generated via a display generation component, A navigation user interface element having a first location corresponding to a first physical location, A means for displaying a user interface, including a content element containing first individual content corresponding to a first view of the first physical location, and a content element displayed with first visual prominence; While the content element having the first visual prominence is being displayed via the display generation component, user input corresponding to a request specifying a second location corresponding to a second physical location is detected via one or more input devices. In response to detecting the user input, the content element displays navigation from the first location to the second location, including displaying representations of one or more locations between the first location and the second location with reduced visual saturation relative to the first visual saturation. An information processing device comprising: displaying navigation from the first location to the second location; and then displaying a second separate piece of content within the content element corresponding to a second view of the second physical location with a visual spiciness greater than the reduced visual spiciness relative to the first visual spiciness.

46. It is an electronic device, One or more processors, Memory and An electronic device comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 28 to 41.

47. A non-temporary computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device causes the electronic device to perform the method according to any one of claims 28 to 41.

48. It is an electronic device, One or more processors, Memory and An electronic device comprising means for carrying out the method described in any one of claims 28 to 41.

49. An information processing device for use in an electronic device, wherein the information processing device is An information processing apparatus comprising means for performing the method described in any one of claims 28 to 41.

Images