Devices, methods, and graphical user interfaces for viewing and interacting with three-dimensional environment

Abstract

A computer system displays a user interface at a first location in a three-dimensional environment and displays a simulated lighting effect within a first sub-region of the user interface. In response to detecting an event corresponding to the movement of the user interface, the computer system moves the user interface to a second location in the three-dimensional environment. Moving the simulated lighting effect within a first sub-region of the user interface if the distance between the first location and the second location is less than a threshold distance; and if the distance between the first location and the second location is at least the threshold distance, moving the simulated lighting effect across a third sub-region located between the first sub-region and the second sub-region to a second sub-region of the user interface, the simulated lighting effect is not displayed in the third sub-region when the user interface is stationary relative to the three-dimensional environment.

Description

[0001] Related patent applications This application is a continuation to U.S. Patent Application No. 18 / 676,171, filed May 28, 2024, which claims priority to U.S. Provisional Patent Application No. 63 / 470,785, filed June 2, 2023. Technical Field

[0002] This disclosure relates in its entirety to computer systems that communicate with display generation components and one or more sensors and optionally with one or more input devices that provide computer-generated experiences, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display. Background Technology

[0003] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices used in computer systems and other electronic computing devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays) are used to interact with the virtual / augmented reality environment. Example virtual elements include virtual objects such as digital images, videos, text, icons, and control elements (such as buttons and other graphics). Summary of the Invention

[0004] Despite advancements in accuracy and visual fidelity in virtual reality and augmented reality environments (e.g., simulating real-world physics and / or lighting effects), work on modifying virtual / augmented reality environments to introduce non-intuitive behaviors (e.g., behaviors that do not reflect real-world objects, light, and / or shadows) to improve device performance, enhance immersion, and / or maintain the clarity of virtual content display has not received sufficient attention. For example, highly accurate simulations of real-world lighting effects (such as light reflections and shadows) can result in large reflections (e.g., along the entire length of the user interface window's boundary), which reduces visibility (e.g., because the displayed reflections would be too bright for comfortable viewing based on real-world lighting) and can lead to performance issues or hardware limitations (e.g., magnification and / or emphasis) due to technical and / or physical limitations of the computer system's display generation components, such as artifacts and / or aliasing. Utilizing non-intuitive behaviors (e.g., for light sources and / or visual effects such as reflections and shadows) allows computer systems to maintain the visibility of displayed content while minimizing the impact of performance and / or hardware limitations. For example, visual effects such as reflections can be removed from certain areas (e.g., to prevent reflections from appearing along the entire edge or side of the user interface), but some behaviors that follow real-world lighting can also be retained (e.g., the positioning and / or size of reflections change as the user interface moves and / or the user's viewpoint changes). This allows for a balance between high performance and visual clarity (e.g., by avoiding or eliminating performance issues, and / or by compensating for hardware limitations by introducing non-intuitive behaviors that do not reflect the behavior of objects, light, and / or shadows in the real world), while maintaining a baseline level of immersion (e.g., by retaining some behaviors that are intuitive and consistent with objects, light, and / or shadows in the real world).

[0005] Therefore, computer systems with improved methods and interfaces are needed to provide users with computer-generated experiences, enabling them to view extended reality environments and interact with computer systems more efficiently and intuitively. Such methods and interfaces can optionally supplement or replace conventional methods used to provide users with extended reality experiences. By helping users understand the relationship between the inputs provided and the device's responses to those inputs, such methods and interfaces reduce the quantity, extent, and / or nature of user input, thus creating a more effective human-computer interface.

[0006] The disclosed system reduces or eliminates the aforementioned defects and other problems associated with the user interface of a computer system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a laptop, tablet, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touchscreen" or "touchscreen display"). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, in addition to display generation components, the computer system also has one or more output devices, including one or more haptic output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, and a program or instruction set stored in memory for performing multiple functions. In some implementations, users interact with the GUI through touch and gestures of a stylus and / or fingers on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body (as captured by cameras and other motion sensors), and / or voice input (as captured by one or more audio input devices). In some implementations, functions performed through interaction may optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving phone calls, video conferencing, sending and receiving emails, instant messaging, test support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions may optionally be included in transient and / or non-transitory computer-readable storage media or other computer program products configured for execution by one or more processors.

[0007] Electronic devices require improved methods and interfaces for viewing and interacting with 3D environments. Such methods and interfaces can complement or replace conventional methods for viewing and interacting with 3D environments. They reduce the amount, extent, and / or nature of user input, resulting in more efficient human-computer interfaces. For battery-powered computing devices, these methods and interfaces save power and increase the time interval between battery charging sessions.

[0008] According to some embodiments, a method is performed at a computer system communicating with a display generation component and one or more sensors. The method includes displaying a first view of a three-dimensional environment via the display generation component, the first view of the three-dimensional environment including a first user interface at a first location in the three-dimensional environment. The first user interface displays a first simulated lighting effect at a first location on the first user interface. The first location is a location within a first sub-region of the first user interface. The method includes detecting an event corresponding to movement of the first user interface; and in response to detecting the event, moving the first user interface from the first location in the three-dimensional environment to a second location in the three-dimensional environment. Moving the first user interface includes: displaying a first user interface with a second simulated lighting effect at the second location on the first user interface based on determining that the distance between the first location and the second location is less than a threshold distance, wherein the second location is different from the first location on the first user interface, and wherein the second location is within a first sub-region of the first user interface; and displaying a first user interface with a third simulated lighting effect at a third location on the first user interface based on determining that the distance between the first location and the second location is at least a threshold distance. The third location on the first user interface is different from both the first location and the second location on the first user interface. The third location is within a second sub-region of the first user interface. The second sub-region of the first user interface is different from the first sub-region of the first user interface; and the first user interface includes a third sub-region located between the first and second sub-regions of the first user interface, wherein the third sub-region is the area where simulated lighting effects are not displayed when the first user interface is stationary relative to the three-dimensional environment.

[0009] According to some embodiments, a method is performed at a computer system communicating with a display generation component and one or more sensors. The method includes: displaying a first user interface in a first view of a three-dimensional environment via the display generation component. The first user interface has a simulated three-dimensional effect when visible in the first view of the three-dimensional environment, and the first user interface has a first spatial relationship with a user's viewpoint. The method includes detecting input corresponding to a request to change the user's viewpoint to have a second spatial relationship with the first user interface. The second spatial relationship differs from the first spatial relationship, and a first portion of the input corresponds to a corresponding change in viewpoint, and a second portion following the first portion corresponds to a corresponding change in viewpoint. The method includes: in response to detecting the first portion of the input: displaying a second view of the three-dimensional environment different from the first view of the three-dimensional environment; and changing the simulated three-dimensional effect by a first amount. The method includes: in response to detecting the second portion of the input: displaying a third view of the three-dimensional environment different from both the first and second views of the three-dimensional environment; and changing the simulated three-dimensional effect by a second amount, wherein the second amount differs from the first amount.

[0010] According to some embodiments, a method is performed at a computer system communicating with a display generation component and one or more sensors. The method includes: displaying a first user interface object in a first orientation and at a first object location in the three-dimensional environment when a first view of a three-dimensional environment is visible via the display generation component; and displaying a first simulated shadow corresponding to the first user interface object at a first shadow location in the three-dimensional environment. The first simulated shadow at the first shadow location has a first spatial relationship with the first user interface object. The method includes: detecting user input pointing to the first user interface object. The method includes: in response to detecting the user input, changing the orientation of the first user interface object from the first orientation to a second orientation, the change including: displaying the first user interface object in the second orientation; and displaying the first simulated shadow at a second shadow location in the three-dimensional environment. The second shadow location in the first view of the three-dimensional environment differs from the first shadow location in the first view of the three-dimensional environment. The first simulated shadow at the second shadow location has a second spatial relationship with the first user interface object; and the second spatial relationship differs from the first spatial relationship.

[0011] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not exhaustive; in particular, many additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, description, and claims. Furthermore, it should be pointed out that the language used in this specification has been chosen in principle for readability and instruction purposes, and such choice may not be necessary to depict or define the subject matter of the invention. Attached Figure Description

[0012] To better understand the various described embodiments, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals in all the drawings indicate corresponding parts.

[0013] Figure 1A This is a block diagram illustrating the operating environment of a computer system for providing extended reality (XR) experiences according to some implementation schemes.

[0014] Figures 1B to 1P It is used in Figure 1A Examples of computer systems that provide XR experiences in the operating environment.

[0015] Figure 2 This is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience, according to some implementation schemes.

[0016] Figure 3 This is a block diagram illustrating a display generation component of a computer system configured to provide an XR experience to a user, according to some implementation schemes.

[0017] Figure 4 This is a block diagram illustrating a hand tracking unit of a computer system configured to capture user gesture input according to some implementation schemes.

[0018] Figure 5 This is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input according to some implementation schemes.

[0019] Figure 6 This is a flowchart illustrating a flare-assisted gaze tracking pipeline according to some implementation schemes.

[0020] Figures 7A to 7L Examples of techniques, according to some implementations, for changing the displayed visual effects as the user interface and / or user interface elements are repositioned and / or as the user's viewpoint changes relative to the user interface and / or user interface elements.

[0021] Figures 8A to 8M Example techniques for repositioning shadows corresponding to the user interface as the user interface is reoriented, according to some implementation schemes, are illustrated.

[0022] Figure 9 This is a flowchart of a method for updating the display of visuals as the user interface and / or user interface elements are repositioned, according to various implementation schemes.

[0023] Figure 10 It is a flowchart of a method for updating the visual effects applied to the user interface and / or user interface elements as the user's viewpoint changes, according to various implementation schemes.

[0024] Figure 11 This is a flowchart of a method for repositioning the shadow corresponding to the user interface as the user interface is reoriented, based on various implementation schemes. Detailed Implementation

[0025] According to some implementations, this disclosure relates to a user interface for providing extended reality (XR) experiences to users.

[0026] The systems, methods, and GUIs described in this paper improve user interface interactions with virtual / augmented reality environments in a variety of ways.

[0027] In some implementations, the computer system displays a first user interface with a first simulated lighting effect in a first sub-region of the first user interface. In response to detecting movement of the first user interface, the computer system moves the first user interface and displays a corresponding simulated lighting effect in either the first sub-region of the first user interface or a second sub-region of the first user interface separated from the first sub-region by a third sub-region, depending on the amount of movement of the first user interface. Conditionally displaying the first user interface with the visual effect in the corresponding sub-region provides improved visual feedback regarding the spatial relationship between the user interface and the three-dimensional environment. This improves the user's contextual awareness of the three-dimensional environment and enhances the user's physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible three-dimensional environment), and also avoids displaying the visual effect (e.g., simulated lighting effect) in a manner that might cause visual artifacts and / or aliasing on the computer system's display.

[0028] In some implementations, the computer system displays a first user interface with a simulated 3D effect. In response to detecting a change in the user's viewpoint, the computer system alters the simulated 3D effect by a first amount during a first portion of the viewpoint change, and by a second amount during a second portion of the viewpoint change. By altering the simulated 3D effect by different amounts (e.g., gradually decreasing or increasing) as the viewpoint moves relative to the user interface, the computer system automatically provides visual feedback regarding the spatial relationship between the user interface and the 3D environment. Furthermore, by gradually reducing the simulated 3D effect as the viewpoint moves, motion and parallax in the 3D environment are reduced, improving the user's contextual awareness of the 3D environment and lowering the likelihood of motion sickness when using the computer system.

[0029] In some implementations, the computer system displays a simulated shadow with a first spatial relationship to a user interface object. In response to detecting a change in the orientation of the user interface object, the computer system displays the user interface object with an updated orientation and the simulated shadow with a second spatial relationship different from the first spatial relationship. Changing the spatial relationship between the simulated shadow and the user interface object based on the change in the orientation of the user interface object relative to the 3D environment provides improved visual feedback regarding the spatial relationship between the user interface object and the 3D environment. This improves the user's contextual awareness of the 3D environment and enhances user physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible 3D environment).

[0030] Figures 1A to 6 A description of a sample computer system for providing XR experiences to users is provided. Figures 7A to 7LExamples of techniques, according to some implementations, for changing the displayed visual effects as the user interface and / or user interface elements are repositioned and / or as the user's viewpoint changes relative to the user interface and / or user interface elements. Figures 8A to 8M Example techniques for repositioning shadows corresponding to the user interface as the user interface is reoriented, according to some implementation schemes, are illustrated. Figure 9 This is a flowchart of a method for updating the display of visuals as the user interface and / or user interface elements are repositioned, according to various implementation schemes. Figure 10 It is a flowchart of a method for updating the visual effects applied to the user interface and / or user interface elements as the user's viewpoint changes, according to various implementation schemes. Figures 7A to 7L The user interface in the example is used to demonstrate Figure 9 and Figure 10 The process in. Figure 11 This is a flowchart of a method for repositioning the shadow corresponding to the user interface as the user interface is reoriented, based on various implementation schemes. Figures 8A to 8M The user interface in the example is used to demonstrate Figure 11 The process in.

[0031] The processes described below enhance device operability and (e.g., by helping users provide appropriate input and reducing user errors when operating / interacting with the device) make the user-device interface more efficient through various technologies. These include providing users with improved visual feedback, reducing the amount of input required to perform operations, providing additional control options without cluttering the user interface with additional display controls, performing operations without further user input when a set of conditions have been met, improving privacy and / or security, providing a richer, more detailed, and / or more realistic user experience while saving storage space, and / or additional technologies. These technologies also reduce power consumption and extend device battery life by enabling users to use the device more quickly and efficiently. This saves battery power and, therefore, weight, and improves the device's ergonomics. These technologies also enable real-time communication, allow the use of fewer and / or less accurate sensors, resulting in a more compact, lighter, and cheaper device, and enabling the device to be used in a variety of lighting conditions. These technologies reduce energy consumption, thereby reducing the heat generated by the device. This is especially important for wearable devices, where if a device generates too much heat, even when operating entirely within the parameters of its components, it can become uncomfortable for the user to wear.

[0032] Furthermore, in a method described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the described method may be repeated in multiple repetitions such that, during the repetitions, all conditions for the steps in the method, depending on the circumstances, are satisfied in different repetitions of the method. For example, if the method requires performing a first step (if the conditions are satisfied) and a second step (if the conditions are not satisfied), those skilled in the art will know that the stated steps are repeated until both conditions are satisfied and conditions are not satisfied (in no particular order). Thus, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing dependent operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions for the steps in the method, depending on the circumstances, are satisfied. Those skilled in the art will also understand that, similar to a method having dependent steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all dependent steps have been performed.

[0033] In some implementation schemes, such as Figure 1A As shown, an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted display (HMD), a monitor, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a haptic output generator 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, haptic sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, speed sensors, etc.), and optionally one or more peripheral devices 195 (e.g., home appliances, wearable devices, etc.). In some embodiments, one or more of the input devices 125, output devices 155, sensors 190, and peripheral devices 195 are integrated with the display generation component 120 (e.g., in a head-mounted or handheld device).

[0034] In describing XR experiences, various terms are used to distinguish several related but different environments that a user can sense and / or interact with (e.g., interacting with inputs detected by the computer system 101 that generates the XR experience, causing the computer system to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101). The following is a subset of these terms: Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through senses such as sight, touch, hearing, taste, and smell.

[0035] Extended Reality: Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In XR, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. For example, an XR system can detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of the properties of virtual objects in the XR environment can be done in response to a representation of physical motion (e.g., a voice command). People can use any of their senses to sense and / or interact with XR objects, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. For example, audio objects can enable audio transparency, which selectively introduces ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, people can sense and / or interact only with audio objects.

[0036] Examples of XR include virtual reality and mixed reality.

[0037] Virtual Reality: A virtual reality (VR) environment is a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing human avatars are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through the simulation of a person's presence within the computer-generated environment and / or through the simulation of a subset of a person's physical movements within the computer-generated environment.

[0038] Mixed Reality: Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments refer to simulated environments designed to incorporate sensory input from the physical environment, or its representations, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between, but not limited to, a purely physical environment as one end and a virtual reality environment as the other. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to present an MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical objects or their representations from the physical environment). For example, a system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0039] Examples of mixed reality include augmented reality and augmented virtual reality.

[0040] Augmented Reality (AR): An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation of the physical environment. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects overlaid on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto the physical environment, such as as a hologram or onto a physical surface, allowing a person to perceive the virtual objects superimposed on the physical environment. Augmented reality environments also refer to simulated environments in which the representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system can transform one or more sensor images to apply a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. As another example, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it, such that the modified portions can be representative but not realistic versions of the original captured image. Furthermore, the representation of the physical environment can be transformed by graphically removing or blurring portions of it.

[0041] Augmented Virtual: An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment combines one or more sensory inputs from a physical environment. Sensory input can be a representation of one or more characteristics of the physical environment. For example, an AV park could have virtual trees and virtual buildings, but a person's face could be realistically reproduced from an image taken of a physical person. Similarly, virtual objects could adopt the shape or color of a physical object imaged by one or more imaging sensors. Furthermore, virtual objects could adopt shadows that correspond to the sun's position within the physical environment.

[0042] In augmented reality, mixed reality, or virtual reality environments, a view of the three-dimensional environment is visible to the user. This view is typically visible to the user via a virtual viewport through one or more display generating components (e.g., a display providing stereoscopic content to different eyes of the same user), which has a viewport boundary that defines the extent of the three-dimensional environment visible to the user via the one or more display generating components. In some embodiments, the area defined by the viewport boundary is smaller than the user's visual field in one or more dimensions (e.g., based on the user's visual field, the size of one or more display generating components, optical properties or other physical characteristics, and / or the position and / or orientation of one or more display generating components relative to the user's eyes). In some embodiments, the area defined by the viewport boundary is larger than the user's visual field in one or more dimensions (e.g., based on the user's visual field, the size of one or more display generating components, optical properties or other physical characteristics, and / or the position and / or orientation of one or more display generating components relative to the user's eyes). The viewport and viewport boundary typically move with the movement of one or more display generating components (e.g., with the user's head for head-mounted devices, or with the user's hand for handheld devices such as tablets or smartphones). The user's viewpoint determines what is visible within the viewport. The viewpoint typically specifies the position and orientation relative to the 3D environment, and as the viewpoint moves, the view of the 3D environment also moves within the viewport. For head-mounted devices, the viewpoint is typically based on the position and orientation of the user's head, face, and / or eyes to provide a perceptually accurate view of the 3D environment that offers an immersive experience when the user is using the head-mounted device. For handheld or fixed devices, the viewpoint shifts with the movement of the handheld or fixed device and / or with changes in the user's positioning relative to the handheld or fixed device (e.g., the user moves towards, away from, up, down, right, and / or left). For devices including display generation components with virtual pass-through, a portion of the physical environment visible (e.g., displayed and / or projected) via one or more display generation components is based on the field of view of one or more cameras communicating with the display generation components, which typically move with the movement of the display generation components (e.g., for head-mounted devices, moving with the movement of the user's head, or for handheld devices such as tablets or smartphones, moving with the movement of the user's hands), because the user's viewpoint moves with the movement of the field of view of the one or more cameras (and the appearance of one or more virtual objects displayed via one or more display generation components is updated based on the user's viewpoint (e.g., the display position and pose of the virtual objects are updated based on the movement of the user's viewpoint)).For a display generating component with optical transparency, portions of the physical environment visible through one or more display generating components (e.g., optically visible through one or more portions or fully transparent portions of the display generating component) are based on the user's field of view through the portion or fully transparent portion of the display generating component (e.g., for a head-mounted device, it moves with the movement of the user's head, or for a handheld device such as a tablet or smartphone, it moves with the movement of the user's hand), because the user's viewpoint moves with the movement of the user's field of view through the portion or fully transparent portion of the display generating component (and the appearance of one or more virtual objects is updated based on the user's viewpoint).

[0043] In some embodiments, the representation of the physical environment (e.g., displayed via virtual passthrough or optical passthrough) may be partially or completely occluded by the virtual environment. In some embodiments, the amount of virtual environment displayed (e.g., the amount of physical environment not displayed) is based on the immersion level of the virtual environment (e.g., relative to the representation of the physical environment). For example, increasing the immersion level may optionally result in more virtual environment being displayed, replacing and / or occluding more physical environment, and decreasing the immersion level may optionally result in less virtual environment being displayed, thereby revealing portions of the physical environment that were previously not displayed and / or occluded. In some embodiments, at a particular immersion level, one or more first background objects (e.g., in the representation of the physical environment) are visually de-emphasized more than one or more second background objects (e.g., dimmed, blurred, and / or displayed with increased transparency), and one or more third background objects are de-emphasized. In some embodiments, the level of immersion includes the associated degree to which virtual content (e.g., a virtual environment and / or virtual content) displayed by the computer system occludes background content (e.g., content other than the virtual environment and / or virtual content) around / behind the virtual environment. Optionally, this includes the number of items in the displayed background content and / or the displayed visual characteristics of the background content (e.g., color, contrast, and / or opacity), the angular range of the virtual content displayed via the display generating component (e.g., 60 degrees for content displayed at low immersion, 120 degrees for content displayed at medium immersion, or 180 degrees for content displayed at high immersion), and / or the proportion of the field of view displayed via the display generating component occupied by the virtual content (e.g., 33% of the field of view occupied by the virtual content at low immersion, 66% of the field of view occupied by the virtual content at medium immersion, or 100% of the field of view occupied by the virtual content at high immersion). In some embodiments, the background content is included within a background on which the virtual content is displayed (e.g., background content in a representation of the physical environment). In some implementations, background content includes user interfaces (e.g., user interfaces corresponding to applications generated by a computer system), virtual objects not associated with or included in the virtual environment and / or virtual content (e.g., files generated by the computer system or other user representations), and / or real objects (e.g., transparent objects representing real objects in the user's surrounding physical environment, visible such that they are displayed via display generation components and / or via transparent or semi-transparent parts of the display generation components, because the computer system does not obscure / impede their visibility through the display generation components). In some implementations, at a low immersion level (e.g., a first immersion level), the background, virtual, and / or real objects are displayed in an unobstructed manner. For example, a virtual environment with a low immersion level may optionally be displayed concurrently with background content, which may optionally be displayed at full brightness, color, and / or semi-transparency.In some implementations, at higher immersion levels (e.g., a second immersion level above the first immersion level), background, virtual, and / or real objects are displayed in an occluded manner (e.g., dimmed, blurred, or removed from the display). For example, a corresponding virtual environment with a high immersion level is displayed without concurrently displaying background content (e.g., in full-screen or fully immersive mode). Alternatively, a virtual environment displayed at a medium immersion level is displayed concurrently with darkened, blurred, or otherwise de-emphasized background content. In some implementations, the visual characteristics of background objects differ between background objects. For example, at a particular immersion level, one or more first background objects are visually de-emphasized more than one or more second background objects (e.g., dimmed, blurred, and / or displayed with increased transparency), and one or more third background objects are stopped from displaying. In some implementations, zero immersion or a zero immersion level corresponds to a virtual environment that is stopped from displaying, and instead, a representation of the physical environment (optionally having one or more virtual objects, such as an application, window, or virtual 3D object) is displayed, and the representation of the physical environment is not occluded by the virtual environment. Using physical input elements to adjust immersion levels provides a quick and efficient way to adjust immersion, which enhances the operability of computer systems and makes user-device interfaces more efficient.

[0044] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same location and / or position within the user's viewpoint, the virtual object remains viewpoint-locked even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the direction forward of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); therefore, the user's viewpoint remains fixed even when the user's gaze shifts without moving the user's head. In embodiments where the computer system has a display generating component (e.g., a display screen) that can be repositioned relative to the user's head, the user's viewpoint is the augmented reality view presented to the user on the computer system's display generating component. For example, a viewpoint-locked virtual object displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint, even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or orientation of a viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In an implementation where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, so the virtual object is also referred to as a "head-locked virtual object".

[0045] Environment-locked visual objects: When a computer system displays a virtual object at a location and / or position within the user's viewpoint, the virtual object is environment-locked (or, "world-locked"), the location and / or position being based on a location and / or object within a three-dimensional environment (e.g., a physical or virtual environment) (e.g., selected and / or anchored to that location and / or object with reference to it). As the user's viewpoint moves, the location and / or object in the environment relative to the user's viewpoint changes, causing the environment-locked virtual object to appear at different locations and / or positions within the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user appears at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) so that the tree is now centered to the left in the user's viewpoint (e.g., the tree's position shifts in the user's viewpoint), the environment-locked virtual object locked to the tree appears centered to the left in the user's viewpoint. In other words, the position and / or orientation of an environment-locked virtual object displayed in the user's viewpoint depends on the position to which the virtual object is locked and / or the orientation and / or orientation of the object within the environment. In some implementations, the computer system uses a stationary frame of reference (e.g., a coordinate system anchored to a fixed position and / or object in the physical environment) to determine the position of the environment-locked virtual object displayed in the user's viewpoint. An environment-locked virtual object may be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object), or it may be locked to a movable part of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot), causing the virtual object to move with the viewpoint or that part of the environment to maintain a fixed relationship between the virtual object and that part of the environment.

[0046] In some implementations, environment-locked or viewpoint-locked virtual objects exhibit lazy following behavior, reducing or delaying their movement relative to the movement of a reference point they are following. In some implementations, when exhibiting lazy following behavior, the computer system intentionally delays the movement of the virtual object when movement of the reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 cm and 300 cm from the viewpoint) is detected. For example, when the reference point (e.g., that portion of the environment or the viewpoint) moves at a first rate, the virtual object is moved by the device to remain locked to the reference point, but moves at a second rate that is slower than the first rate (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up). In some implementations, when the virtual object exhibits lazy following behavior, the device ignores small movements of the reference point (e.g., ignores movements of the reference point below a threshold amount, such as 0 to 5 degrees or 0 cm to 50 cm). For example, when the reference point (e.g., a portion of the environment or viewpoint to which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed to maintain a fixed or substantially fixed position relative to a portion of the viewpoint or environment to which the virtual object is locked), and when the reference point (e.g., a portion of the environment or viewpoint to which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object first increases (e.g., because the virtual object is being displayed to maintain a fixed or substantially fixed position relative to a portion of the viewpoint or environment to which the virtual object is locked), and then decreases when the amount of movement of the reference point increases to above a threshold (e.g., a "lazy following" threshold), because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some implementations, maintaining a substantially fixed position of the virtual object relative to a reference point includes displaying the virtual object within a threshold distance (e.g., 1cm, 2cm, 3cm, 5cm, 15cm, 20cm, 50cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the reference point).

[0047] Hardware: Many different types of electronic systems enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablet devices, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection techniques that project graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into a physical environment, such as as holograms or onto a physical surface. In some embodiments, controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. The following is relative to... Figure 2The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device located locally or remotely relative to scene 105 (e.g., physical environment). For example, the controller 110 is a local server located within scene 105. Alternatively, the controller 110 is a remote server (e.g., a cloud server, central server, etc.) located outside scene 105. In some embodiments, the controller 110 is communicatively coupled to display generation components 120 (e.g., HMD, monitor, projector, touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, controller 110 is included within the housing (e.g., physical enclosure) of display generation component 120 (e.g., HMD or portable electronic device including display and one or more processors), one or more input devices in input device 125, one or more output devices in output device 155, one or more sensors in sensor 190, and / or one or more peripheral devices in peripheral device 195, or shares the same physical housing or support structure with one or more of the aforementioned devices.

[0048] In some implementations, display generation component 120 is configured to provide an XR experience to a user (e.g., at least the visual component of the XR experience). In some implementations, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. The following is relative to... Figure 3 The display generation component 120 is described in more detail. In some embodiments, the functionality of the controller 110 is provided by and / or combined with the display generation component 120.

[0049] According to some implementation schemes, when a user is virtually and / or physically present within scene 105, display generation component 120 provides the user with an XR experience.

[0050] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head, his / her hand, etc.). Thus, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds the device having a display facing the user's field of view and a camera facing scene 105. In some embodiments, the handheld device is optionally placed within a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is an XR chamber, housing, or room configured to present XR content, wherein the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware used for displaying XR content (e.g., a handheld device or a tripod-mounted device) can be implemented on another type of hardware used for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface illustrating interaction with XR content triggered by an interaction occurring in the space in front of a handheld device or tripod-mounted device can be similarly implemented using an HMD, where the interaction occurs in the space in front of the HMD and the response to the XR content is displayed via the HMD. Similarly, a user interface illustrating interaction with XR content triggered by movement of a handheld device or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)) can be similarly implemented using an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)).

[0051] Despite Figure 1A The relevant features of the operating environment 100 are illustrated herein, but those skilled in the art will understand from this disclosure that various other features are not illustrated for the sake of brevity and to avoid obscuring further relevant aspects of the exemplary embodiments disclosed herein.

[0052] Figures 1A to 1PVarious examples of computer systems for performing methods and providing audio, visual, and / or haptic feedback as part of the user interface described herein are illustrated. In some embodiments, the computer system includes one or more display generation components (e.g., first and second display components 1-120a, 1-120b and / or first and second optical modules 11.1.1-104a and 11.1.1-104b) for displaying virtual elements and / or representations of the physical environment to a user of the computer system, the virtual elements and / or the representations of the physical environment optionally being generated based on detected events and / or user input detected by the computer system. The user interface generated by the computer system is optionally corrected by one or more corrective lenses 11.3.2-216 to make it easier for a user who would otherwise use glasses or contact lenses to correct their vision to view the user interface, the one or more corrective lenses optionally being removably attached to one or more optical modules in the optical modules. While many user interfaces illustrated herein represent a single view of the user interface, user interfaces in an HMD may optionally use two optical modules (e.g., first display component 1-120a and second display component 1-120b and / or first optical module 11.1.1-104a and second optical module 11.1.1-104b) for display, one optical module for the user's right eye and a different optical module for the user's left eye, presenting slightly different images to the two different eyes to generate the illusion of stereoscopic depth. A single view of the user interface is typically a right-eye view or a left-eye view; the depth effect is explained in text or using other diagrams or views. In some embodiments, the computer system includes one or more external displays (e.g., display component 1-108) for displaying status information of the computer system to the user of the computer system (when the computer system is not worn) and / or to others near the computer system. This status information may optionally be generated based on detected events and / or user input detected by the computer system. In some embodiments, the computer system includes one or more audio output components (e.g., electronic components 1-112) for generating audio feedback, which may optionally be generated based on detected events and / or user input detected by the computer system. In some embodiments, the computer system includes one or more input devices for detecting input, such as one or more sensors (e.g., sensor components 1-356 and / or...) for detecting information about the physical environment of the device. Figure 1I One or more sensors), which can be used (optionally with one or more illuminators, such as Figure 1IThe system combines the illuminators described herein to generate digital pass-through images, capture visual media (e.g., photographs and / or videos) corresponding to the physical environment, or determine the pose (e.g., positioning and / or orientation) of physical objects and / or surfaces in the physical environment, enabling the placement of virtual objects based on the detected pose of the physical objects and / or surfaces. In some embodiments, the computer system includes one or more input devices for detecting input, such as one or more sensors (e.g., sensor assemblies 1-356 and / or...) for detecting hand positioning and / or movement. Figure 1I One or more sensors), which can be used (optionally with one or more illuminators, such as Figure 1I The illuminators 6-124 described herein (in combination) determine when one or more air gestures are performed. In some embodiments, the computer system includes one or more input devices for detecting input, such as one or more sensors for detecting eye movement (e.g., Figure 1I (Eye-tracking and gaze-tracking sensors in the system), these sensors can be used (optionally combined with one or more lights, such as...) Figure 10The light (11.3.2-110) in the image determines attention or gaze localization and / or gaze movement, which can optionally be used to detect gaze-only input based on gaze movement and / or dwell. Combinations of the various sensors described above can be used to determine user facial expressions and / or hand movements for generating an avatar or representation of the user, such as an anthropomorphic avatar or representation for real-time communication sessions, wherein the avatar has facial expressions, hand movements, and / or body movements detected by the user based on or similar to the device. Gaze and / or attention information may optionally be combined with hand tracking information to determine user interaction with one or more user interfaces based on direct and / or indirect input, such as air gestures or input using one or more hardware input devices, such as one or more buttons (e.g., first buttons 1-128, buttons 11.1.1-114, second buttons 1-132 and / or dials or buttons 1-328), knobs (e.g., first buttons 1-128, buttons 11.1.1-114 and / or dials or buttons 1-328), digital crowns (e.g., pressable and twistable or rotatable first buttons 1-128, buttons 11.1.1-114 and / or dials or buttons 1-328), touchpads, touchscreens, keyboards, mice and / or other input devices. One or more buttons (e.g., first buttons 1-128, buttons 11.1.1-114, second buttons 1-132, and / or dials or buttons 1-328) may optionally be used to perform system operations, such as recentering content in the user-visible 3D environment of the device, displaying the main user interface for launching an application, initiating a real-time communication session, or initiating the display of a virtual 3D background. A knob or digital crown (e.g., pressable and twistable or rotatable first buttons 1-128, buttons 11.1.1-114, and / or dials or buttons 1-328) may optionally be rotatable to adjust parameters of the visual content, such as the level of immersion of the virtual 3D environment (e.g., the extent to which the virtual content occupies the user's viewport in the 3D environment) or other parameters associated with the 3D environment and the virtual content displayed via optical modules (e.g., first display components 1-120a and second display components 1-120b and / or first optical modules 11.1.1-104a and second optical modules 11.1.1-104b).

[0053] Figure 1BExamples of head-mounted display (HMD) devices 1-100 configured to be worn by a user and provide virtual and altered / mixed reality (VR / AR) experiences are illustrated in front, top, and perspective views. The HMD 1-100 may include a display unit 1-102 or assembly, an electronic strip assembly 1-104 connected to and extending from the display unit 1-102, and a strap assembly 1-106 secured at either end to the electronic strip assembly 1-104. The electronic strip assembly 1-104 and the strap 1-106 may be part of a retention assembly configured to wrap around the user's head to hold the display unit 1-102 against the user's face.

[0054] In at least one example, the band assembly 1-106 may include a first band 1-116 configured to wrap around the back of the user's head and a second band 1-117 configured to extend above the top of the user's head. As shown, the second band may extend between the first electronic band 1-105a and the second electronic band 1-105b of the electronic band assembly 1-104. The band assembly 1-104 and the band assembly 1-106 may be part of a fixing mechanism that extends rearward from the display unit 1-102 and is configured to hold the display unit 1-102 against the user's face.

[0055] In at least one example, the fixing mechanism includes a first electronic strip 1-105a, which includes a first proximal end 1-134 coupled to a display unit 1-102 (e.g., a housing 1-150 of the display unit 1-102) and a first distal end 1-136 opposite to the first proximal end 1-134. The fixing mechanism may also include a second electronic strip 1-105b, which includes a second proximal end 1-138 coupled to the housing 1-150 of the display unit 1-102 and a second distal end 1-140 opposite to the second proximal end 1-138. The fixing mechanism may also include a first strip 1-116 and a second strip 1-117, the first strip including a first end 1-142 coupled to the first distal end 1-136 and a second end 1-144 coupled to the second distal end 1-140, and the second strip extending between the first electronic strip 1-105a and the second electronic strip 1-105b. Strips 1-105a to b and strip 1-116 may be coupled via a connecting mechanism or component 1-114. In at least one example, the second strip 1-117 includes a first end 1-146 coupled to the first electronic strip 1-105a between a first proximal end 1-134 and a first distal end 1-136, and a second end 1-148 coupled to the second electronic strip 1-105b between a second proximal end 1-138 and a second distal end 1-140.

[0056] In at least one example, the first electronic strip and the second electronic strips 1-105a to b comprise plastic, metal, or other structural materials forming the shape of the substantially rigid strips 1-105a to b. In at least one example, the first strip and the second strips 1-116, 1-117 are formed of an elastic flexible material (including woven textiles, rubber, etc.). The first strip 1-116 and the second strip 1-117 may be flexible enough to conform to the shape of the user's head when wearing the HMD 1-100.

[0057] In at least one example, one or more of the first electronic stripe and the second electronic stripe 1-105a to b may define an inner stripe volume and include one or more electronic components disposed within the inner stripe volume. In one example, such as Figure 1B As shown, the first electronic strip 1-105a may include electronic components 1-112. In one example, electronic components 1-112 may include a speaker. In another example, electronic components 1-112 may include computing components, such as a processor.

[0058] In at least one example, the housing 1-150 defines a first front opening 1-152. The front opening is located in... Figure 1B The section marked 1-152 with dashed lines is because the display assembly 1-108 is configured to obscure the first opening 1-152 from a view when the HMD 1-100 is assembled. The housing 1-150 may also define a rearward second opening 1-154. The housing 1-150 also defines an internal volume between the first opening 1-152 and the second opening 1-154. In at least one example, the HMD 1-100 includes a display assembly 1-108, which may include a front cover disposed in or across the front opening 1-152 to obscure the front opening 1-152 and a display screen (shown in other figures). In at least one example, the display screen of the display assembly 1-108, and the display assembly 1-108 in general, has a curvature configured to follow the curvature of the user's face. The display screen of display unit 1-108 can be bent as shown to complement the user's facial features and the overall curvature from one side of the face to the other, such as from left to right and / or from top to bottom, wherein display unit 1-102 is pressed.

[0059] In at least one example, the housing 1-150 may define a first hole 1-126 between a first opening 1-152 and a second opening 1-154, and a second hole 1-130 between the first opening 1-152 and the second opening 1-154. The HMD 1-100 may also include a first button 1-126 disposed in the first hole 1-128, and a second button 1-132 disposed in the second hole 1-130. The first button 1-128 and the second button 1-132 are pressable through their respective holes 1-126 and 1-130. In at least one example, the first button 1-126 and / or the second button 1-132 may be a rotary dial and a pressable button. In at least one example, the first button 1-128 is a pressable and rotary dial button, and the second button 1-132 is a pressable button.

[0060] Figure 1C A rear perspective view of HMD 1-100 is illustrated. HMD 1-100 may include a light seal 1-110 extending rearwardly around the periphery of housing 1-150 of display assembly 1-108, as shown. The light seal 1-110 may be configured to extend from housing 1-150 to the user's face, surrounding the user's eyes, to block external light from being visible. In one example, HMD 1-100 may include a first display assembly 1-120a and a second display assembly 1-120b disposed at or within a rearwardly facing second opening 1-154 defined by housing 1-150 and / or disposed within the internal volume of housing 1-150 and configured to project light through the second opening 1-154. In at least one example, each display assembly 1-120a to b may include a corresponding display screen 1-122a, 1-122b configured to project light toward the user's eyes in a rearward direction through the second opening 1-154.

[0061] In at least one example, reference Figure 1B and Figure 1C Both, the display assembly 1-108 can be a front-facing display assembly including a display screen configured to project light in a first forward direction, and the rear display screens 1-122a to b can be configured to project light in a second rearward direction opposite to the first direction. As described above, the light seal 1-110 can be configured to block light from outside the HMD 1-100 from reaching the user's eyes, including a component made of... Figure 1B The front perspective view shows the light projected by the front display screen of the display assembly 1-108. In at least one example, the HMD 1-100 may also include a curtain 1-124 that blocks the second opening 1-154 between the housing 1-150 and the rear display assemblies 1-120a to b. In at least one example, the curtain 1-124 may be elastic or at least partially elastic.

[0062] Figure 1B and Figure 1C Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1D to 1F In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1D to 1F Any of the features, components and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1B and Figure 1C Examples of devices, features, components, and parts are shown.

[0063] Figure 1D An exploded view of an example HMD 1-200 including its various parts or components, separated according to the modularity and selective coupling of these components. For example, HMD 1-200 may include a strip 1-216 selectively coupled to a first electronic strip 1-205a and a second electronic strip 1-205b. The first fixed strip 1-205a may include a first electronic component 1-212a, and the second fixed strip 1-205b may include a second electronic component 1-212b. In at least one example, the first strip and the second strips 1-205a to 1-205b are removably coupled to a display unit 1-202.

[0064] Furthermore, HMD 1-200 may include a light-sealing member 1-210 configured to be removably coupled to display unit 1-202. HMD 1-200 may also include a lens 1-218, which may be removably coupled to display unit 1-202, for example, on a first display assembly and a second display assembly including a display screen. Lens 1-218 may include a custom prescription lens configured for vision correction. As noted, in Figure 1D The exploded view shows that each component described above can be removably coupled, attached, reattached, and replaced to update the component, or replaced for different users. For example, belts such as belt 1-216, light seals such as light seal 1-210, lenses such as lens 1-218, and electronic strips such as electronic strips 1-205a to b can be replaced according to the user, so that these parts are customized to fit and correspond to a single user of HMD 1-200.

[0065] Figure 1D Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figure 1B , Figure 1C and Figures 1E to 1FIn any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figure 1B , Figure 1C and Figures 1E to 1F Any of the features, components and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1D Examples of devices, features, components, and parts are shown.

[0066] Figure 1E An exploded view illustrating an example of a display unit 1-306 of an HMD is shown. The display unit 1-306 may include a front display assembly 1-308, a frame / housing assembly 1-350, and a curtain assembly 1-324. The display unit 1-306 may also include a sensor assembly 1-356, a logic board assembly 1-358, and a cooling assembly 1-360 disposed between the frame assembly 1-350 and the front display assembly 1-308. In at least one example, the display unit 1-306 may also include a rear display assembly 1-320, which includes a first rear display screen 1-322a and a second rear display screen 1-322b disposed between the frame 1-350 and the curtain assembly 1-324.

[0067] In at least one example, the display unit 1-306 may further include a motor assembly 1-362 configured as an adjustment mechanism for adjusting the positioning of the display screens 1-322a to b of the display unit 1-320 relative to the frame 1-350. In at least one example, the display unit 1-320 is mechanically coupled to the motor assembly 1-362, and each display screen 1-322a to b has at least one motor, such that the motor is capable of translating the display screens 1-322a to b to match the interpupillary distance of the user's eyes.

[0068] In at least one example, display unit 1-306 may include a dial or button 1-328 that is pressable relative to frame 1-350 and accessible to a user outside frame 1-350. Button 1-328 may be electrically connected to motor assembly 1-362 via a controller, such that button 1-328 can be operated by a user to cause the motor of motor assembly 1-362 to adjust the positioning of display screens 1-322a to b.

[0069] Figure 1E Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1B to 1D and Figure 1F In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1B to 1D and Figure 1FAny of the features, components and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1E Examples of devices, features, components, and parts are shown.

[0070] Figure 1F An exploded view of another example of a display unit 1-406 of an HMD device similar to other HMD devices described herein is illustrated. The display unit 1-406 may include a front display assembly 1-402, a sensor assembly 1-456, a logic board assembly 1-458, a cooling assembly 1-460, a frame assembly 1-450, a rear display assembly 1-421, and a curtain assembly 1-424. The display unit 1-406 may also include a motor assembly 1-462 for adjusting the positioning of a first display sub-assembly 1-420a and a second display sub-assembly 1-420b of the rear display assembly 1-421, including a first and second corresponding display screen for interpupillary adjustment, as described above.

[0071] References in this article Figures 1B to 1E The following figures, which are referenced in this disclosure, will be used to describe the subject in more detail. Figure 1F The exploded view shows the various parts, systems, and components. Figure 1F The display unit 1-406 shown can be connected with Figures 1B to 1E The shown fixture assembly and integration includes electronic strips, belts, and other components (including light seals, connecting components, etc.).

[0072] Figure 1F Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1B to 1E In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1B to 1E Any of the features, components and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1F Examples of devices, features, components, and parts are shown.

[0073] Figure 1G An example is the front cover assembly 3-100 of the HMD device described herein (e.g., Figure 1G An exploded perspective view of the front cover assembly 3-1) of the HMD 3-100 shown or any other HMD device shown and described herein. Figure 1GThe front cover assembly 3-100 shown may include a transparent or translucent cover 3-102, a shield 3-104 (or “cover”), an adhesive layer 3-106, a display assembly 3-108 including a biconvex lens panel or array 3-110, and a structural decorative element 3-112. The adhesive layer 3-106 secures the shield 3-104 and / or the transparent cover 3-102 to the display assembly 3-108 and / or the decorative element 3-112. The decorative element 3-112 secures various components of the front cover assembly 3-100 to the frame or base of the HMD device.

[0074] In at least one example, such as Figure 1G As shown, the transparent cover 3-102, the protective cover 3-104, and the display assembly 3-108 including a biconvex lens array 3-110 can be bent to adapt to the curvature of a user's face. The transparent cover 3-102 and the protective cover 3-104 can be bent in two or three dimensions, for example, vertically in and out of the Z-plane along the Z direction, and horizontally in and out of the ZX-plane along the X direction. In at least one example, the display assembly 3-108 may include the biconvex lens array 3-110 and a display panel with pixels configured to project light through the protective cover 3-104 and the transparent cover 3-102. The display assembly 3-108 can be bent in at least one direction (e.g., the horizontal direction) to adapt to the curvature of a user's face from one side (e.g., the left) to the other (e.g., the right). In at least one example, each layer or component of the display assembly 3-108 (which will be shown and described in more detail in the following figures, but may include the biconvex lens array 3-110 and the display layer) may be similarly or concentrically curved in the horizontal direction to accommodate the curvature of the user's face.

[0075] In at least one example, the cover 3-104 may include a transparent or translucent material through which the display component 3-108 projects light. In one example, the cover 3-104 may include one or more opaque portions, such as opaque ink-printed portions or other opaque film portions on the back of the cover 3-104. When the HMD device is worn, the rear surface may be the surface of the cover 3-104 facing the user's eyes. In at least one example, the opaque portion may be on the front surface of the cover 3-104 opposite the rear surface. In at least one example, one or more opaque portions of the cover 3-104 may include peripheral portions that visually conceal any components surrounding the outer periphery of the display screen of the display component 3-108. In this way, the opaque portions of the cover conceal any other components of the HMD device that would otherwise be visible through the transparent or translucent cover 3-102 and / or the cover 3-104, including electronic components, structural components, etc.

[0076] In at least one example, the housing 3-104 may define one or more transparent aperture portions 3-120 through which sensors can transmit and receive signals. In one example, portion 3-120 is an aperture through which sensors can extend or transmit and receive signals. In one example, portion 3-120 is a transparent portion, or a portion more transparent than the surrounding translucent or opaque portion of the housing, through which sensors can transmit and receive signals through the housing and via transparent cover 3-102. In one example, the sensor may include a camera, an IR sensor, a LUX sensor, or any other visual or non-visual environmental sensor of the HMD device.

[0077] Figure 1G Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included, individually or in any combination, in any other example of the devices, features, components, and parts described herein. Similarly, any of the features, components, and / or parts shown and described herein (including their arrangement and configuration) may be included, individually or in any combination. Figure 1G Examples of devices, features, components, and parts are shown.

[0078] Figure 1H An exploded view of an example HMD device 6-100 is shown. The HMD device 6-100 may include a sensor array or system 6-102, which includes one or more sensors, cameras, projectors, etc., mounted to one or more components of the HMD 6-100. In at least one example, the sensor system 6-102 may include a bracket 1-338 on which one or more sensors of the sensor system 6-102 may be fixed / secured.

[0079] Figure 1I A portion of an HMD device 6-100, including a front transparent cover 6-104 and a sensor system 6-102, is illustrated. The sensor system 6-102 may include multiple different sensors, transmitters, and receivers, including cameras, IR sensors, projectors, etc. The transparent cover 6-104 is illustrated on the front of the sensor system 6-102 to illustrate the relative positioning of the various sensors and transmitters and the orientation of each sensor / transmitter in system 6-102. As referenced herein, "side," "side," "lateral," "horizontal," and other similar terms refer to... Figure 1J The orientation or direction indicated by the X-axis. Terms such as "vertical," "upward," "downward," and similar terms refer to the orientation or direction indicated by... Figure 1JThe orientation or direction indicated by the Z-axis. Terms such as "frontward," "rearward," "forward," "backward," and similar terms refer to the orientation or direction indicated by the Z-axis. Figure 1J The orientation or direction indicated by the Y-axis shown.

[0080] In at least one example, a transparent cover 6-104 may define the front outer surface of an HMD device 6-100, and a sensor system 6-102, including various sensors and their components, may be positioned behind the cover 6-104 in the Y-axis / direction. The cover 6-104 may be transparent or translucent to allow light to pass through it, including both light detected by the sensor system 6-102 and light emitted therefrom.

[0081] As described elsewhere herein, the HMD device 6-100 may include one or more controllers, which include processors for electrically coupling various sensors and transmitters of the sensor system 6-102 to one or more motherboards, processing units, and other electronic devices such as displays. Furthermore, as will be shown in more detail below with reference to other accompanying drawings, various sensors, transmitters, and other components of the sensor system 6-102 may be coupled to the HMD device 6-100. Figure 1I Various structural frame components, brackets, etc., not shown. For clarity, Figure 1I The components of the sensor system 6-102 are shown, which are not attached to or electrically coupled to other components.

[0082] In at least one example, the device may include one or more controllers having a processor configured to execute instructions stored on a memory component electrically coupled to the processor. These instructions may include, or cause the processor to execute, one or more algorithms for self-correcting the angle and position of the various cameras described herein as the camera's initial position, angle, or orientation is affected by collisions or deformations due to accidental drop events or other events over time.

[0083] In at least one example, the sensor system 6-102 may include one or more scene cameras 6-106. System 6-102 may include two scene cameras 6-102, respectively positioned on either side of the nose bridge or arched structure of the HMD device 6-100, such that each of the two cameras 6-106 approximately corresponds to the positioning of the user's left and right eyes behind the cover 6-103. In at least one example, the scene cameras 6-106 are generally oriented forward in the Y direction to capture images in front of the user during use of the HMD 6-100. In at least one example, the scene cameras are color cameras and, when the HMD device 6-100 is used, provide images and content for MR video pass-through to a display screen facing the user's eyes. The scene cameras 6-106 may also be used for environment and object reconstruction.

[0084] In at least one example, the sensor system 6-102 may include a first depth sensor 6-108 that is generally pointing forward in the Y direction. In at least one example, the first depth sensor 6-108 may be used for environment and object reconstruction as well as user hand and body tracking. In at least one example, the sensor system 6-102 may include a second depth sensor 6-110 centrally located along the width of the HMD device 6-100 (e.g., along the X-axis). For example, the second depth sensor 6-110 may be located above the central bridge of the nose or on an adapter structure above the nose when the user wears the HMD 6-100. In at least one example, the second depth sensor 6-110 may be used for environment and object reconstruction as well as hand and body tracking. In at least one example, the second depth sensor may include a LiDAR sensor.

[0085] In at least one example, the sensor system 6-102 may include a depth projector 6-112, which is typically forward-facing to project electromagnetic waves (e.g., in the form of a predetermined spot pattern) into or within the field of view of the user and / or scene camera 6-106, or into or beyond the field of view of the user and / or scene camera 6-106. In at least one example, the depth projector is capable of projecting electromagnetic waves of light in the form of a spot pattern, which are reflected from an object and back into the aforementioned depth sensors, including depth sensors 6-108 and 6-110. In at least one example, the depth projector 6-112 may be used for environment and object reconstruction, as well as hand and body tracking.

[0086] In at least one example, the sensor system 6-102 may include a downward-facing camera 6-114, whose field of view is generally directed downwards relative to the HMD device 6-100 on the Z-axis. In at least one example, the downward-facing camera 6-114 may be positioned as shown on the left and right sides of the HMD device 6-100 and used for hand and body tracking, head-mounted device tracking, and facial avatar detection and creation for displaying a user avatar on the front display screen of the HMD device 6-100 as described elsewhere herein. For example, the downward-facing camera 6-114 may be used to capture facial expressions and movements of the user's face below the HMD device 6-100, including the cheeks, mouth, and chin.

[0087] In at least one example, the sensor system 6-102 may include a jaw camera 6-116. In at least one example, the jaw camera 6-116 may be positioned as shown on the left and right sides of the HMD device 6-100 and used for hand and body tracking, head-mounted device tracking, and facial avatar detection and creation for displaying a user avatar on the front display screen of the HMD device 6-100 as described elsewhere herein. For example, the jaw camera 6-116 may be used to capture facial expressions and movements of the user's face (including the user's jaw, cheeks, mouth, and chin) below the HMD device 6-100. For hand and body tracking, head-mounted device tracking, and facial avatar detection... In at least one example, the sensor system 6-102 may include a side camera 6-118. The side camera 6-118 may be oriented to capture left and right views along the X-axis or in a direction relative to the HMD device 6-100. In at least one example, the side camera 6-118 may be used for hand and body tracking, head-mounted device tracking, and facial avatar detection and reconstruction.

[0088] In at least one example, the sensor system 6-102 may include multiple eye-tracking and gaze-tracking sensors for determining identity, status, and the user's gaze direction during and / or prior to use. In at least one example, the eye / gaze-tracking sensor may include a nose-eye camera 6-120 positioned on either side of the user's nose and adjacent to the user's nose when wearing the HMD device 6-100. The eye / gaze sensor may also include a bottom eye camera 6-122 positioned below the respective user's eye for capturing images of the eye for use in facial avatar detection and creation, gaze tracking, and iris identification functions.

[0089] In at least one example, sensor system 6-102 may include an infrared illuminator 6-124 that is pointed outward from HMD device 6-100 to illuminate the external environment and any objects therein with IR light for IR detection using one or more IR sensors of sensor system 6-102. In at least one example, sensor system 6-102 may include a flicker sensor 6-126 and an ambient light sensor 6-128. In at least one example, flicker sensor 6-126 may detect the refresh rate of the overhead light to avoid display flicker. In one example, infrared illuminator 6-124 may include a light-emitting diode and may be specifically designed for low-light environments to illuminate a user's hands and other objects in low light for detection by the infrared sensors of sensor system 6-102.

[0090] In at least one example, multiple sensors (including scene camera 6-106, downward camera 6-114, chin camera 6-116, side camera 6-118, depth projector 6-112, and depth sensors 6-108, 6-110) can be used in combination with an electrically coupled controller to combine depth data with camera data for hand tracking and for sizing, thereby improving the hand tracking and object recognition and tracking functions of the HMD device 6-100. In at least one example, the above-described and Figure 1I The downward-facing camera 6-114, the jaw camera 6-116, and the side camera 6-118 shown can be wide-angle cameras capable of operating in both the visible and infrared spectra. In at least one example, these cameras 6-114, 6-116, and 6-118 can operate solely in black-and-white light detection to simplify image processing and achieve sensitivity.

[0091] Figure 1I Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1J to 1L In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1J to 1L Any of the features, components and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1I Examples of devices, features, components, and parts are shown.

[0092] Figure 1JA lower perspective view of an example HMD 6-200 including a cover or shield 6-204 fixed to a frame 6-230 is shown. In at least one example, a sensor 6-203 of a sensor system 6-202 may be disposed around the periphery of the HMD 6-200 such that the sensor 6-203 is disposed outwardly around the periphery of the display area or region 6-232 so as not to obstruct the view of the displayed light. In at least one example, the sensor may be disposed behind the shield 6-204 and aligned with a transparent portion of the shield, thereby allowing light to pass back and forth through the shield 6-204 by the sensor and the projector. In at least one example, an opaque ink or other opaque material or film / layer may be disposed on the shield 6-204 around the display area 6-232 to conceal components of the HMD 6-200 outside the display area 6-232 rather than through a transparent portion defined by the opaque portion through which the sensor and the projector transmit and receive light and electromagnetic signals during operation. In at least one example, the shield 6-204 allows light to pass through the display (e.g., within the display area 6-232), but does not allow light to pass radially outward from the display area surrounding the periphery of the display and the shield 6-204.

[0093] In some examples, the shield 6-204 includes a transparent portion 6-205 and an opaque portion 6-207, as described above and elsewhere herein. In at least one example, the opaque portion 6-207 of the shield 6-204 may define one or more transparent areas 6-209 through which the sensor 6-203 of the sensor system 6-202 transmits and receives signals. In the illustrated examples, the sensor 6-203 of the sensor system 6-202, which transmits and receives signals through the shield 6-204, or more specifically through the transparent area 6-209 defined by the opaque portion 6-207 of the shield 6-204, may include... Figure 1I The examples illustrate those same or similar sensors, such as depth sensors 6-108 and 6-110, depth projector 6-112, first scene camera and second scene camera 6-106, first downward camera and second downward camera 6-114, first side camera and second side camera 6-118, and first infrared illuminator and second infrared illuminator 6-124. These sensors also... Figure 1K and Figure 1L The example is shown. Other sensors, sensor types, number of sensors, and their relative positioning can be included in one or more other examples of the HMD.

[0094] Figure 1J Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figure 1I and Figures 1K to 1LIn any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figure 1I and Figures 1K to 1L Any of the features, components and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1J Examples of devices, features, components, and parts are shown.

[0095] Figure 1K A front view of a portion of an example of an HMD device 6-300, including a display 6-334, brackets 6-336, 6-338, and a frame or housing 6-330, is shown. Figure 1K The examples shown do not include a front cover or shield to illustrate brackets 6-336 and 6-338. For example, Figure 1J The shield 6-204 shown includes an opaque portion 6-207 that visually covers / blocks the view of anything outside the display / display area 6-334 (e.g., radially / peripherally outside the display / display area), including the sensor 6-303 and the bracket 6-338.

[0096] In at least one example, various sensors of sensor system 6-302 are coupled to brackets 6-336, 6-338. In at least one example, scene camera 6-306 includes strict tolerances for angles relative to each other. For example, the tolerance for the mounting angle between two scene cameras 6-306 may be 0.5 degrees or less, such as 0.3 degrees or less. To achieve and maintain such strict tolerances, in one example, scene camera 6-306 may be mounted to bracket 6-338 instead of a housing. The bracket may include a cantilever on which scene camera 6-306 and other sensors of sensor system 6-302 may be mounted to maintain their positioning and orientation in the event of a drop event caused by a user that results in any deformation of other brackets 6-226, housing 6-330, and / or housing.

[0097] Figure 1K Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1I to 1J and Figure 1L In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1I to 1J and Figure 1L Any of the features, components and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1K Examples of devices, features, components, and parts are shown.

[0098] Figure 1L A bottom view illustrating an example of an HMD 6-400 including a front display / cover assembly 6-404 and a sensor system 6-402 is shown. The sensor system 6-402 is compatible with the above and other parts of this document (including references). Figures 1I to 1K Other sensor systems described are similar. In at least one example, the jaw camera 6-416 may be oriented downwards to capture images of the user's lower facial features. In one example, the jaw camera 6-416 may be directly coupled to a frame or housing 6-430 or one or more internal brackets that are directly coupled to the frame or housing 6-430 shown. The frame or housing 6-430 may include one or more holes / openings 6-415 through which the jaw camera 6-416 transmits and receives signals.

[0099] Figure 1L Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figures 1I to 1K In any other example of the devices, features, components, and parts shown and described herein. Similarly, refer to... Figures 1I to 1K Any of the features, components and / or parts shown and described (including their arrangement and configuration) may be included individually or in any combination. Figure 1L Examples of devices, features, components, and parts are shown.

[0100] Figure 1M A rear perspective view of an interpupillary distance (IPD) adjustment system 11.1.1-102 is illustrated. This IPD adjustment system includes a first optical module and a second optical module 11.1.1-104a-104a-104a-105a, slidably engaged / coupled to corresponding guide rods 11.1.1-108a ... In at least one example, buttons 11.1.1-114 can be electrically communicated with the first motor and the second motors 11.1.1-110a to b via a processor or other circuit components to activate the first motor and the second motors 11.1.1-110a to b and respectively cause the first optical module and the second optical modules 11.1.1-104a to b to change their positions relative to each other.

[0101] In at least one example, the first and second optical modules 11.1.1-104a to b may include corresponding display screens configured to project light toward the user's eyes when the HMD 11.1.1-100 is worn. In at least one example, the user can manipulate (e.g., press and / or rotate) buttons 11.1.1-114 to activate positional adjustment of the optical modules 11.1.1-104a to b to match the interpupillary distance of the user's eyes. The optical modules 11.1.1-104a to b may also include one or more cameras or other sensors / sensor systems for imaging and measuring the user's IPD, such that the optical modules 11.1.1-104a to b can be adjusted to match the IPD.

[0102] In one example, a user can manipulate buttons 11.1.1-114 to cause automatic positional adjustment of the first and second optical modules 11.1.1-104a to b. In another example, a user can manipulate buttons 11.1.1-114 to cause manual adjustment, moving the optical modules 11.1.1-104a to b further or closer (e.g., when the user rotates buttons 11.1.1-114 in one way or another) until the user visually matches their own IPD. In one example, manual adjustment is communicated electronically via one or more circuits, and the power for moving the optical modules 11.1.1-104a to b via motors 11.1.1-110a to b is supplied by a power source. In another example, the adjustment and movement of the optical modules 11.1.1-104a to b via the manipulation buttons 11.1.1-114 are mechanically actuated via the movement buttons 11.1.1-114.

[0103] Figure 1M Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included, individually or in any combination, in any other example of the devices, features, components, and parts shown and described herein in any other illustrated figures. Similarly, any of the features, components, and / or parts shown and described herein with reference to any other illustrated figures (including their arrangement and configuration) may be included, individually or in any combination, in any other example of the devices, features, components, and / or parts shown and / or described herein. Figure 1M Examples of devices, features, components, and parts are shown.

[0104] Figure 1N A front perspective view of a portion of HMD 11.1.2-100 is shown, including an outer structural frame 11.1.2-102 defining first and second holes 11.1.2-106a, 11.1.2-106b, and an inner or intermediate structural frame 11.1.2-104. Holes 11.1.2-106a to b are located in... Figure 1NThe holes 11.1.2-106a to b are shown in dashed lines because viewing the HMD 11.1.2-100 may be obstructed by one or more other components coupled to the inner frame 11.1.2-104 and / or the outer frame 11.1.2-102, as illustrated. In at least one example, the HMD 11.1.2-100 may include a first mounting bracket 11.1.2-108 coupled to the inner frame 11.1.2-104. In at least one example, the mounting bracket 11.1.2-108 is coupled to the inner frame 11.1.2-104 between the first and second holes 11.1.2-106a to b.

[0105] Mounting brackets 11.1.2-108 may include intermediate or central portions 11.1.2-109 coupled to the inner frame 11.1.2-104. In some examples, the intermediate or central portions 11.1.2-109 may not be the geometric center or middle of the brackets 11.1.2-108. Instead, the intermediate / central portions 11.1.2-109 may be positioned between a first cantilever extension arm and a second cantilever extension arm extending away from the intermediate portions 11.1.2-109. In at least one example, mounting bracket 108 includes first cantilever arms 11.1.2-112 and second cantilever arms 11.1.2-114 extending away from the intermediate portions 11.1.2-109 of the mounting brackets 11.1.2-108 coupled to the inner frame 11.1.2-104.

[0106] like Figure 1N As shown, the outer frame 11.1.2-102 may define a curved geometry on its lower side to adapt to the user's nose when the user wears the HMD 11.1.2-100. This curved geometry may be referred to as the bridge of the nose 11.1.2-111 and is centrally located on the lower side of the HMD 11.1.2-100 as shown. In at least one example, the mounting bracket 11.1.2-108 may be connected to the inner frame 11.1.2-104 between holes 11.1.2-106a and b, such that the cantilever 11.1.2-112, 11.1.2-114 extend downward and laterally outward away from the central portion 11.1.2-109 to complement the nose bridge geometry of the outer frame 11.1.2-102. In this way, the mounting bracket 11.1.2-108 is configured to adapt to the user's nose, as described above. The geometry of the bridge of the nose 11.1.2-111 adapts to the nose, as it provides a curvature that conforms to the shape of the user's nose, offering a comfortable fit from above, above, and around.

[0107] The first cantilever 11.1.2-112 may extend in a first direction away from the middle portion 11.1.2-109 of the mounting bracket 11.1.2-108, and the second cantilever 11.1.2-114 may extend in a second direction opposite to the first direction away from the middle portion 11.1.2-109 of the mounting bracket 11.1.2-108. The first cantilever 11.1.2-112 and the second cantilever 11.1.2-114 are referred to as “cantilever” or “cantilever” arms because each arm 11.1.2-112, 11.1.2-114 includes free distal ends 11.1.2-116, 11.1.2-118, respectively, which are not attached to the inner frame 11.1.2-102 and the outer frame 11.1.2-104. In this way, arms 11.1.2-112 and 11.1.2-114 extend from the middle section 11.1.2-109, which can be connected to the inner frame 11.1.2-104, while the distal ends 11.1.2-102 and 11.1.2-104 are not attached.

[0108] In at least one example, the HMD 11.1.2-100 may include one or more components coupled to the mounting bracket 11.1.2-108. In one example, the components include multiple sensors 11.1.2-110a-f. Each of the multiple sensors 11.1.2-110a-f may include various types of sensors, including cameras, IR sensors, etc. In some examples, one or more of the sensors 11.1.2-110a-f may be used for object recognition in three-dimensional space, making it important to maintain the precise relative positioning of two or more of the multiple sensors 11.1.2-110a-f. The cantilever nature of the mounting bracket 11.1.2-108 protects the sensors 11.1.2-110a-f from damage and displacement in the event of an accidental drop by the user. Because the sensors 11.1.2-110a-f cantilevered on the arms 11.1.2-112 and 11.1.2-114 of the mounting bracket 11.1.2-108, the stress and deformation of the internal frame and / or the external frames 11.1.2-104 and 11.1.2-102 are not transmitted to the cantilever arms 11.1.2-112 and 11.1.2-114, and therefore do not affect the relative position of the sensors 11.1.2-110a-f coupled to / mounted to the mounting bracket 11.1.2-108.

[0109] Figure 1NAny of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination in any other example of the devices, features, components, and other examples described herein. Similarly, any of the features, components, and / or parts shown and described herein (including their arrangement and configuration) may be included individually or in any combination. Figure 1N Examples of devices, features, components, and parts are shown.

[0110] Figure 10 An example of optical modules 11.3.2-100 for use in electronic devices, such as HMDs, including the HMD devices described herein, is illustrated. As shown in one or more other examples described herein, optical modules 11.3.2-100 may be one of two optical modules within an HMD, wherein each optical module is aligned to project light toward a user's eye. In this way, a first optical module may project light toward a user's first eye via a display screen, and a second optical module of the same device may project light toward a user's second eye via another display screen.

[0111] In at least one example, the optical module 11.3.2-100 may include an optical frame or housing 11.3.2-102, which may also be referred to as a tube or optical module tube. The optical module 11.3.2-100 may also include a display 11.3.2-104 coupled to the housing 11.3.2-102, the display including one or more display screens. The display 11.3.2-104 may be coupled to the housing 11.3.2-102 such that the display 11.3.2-104 is configured to project light toward the user's eyes when the HMD to which the display module 11.3.2-100 belongs is worn during use. In at least one example, the housing 11.3.2-102 may surround the display 11.3.2-104 and provide connection features for coupling other components of the optical module described herein.

[0112] In one example, the optical module 11.3.2-100 may include one or more cameras 11.3.2-106 coupled to the housing 11.3.2-102. The cameras 11.3.2-106 may be positioned relative to the display 11.3.2-104 and the housing 11.3.2-102 such that the cameras 11.3.2-106 are configured to capture one or more images of a user's eye during use. In at least one example, the optical module 11.3.2-100 may also include a light strip 11.3.2-108 surrounding the display 11.3.2-104. In one example, the light strip 11.3.2-108 is disposed between the display 11.3.2-104 and the camera 11.3.2-106. The light strip 11.3.2-108 may include a plurality of lights 11.3.2-110. The plurality of lights may include one or more light-emitting diodes (LEDs) or other lights configured to project light toward the user's eyes when the HMD is worn. The individual lights 11.3.2-110 in the light strips 11.3.2-108 may be spaced apart around the light strips 11.3.2-108, and are therefore uniformly or non-uniformly spaced around the display 11.3.2-104 at various locations on the light strips 11.3.2-108 and around the display 11.3.2-104.

[0113] In at least one example, the housing 11.3.2-102 defines a viewing opening 11.3.2-101 through which a user can view the display 11.3.2-104 when wearing the HMD device. In at least one example, the LEDs are configured and arranged to emit light onto the user's eyes through the viewing opening 11.3.2-101. In one example, a camera 11.3.2-106 is configured to capture one or more images of the user's eyes through the viewing opening 11.3.2-101.

[0114] As mentioned above, Figure 10 Each of the components and features of the optical modules 11.3.2-100 shown can be replicated in another (e.g., a second) optical module set up with the HMD to interact with the user’s other eye (e.g., project light and capture images).

[0115] Figure 10 Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination. Figure 1P Any other example of the device, feature, component, and part shown or otherwise described herein. Similarly, refer to... Figure 1P Any of the features, components, and / or parts shown, described, or otherwise described herein (including their arrangement and configuration) may be included individually or in any combination. Figure 10 Examples of devices, features, components, and parts are shown.

[0116] Figure 1P A cross-sectional view of an example optical module 11.3.2-200 is shown, which includes a housing 11.3.2-202, a display assembly 11.3.2-204 coupled to the housing 11.3.2-202, and a lens 11.3.2-216 coupled to the housing 11.3.2-202. In at least one example, the housing 11.3.2-202 defines a first aperture or channel 11.3.2-212 and a second aperture or channel 11.3.2-214. Channels 11.3.2-212 and 11.3.2-214 can be configured to slidably engage corresponding tracks or guides of an HMD device to allow the optical module 11.3.2-200 to be adjusted and positioned relative to the user's eye to match the user's interpupillary distance (IPD). The housing 11.3.2-202 can slidably engage the guide rod to secure the optical module 11.3.2-200 in the appropriate position within the HMD.

[0117] In at least one example, the optical module 11.3.2-200 may further include a lens 11.3.2-216 coupled to the housing 11.3.2-202 and positioned between the display assembly 11.3.2-204 and the user's eye when the HMD is worn. The lens 11.3.2-216 may be configured to direct light from the display assembly 11.3.2-204 to the user's eye. In at least one example, the lens 11.3.2-216 may be part of a lens assembly including a corrective lens removably attached to the optical module 11.3.2-200. In at least one example, lenses 11.3.2-216 are positioned above light strips 11.3.2-208 and one or more eye-tracking cameras 11.3.2-206, such that cameras 11.3.2-206 are configured to capture an image of a user's eye through lenses 11.3.2-216, and light strips 11.3.2-208 include lamps configured to project light onto the user's eye through lenses 11.3.2-216 during use.

[0118] Figure 1P Any of the features, components, and / or parts shown (including their arrangement and configuration) may be included individually or in any combination in any other example of the devices, features, components, and parts described herein and in any other example. Similarly, any of the features, components, and / or parts shown and described herein (including their arrangement and configuration) may be included individually or in any combination. Figure 1P Examples of devices, features, components, and parts are shown.

[0119] Figure 2This is a block diagram of an example controller 110 according to some implementation schemes. Although some specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features have not been illustrated for the sake of brevity and to avoid obscuring further relevant aspects of the implementation schemes disclosed herein. Therefore, as a non-limiting example, in some embodiments, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0120] In some embodiments, one or more communication buses 204 include circuitry for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0121] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory 220 may optionally include one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures, or subsets thereof, including optional operating system 230 and XR experience module 240.

[0122] Operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR experience module 240 is configured to manage and coordinate single or multiple XR experiences for one or more users (e.g., single XR experiences for one or more users, or multiple XR experiences for corresponding groups of one or more users). To this end, in various embodiments, XR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

[0123] In some implementations, the data acquisition unit 242 is configured to acquire data from... Figure 1A The data acquisition unit 242 includes at least the display generation component 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data acquisition unit 242 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0124] In some implementations, the tracking unit 244 is configured to map scene 105, and the tracking at least shows the generating component 120 relative to... Figure 1A The tracking unit 244 tracks the location / position of scenario 105, and optionally tracks the position of one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. To this end, in various embodiments, the tracking unit 244 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics. In some embodiments, the tracking unit 244 includes a hand tracking unit 245 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 245 is configured to track the location / position of one or more portions of the user's hand, and / or the location of one or more portions of the user's hand relative to the user's hand. Figure 1A The motion of scene 105 relative to the display generation component 120 and / or relative to a coordinate system (defined relative to the user's hand). The following refers to the motion relative to... Figure 4 The hand tracking unit 245 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the user's gaze (or more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hand)) or relative to XR content displayed via display generation component 120. The following description is relative to... Figure 5 The eye-tracking unit 243 is described in more detail.

[0125] In some implementations, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripheral device 195. To this end, in various implementations, coordination unit 246 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0126] In some embodiments, the data sending unit 248 is configured to send data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data sending unit 248 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0127] Although the data acquisition unit 242, the tracking unit 244 (e.g., including eye tracking unit 243 and hand tracking unit 245), the coordination unit 246, and the data transmission unit 248 are shown residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including eye tracking unit 243 and hand tracking unit 245), the coordination unit 246, and the data transmission unit 248 may reside in a separate computing device.

[0128] also, Figure 2 This is used more as a functional description of various features that can exist in a particular specific implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, individually shown items can be combined, and some items can be separated. For example, Figure 2 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0129] Figure 3This is a block diagram illustrating an example of generating component 120 according to some embodiments. Although some specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features have not been illustrated for the sake of brevity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the display generation component 120 (e.g., HMD) includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, and / or similar interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal and / or external image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0130] In some embodiments, one or more communication buses 304 include circuitry for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include inertial measurement units (IMUs), accelerometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, and / or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0131] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to a user. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some embodiments, one or more XR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, display generation component 120 (e.g., HMD) includes a single XR display. Alternatively, display generation component 120 may include XR displays for each of the user's eyes. In some embodiments, one or more XR displays 312 are capable of presenting MR and VR content. In some embodiments, one or more XR displays 312 are capable of presenting either MR or VR content.

[0132] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand and optionally the user's arm (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward to acquire image data corresponding to the scene the user would see in the absence of a display generation component 120 (e.g., an HMD) (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0133] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 may optionally include one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330 and XR rendering module 340.

[0134] Operating system 330 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR rendering module 340 is configured to present XR content to a user via one or more XR displays 312. For the aforementioned purposes, in various embodiments, XR rendering module 340 includes data acquisition unit 342, XR rendering unit 344, XR mapping generation unit 346, and data transmission unit 348.

[0135] In some implementations, the data acquisition unit 342 is configured to acquire data from at least... Figure 1A The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). To this end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0136] In some implementations, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. To this end, in various implementations, the XR rendering unit 344 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0137] In some implementations, the XR mapping generation unit 346 is configured to generate XR maps based on media content data (e.g., 3D maps of mixed reality scenes or maps in which computer-generated objects can be placed to generate extended reality physical environments). To this end, in various implementations, the XR mapping generation unit 346 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0138] In some implementations, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various implementations, the data transmission unit 348 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0139] Although the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data sending unit 348 are shown residing in a single device (e.g., Figure 1A The data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data sending unit 348 are located on the display generation component 120, but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data sending unit 348 may be located in a separate computing device.

[0140] also, Figure 3 This serves more as a functional description of various features that may exist in a particular specific implementation, and differs from the structural schematic diagram of the implementation described herein. As those skilled in the art will recognize, individually shown items can be combined, and some items can be separated. For example, Figure 3 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0141] Figure 4 This is a schematic illustration of an example embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1A ) by hand tracking unit 245 ( Figure 2 To control and track the location / position of one or more parts of the user's hand, and / or the location of one or more parts of the user's hand relative to the user's hand. Figure 1AThe scenario 105 refers to movement relative to a portion of the user's surrounding physical environment, relative to display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hand. In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0142] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures at least three-dimensional scene information including the human user's hand 406. The image sensor 404 captures images of the hand at sufficient resolution to distinguish the fingers and their corresponding positions. The image sensor 404 typically captures images of other parts of the user's body, or possibly all parts of the body, and may have scaling capabilities or be a dedicated sensor with increased magnification to capture images of the hand at the desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of scene 105, or serves as the image sensor for capturing the physical environment of scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a way that uses the field of view of the image sensor 404 or a portion thereof to define an interaction space in which hand movements captured by the image sensor are considered input to the controller 110.

[0143] In some implementations, image sensor 404 outputs a sequence of frames containing 3D image data (and, in addition, possibly color image data) to controller 110, which extracts high-level information from the image data. This high-level information is typically provided via an application programming interface (API) to an application running on the controller, which in turn drives display generation component 120. For example, a user can interact with software running on controller 110 by moving their hand 406 and / or changing their hand gestures.

[0144] In some embodiments, image sensor 404 projects a speckle pattern onto a scene containing hand 406 and captures an image of the projected pattern. In some embodiments, controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) via triangulation based on the lateral offset of the specks in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. This method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from image sensor 404. In this disclosure, it is assumed that image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis such that the depth coordinates of points in the scene correspond to the z-component measured by the image sensor. Alternatively, image sensor 404 (e.g., a hand-tracking device) may use other 3D mapping methods, such as stereo imaging or time-of-flight measurement, based on a single or multiple cameras or other types of sensors.

[0145] In some implementations, hand tracking device 140 captures and processes time-series depth maps containing the user's hand as the user moves their hand (e.g., the entire hand or one or more fingers). Software running on a processor in image sensor 404 and / or controller 110 processes the 3D map data to extract image block descriptors of the hand from these depth maps. The software may match these descriptors with image block descriptors stored in database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D positioning of the user's hand joints and fingertips.

[0146] The software can also analyze the trajectories of the hand and / or fingers across multiple frames in a sequence to identify gestures. The pose estimation function described herein can be alternated with motion tracking, such that patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to find pose changes occurring in the remaining frames. Pose, motion, and gesture information is provided to an application running on controller 110 via the aforementioned API. The application can, for example, move and modify the image presented on display generation component 120 in response to pose and / or gesture information, or perform other functions.

[0147] In some implementations, gestures include air gestures. An air gesture is a gesture detected without the user touching an input element that is part of the device (e.g., computer system 101, one or more input devices 125 and / or hand tracking device 140) (or independent of an input element that is part of the device) and based on the detected movement of a part of the user's body (e.g., head, one or two arms, one or two hands, one or more fingers and / or one or two legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., including a tapping gesture in which the hand moves a predetermined amount and / or speed in a predetermined pose, or a shaking gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0148] In some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with an XR environment (e.g., a virtual or mixed reality environment) performed by the movement of a user's fingers relative to other fingers or portions of the user's hand. In some embodiments, air gestures are detected without the user touching an input element that is part of the device (or independently of an input element that is part of the device) and are based on the detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or portion of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tapping gesture that includes the hand moving a predetermined amount and / or speed in a predetermined pose, or a shaking gesture that includes a predetermined speed or amount of rotation of a part of the user's body)).

[0149] In some implementations where the input gesture is an air gesture (e.g., where the input device provides information to the computer system about which user interface element is the target of the user input in the absence of physical contact, such as contact with a user interface element displayed on a touchscreen, or contact with a mouse or touchpad to move the cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Therefore, in implementations involving air gestures, for example, the input gesture combined with (e.g., concurrently) the movement of the user's fingers and / or hand detects attention (e.g., gaze) toward a user interface element to perform pinch and / or tap input, as described below.

[0150] In some implementations, input gestures directed to a user interface object are performed, either directly or indirectly, by referencing the user interface object. For example, user input is performed directly on the user interface object based on the user's hand performing an input gesture at a location corresponding to the user interface object's position in the three-dimensional environment (e.g., determined based on the user's current viewpoint). In some implementations, when user attention to the user interface object (e.g., gazing) is detected, input gestures are performed indirectly on the user interface object based on the user's hand not being positioned at a location corresponding to the user interface object's position in the three-dimensional environment while the user is performing the input gesture. For example, for direct input gestures, the user can guide their input to the user interface object by initiating a gesture at or near a location corresponding to the user interface object's display position (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge or center of the option). For indirect input gestures, the user can guide their input to the user interface object by focusing on it (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location not corresponding to the user interface object's display position).

[0151] In some implementations, the input gestures (e.g., air gestures) used in the various examples and implementations described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0152] In some implementations, pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes the movement of two or more fingers of the hand to contact each other, i.e., optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact. A long pinch gesture as an air gesture includes the movement of two or more fingers of the hand to contact each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact is detected. For example, a long pinch gesture includes the user holding a pinch gesture (e.g., where two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some implementations, a double pinch gesture as an air gesture includes two (e.g., more) pinch inputs (e.g., performed by the same hand) that are detected consecutively with each other immediately (e.g., within a predefined time period). For example, a user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts the contact between two or more fingers), and performs a second pinch input within a predefined time period after releasing the first pinch input (e.g., within 1 second or within 2 seconds).

[0153] In some embodiments, pinch and drag gestures as air gestures include pinch gestures (e.g., pinching gestures or long pinch gestures) performed in conjunction with (e.g., following) drag input that changes the user's hand position from a first position (e.g., the start position of the drag) to a second position (e.g., the end position of the drag). In some embodiments, the user holds the pinch gesture while performing the drag input and releases the pinch gesture (e.g., opening two or more of their fingers) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and drag input are performed by the same hand (e.g., the user pinches two or more fingers together to touch each other and uses the drag gesture to move the same hand to the second position in the air). In some implementations, pinch input is performed by the user's first hand, and drag input is performed by the user's second hand (e.g., while the user continues pinch input with the user's first hand, the user's second hand moves in the air from a first position to a second position). In some implementations, input gestures as air gestures include inputs performed using both of the user's hands (e.g., pinch and / or tap inputs). For example, input gestures include two (e.g., more) pinch inputs performed in combination with each other (e.g., concurrently or within a predefined time period). For example, a first pinch gesture (e.g., pinch input, long pinch input, or pinch and drag input) is performed using the user's first hand, and a second pinch input is performed using the other hand (e.g., the second hand in both of the user's hands). In some implementations, movement between the user's two hands is performed (e.g., increasing and / or decreasing the distance or relative orientation between the user's two hands).

[0154] In some implementations, a tap input performed as an air gesture (e.g., pointing at a user interface element) includes movement of a user's finger toward the user interface element, movement of a user's hand toward the user interface element (optionally, the user's finger extends toward the user interface element), downward movement of a user's finger (e.g., mimicking a mouse click or a tap on a touchscreen), or other predefined movements of the user's hand. In some implementations, the tap input performed as an air gesture is detected based on the movement characteristics of the finger or hand performing the tap gesture movement, which is the finger or hand moving away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by the end of the movement. In some implementations, the end of the movement is detected based on changes in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of finger or hand movement, and / or a reversal of the acceleration direction of finger or hand movement).

[0155] In some implementations, the user's attention is determined to be directed to a portion of the 3D environment based on the detection of a gaze directed to that portion of the 3D environment (optionally, no other conditions are required). In some implementations, the user's attention is determined to be directed to that portion of the 3D environment based on the detection of a gaze directed to that portion of the 3D environment using one or more additional conditions, such as requiring the gaze to be directed to that portion of the 3D environment for at least a threshold duration (e.g., dwell time) and / or requiring the gaze to be directed to that portion of the 3D environment when the user's viewpoint is within a distance threshold from that portion of the 3D environment, so that the device determines that the user's attention is directed to that portion of the 3D environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the 3D environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0156] In some implementations, the detection of the readiness configuration of a user or a portion of a user is performed by a computer system. The detection of the hand's readiness configuration is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the readiness of the hand is determined based on whether it has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart in preparation for a pinch or grasping gesture, or a pre-tap with one or more fingers extended and the back of the hand facing the user), whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular manner (e.g., moving towards an area in front of the user above the user's waist and below the user's head, or moving away from the user's body or legs). In some implementations, the readiness state is used to determine whether an interactive element of the user interface responds to attentional (e.g., gaze) input.

[0157] In scenarios where input is described by reference to air gestures, it should be understood that hardware input devices attached to or held by one or both of the user's hands can be used to detect such gestures. Optical tracking, one or more accelerometers, one or more gyroscopes, one or more magnetometers and / or one or more inertial measurement units can be used to track the spatial positioning of the hardware input device, and the positioning and / or movement of the hardware input device can be used in place of the positioning and / or movement of one or both hands in relation to the corresponding air gesture. In scenarios describing input using air gestures, it should be understood that similar gestures can be detected using hardware input devices attached to or held by one or both of the user's hands. User input can be detected using controls contained within the hardware input device, such as one or more touch-sensitive input elements, one or more pressure-sensitive input elements, one or more buttons, one or more knobs, one or more dials, one or more joysticks, one or more hand or finger covers that can detect the positioning or changes in positioning of parts of the hand and / or fingers relative to each other, relative to the user's body, and / or relative to the user's physical environment, and / or other hardware input device controls. User input using controls contained within the hardware input device replaces hand and / or finger gestures such as air taps or air pinches in the corresponding air gesture. For example, a selection input described as being performed using an air tap or air pinch input can alternatively be detected using button presses, taps on touch-sensitive surfaces, presses on pressure-sensitive surfaces, or other hardware inputs. As another example, motion input described as being performed using air pinch and drag can be optionally detected based on interaction with hardware input controls, such as pressing and holding a button, touching a touch on a touch-sensitive surface, pressing a pressure-sensitive surface, or other hardware input following movement of a hardware input device (e.g., a hand associated with the hardware input device) through space. Similarly, two-handed input involving movement of hands relative to each other can be performed using an air gesture and a hardware input device not in the hand performing the air gesture, two hardware input devices held in different hands, or two air gestures performed by different hands using air gestures and / or inputs detected by one or more of the aforementioned hardware input devices.

[0158] In some embodiments, the software may be downloaded to controller 110 electronically, for example, via a network, or alternatively, may be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, database 408 is also stored in memory associated with controller 110. Alternatively or additionally, some or all of the described functions of the computer may be implemented in dedicated hardware, such as custom or semi-custom integrated circuits or programmable digital signal processors (DSPs). Although in Figure 4The controller 110 is shown, but for example, as a separate unit from the image sensor 404, some or all of the controller's processing functions may be performed by a suitable microprocessor and software, or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand-tracking device), or by other devices associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing function of the image sensor 404 may also be integrated into a computer or other computerized device controlled by the sensor output.

[0159] Figure 4 It also includes a schematic diagram of a depth map 410 captured by image sensor 404 according to some embodiments. As described above, the depth map comprises a matrix of pixels with corresponding depth values. Pixel 412 corresponding to hand 406 has been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z-distance from image sensor 404), where gray shadows become darker as depth increases. Controller 110 processes these depth values ​​to identify and segment components of the image that have human hand characteristics (i.e., a group of adjacent pixels). These characteristics may include, for example, overall size, shape, and frame-to-frame motion from the depth map sequence.

[0160] Figure 4 The controller 110 also schematically illustrates, according to some embodiments, the hand skeleton 414 ultimately extracted from the depth map 410 of the hand 406. Figure 4 In this configuration, the hand skeleton 414 is superimposed on the hand background 416, which has already been segmented from the original depth map. In some embodiments, key feature points of the hand, and optionally on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, fingertips, the center of the palm, the end of the hand connecting to the wrist, etc.), are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points across multiple image frames to determine, according to some embodiments, the gesture performed by the hand or the current state of the hand.

[0161] Figure 5 An eye-tracking device 130 is illustrated. Figure 1A Example implementation of ). In some implementations, the eye-tracking device 130 consists of an eye-tracking unit 243 ( Figure 2The eye-tracking device 130 is controlled to track the positioning and movement of a user's gaze relative to scene 105 or relative to XR content displayed via display generation component 120. In some embodiments, the eye-tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as a head-mounted device, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for the user to view and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye-tracking device 130 may optionally be a separate device from the handheld device or XR room. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 may optionally be used in conjunction with a display generation component that is also head-mounted or not head-mounted. In some embodiments, the eye-tracking device 130 is not a head-mounted device and may optionally be used in conjunction with a head-mounted display generation component. In some embodiments, the eye-tracking device 130 is not a head-mounted device and may optionally be part of a non-head-mounted display generation component.

[0162] In some embodiments, the display generation component 120 uses display mechanisms (e.g., a left near-eye display panel and a right near-eye display panel) to display frames including left and right images in front of the user's eyes, thereby providing the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or semi-transparent display on which virtual objects are displayed, allowing the user to view the physical environment directly through the transparent or semi-transparent display. In some embodiments, the display generation component projects virtual objects onto the physical environment. The virtual objects may, for example, be projected onto a physical surface or as holograms, allowing an individual to observe virtual objects superimposed on the physical environment using the system. In this case, separate display panels and image frames for the left and right eyes may not be necessary.

[0163] like Figure 5As shown, in some embodiments, eye-tracking device 130 (e.g., gaze tracking device) includes at least one eye-tracking camera (e.g., an infrared (IR) or near-infrared (NIR) camera) and an illumination source (e.g., an array or ring of IR or NIR light sources, such as LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be pointed at the user's eye to receive IR or NIR light reflected directly from the eye, or alternatively, it may be pointed at "hot" mirrors located between the user's eye and the display panel, which reflect the IR or NIR light from the eye back to the eye-tracking camera while allowing visible light to pass through. Eye-tracking device 130 may optionally capture images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps), analyze these images to generate gaze tracking information, and transmit the gaze tracking information to controller 110. In some embodiments, the user's two eyes are tracked separately using corresponding eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked using corresponding eye-tracking cameras and illumination sources.

[0164] In some implementations, a device-specific calibration procedure is used to calibrate the eye-tracking device 130 to determine parameters for the eye-tracking device in a specific operating environment 100, such as the 3D geometry and parameters of the LEDs, camera, thermal mirror (if present), eye lenses, and display. The device-specific calibration procedure can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration procedure can be automated or manual. According to some implementations, a user-specific calibration procedure may include estimations of eye parameters for a specific user, such as pupil position, foveal position, optical axis, visual axis, interocular distance, etc. According to some implementations, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, a flash-assisted method can be used to process the images captured by the eye-tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0165] like Figure 5As shown, the eye-tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system. The gaze tracking system includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-infrared (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye-tracking camera 540 may be pointed toward a mirror 550 located between the user's eye 592 and a display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, projector, etc.). These mirrors reflect the IR or NIR light from the eye 592 while allowing visible light to pass through. Figure 5 (as shown in the top portion), or alternatively, it can be pointed towards the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion), Figure 5 (As shown in the bottom part).

[0166] In some implementations, controller 110 renders AR or VR frames 562 (e.g., left and right frames for the left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye-tracking camera 540 for various purposes, such as processing frame 562 for display. Controller 110 may optionally estimate the user's gaze point on display 510 based on the gaze tracking input 542 obtained from eye-tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated based on gaze tracking input 542 may optionally be used to determine the direction the user is currently looking.

[0167] The following describes several possible use cases for the user's current gaze direction and is not intended to be limiting. As an example use case, controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, controller 110 may generate virtual content at a higher resolution in the concave region determined according to the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content in the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content in the view based at least partially on the user's current gaze direction. As another example use case in an AR application, controller 110 may guide an external camera used to capture the physical environment of an XR experience to focus in the determined direction. The external camera's autofocus mechanism may then focus on an object or surface in the environment that the user is currently looking at on display 510. As another example use case, eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of eye lens 520 so that the virtual object the user is currently looking at has appropriate convergence / divergence to match the convergence of the user's eyes 592. The controller 110 can use gaze tracking information to guide the eye lens 520 to adjust its focus so that the nearby object that the user is looking at appears at the correct distance.

[0168] In some embodiments, the eye-tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens 520), an eye-tracking camera (e.g., eye-tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)). The light source emits light (e.g., IR or NIR light) toward the user's eyes 592. In some embodiments, the light source may be arranged in a ring or circle around each lens in the head-mounted device, such as... Figure 5 As shown. In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0169] In some embodiments, the display 510 emits light in the visible light range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. It should be noted that the positions and angles of the eye-tracking camera 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, cameras 540 with a wider field of view (FOV) and cameras 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, cameras 540 operating at one wavelength (e.g., 850 nm) and cameras 540 operating at different wavelengths (e.g., 940 nm) may be used on each side of the user's face.

[0170] like Figure 5 The gaze tracking system implementations illustrated herein can be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide users with computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experiences.

[0171] Figure 6 Examples of flash-assisted gaze tracking pipelines according to some embodiments are illustrated. In some embodiments, the gaze tracking pipeline uses a flash-assisted gaze tracking system (e.g., such as...) Figure 1A and Figure 5 The eye-tracking device 130 shown is used to implement this. The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and flash in the current frame. When not in tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and flash in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in tracking state.

[0172] like Figure 6 As shown, the gaze-tracking camera captures left and right images of the user's left and right eyes. The captured images are then fed into a gaze-tracking pipeline for processing to begin at 610. As indicated by the arrow returning to element 600, the gaze-tracking system can continue capturing images of the user's eyes, for example, at a rate of 60 to 120 frames per second. In some embodiments, each set of captured images can be fed into the pipeline for processing. However, in some embodiments or under certain conditions, not all captured frames are processed by the pipeline.

[0173] At 610, for the currently captured image, if the tracking state is yes, the method proceeds to element 640. At 610, if the tracking state is no, the image is analyzed to detect the user's pupil and flash, as indicated at 620. At 630, if the pupil and flash are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0174] At 640, if proceeding from element 610, the current frame is analyzed to track the pupil and flashes in part based on previous information from the previous frame. At 640, if proceeding from element 630, the tracking state is initialized based on the pupil and flashes detected in the current frame. The processing result at element 640 is checked to verify that the tracking or detection result is credible. For example, the result may be checked to determine whether a sufficient number of pupils and flashes used for gaze estimation were successfully tracked or detected in the current frame. At 650, if the result is not credible, the tracking state is set to no at element 660, and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is credible, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes), and the pupil and flash information is passed to element 680 to estimate the user's gaze point.

[0175] Figure 6 This is intended as an example of an eye-tracking technology that can be used in a particular specific implementation. As will be recognized by those skilled in the art, in a computer system 101 for providing an XR experience to a user, other eye-tracking technologies that are currently available or will be developed in the future may be used to replace or in combination with the flash-assisted eye-tracking technology described herein, depending on the various implementations.

[0176] In some implementations, a portion of the captured real-world environment 602 is used to provide an XR experience to the user, such as a mixed reality environment in which one or more virtual objects are overlaid on a representation of the real-world environment 602.

[0177] Therefore, this description describes some embodiments of a three-dimensional environment (e.g., an XR environment) that includes representations of real-world objects and virtual objects. For example, a three-dimensional environment may optionally include a representation of a table existing in a physical environment, which is captured and displayed in the three-dimensional environment (e.g., actively displayed via a camera and display of a computer system or passively displayed via a transparent or semi-transparent display of a computer system). As previously described, the three-dimensional environment may optionally be a mixed reality system, wherein the three-dimensional environment is based on a physical environment captured by one or more sensors of a computer system and displayed via a display generation component. As a mixed reality system, the computer system may optionally be able to selectively display portions and / or objects of the physical environment such that the corresponding portions and / or objects of the physical environment appear as if they exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system may optionally be able to display virtual objects in the three-dimensional environment to appear as if the virtual objects exist in the real world (e.g., the physical environment) by placing virtual objects in the three-dimensional environment at corresponding locations in the real world that have corresponding positions in the three-dimensional environment. For example, the computer system may optionally display a vase such that the vase appears as if a real vase were placed on top of a table in the physical environment. In some implementations, a corresponding location in the three-dimensional environment has a corresponding location in the physical environment. Therefore, when a computer system is described as displaying a virtual object at a corresponding location relative to a physical object (e.g., such as at or near a user's hand or at or near a physical table), the computer system displays the virtual object at a specific location in the three-dimensional environment such that it appears as if the virtual object were at or near a physical object in the physical environment (e.g., the virtual object is displayed in the three-dimensional environment at a location in the physical environment that would be displayed if the virtual object were a real object at that specific location).

[0178] In some implementations, real-world objects that exist in the physical environment and are displayed in a 3D environment (e.g., and / or visible via a display generation component) can interact with virtual objects that exist only in the 3D environment. For example, the 3D environment may include a table and a vase placed on top of the table, where the table is a view (or representation) of a physical table in the physical environment, and the vase is a virtual object.

[0179] In a three-dimensional environment (e.g., a real environment, a virtual environment, or a hybrid environment including both real and virtual objects), an object is sometimes referred to as having depth or simulated depth, or as being visible, displayed, or placed at different depths. In this context, depth refers to a dimension other than height or width. In some embodiments, depth is defined relative to a fixed set of coordinates (e.g., where a room or object has a height, depth, and width defined relative to a fixed set of coordinates). In some embodiments, depth is defined relative to a user's position or viewpoint, in which case the depth dimension varies based on the user's position and / or the position and angle of the user's viewpoint. In some embodiments where depth is defined relative to the user's location relative to a surface of the environment (e.g., the surface of the environment's floor or ground), objects further away from the user along lines extending parallel to the surface are considered to have greater depth in the environment, and / or the depth of an object is measured along an axis extending outward from the user's position and parallel to the surface of the environment (e.g., depth is defined in a cylindrical or substantially cylindrical coordinate system, where the user's position is at the center of a cylinder extending from the user's head toward the user's feet). In some embodiments where depth is defined relative to the user's viewpoint (e.g., a direction relative to a point in space that determines which part of the environment is visible via a head-mounted device or other display), objects further away from the user's viewpoint along a line extending parallel to the user's viewpoint are considered to have greater depth in the environment, and / or the depth of an object is measured along an axis extending outward from a line extending from and parallel to the user's viewpoint (e.g., defining depth in a spherical or substantially spherical coordinate system, where the origin of the viewpoint is at the center of a sphere extending outward from the user's head). In some embodiments, depth is defined relative to a user interface container (e.g., a window or application displaying application and / or system content), where the user interface container has a height and / or width, and depth is a dimension orthogonal to the height and / or width of the user interface container. In some implementations, when a depth is defined relative to a user interface container, when the container is placed in a three-dimensional environment or initially displayed (e.g., such that the container's depth dimension extends outward away from the user or the user's viewpoint), the container's height and / or width are typically orthogonal or substantially orthogonal to a straight line extending from the user's location (e.g., the user's viewpoint or the user's position) to the user interface container (e.g., the center of the user interface container or another feature point of the user interface container). In some implementations, when a depth is defined relative to a user interface container, the object's depth relative to the user interface container refers to the object's positioning along the depth dimension of the user interface container. In some implementations, multiple different containers may have different depth dimensions (e.g., different depth dimensions extending away from the user or the user's viewpoint in different directions and / or from different starting points).In some implementations, when depth is defined relative to a user interface container, the orientation of the depth dimension remains constant relative to the user interface container as the position of the user interface container changes, or as the user and / or the user's viewpoint changes (e.g., when multiple different viewers are viewing the same container in a 3D environment, such as during a collaborative session and / or when multiple participants are in a real-time communication session with shared virtual content including the container). In some implementations, for curved containers (e.g., containers including areas with curved surfaces or curved contents), the depth dimension may optionally extend into the surface of the curved container. In some cases, z-interval (e.g., the distance between two objects in the depth dimension), z-height (e.g., the distance of one object from another in the depth dimension), z-position (e.g., the position of an object in the depth dimension), z-depth (e.g., the position of an object in the depth dimension), or simulated z-dimensionality (e.g., depth used as a dimension of an object, a dimension of the environment, an orientation in space, and / or an orientation in simulated space) are used to refer to the concept of depth as described above.

[0180] In some implementations, a user may optionally be able to interact with virtual objects in a 3D environment using one or both hands as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the computer system may optionally capture one or both of the user's hands and display a representation of the user's hands in the 3D environment (e.g., in a manner similar to displaying real-world objects in the 3D environment described above). Alternatively, in some implementations, the user's hands may be visible via the display generation component, through the ability to see the physical environment through the user interface, due to the transparency / semi-transparency of a portion of the user interface being displayed by the display generation component, or due to the projection of the user interface onto a transparent / semi-transparent surface or onto the user's eyes or into the user's field of view. Thus, in some implementations, the user's hands are displayed at corresponding locations in the 3D environment and are treated as if they were objects in the 3D environment that could interact with virtual objects in the 3D environment as if these virtual objects were physical objects in the physical environment. In some implementations, the computer system may update the display of the user's hand representation in the 3D environment in conjunction with the movement of the user's hands in the physical environment.

[0181] In some embodiments described below, the computer system may optionally determine the “effective” distance between a physical object in the physical world and a virtual object in a three-dimensional environment, for example, to determine whether the physical object is directly interacting with the virtual object (e.g., whether a hand is touching, grasping, holding, or within a threshold distance of the virtual object). For example, a hand directly interacting with a virtual object may optionally include one or more of the following: a finger pressing a virtual button, a user’s hand grasping a virtual vase, a user’s hand clasped together to pinch / hold the application’s user interface, and two fingers performing any other type of interaction described herein. For example, the computer system may optionally determine the distance between the user’s hand and the virtual object when determining whether and / or how the user is interacting with the virtual object. In some embodiments, the computer system determines the distance between the user’s hand and the virtual object by determining the distance between the position of the hand in the three-dimensional environment and the position of the virtual object of interest in the three-dimensional environment. For example, if a user's one or both hands are located at a specific location in the physical world, the computer system may optionally capture the one or both hands and display them at a specific corresponding location in a three-dimensional environment (e.g., the location where the hand would be displayed in the three-dimensional environment if it were a virtual hand rather than a physical hand). Optionally, the location of the hand in the three-dimensional environment may be compared with the location of a virtual object of interest in the three-dimensional environment to determine the distance between the user's one or both hands and the virtual object. In some embodiments, the computer system may optionally determine the distance between a physical object and a virtual object by comparing locations in the physical world (e.g., rather than comparing locations in the three-dimensional environment). For example, when determining the distance between a user's one or both hands and a virtual object, the computer system may optionally determine the corresponding location of the virtual object in the physical world (e.g., the location where the virtual object would be located in the physical world if it were a physical object rather than a virtual object), and then determine the distance between the corresponding physical location and the user's one or both hands. In some embodiments, the same technique may optionally be used to determine the distance between any physical object and any virtual object. Therefore, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, the computer system may optionally perform any of the techniques described above to map the position of the physical object to the three-dimensional environment and / or map the position of the virtual object to the physical environment.

[0182] In some implementations, the same or similar techniques are used to determine where and what the user's gaze is directed at, and / or where and what the physical stylus held by the user is pointing at. For example, if the user's gaze is directed at a specific location in the physical environment, the computer system may optionally determine a corresponding location in the three-dimensional environment (e.g., a virtual location of the gaze), and if a virtual object is located at that corresponding virtual location, the computer system may optionally determine that the user's gaze is directed at that virtual object. Similarly, the computer system may optionally be able to determine the direction in which the stylus is pointing in the physical environment based on the orientation of the physical stylus. In some implementations, based on this determination, the computer system determines a corresponding virtual location in the three-dimensional environment corresponding to the location pointed at by the stylus in the physical environment, and optionally determines that the stylus is pointing at the corresponding virtual location in the three-dimensional environment.

[0183] Similarly, the embodiments described herein may refer to the location of a user (e.g., a user of a computer system) in a three-dimensional environment and / or the location of the computer system in a three-dimensional environment. In some embodiments, the user of the computer system is holding, wearing, or otherwise located at or near the computer system. Thus, in some embodiments, the location of the computer system serves as a proxy for the location of the user. In some embodiments, the location of the computer system and / or the user in the physical environment corresponds to a corresponding location in the three-dimensional environment. For example, the location of the computer system would be its location in the physical environment (and its corresponding location in the three-dimensional environment) such that, if the user stands at that location facing the corresponding portion of the physical environment visible via the display generation component, the user will see from that location objects in the physical environment that are positioned, oriented, and / or sized (e.g., in an absolute sense and / or relative to each other) in the same way as objects displayed or visible in the three-dimensional environment by or via the display generation component of the computer system. Similarly, if the virtual objects displayed in a 3D environment are physical objects in the physical environment (e.g., physical objects placed in the physical environment at the same location as these virtual objects in the 3D environment, and physical objects in the physical environment having the same size and orientation as in the 3D environment), then the position of the computer system and / or the user is the position from which the user will see these virtual objects in the physical environment at the same location, orientation, and / or size (e.g., in an absolute sense and / or relative to each other and real-world objects) as the virtual objects displayed in the 3D environment by the display generation components of the computer system.

[0184] In this disclosure, various input methods are described in relation to interaction with a computer system. When an example is provided using one input device or method, and another example is provided using another input device or method, it should be understood that each example is compatible with and optionally utilizes the input device or method described relative to the other example. Similarly, various output methods are described in relation to interaction with a computer system. When an example is provided using one output device or method, and another example is provided using another output device or method, it should be understood that each example is compatible with and optionally utilizes the output device or method described relative to the other example. Similarly, various methods are described in relation to interaction with a virtual or mixed reality environment via a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and optionally utilizes the methods described relative to the other example. Therefore, this disclosure discloses embodiments that are combinations of features of multiple examples without exhaustively listing all features of the embodiments in the description of each example embodiment.

[0185] User interface and related processes Now turn attention to implementations of user interfaces (“UIs”) and associated processes that can be implemented on computer systems, such as portable multifunction devices or head-mounted devices, in communication with display generation components and one or more sensors and optionally one or more input devices.

[0186] Figures 7A to 7L as well as Figures 8A to 8MThis includes an example of a three-dimensional environment visible via a display generation component (e.g., display generation component 7100 or display generation component 120) of a computer system (e.g., computer system 101) and interactions occurring within the three-dimensional environment caused by user input directed towards the three-dimensional environment and / or input received from other computer systems and / or sensors. In some embodiments, input is directed towards a virtual object within the three-dimensional environment by a user gaze detected in an area occupied by the virtual object or by a hand gesture performed at a location in the physical environment corresponding to the area of ​​the virtual object. In some embodiments, input is directed towards a virtual object within the three-dimensional environment by a hand gesture performed when the virtual object has input focus (e.g., when the virtual object has been selected by concurrently and / or previously detected gaze input, by concurrently or previously detected pointer input, and / or by concurrently and / or previously detected gesture input) (e.g., optionally, at a location in the physical environment independent of the area of ​​the virtual object in the three-dimensional environment). In some embodiments, input is directed towards a virtual object within the three-dimensional environment by an input device that has positioned a focus selector object (e.g., a pointer object or selector object) at the location of the virtual object. In some embodiments, input is directed to virtual objects within the three-dimensional environment via other components (e.g., voice and / or control buttons). In some embodiments, input is directed to a physical object or a representation of a virtual object corresponding to a physical object via user hand movements (e.g., whole hand movement, whole hand movement in a corresponding posture, movement of a portion of the user's hand relative to another portion of the hand, and / or relative movement between the hands) and / or manipulations relative to a physical object (e.g., touch, swipe, tap, open, toward movement, and / or relative movement). In some embodiments, the computer system displays changes in the three-dimensional environment based on input from sensors (e.g., image sensors, temperature sensors, biometric sensors, motion sensors, and / or proximity sensors) and contextual conditions (e.g., location, time, and / or the presence of other people in the environment) (e.g., displaying additional virtual content, stopping the display of existing virtual content, and / or transitioning between different levels of immersion in displaying visual content). In some implementations, the computer system displays some changes in the 3D environment based on input from other computers used by other users sharing the computer-generated environment with the user of the computer system (e.g., in a shared computer-generated experience, in a shared virtual environment, and / or in a shared virtual or augmented reality environment in a communication session). These changes include displaying additional virtual content, stopping the display of existing virtual content, and / or transitioning between different levels of immersion in the display of visual content.In some implementations, the computer system displays changes in a three-dimensional environment based on input from sensors (e.g., changes in the movement, deformation, and / or visual characteristics of user interfaces, virtual surfaces, user interface objects, and / or virtual landscapes), the sensors detecting movement of other people and objects, as well as movement of the user that may not conform to the criteria of recognized gesture inputs that trigger associated operations of the computer system.

[0187] In some embodiments, the 3D environment visible via the display generation components described herein is a virtual 3D environment that includes virtual objects and content at different virtual locations within a 3D environment without a representation of a physical environment. In some embodiments, the 3D environment is a mixed reality environment that displays virtual objects at different virtual locations within a 3D environment constrained by one or more physical aspects of the physical environment (e.g., the location and orientation of walls, floors, surfaces, direction of gravity, time of day, and / or spatial relationships between physical objects). In some embodiments, the 3D environment is an augmented reality environment that includes a representation of a physical environment. In some embodiments, the representation of the physical environment includes corresponding representations of physical objects and surfaces at different locations within the 3D environment, such that spatial relationships between different physical objects and surfaces in the physical environment are reflected by spatial relationships between representations of physical objects and surfaces in the 3D environment. In some embodiments, when a virtual object is positioned relative to a representation of physical objects and surfaces in the 3D environment, the virtual object appears to have a corresponding spatial relationship with the physical objects and surfaces in the physical environment. In some implementations, the computer system transitions between displaying different types of environments based on user input and / or contextual conditions (e.g., transitioning between presenting computer-generated environments or experiences with different levels of immersion, and adjusting the relative salience of audio / visual sensory inputs from virtual content and representations from the physical environment).

[0188] In some embodiments, the display generation component includes a pass-through portion in which a representation of the physical environment is displayed or visible. In some embodiments, the pass-through portion of the display generation component is a transparent or translucent (e.g., see-through) portion of the display generation component that displays at least a portion of the physical environment surrounding the user or within the user's field of view (sometimes referred to as "optical pass-through"). For example, the pass-through portion is a portion of a head-mounted display or head-up display that is made translucent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent, allowing the user to view the real world around them without removing the head-mounted display or moving away from the head-up display. In some embodiments, when displaying a virtual or mixed reality environment, the pass-through portion gradually transitions from translucent or transparent to completely opaque. In some embodiments, the pass-through portion of the display generation component displays a real-time feed (sometimes referred to as "digital pass-through") of images or video of at least a portion of the physical environment captured by one or more cameras (e.g., a rear-facing camera on a mobile device or associated with a head-mounted display, or other cameras that feed image data to a computer system). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is directly in front of the user's eyes (e.g., behind the display generating component relative to the user). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is not directly in front of the user's eyes (e.g., in a different physical environment, or to the side or behind the user).

[0189] In some embodiments, when virtual objects are displayed at locations corresponding to the positions of one or more physical objects in a physical environment (e.g., locations in a virtual reality, mixed reality, or augmented reality environment), at least some of the virtual objects are displayed to replace a portion of the camera's live view (e.g., a portion of the physical environment captured in the live view) (e.g., replacing its display). In some embodiments, at least some of the virtual objects and content are projected onto a physical surface or blank space in the physical environment and are visible through a transparent portion of the display generating component (e.g., visible as part of the camera view of the physical environment, or visible through a transparent or semi-transparent portion of the display generating component). In some embodiments, at least some of the virtual objects and content are displayed as portions covering the display and obscuring at least a portion of the view of the physical environment visible through the transparent or semi-transparent portion of the display generating component.

[0190] In some implementations, the display generation component displays different views of the 3D environment based on user input or movement that alters the virtual positioning of the viewpoint relative to the 3D environment, changing the currently displayed view of the 3D environment. In some implementations, when the 3D environment is virtual, the viewpoint moves based on navigation or motion requests (e.g., hand gestures in the air and / or gestures performed by moving one part of the hand relative to another part of the hand), without requiring movement of the user's head, torso, and / or the display generation component in the physical environment. In some implementations, movement of the user's head and / or torso, and / or movement of the display generation component or other position sensing elements of the computer system (e.g., due to the user gripping the display generation component or wearing an HMD) relative to the physical environment causes a corresponding movement of the viewpoint relative to the 3D environment (e.g., with a corresponding change in direction, distance, speed, and / or orientation), resulting in a corresponding change in the currently displayed view of the 3D environment. In some implementations, when a virtual object has a preset spatial relationship relative to a viewpoint (e.g., anchored or fixed to the viewpoint), movement of the viewpoint relative to the 3D environment will cause movement of the virtual object relative to the 3D environment while maintaining the virtual object's position within the field of view (e.g., the virtual object is referred to as head-locked). In some implementations, the virtual object is body-locked to the user and moves relative to the 3D environment as the user moves as a whole in the physical environment (e.g., carrying or wearing display generation components and / or other position sensing components of the computer system), but will not move in the 3D environment in response to individual user head movements (e.g., the display generation components and / or other position sensing components of the computer system rotate around a fixed position of the user in the physical environment). In some implementations, the virtual object may optionally be locked to another part of the user, such as the user's hand or wrist, and moves in the 3D environment according to movement of that part of the user in the physical environment to maintain a preset spatial relationship between the virtual object's position and the virtual position of that part of the user in the 3D environment. In some implementations, virtual objects are locked to a preset portion of the field of view provided by the display generation component and move in the three-dimensional environment according to the movement of the field of view, regardless of the movement of the user that does not cause a change in the field of view.

[0191] In some implementation schemes, such as Figures 7A to 7L as well as Figures 8A to 8MAs shown, the view of the 3D environment sometimes does not include a representation of the user's hands, arms, and / or wrists. In some embodiments, a representation of the user's hands, arms, and / or wrists is included in the view of the 3D environment. In some embodiments, a representation of the user's hands, arms, and / or wrists is included in the view of the 3D environment as part of a representation of the physical environment provided via a display generation component. In some embodiments, these representations are not part of the representation of the physical environment and are captured separately (e.g., pointed at the user's hands, arms, and wrists by one or more cameras) and displayed in the 3D environment independently of the currently displayed view of the 3D environment. In some embodiments, these representations include camera images captured by one or more cameras of a computer system or stylized versions of the arms, wrists, and / or hands based on information captured by various sensors. In some embodiments, these representations replace a portion of the display of the representation of the physical environment, overlay that portion of the representation of the physical environment, or obscure that portion of the view of the representation of the physical environment. In some embodiments, when the display generation component does not provide a view of the physical environment and provides a completely virtual environment (e.g., no camera view and no transparent pass-through portion), a real-time visual representation (e.g., a stylized representation or segmented camera image) of one or both of the user's arms, wrists, and / or hands may optionally still be displayed in the virtual environment. In some embodiments, if no representation of the user's hands is provided in the view of the three-dimensional environment, the positioning corresponding to the user's hands may optionally be indicated in the three-dimensional environment, for example, by altering the appearance of the virtual content at the positioning in the three-dimensional environment corresponding to the position of the user's hands in the physical environment (e.g., by altering the translucency and / or simulating changes in reflectivity). In some embodiments, the representation of the user's hands or wrists is outside the currently displayed view of the three-dimensional environment, while the virtual positioning in the three-dimensional environment corresponding to the position of the user's hands or wrists is outside the current field of view provided via the display generation component; and in response to the virtual positioning corresponding to the position of the user's hands or wrists moving within the current field of view due to movement of the display generation component, the user's hands or wrists, the user's head, and / or the user as a whole, the representation of the user's hands or wrists becomes visible in the view of the three-dimensional environment.

[0192] Figures 7A to 7L Examples are given of how the displayed visuals change as the user interface and / or user interface elements are repositioned and / or as the user's viewpoint changes relative to the user interface and / or user interface elements. Figure 9 This is a flowchart of an exemplary method 900 for updating the display of visuals as the user interface and / or user interface elements are repositioned. Figure 10 This is a flowchart of an exemplary method 1000 for updating the display of visual effects applied to the user interface and / or user interface elements as the user's viewpoint changes. Figures 7A to 7L The user interface in the document is used to illustrate the processes described below, including Figure 9 and Figure 10 The process in.

[0193] like Figures 7A to 7L As shown in the example, the display generating component 7100 of computer system 101 is a touchscreen held by user 7002. In some embodiments, the display generating component of computer system 101 is a head-mounted display worn on the head of user 7002 (e.g., head-mounted display 7100a, such as...). Figures 7E1 to 7E2 , Figures 8D2 to 8D3 and Figures 8E1 to 8E2 As shown (for example, in) Figures 7A to 7LThe content shown as visible via the display generation component 7100 of computer system 101 corresponds to the field of view of user 7002 while wearing the head-mounted display. In some embodiments, the display generation component is a stand-alone display, projector, or another type of display. In some embodiments, the computer system communicates with one or more input devices, including cameras or other sensors and input devices that detect movement of the user's hands, movement of the user's whole body, and / or movement of the user's head in the physical environment. In some embodiments, the one or more input devices detect movement of the user's hands, face, and / or whole body and current posture, orientation, and positioning. For example, in some embodiments, when the user's hand 7020 is within the field of view of one or more sensors of HMD 7100a (e.g., within the user's field of view), a representation of the user's hand 7020' is displayed on the user interface on the display of HMD 7100a (e.g., as a pass-through representation and / or as a virtual representation of the user's hand 7020). In some embodiments, when the user's hand 7022 is within the field of view of one or more sensors of the HMD 7100a (e.g., within the user's field of view), a representation of the user's hand 7022' is displayed in the user interface shown on the display of the HMD 7100a (e.g., as a pass-through representation and / or as a virtual representation of the user's hand 7022). In some embodiments, the user's hand 7020 and / or the user's hand 7022 is used to optionally combine gaze input to perform one or more gestures (e.g., one or more air gestures). In some embodiments, one or more gestures performed using the user's hand 7020 and / or 7022 include direct air gesture input based on the positioning of the representation of the user's hand 7020' and / or 7022' displayed within the user interface on the display of the HMD 7100a. For example, direct air gesture input is determined to point to a user interface object displayed at a location that intersects with the displayed positioning of the representation of the user's hand 7020' and / or 7022' in the user interface. In some implementations, one or more gestures performed using the user's hands 7020 and / or 7022 include indirect air gesture input, which is based on a virtual object displayed at a location corresponding to the location where the user's attention is currently detected (e.g., and / or optionally not based on the location of a representation of the user's hands 7020' and / or 7022' displayed within the user interface). For example, when the user's attention to a user interface object is detected (e.g., based on a gaze or other indication of the user's attention), indirect air gestures, such as gazing and pinching (e.g., or other gestures performed using the user's hands), are performed relative to the user interface object.

[0194] In some embodiments, user input is detected via a touch-sensitive surface or touchscreen. In some embodiments, one or more input devices include an eye-tracking component that detects the location and movement of the user's gaze. In some embodiments, a display generation component and optionally one or more input devices, along with a computer system, are part of a head-mounted device that moves and rotates with the user's head in the physical environment and changes the user's viewpoint in a three-dimensional environment provided via the display generation component. In some embodiments, the display generation component is a heads-up display that does not move or rotate with the user's head or the user's entire body, but optionally changes the user's viewpoint in a three-dimensional environment based on the movement of the user's head or body relative to the display generation component. In some embodiments, the display generation component (e.g., a touchscreen) may optionally be moved and rotated by the user's hand relative to the physical environment or relative to the user's head, and changes the user's viewpoint in a three-dimensional environment based on the movement of the display generation component relative to the user's head or face or relative to the physical environment.

[0195] In some implementations, the display generation component 7100 includes a head-mounted display (HMD) 7100a. For example, such as Figure 7E1 (For example, and) Figure 8D2 and Figure 8E1As illustrated, the head-mounted display 7100a includes one or more displays that show a representation of a portion of a three-dimensional environment 7000' corresponding to the user's viewpoint. While HMDs typically include multiple displays, including a display for the right eye and a separate display for the left eye, displaying slightly different images to generate a user interface with stereoscopic depth, the figure shows a single image corresponding to the image for one eye, and depth information is indicated by other annotations or descriptions in the figure. In some embodiments, HMD 7100a includes one or more sensors (e.g., one or more inward-facing and / or outward-facing image sensors 314), such as sensors 7101a, 7101b, and / or 7101c, for detecting the user's state, including facial and / or eye tracking (e.g., using one or more inward-facing sensors 7101a and / or 7101b) and / or tracking the user's hands, torso, or other movements (e.g., using one or more outward-facing sensors 7101c). In some embodiments, the HMD 7100a includes one or more input devices optionally located on the housing of the HMD 7100a, such as one or more buttons, a touchpad, a touchscreen, a scroll wheel, a rotatable and pressable digital crown, or other input devices. In some embodiments, the input element is a mechanical input element; in other embodiments, the input element is a solid-state input element that responds to a press input based on detected pressure or intensity. For example, in Figure 7E1 (For example, and) Figure 8D2 and Figure 8E1 In HMD 7100a, one or more of buttons 701a, buttons 701b, and a digital crown 703 are included for providing input to HMD 7100a. It should be understood that additional and / or alternative input devices may be included in HMD 7100a.

[0196] Figure 7E2 (For example, and) Figure 8D3 and Figure 8E2 The image shows a top view of user 7002 in physical environment 7000. For example, user 7002 is wearing HMD 7100a, such that user's hands 7020 and / or 7022 (e.g., which may be used to provide air gestures or other user input) are physically present within physical environment 7000 behind the display of HMD 7100a.

[0197] Figure 7E1 (For example, and) Figure 8D2 and Figure 8E1 Examples are given for comparison with Figures 7A to 7D , Figures 7E3 to 8D1 and Figures 8E3 to 8MThe illustrated displays represent alternative display generation components for different computer systems. It should be understood that this document references... Figures 7A to 7D , Figures 7E3 to 8D1 and Figures 8E3 to 8M The processes, features, and functions described in the display generation component 7100 also apply to Figures 7E1 to 7E2 , Figures 8D2 to 8D3 and Figures 8E1 to 8E2 Illustrated HMD 7100a.

[0198] Figure 7A A physical environment 7000 is illustrated, which includes a first wall 7004, a second wall 7006, a floor 7008, and a physical object 7014. User 7002 has a left hand 7020 and a right hand 7022, and user 7002 is holding computer system 101 (e.g., ...). Figure 1A The display generation component 7100 (as shown). Figure 7A It also includes a top view of user 7002, and arrows indicating the direction user 7002 is facing (e.g., in...). Figure 7A In the top view of user 7002, the user is facing the direction of physical object 7014. The top view of user 7002 also includes virtual object 7016, which does not exist in the physical environment 7000.

[0199] Figure 7B A view of a three-dimensional environment visible via the display generation component 7100 is illustrated. In some embodiments, the three-dimensional environment includes elements of a physical environment 7000, such as a representation 7014' of a physical object 7014, a representation 7004' of a first wall 7004, and a representation 7008' of a floor 7008. In some embodiments, the three-dimensional environment includes one or more virtual elements, such as virtual elements 7024, 7026, 7028, 7030, 7032, 7034, 7036, and 7038. In some embodiments, virtual elements 7024, 7026, 7028, 7030, 7032, 7034, 7036, and 7038 are application launch capabilities (e.g., and optionally, this set of application launch capabilities includes a main user interface for launching and / or accessing applications on computer system 101), and user 7002 can launch different applications by interacting with the respective application launch capabilities. In some implementations, interactive elements 7024, 7026, 7028, 7030, 7032, 7034, 7036, and 7038 are collectively referred to as the "main user interface".

[0200] Figure 7B It also includes a top-down view of user 7002, which is similar to Figure 7AThe top view shows the position of virtual elements (e.g., virtual elements 7024, 7026, 7032 (partially obscured by virtual element 7026 from the top view) and 7028 (other virtual elements of the main user interface are obscured from the top view)) relative to user 7002. Figure 7B The top view also includes a virtual object 7016 that is currently not visible via the display generation component 7100 of the computer system 101 (e.g., because the virtual object 7016 is behind the user 7002 and outside the user 7002's field of vision).

[0201] Figure 7C A user interface 7032 in a three-dimensional environment is illustrated. In some embodiments, the user interface 7032 is an application user interface (e.g., a user interacts with one of virtual elements 7024, 7026, 7028, 7030, 7032, 7034, 7036, or 7038 to display the user interface 7032). In some embodiments, the user interface 7032 includes interactive elements 7001, 7003, 7005, and 7007. In some embodiments, interactive elements are buttons or indicator lights, sliders, toggle devices, text input fields, or other suitable interactive elements that enable user 7002 to interact with the user interface 7032. Figure 7C In this embodiment, user interface 7032 is shown as having four interactive elements, but in some embodiments, user interface 7032 includes additional or fewer interactive elements. The following description describes various behaviors and appearances of user interface 7032, but in some embodiments, the described behaviors and appearances also apply to the main user interface (e.g., Figure 7B The main user interface includes interactive elements 7024, 7026, 7028, 7030, 7032, 7034, 7036 and 7038 (for example, user interface 7032 and the main user interface are interchangeable in the figures described below).

[0202] In some implementations, when the computer system 101 displays the user interface 7032 (e.g., in response to detecting a connection with...), Figure 7B(For user interaction with corresponding virtual elements in the interface), computer system 101 also positions light sources 7042 and 7044 relative to user interface 7032. In some embodiments, user interface 7032 is a reference point for light sources 7042 and 7044 (e.g., light sources 7042 and 7044 are positioned relative to this reference point, and this reference point is a point in or along the user interface 7032 (e.g., a center point)). In some embodiments, light sources 7042 and 7044 influence other user interface elements (e.g., user interface 7050, as described further below) (e.g., projecting light onto it), such that light sources 7042 and 7044 are initially positioned relative to user interface 7032, but can interact with other (e.g., or all) user interface elements. In some embodiments, the main user interface (e.g., ... Figure 7B The main user interface, including interactive elements 7024, 7026, 7028, 7030, 7032, 7034, 7036, and 7038, serves as a reference point for light sources 7042 and 7044 (e.g., and the description of the behavior of light sources 7042 and 7044 with reference to user interface 7032 herein similarly applies to the main user interface). In some embodiments, the aforementioned "reference point" corresponds to the initial positioning of light sources 7042 and 7044, and light sources 7042 and / or 7044 may be repositioned (e.g., automatically) based on the movement of different user interfaces and / or user interface elements (e.g., the main user interface, user interface 7032, or user interface 7050, as described further in detail below). In some implementations, the aforementioned “reference point” applies to the initial positioning of light sources 7042 and 7044, and is not referenced after the initial positioning of light sources 7042 and 7044, unless the user interface 7032 (e.g., or the main user interface) is re-displayed at the new location (e.g., in this case, light sources 7042 and 7044 are repositioned at a predefined position relative to the new position of the user interface 7032) (e.g., optionally, regardless of whether and / or how light sources 7042 and 7044 are repositioned according to the movement of other user interfaces or other user interface elements, as described below).

[0203] In some implementations, light sources 7042 and 7044 are placed opposite each other, positioning the user interface 7032 between the two light sources. For example, as Figure 7CAs shown, light source 7042 is located in the upper left corner, and light source 7044 is located in the lower right corner. In some embodiments, light sources 7042 and 7044 are positioned relative to the user interface 7032 at a default location. For example, light sources 7042 and 7044 are positioned at a 45-degree angle (e.g., relative to the horizontal direction or axis of the user interface 7032) and at a preset distance from the user interface 7032 (e.g., the center point of the user interface 7032).

[0204] In some embodiments, the user interface 7032 is substantially planar, and light sources 7042 and 7044 are positioned in the same plane as the user interface 7032. In some embodiments, light sources 7042 and 7044 are positioned in the same plane defined by the display generation component 7100 of the computer system 101 (e.g., in cases where the user interface 7032 has a non-planar or otherwise irregular shape).

[0205] User interface 7032 displays visual effects 7046 and 7048. In some embodiments, visual effect 7046 corresponds to light source 7042 (e.g., originating from the light source and / or having an appearance depending on the (e.g., spatial) relationship between visual effect 7046 and the light source), and visual effect 7048 corresponds to light source 7044 (e.g., originating from the light source and / or having an appearance depending on the (e.g., spatial) relationship between visual effect 7046 and the light source). In some implementations, visual effect 7046 corresponds to both light source 7042 and light source 7044 (e.g., the appearance of visual effect 7046 depends on the (e.g., spatial) relationship between visual effect 7046 and light source 7042, and also depends on the (e.g., spatial) relationship between visual effect 7046 and light source 7044), and visual effect 7048 corresponds to both light source 7042 and light source 7044 (e.g., the appearance of visual effect 7048 depends on the (e.g., spatial) relationship between visual effect 7048 and light source 7042, and also depends on the (e.g., spatial) relationship between visual effect 7048 and light source 7044).

[0206] In some implementations, visual effects 7046 and 7048 are reflection effects, glow effects, or shadow effects. The following description of visual effects 7046 and 7048 applies to any suitable visual effect.

[0207] Figure 7CThe top view shows user 7002 facing user interface 7032, which is located between user 7002 and physical object 7014 (representation). Light sources 7042 and 7044 are positioned in the same plane as user interface 7032 (e.g., an imaginary plane that includes user interface 7032 and extends beyond the boundary of user interface 7032).

[0208] Figure 7D An example is shown of a user interface 7050 displayed concurrently with user interface 7032 (e.g., and as shown in the example). Figure 7D As shown, the user interface is displayed as a portion of the user interface 7032 that covers or obscures the viewpoint of the user 7002 (shown via the display generation component 7100 of the computer system 101). In some embodiments, in response to detecting user input (e.g., pointing to...), Figure 7C User input, or pointers to interactive elements in the user interface 7032 Figure 7B The user interface 7050 is displayed using virtual elements (user input).

[0209] User interface 7050 includes interactive elements such as text input field 7056, button 7009, and button 7011. In some embodiments, user interface 7050 includes additional interactive elements, optionally in addition to... Figure 7D In addition to one or more interactive elements shown, and in some embodiments, the user interface 7050 includes more than Figure 7D Fewer interactive elements are shown. In some implementations, one or more visual effects (e.g., reflection effects, glow effects, and / or shadow effects) are applied to one or more user interface elements of user interface 7050. For example, Figure 7D A text input field 7056 with visual effect 7058 is shown, and additional visual effects are optionally applied to buttons 7009 and 7011 (e.g., in addition to visual effect 7058 being applied to text input field 7056). In some embodiments, visual effects applied to other user interface elements (e.g., buttons 7009 and / or 7011) have similar behavior to the visual effects described below.

[0210] User interface 7050 displays visual effect 7052, which appears in the upper left corner of user interface 7050 and corresponds to light source 7042 (e.g., and / or light source 7044) (e.g., having an appearance depending on the (e.g., spatial) relationship between visual effect 7052 and the light source). User interface 7050 displays visual effect 7054, which appears in the lower left corner of user interface 7050 and corresponds to light source 7044 (e.g., and / or light source 7042) (e.g., having an appearance depending on the (e.g., spatial) relationship between visual effect 7054 and the light source). In some embodiments, visual effects 7052 and 7054 are reflection effects, glow effects, or shadow effects. The following description of visual effects 7052 and 7054 applies to any suitable visual effect.

[0211] In some implementations, visual effects 7052 and 7054 correspond to light sources other than light source 7042 and / or light source 7044. For example, visual effect 7052 corresponds to a first light source (e.g., other than light source 7042 and light source 7044) and / or a second light source (e.g., other than the first light source, light source 7042, and light source 7044), and visual effect 7054 corresponds to the second light source and / or the first light source. The first and second light sources are similar to light source 7042 and light source 7044, respectively, but are positioned at predefined locations relative to user interface 7050 (e.g., instead of user interface 7032, as in the case of light source 7042 and light source 7044).

[0212] In some embodiments, the user interface 7050 includes interactive elements with simulated three-dimensional effects. For example, text input field 7056 is displayed with an appearance that includes simulated depth effects (e.g., to give the appearance that text input field 7056 is recessed into the surface of user interface 7050). In some embodiments, user interface 7050 includes a mix of interactive elements with and without simulated three-dimensional effects (e.g., text input field 7056 includes simulated three-dimensional effects, but buttons 7009 and 7011 do not). In some embodiments, interactive elements displaying simulated three-dimensional effects also display visual effects corresponding to one or more available light sources.

[0213] For example, text input field 7056 displays a visual effect 7058, which corresponds to light source 7042 (e.g., and / or light source 7044). In some embodiments, visual effect 7058 is a reflection effect, a glow effect, and / or a shadow effect.

[0214] Figure 7D The top view is similar to Figure 7CThe view includes a top view, but also a user interface 7050, which is displayed from a viewpoint closer to user 7002 (e.g., between user interface 7032 and user 7002).

[0215] In Figure 7E (for example, Figure 7E1 , Figure 7E2 and Figure 7E3 (e.g., where in) Figure 7E1 The HMD 7100a shows something similar to Figure 7E3 The user interface described herein), user interface 7050 relative to user interface 7032 (e.g., and light sources 7042 and 7044, user 7002, Figure 7B The main user interface (e.g., which may optionally replace user interface 7032) and / or the 3D environment are repositioned.

[0216] Figure 7E (for example, Figure 7E1 , Figure 7E2 and Figure 7E3 Visual effects 7052, 7054, and 7058 are illustrated with an updated appearance to reflect the new relative positioning of the user interface 7050 with respect to light sources 7042 and 7044. For example, as the user interface 7050 is repositioned (e.g., moved horizontally relative to the viewpoint of user 7002, as shown in FIG. 7E), visual effect 7052 is displayed in different locations (e.g., repositioned along the edge of the user interface 7050 depending on the change in the position of the user interface 7050) and / or has different sizes (e.g., to simulate the amount and / or direction of light reaching the user interface 7050, which would produce visual effect 7052 in a real-world scene). In some embodiments, if the user interface 7050 is repositioned in the opposite direction to that shown in FIG. 7E, visual effects 7052, 7054, and 7055 are updated in the opposite or reverse manner (e.g., moved or expanded to the left, rather than moved or expanded to the right).

[0217] In some implementations, the visual effect 7058 of the text input field 7056 changes appearance by a different amount compared to the changes in visual effects 7052 and 7054 of the user interface 7050 more overall. For example, visual effect 7052 (e.g., to the right) moves by a first amount, and visual effect 7054 (e.g., to the left) moves by a first amount, and visual effect 7058 moves by a second amount different from the first amount (e.g., a larger amount, as shown in Figure 7E). In some implementations, visual effect 7058 changes appearance in a first direction (e.g., horizontal or x-direction) but not in a second direction (e.g., vertical or y-direction). For example, compared with Figure 7DCompared to visual effect 7058 in Figure 7E, visual effect 7058 in Figure 7E has extended along the bottom surface of text input field 7056, but the amount of visual effect 7058 displayed along the right surface of text input field 7056 is less. Figure 7D This is the same as in Figure 7E (e.g., visual effect 7058 does not change the appearance along the right surface of text input field 7056 and only changes the appearance along the bottom surface of text input field 7058). In some embodiments, visual effect 7058 changes the appearance in both the first and second directions, but the amount of appearance change in the first direction is different from the amount of appearance change in the second direction (e.g., the amount by which visual effect 7058 expands (or contracts) along the bottom surface of the text input field is greater than the amount by which visual effect 7058 expands or contracts along the right surface of text input field 7056, and vice versa).

[0218] In some implementations, when the user interface 7050 is repositioned, visual effects 7052 and 7054 are displayed with different appearances (e.g., visually de-emphasizing the different appearances of visual effects 7052 and 7054, such as a darker appearance, a brighter appearance, and / or a more blurred appearance). In some implementations, the user interface 7050 is displayed with a different appearance (e.g., visually emphasizing different appearances of the user interface 7050, such as a darker appearance, a brighter appearance, and / or a more blurred appearance), and visual effects 7052 and 7054 are displayed with a different appearance that visually emphasizes visual effects 7052 and 7054 by a greater amount than that of the user interface 7050 (e.g., when the user interface 7050 is repositioned, the user interface 7050 is darkened, brightened, and / or blurred, and when the user interface 7050 is repositioned, visual effects 7052 and 7054 are darkened, brightened, and / or blurred by a greater amount than that of the user interface 7050) (e.g., such that visual effects 7052 and 7054 are visually emphasized relative to (e.g., visually emphasized) the different appearances of the user interface 7050). In some embodiments, in response to the detection of user input pointing to a motion indicator (e.g., a grabber or other motion and / or resizing indicator) directed at the user interface 7050, the user interface 7050 is displayed with a different appearance (e.g., visually de-emphasizing the user interface 7050) (e.g., even if the user interface 7050 has not been repositioned). In some embodiments, once the user interface 7050 is no longer repositioned (e.g., substantially stationary), the user interface 7050, visual effects 7052, and / or visual effects 7054 are displayed with their original or default appearance (e.g., similar to...). Figure 7D (Same appearance as in the text) is displayed.

[0219] In some implementations, when the user interface 7050 is repositioned, if visual effects 7052 and 7054 are displayed with reduced visual salience, the computer system 101 displays an animated transition of visual effects 7052 and / or 7054 sliding into place via the display generation component 7100 (e.g., visual effects 7052 and / or 7054 from...). Figure 7D (Animated transitions of position sliding and / or expanding to the position in Figure 7E). In some embodiments where visual effects 7052 and 7054 are shadow effects, as the user interface 7050 is repositioned, the computer system 101 displays visual effects 7052 and / or visual effects 7054 via display generation component 7100 through multiple intermediate appearance transitions corresponding to occlusion of light source 7042 and / or light source 7044 (e.g., occlusion by one or more visible or invisible user interface objects).

[0220] In some embodiments where visual effects 7052 and 7054 are shadow effects, the shadow effects of visual effects 7052 and 7054 remain visible when the user interface 7050 is repositioned (e.g., even if the user interface 7050 is displayed with reduced visual salience when it is repositioned). In some embodiments where visual effects 7052 and 7054 are shadow effects, as the user interface 7050 is repositioned, visual effects 7052 and 7054 are displayed with reduced visual salience (e.g., as described above in the preceding paragraph).

[0221] exist Figure 7F In this view, the user interface 7050 continues to be repositioned (e.g., moved further along the same direction as in Figure 7E). This top view shows the new relative positioning of the user interface 7050 in the three-dimensional environment. Based on the repositioning of the user interface 7050, visual effects 7052, 7054, and 7058 are again updated to a new appearance (e.g., updated to a newer position).

[0222] In some implementations, the user interface 7050 is in Figure 7D Between Figure 7E and Figure 7E Figure 7F Reposition the same quantity. Visual effects 7052 and 7054 in Figure 7D Between and Figure 7E and in Figure 7E and Figure 7F The same amount of movement (e.g., and / or size change) between them. In contrast, visual effect 7058 in Figure 7D A larger amount of movement (e.g., and / or dimensional change) between and Figure 7E, and in Figure 7E and Figure 7FThe amount of movement between them is less. In other words, in some implementations, the visual effect applied to user interface elements (e.g., text input field 7056) within the user interface (e.g., user interface 7050) reflects a greater degree of change in response to the first part of the movement, and the degree of change decreases (e.g., gradually decreases) in response to the second part of the movement (e.g., which optionally includes the same amount of movement as the first part of the movement).

[0223] Figure 7F Four regions are illustrated, including region 7060 along the top surface of user interface 7050, region 7062 along the left surface of user interface 7050, region 7064 along the right surface of user interface 7050, and region 7066 along the bottom surface of user interface 7050. Regions 7060, 7062, 7064, and 7066 may optionally not be visually displayed (e.g., illustrated for reference but not rendered by the display generation component 7100 of computer system 101), and represent areas in which no visual effects are displayed. For example, visual effect 7052 continues to move or extend to the right along the top edge of user interface 7050, but visual effect 7052 cannot move or extend into region 7060 (e.g., visual effect 7052 moves or extends until it is adjacent to region 7060, such as...). Figure 7F As shown, visual effect 7052 cannot move or extend beyond the boundary of region 7060. Similarly, visual effect 7054 extends along the bottom surface of user interface 7050 until it is adjacent to region 7066, but does not extend beyond the boundary of region 7066.

[0224] Regions 7060, 7062, 7064, and 7066 prevent visual effects 7052 and 7054 from accurately simulating the visual effects caused by light sources 7042 and 7044 (e.g., if visual effect 7042 is a reflection of light from light source 7042, then an accurate simulation of the reflected light would be able to extend into and through region 7060). However, regions 7060, 7062, 7064, and 7066 allow computer system 101 to maintain consistent visual sharpness. For example, in some cases, a truly accurate simulation of real-world lighting would result in visual effect 7052 (e.g., reflection) spanning the entire top surface of user interface 7050. In practice, this could cause visual artifacts (e.g., undesirable aliasing) in the displayed view, and displaying visual effects spanning large areas (e.g., even if accurately rendered) could negatively impact the visibility (e.g., and therefore accessibility) of the user interface and / or user interface elements.

[0225] In some implementations, regions 7060, 7062, 7064, and 7066 are associated with a distance threshold (e.g., threshold movement amount). If the user interface 7050 is repositioned to a new location less than the threshold distance from the original location, visual representations 7052, 7054, and 7058 are updated, such as... Figures 7D to 7F As shown. In some implementations, Figure 7F The state shown is that the user interface 7050 has been moved to a new location that is just below a threshold distance from the original location (e.g., the maximum distance before reaching or exceeding the threshold distance). In some embodiments, as long as the user interface 7050 is repositioned to a new location that is less than a threshold distance from the original location, the light sources 7042 and 7044 will not change their position or orientation as the user interface 7050 is repositioned.

[0226] exist Figure 7G In the process, the user interface 7050 continues to be repositioned and has moved to a position consistent with its original location (e.g., ...). Figure 7D The user interface 7050 is now positioned at a new location that is a threshold distance (e.g., or greater than a threshold distance). As shown in the top view, the user interface 7050 is now rotated (e.g., rotated with the user 7002's viewpoint as the user 7002 rotates to face a different direction). The view of the 3D environment visible via the display generation component 7100 of the computer system 101 is updated to display representations 7004' of wall 7004, 7006' of wall 7006, and 7008' of floor 7008, consistent with the user 7002's current viewpoint; the representation 7014' of physical object 7014 is no longer visible.

[0227] In response to the detection that the user interface 7050 has moved to a new location that is a threshold distance (e.g., or greater than the threshold distance) from its original location (e.g., and because the visual effect is not displayed in areas 7060, 7062, 7064, and 7066), light sources 7042 and 7044 are repositioned. In some embodiments, light sources 7042 and 7044 are "flipped" relative to their default locations around (e.g., the vertical axis or y-axis of the user interface 7050). In other words, if the default locations of light sources 7042 and 7044 are at the top left and bottom right corners, respectively, then once "flipped" around the vertical axis or y-axis (e.g., according to movement along the horizontal axis or x-axis), the new locations of light sources 7042 and 7044 are at the top right and bottom left corners, respectively (e.g., and because the visual effect is not displayed in areas 7060, 7062, 7064, and 7066). Figure 7G (As shown).

[0228] Visual effects 7052, 7054, and 7068 are updated according to the movement of the user interface 7050. In some embodiments, since visual effects 7052 and 7054 correspond to light sources 7042 and 7044, respectively, when the light sources are "flipped," visual effects 7052 and 7054 are also "flipped" (e.g., displayed on opposite sides of regions 7060 and 7066, respectively). In some embodiments, before the user interface 7050 moves to a location within a threshold distance from the original location, visual effects 7052 and 7054 move by a first amount when the user interface 7050 moves by a first amount, and when the user interface 7050 moves to a location within a threshold distance (e.g., or greater than the threshold distance) from the original location, visual effects 7052 and 7054 move by a second amount (e.g., greater than the first amount) when the user interface 7050 moves by a second amount (e.g., because visual effects 7052 and 7054 cannot be displayed in areas 7060 and 7066). In some embodiments, for a corresponding amount of movement of the user interface 7050 to a location within a threshold distance from the original location, the amount of movement of visual effects 7052 and 7054 is less than the same corresponding amount of movement of the user interface to a location at least a threshold distance from the original location.

[0229] In some implementations, even if light sources 7042 and 7044 are repositioned relative to the user interface 7050, the visual effect 7058 will not "flip" (e.g., the visual effect of user interface elements within the user interface 7050 behaves differently than the visual effect applied to the user interface 7050 itself). Instead, as Figure 7G As shown, visual effect 7058 continues to be similar to... Figures 7D to 7F Updated in a similar manner as shown and described with reference to these figures, but with a smaller degree of variation (e.g., smaller movement and / or size changes).

[0230] exist Figure 7H In this process, the user interface 7050 continues to be repositioned. As shown in the top view, the user interface 7050 continues to be rotated (e.g., rotating with the user 7002's viewpoint as the user 7002 rotates to face a different direction). The view of the 3D environment visible via the display generation component 7100 of the computer system 101 is updated to display a virtual object 7016 consistent with the user 7002's current viewpoint (e.g., which is now located in the viewport of the 3D environment visible to the user 7002), a representation of the walls 7006', and a representation of the floor 7008'; the representation 7004' of the wall 7004 is no longer visible.

[0231] Because the user interface 7050 is moved to a location previously occupied (e.g., Figure 7G The new positioning of the user interface 7050 is less than a threshold distance, therefore visual effects 7052 and 7054 are in accordance with the above reference. Figures 7D to 7F Updated in a similar manner to the description. Visual effect 7058 continues in a manner similar to... Figures 7D to 7F Updated in a similar manner to those shown and described, but with a smaller degree of change (e.g., smaller movement and / or size changes, which may optionally be smaller than those from...). Figures 7F to 7G The amount of movement and / or size change). In some implementations, if the user interface 7050 moves to a position previously occupied (e.g., Figure 7G If the user interface 7050 is repositioned at a new location that is a threshold distance (e.g., or greater than a threshold distance) from the previous location of the user interface 7050, then visual effects 7052 and 7054 are again "flipped" around the y-axis to the opposite side (e.g., switched to the opposite side of regions 7060 and 7066, respectively). In this way, as the user interface 7050 continues to be repositioned, each time the user interface 7050 moves a threshold distance (e.g., moves to a new location that is a threshold distance from the previous location of visual effects 7052 and 7054), visual effects 7052 and 7054 continue to "flip" around the y-axis of the user interface 7050 (e.g., flipped to the alternating side relative to regions 7060 and 7066, respectively). In some embodiments, the threshold distance is defined such that if the user interface 7050 moves by a threshold distance from the previous location of the user interface 7050, then visual effects 7052 and 7054 are again "flipped" around the y-axis of the user interface 7050 (e.g., flipped to the alternating side relative to regions 7060 and 7066, respectively). Figures 7F to 7G In a similar manner to the example shown, the user interface 7050 is rotated for repositioning. Each time the user interface 7050 is rotated, visual effects 7052 and 7054 continue to "flip" around the y-axis of the user interface 7050 (e.g., the rotation causes the user interface 7050 to move to a new position a threshold distance from the previous position of visual effects 7052 and 7054 after their last "flip"). Furthermore, if the user interface 7050 rotates far enough that it returns to its original position... Figure 7D or Figure 7F If the user interface 7050 is positioned correctly, then visual effects 7052 and 7054 will be displayed in the same location. Figure 7D and Figure 7F The same location (e.g., or locations on the same side of regions 7060 and 7066 respectively) (e.g., such as...) Figure 7DAs shown, visual effect 7052 begins on the left side of region 7060 and visual effect 7054 begins on the right side of region 7066. Visual effects 7052 and 7054 are "flipped" once every 45 degrees (or other angular intervals, such as 15 to 120 degrees) of user interface rotation, so that when user interface 7050 has rotated a full 360 degrees (e.g., returned to its original position), visual effects 7052 and 7054 are again located on the left side of region 7060 and the right side of region 7066, respectively.

[0232] exist Figure 7I In the process, the viewpoint of user 7002 changes (e.g., user 7002 repositions display generation component 7100 to display a different view of the 3D environment), but user 7002 does not reposition user interface 7050 (e.g., user interface 7050 is in...). Figure 7H and Figure 7I Both are in the same position in the three-dimensional environment. As shown in the top view, user 7002 has moved to the intermediate position 7068, and then to a new position with a new perspective.

[0233] Visual effects 7052 and 7054 are updated to reflect changes in the user's (7002's) viewpoint. For example, visual effect 7052 continues to move along the top surface of the user interface 7050 (e.g., and moves almost completely away from the top surface of the user interface 7050), and visual effect 7054 has moved and / or extended to the boundary of region 7062 (e.g., but cannot move and / or extend into or beyond region 7062).

[0234] In some implementations, for certain types of visual effects (e.g., shadow effects), visual effects 7052 and 7054 are not updated to reflect changes in the user 7002's viewpoint. Instead, for these types of visual effects... Figure 7I The positioning of visual effects 7052 and 7054 shown is such that visual effects 7052 and 7054 are alternatively positioned relative to... Figure 7H The same location (e.g., relative to user interface 7050).

[0235] In some implementations, visual effects applied to user interface elements (e.g., visual effect 7058 corresponding to text input field 7056) have different behaviors than visual effects applied to user interface 7050 (e.g., its boundaries and / or window) (e.g., visual effects 7052 and 7054). For example, the appearance of visual effect 7058 does not change when the viewpoint of user 7002 changes (e.g., as...). Figures 7H to 7IAs shown), the appearance of visual effects 7052 and 7054 does change when the user's viewpoint changes (e.g., and / or the user interface 7050 is repositioned). In some embodiments, visual effect 7058 changes its appearance when the user's viewpoint changes and does not change its appearance when the user interface 7050 is repositioned, while visual effects 7052 and 7054 change their appearance both when the user's viewpoint changes and when the user interface 7050 is repositioned.

[0236] In some implementations, visual effect 7058 changes the appearance based on changes in the user's viewpoint in a first direction (e.g., horizontal), but not based on changes in the user's viewpoint in a second direction other than the first direction (e.g., vertical). In some implementations, visual effects 7052 and 7054 change the appearance based on changes in the user's viewpoint in the first direction and also based on changes in the user's viewpoint in the second direction.

[0237] In some implementations, the movement threshold used to determine whether to "flip" light sources 7042 and 7044 is applied only to the movement of user interface 7050 within (e.g., relative to) the 3D environment, and not to the movement of user 7002 (e.g., even if this might change the relative positioning of user interface 7050 relative to user 7002). Thus, even if user 7002 moves significantly and / or moves to extreme viewpoints, user 7002 cannot cause light sources 7042 and 7044 to "flip" back to their original orientation (e.g., light source 7042 is in the upper left corner and light source 7044 is in the lower right corner). In some implementations, computer system 101 continues to update visual effects 7052 and 7054, but without performing any "flipping". For example, visual effect 7054 is displayed at the boundary of region 7062 and can be resized according to the movement of user 7002, but will not enter or overlap with region 7062.

[0238] In some implementations, if user 7002 moves such that user interface 7050 is displayed at an extreme viewing angle (e.g., the user is almost directly looking at the left edge of user interface 7050, which includes visual effect 7054 and area 7062, and only a very small portion of the “front” surface of user interface 7050, including text input field 7056, button 7009, and button 7011, is visible), then computer system 101 stops displaying visual effects 7052 and 7054 (e.g., and optionally, also stops displaying visual effect 7058) (e.g., to prevent any visual artifacts from appearing at extreme viewing angles). For example, when user 7002 is in the intermediate position 7068, computer system 101 continues to display visual effects 7052 and 7054, but when user 7002 is in a more extreme position... Figure 7I When the position is indicated, computer system 7100 stops displaying visual effects 7052 and 7054.

[0239] exist Figure 7J In the middle, user 7002 returns to the original location (for example, Figure 7H The same location as user 7002 in the image). Visual effects 7052, 7054, and 7058 are updated to reflect the change in user 7002's location (e.g., each visual effect in the image). Figure 7J It has with Figure 7H The two images show the same appearance because user 7002 is in the same position in both images, and user interface 7050 is in the same position.

[0240] exist Figure 7K In this case, user 7002 redisplays user interface 7032. As shown in the top view, user interface 7032 is displayed between user interface 7050 and user 7002 (e.g., such that user 7002 cannot currently see user interface 7050 via display generation component 7100 of computer system 101). In some embodiments, user 7002 redisplays (e.g., Figure 7B The main user interface is used instead of (for example, or displayed in addition to the user interface 7032).

[0241] As referenced above Figure 7C As described, in some embodiments, light sources 7042 and 7044 are positioned relative to a reference point in the user interface 7032 (or relative to the user interface 7032, which itself serves as a reference point). Figure 7KAs shown, when the user interface 7032 is redisplayed, light sources 7042 and 7044 are automatically repositioned relative to the reference point in the user interface 7032 at their default positions. In some embodiments, light sources 7042 and 7044 are repositioned regardless of the positioning and / or previous movement of the user interface 7050 (e.g., although in Figure 7G and Figure 7H In this context, the movement of the user interface 7050 causes visual effects 7052 and 7054 to change their positioning (e.g., "flip"), but light sources 7042 and 7044 return to their default positioning (e.g., revert to "not flipped" and return to their original positioning relative to the user interface 7032, as shown in the image). Figure 7C (As shown). Visual effects 7046 and 7048 applied to user interface 7032 are... Figure 7K It has with Figure 7C They have the same appearance (for example, because light source 7042 and light source 7044 have the same relative positioning with respect to user interface 7032 in both figures).

[0242] exist Figure 7L In this process, user 7002 repositions (e.g., moves, restarts, or otherwise recalls) user interface 7050 such that user interface 7050 is displayed between user interface 7032 and user 7002 (e.g., as shown in a top view). Because light sources 7042 and 7044 are as referenced... Figure 7K The description is used for repositioning, therefore visual effects 7052, 7054, and 7058 are in Figure 7L It has with Figure 7D It has the same appearance as the Chinese version.

[0243] While the above description generally does not depend on the type of visual effect and / or combination of visual effect types (e.g., and can be applied to any suitable type of visual effect and / or combination of visual effect types), in some embodiments, one or more of visual effects 7046, 7048, 7052, 7054, and / or 7058 include multiple visual effect types (e.g., a combination of reflection and shadow effects, or a combination of glow and shadow effects). Optionally, the different visual effect types that together constitute a corresponding visual effect (e.g., visual effect 7046, visual effect 7048, visual effect 7052, visual effect 7054, or visual effect 7058) have different positioning relative to the user interface (e.g., user interface 7032 or user interface 7050) corresponding to the corresponding visual effect. For example, if a visual effect representing a simulated lighting effect is displayed in the upper left corner of the user interface, the corresponding simulated shadow is displayed in the lower right corner of the user interface (e.g., consistent with the user interface that occludes the simulated light that produces the simulated lighting effect). In some embodiments, each corresponding visual effect of each corresponding visual effect type changes and / or updates (e.g., as described above, without specifically mentioning the type and / or combination of visual effect types) by the same amount and / or degree (e.g., as the user interface 7050 is repositioned). In some embodiments, when the corresponding visual effect includes a combination of visual effect types, multiple visual effects of different visual effect types (e.g., each corresponding visual effect including each corresponding visual effect type) change and / or update based on the amount of progress along an animation curve. For example, in response to a first amount of movement of the user interface 7050 (e.g., a first portion of movement of the user interface 7050), each corresponding visual effect changes and / or updates by the same first amount along the animation curve, and in response to a second amount of movement of the user interface 7050 (e.g., a second portion of movement of the user interface 7050 following the first portion of movement), each corresponding visual effect changes and / or updates by the same second amount along the animation curve (e.g., which may optionally be different from the first amount).

[0244] The following reference is relative to Figure 9 The described method 900 and relative to Figure 10 Method 1000 described provides information about Figures 7A to 7L Additional description.

[0245] Figures 8A to 8M An example is shown where the shadow corresponding to the user interface is repositioned as the user interface is reoriented. Figure 11 This is a flowchart of an exemplary method 1100 for repositioning the shadow corresponding to the user interface as the user interface is reoriented. Figures 8A to 8M The user interface in the document is used to illustrate the processes described below, including Figure 11 The process in.

[0246] like Figures 8A to 8M As shown in the example, content visible via the display generating component 7100 of computer system 101 is displayed on a touchscreen held by user 7002. In some embodiments, the display generating component 7100 of computer system 101 is a head-mounted display worn on the head of user 7002 (e.g., in...). Figures 8A to 8M The content shown as visible via the display generation component 7100 of the computer system 101 corresponds to the field of view of the user 7002 when wearing a head-mounted display.

[0247] Figure 8A A first view of a three-dimensional environment displayed via a display generation component 7100 of a computer system 101 is illustrated. The first view includes a surface 8000 (e.g., a virtual surface or a perspective view of a physical surface in a corresponding physical environment) and a user interface 8002. Figure 8A The document also includes a side view of the three-dimensional environment, illustrating the relative positioning and orientation of surface 8000 and user interface 8002, to further illustrate the relative depth of user interface 8002 in the first view of the three-dimensional environment. User interface 8002 can be repositioned, reoriented, and / or resized via gripper 8004. While gripper 8004 is illustrated as three bars centered below user interface 8002, gripper 8004 can have any suitable size (e.g., larger than...). Figure 8A The grabber 8004 illustrated may be longer, shorter, wider and / or thinner) and / or shaped (e.g., a single bar, two bars, a circle, a triangle, a rectangle or other shape or graphic), and may be located in any suitable area close to the user interface 8002 (e.g., centered above the user interface 8002, or to the right of the user interface 8002, to the left of the user interface 8002, or other arrangements).

[0248] The light source 8008 is oriented and positioned such that the user interface 8002 casts a shadow 8006 on the surface 8000. As shown in the side view, the shadow 8006 is directly below the user interface 8002. When the user's attention 8010 is not directed at the user interface 8002 or the grabber 8004, the shadow 8006 has a first appearance (e.g., the default appearance).

[0249] exist Figure 8B In the middle, the user's attention 8010 moves to the user interface 8002. In response to detecting that the user's attention 8010 is directed to the user interface 8002, the appearance of the shadow 8006 changes to a different appearance. For example, in Figure 8B In the middle, Shadow 8006 looks better than Figure 8AThe shadow 8006 in the image is darker and / or more opaque to indicate (e.g., and / or provide visual feedback to show) that the user's attention is directed to the user interface 8002.

[0250] exist Figure 8C In this scenario, the user's attention 8010 moves to the grasper 8004, and the user 7002 performs a predefined gesture (e.g., an air tap, an air pinch, or another air gesture) to select the grasper 8004. In response to detecting that the user's attention 8010 is directed at the grasper 8004 (e.g., in conjunction with a predefined gesture), the computer system 101 updates the appearance of the shadow 8006 to a different appearance via the display generation component 7100. In some embodiments, this different appearance (e.g., Figure 8C The appearance of the shadow in 8006) and the default appearance (e.g., Figure 8A The appearance of the shadow 8006 in the image is the same. In some embodiments, the different appearance is the same as... Figure 8A and Figure 8B The shadow 8006 has a different appearance than the other two. In some embodiments, once the computer system 101 detects that the user's attention 8010 is directed towards the grasper 8004 in conjunction with a predefined gesture, the computer system 101 updates the display of the shadow 8006 (e.g., even if the user 7002 has not begun to reposition, reorient, and / or resize the user interface 8002). In some embodiments, the appearance of the shadow 8006 does not change (e.g., remains the same as the previous one). Figure 8B (The same appearance in the previous text), and maintain the same appearance when the user interface 8002 is reoriented (e.g., and Figure 8B (same as above).

[0251] In Figure 8D (for example, Figure 8D1 , Figure 8D2 and Figure 8D3 (e.g., where in) Figure 8D2 The HMD 7100a shows something similar to Figure 8D1 The user interface 8002 (as described in the user interface description) is reoriented by user 7002 (e.g., using an indirect air gesture performed by user's hand 7020 when user's attention 8010 is directed towards grasper 8004, such that the indirect air gesture is not based on the positioning of the user's hand representation 7020'), causing the bottom edge of user interface 8002 to move closer to user 7002's viewpoint and the top edge of user interface 8002 to move further away from user 7002's viewpoint. The positioning of shadow 8002 also changes. Outline 8012 illustrates the original positioning of shadow 8006 (e.g., Figures 8A to 8CThe side view also shows the new position of shadow 8006 (e.g., instead of as shown in the viewpoint of user 7002), and shadow 8006 has a current position that is closer to user 7002 (the viewpoint of user 7002). Figure 8C As shown in Figure 8D, it is located directly below the center of the user interface 8002. Figure 8D1 , Figure 8D2 and Figure 8D3 The shadow 8006 in the image has moved closer to the user's viewpoint than the center of the user interface 8002 is to the user's viewpoint. Although Figure 8D shows the shadow 8006 moving closer to the user's viewpoint, in some implementations, the shadow 8006 moves in another suitable direction (e.g., further away from the user's viewpoint).

[0252] In some implementations, shadow 8006 changes its positioning (e.g., closer to or further away from the user's viewpoint 7002) without changing its size (e.g., width and / or thickness) (e.g., the actual shadow of a real object tilted or reoriented in the same way as user interface 8002 would change size). In some implementations, shadow 8006 has a similar... Figure 8C The shadow 8006 has the same appearance as the default appearance and maintains the same appearance when the user interface 8002 is repositioned, reoriented, and / or resized. In some embodiments, the shadow 8006 has (e.g., different from) the default appearance. Figure 8A , Figure 8B and Figure 8C The fourth appearance is the appearance in the interface 8002, and the fourth appearance is maintained when the user interface 8002 is repositioned, reoriented and / or resized.

[0253] In some implementations, as the user interface 8002 is repositioned, the position of the shadow 8006 is updated in real time (e.g., or at predefined time intervals). In some implementations, the amount of change in the position of the shadow 8006 reflects the amount of change in the orientation of the user interface 8002 (e.g., a small change in the orientation of the user interface 8002 results in a small change in the position of the shadow 8006, and a large change in the orientation of the user interface 8002 results in a large change in the position of the shadow 8006).

[0254] In Figure 8E (for example, Figure 8E1 , Figure 8E2 and Figure 8E3 (for example, where) Figure 8E1 The user interface shown on the HMD 7100a is similar to Figure 8E3(As described in the user interface), user 7002 stops reorienting user interface 8002 (e.g., as shown in Figure 8E where a hand performing a predefined gesture is not present). In response to detecting that the grabber 8004 is no longer selected, computer system 101 updates the appearance of shadow 8006 (e.g., returns to the appearance of the user interface 8002). Figure 8B The same appearance as in the previous example, where the user's attention 8010 is directed to the user interface 8002, and the user 7002 is not repositioned, reoriented, or resized. In some embodiments, the shadow 8006 in Figure 8E has the same appearance as in the previous example. Figure 8B The shadow 8006 has the same appearance (for example, the shadow 8006 is not displayed with the default appearance because the user's attention 8010 is still directed to the grabber 8004 corresponding to the user interface 8002).

[0255] Because the user interface 8002 is further reoriented before the user 7002 stops performing the predefined gesture (e.g., the orientation of the user interface 8002 in FIG. 8E differs from that in FIG. 8D, as shown in the side view), the positioning of the shadow 8006 is also updated (e.g., the shadow 8006 appears closer to the user 7002's viewpoint in FIG. 8E compared to FIG. 8D). In some embodiments, the amount of movement of the shadow 8006 is proportional to the amount by which the user interface 8002 is reoriented (e.g., when the user interface 8002 is reoriented by a larger amount, the shadow 8006 moves a greater distance (e.g., rotates or tilts a greater amount), and when the user interface 8002 is reoriented by a smaller amount, the shadow 8006 moves a smaller distance).

[0256] exist Figure 8F In this state, the user's attention 8010 is no longer directed to the user interface 8002, nor to the grabber 8004, and the computer system 101 updates the appearance of the shadow 8006 to the default appearance (e.g., with...). Figure 8A (Same appearance as in the text).

[0257] Figure 8G Examples illustrate that in some embodiments, user 7002 does not manually adjust the orientation of user interface 8002. In some embodiments, user 7002 repositions user interface 8002 (e.g., changes the position of user interface 8002 without intentionally adjusting its orientation), and computer system 101 automatically repositions user interface 8002 (e.g., to ensure that the content in user interface 8002 remains visible to user 7002 as user interface 8002 moves). In some embodiments, a shadow corresponding to user interface 8002 is displayed (e.g., with...). Figures 8A to 8F The shadow in the image (similar to shadow 8006) has the same characteristics as the reference above. Figures 8A to 8FThe described similar behavior (e.g., the shadow behaves in a similar manner to that described in reference shadow 8006 as the user interface 8002 is automatically repositioned). In some embodiments, when the user interface 8002 is repositioned (e.g., and before the user interface 8002 is automatically repositioned), the shadow moves by a proportional amount (e.g., such that the shadow is always directly below the user interface 8002); when the user interface 8002 begins (e.g., automatically) to reposition, the shadow moves by an additional amount beyond that proportional amount (e.g., compared to the previous amount). Figures 8A to 8G The behavior of the shadow 8006 is consistent with the description.

[0258] For example, Figure 8G The region 8016 around the viewpoint of user 7002 is shown (e.g., in...). Figure 8G (represented by the head of user 7002), where outline 8014 represents the original position of user interface 8002 before it is repositioned.

[0259] In the example above, user interface 8002 begins at a relatively low position in the viewpoint of user 7002. If user 7002 repositions user interface 8002 to move it closer to user 7002's viewpoint, some portions of user interface 8002 may become invisible due to the physical constraints of the display generation component 7100 of computer system 101. To maintain the visibility of these portions of user interface 8002, computer system 101 automatically reorients user interface 8002 to maintain visibility (e.g., by tilting user interface 8002 "up," or described as reorienting user interface 8002 so that the bottom edge of user interface 8002 is closer to user 7002's viewpoint and the top edge of user interface 8002 is further away from user 7002's viewpoint). Geometrically, when user interface 8002 is repositioned so that it will enter region 8016, computer system 101 automatically reorients user interface 8002 so that the surface of user interface 8002 is substantially tangent to the circle defined by region 8016.

[0260] In the intermediate example, the user interface 8002 begins at a lower position in the viewpoint of user 7002, but not as low as in the top example. When the user interface 8002 approaches the viewpoint of user 7002 (e.g., approximately as close as in the top example), but to a lesser extent than in the top example (e.g., the user interface 8002 is not tilted as in the top example), the computer system 101 automatically reorients the user interface 8002.

[0261] In the bottom example, the user interface 8002 begins at a higher position in the viewpoint of user 7002. As the user interface 8002 approaches the user's viewpoint (e.g., approximately as close as in the top and middle examples), the computer system 101 automatically reorients the user interface 8002 (e.g., by tilting the user interface 8002 "down," alternatively described as reorienting the user interface 8002 such that the bottom edge of the user interface 8002 is further away from the viewpoint of user 7002, and the top edge of the user interface 8002 is closer to the viewpoint of user 7002).

[0262] Figure 8H Similar to Figure 8F The difference is in Figure 8H In the middle, the user's attention 8010 is directed at the grasper 8004 and the user 7002 performs a predefined gesture. Since the computer system 101 detects that the user's attention 8010 is directed at the grasper 8004 and that the user 7002 is performing a predefined gesture, shadow 8006 is posed to... Figure 8C The same appearance is shown in Figure 8D.

[0263] exist Figure 8I In the middle, user 7002 is with Figures 8C to 8F In contrast, the user interface 8002 is reoriented in the opposite direction. Figure 8I In this process, the user interface 8002 has been reoriented to have the same characteristics as... Figure 8A The same orientation as the user interface 8002 (e.g., user 7002 "untilted" or straightened the user interface 8002, and reversed it). Figures 8C to 8F (The orientation change executed in the middle). Shadow 8006 moves back to the position of... Figure 8A The same positioning in the middle (e.g., also as shown in the side view).

[0264] exist Figure 8J In this process, user 7002 continues to reorient user interface 8002 in the opposite direction (e.g., the top edge of user interface 8002 moves closer to user 7002's viewpoint, and the bottom edge of user interface 8002 moves further away from user 7002's viewpoint).

[0265] In some implementation schemes, such as Figure 8JAs shown, shadow 8006 moves closer to the viewpoint of user 7002 (e.g., shadow 8006 always moves closer to the viewpoint of user 7002, regardless of whether user interface 8002 is tilted toward or away from the viewpoint of user 7002). In some embodiments, shadow 8006 moves further away from the viewpoint of user 7002 (e.g., tilting user interface 8002 in a first direction causes shadow 8006 to move in the first direction, and tilting user interface 8002 in the opposite direction causes shadow 8006 to move in different (e.g., opposite) directions).

[0266] exist Figure 8K In the middle, user 7002 stops redirecting user interface 8002 (e.g., as...). Figure 8K There is no user hand that performs a predefined gesture (such as...) Figure 8J As shown in the diagram). Because the user's attention 8010 remains focused on the grabber 8004, the shadow 8006 is thus aligned with... Figure 8B The same appearance is shown in Figure 8E.

[0267] exist Figure 8L In this scenario, user 7002 repositions user interface 8002 (e.g., without changing the orientation of user interface 8002), causing user interface 8002 to move further away from surface 8000. In response to detecting the change in the positioning of user interface 8002, computer system 101 updates the appearance of shadow 8006. In some embodiments, shadow 8006 is displayed with an appearance different from that of shadow 8006 in any of the previously described figures (e.g., Figure 8L The appearance of the shadow 8006 is different Figure 8A The appearance of the shadow 8006 and Figure 8B The appearance of shadow 8006 in both. In some embodiments, the size (e.g., width and / or thickness) of shadow 8006 does not change (e.g., even if the shadow cast by a real-world object changes size in a manner similar to the repositioning of user interface 8002). In some embodiments, shadow 8006 is updated to have a visually de-emphasized appearance relative to the default appearance of shadow 8006 (e.g., darker, blurrier, more transparent, and / or other visually de-emphasized).

[0268] Figure 8M This example illustrates what happens if the user interface 8002 moves beyond a threshold height H. Th(For example, where the threshold height is a predefined distance from surface 8000), then computer system 101 stops displaying shadow 8006. In some embodiments, as user interface 8002 is gradually repositioned (e.g., moving upwards relative to surface 8000), computer system 101 displays a gradual change in the appearance of shadow 8006 (e.g., shadow 8006 is displayed with a gradually becoming blurrier and / or brighter appearance as user interface 8002 moves), until user interface 8002 moves beyond the threshold distance H. Th At this point, computer system 101 stops displaying shadow 8006.

[0269] The following reference is relative to Figure 11 The described method 1100 is used to provide information about Figures 8A to 8M Additional description.

[0270] Figure 9 This is a flowchart of an exemplary method 900 for repositioning a shadow corresponding to a user interface as the user interface is reoriented, according to some implementations. In some implementations, method 900 is used in computer systems (e.g., Figure 1A The computer system 101 in the display generation component (e.g., ...) executes the operation at the computer system 101 in the display generation component. Figure 1A , Figure 3 and Figure 4 Display generation component 120 or Figure 7A The display generating component 7100 (e.g., a head-up display, monitor, touchscreen, projector, or other display device) and one or more sensors (e.g., one or more cameras (e.g., color sensors, infrared sensors, and / or other depth-sensing cameras), touch-sensitive surfaces, motion sensors, and / or orientation sensors) communicate. In some embodiments, method 900 is stored in a non-transitory (or transient) computer-readable storage medium and is communicated by one or more processors of a computer system (e.g., one or more processors 202 of computer system 101). Figure 1A The instructions executed by the control 110 in the method are controlled by the control. Some operations in method 900 may be combined, and / or the order of some operations may be changed.

[0271] The computer system displays (902) a first view of a three-dimensional environment via the display generation component, the first view of the three-dimensional environment including a first user interface (e.g., at a first location in the three-dimensional environment (e.g., a first location in the three-dimensional environment that has a first spatial relationship with a first viewpoint associated with the first view of the three-dimensional environment) at a first location in the three-dimensional environment. Figure 7D The user interface 7050 in the first user interface displays a first simulated lighting effect at a first position on the first user interface (e.g., visual effect 7052 is displayed). Figure 7DThe first position is the location within the first sub-region of the first user interface (e.g., the first corner or the first edge) (e.g., visual effect 7052 is in the upper left corner of the user interface 7050, and visual effect 7054 is displayed in the lower right corner of the user interface 7050).

[0272] The computer system detects (904) an event corresponding to the movement of the first user interface (e.g., an input pointing to the first user interface corresponding to a request to move the first user interface, or an input corresponding to a request to move multiple user interfaces including the first user interface, or a request from another participant in a real-time communication session in which the user of the computer system is participating to move the first user interface, or a state change event of the computer system or the first user interface that causes the first user interface to be moved) (e.g., user interface 7050 in Figures 7E to 799). Figure 7G (It has been repositioned).

[0273] In response to the detection of the event, the computer system moves the first user interface from a first location in the three-dimensional environment (906) to a second location in the three-dimensional environment (e.g., the first user interface moves from the first location to the second location in the view of the three-dimensional environment without changing the viewpoint, or the viewpoint changes from a first viewpoint associated with the first view of the three-dimensional environment to a second viewpoint associated with the second view of the three-dimensional environment, which is different from the first view of the three-dimensional environment).

[0274] The mobile first user interface includes: determining that the distance between the first location and the second location is less than a threshold distance (e.g., in...). Figure 7F In the middle, the user interface 7050 moves to the first position (e.g., Figure 7D The second position (located less than a threshold distance) is displayed (908) at the second position on the first user interface, where the first user interface has a second simulated lighting effect. The second position is different from the first position on the first user interface, and the second position is within a first sub-region of the first user interface (e.g., the simulated lighting effect is slightly adjusted along a first corner or first edge of the first user interface). Figure 7F In the middle, visual effect 7052 is displayed in the second position, but still within the upper left corner of user interface 7050, and visual effect 7054 is displayed in the second position, but still within the lower right corner of user interface 7050.

[0275] Furthermore, the mobile first user interface includes: determining that the distance between the first location and the second location is at least a threshold distance (e.g., in...). Figure 7G In the middle, the user interface 7050 moves to the first position (e.g., Figure 7DThe third position on the first user interface is a second location (a distance greater than a threshold distance) displayed (910) with a third simulated lighting effect. The third position on the first user interface is different from the first position and the second position on the first user interface. The third position is within a second sub-region of the first user interface (e.g., a second corner or second edge that is different from the first corner or the first edge) (e.g., Figure 7G The visual effect 7052 is located within the top edge of the upper right corner of the user interface 7050, and the second sub-region of the first user interface is different from the first sub-region of the first user interface (e.g., Figure 7G The visual effect 7052 is displayed near the top right edge of the user interface, or more generally, the top right corner is adjacent to the top edge of the interface. Figure 7D The user interface 7052 displays different sub-regions in the upper left corner. The first user interface includes a third sub-region located between the first sub-region and the second sub-region of the first user interface (e.g., Figure 7G Regions 7060, 7062, 7064, and / or 7066 (e.g., region 7060 is between the upper left and upper right corners of user interface 7050), and the third sub-region is the region that does not display simulated lighting effects (or prevents the display of simulated lighting effects) when the first user interface is stationary relative to the 3D environment (e.g., as user interface 7050 moves from...). Figure 7D Repositioning in Figure 7G (In the positioning, visual effect 7052 is never displayed within the boundary of region 7060).

[0276] In some implementations, the third sub-region is an area where simulated lighting effects may not be displayed when the first user interface is stationary relative to the 3D environment. In some implementations, the third sub-region includes multiple areas of the first user interface where simulated lighting effects are not displayed when the first user interface is stationary relative to the 3D environment. In some implementations, the third sub-region is an area where simulated lighting effects are displayed at most momentarily, such as when transitioning between simulated lighting effects in the first and second sub-regions (e.g., when animing the movement of simulated lighting effects from the first sub-region to the second sub-region, or vice versa).

[0277] Displaying a user interface with simulated lighting effects (e.g., specular reflections) in a 3D environment, and moving the simulated lighting effects according to the movement of the user interface relative to the 3D environment—specifically, moving the simulated lighting effects within the same area of ​​the user interface (e.g., the same edge or corner) when the user interface is moved less than a threshold amount, rather than moving the simulated lighting effects across the middle restricted area of ​​the user interface to different areas (e.g., relative edges or corners) when the user interface is moved more than a threshold amount—and restricting the display of the simulated lighting effects in the restricted areas when the user interface is stationary, allows the computer system to automatically provide visual feedback about the spatial relationship between the user interface and the 3D environment. This improves the user's contextual awareness of the 3D environment. Providing visual cues (even subtle ones) about the 3D environment can help and improve user physiological comfort by avoiding physiological discomfort associated with the user's body movements (which are mismatched with the environmental response in the user's visible 3D environment). User body movements that are mismatched with the environmental response in the user's visible 3D environment (e.g., in a car, airplane, ship, carnival ride, or other experience) can sometimes cause physiological responses, sometimes referred to as motion sickness. Improving user comfort is an important consideration when creating MR experiences, as reduced comfort may cause users to leave the MR experience and re-enter it or enable and disable features, which increases power consumption and reduces battery life (for battery-powered devices). Conversely, if users are physiologically comfortable and can interact with the device quickly and effectively to perform necessary operations, power consumption is reduced and battery life is extended (for battery-powered devices).

[0278] In some embodiments, moving the first user interface from a first location in the 3D environment to a second location in the 3D environment relative to a reference point in the 3D environment constitutes moving the first user interface. In some embodiments, determining that the distance between the first and second locations is less than a threshold distance corresponds to determining that the first user interface has moved less than a threshold amount relative to the reference point. In some embodiments, determining that the distance between the first and second locations is at least a threshold distance corresponds to determining that the first user interface has moved at least a threshold amount relative to the reference point. For example, in Figure 7E and... Figure 7F In this context, the user interface 7050 moves relative to its representation 7014' of the physical object 7014, which is also displayed in the 3D environment. For example, in Figure 7E, the virtual object 7016 is not visible, and... Figure 7H In this context, the user interface 7050 is moved to a position closer to the virtual object 7016 (e.g., the user interface 7050 moves relative to the virtual object 7016). Furthermore, in Figures 7E to... Figure 7J In the middle, the user interface 7050 is relative to (for example, Figure 7BThe positioning of the main menu user interface or the positioning of user interface 7032 (e.g., the current positioning of the user interface, or its last displayed position if the user interface is not currently displayed) is moved, and the positioning of light sources 7042 and 7044 is based on these positioning. The simulated lighting effect of moving the user interface according to its movement relative to the three-dimensional environment, and specifically relative to a reference point in the three-dimensional environment, enables the computer system to automatically provide visual feedback on the spatial relationship between the user interface and the three-dimensional environment. This improves the user's contextual awareness of the three-dimensional environment and enhances the user's physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible three-dimensional environment).

[0279] In some embodiments, the reference point corresponds to the first main menu location of the main menu user interface in the three-dimensional environment (e.g., the main menu user interface currently or recently displayed in the three-dimensional environment). In some embodiments, the main menu user interface provides a default or main user interface for the computer system and may optionally include an icon for launching an application, an icon for starting a communication session (e.g., an icon corresponding to a different user other than the user in question), an icon for starting a computer-generated experience, an icon for opening a file or other content, container objects such as folders or groups of icons for different categories of user interface objects, a main button, a dock, a root menu, or other user interface elements for performing operations and interacting with the three-dimensional environment. For example, as described with reference to FIG7E, in some embodiments, user interface 7050 is relative to (e.g., Figure 7B The main user interface is repositioned, replacing user interface 7032 in Figure 7E. Simulated lighting effects that move the user interface based on its position relative to the main menu user interface in the 3D environment (e.g., current position (if displayed) or most recently positioned (if not displayed)) enable the computer system to automatically use a consistent reference point when providing visual feedback on the spatial relationship between the user interface and the 3D environment. This further improves the user's contextual awareness of the 3D environment and enhances user physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible 3D environment).

[0280] In some embodiments, the computer system detects one or more inputs corresponding to a request to reposition the main menu user interface at a second main menu location in a three-dimensional environment. In some embodiments, the one or more inputs include movement of the user's viewport relative to the first main menu location (optionally, this movement exceeds a threshold amount of movement required to reposition the main menu user interface), followed by input corresponding to a request to display the main menu user interface. In some embodiments, the one or more inputs include input pointing to the main menu user interface for moving the main menu user interface. In some embodiments, in response to detecting the one or more inputs, the computer system repositions the main menu user interface at the second main menu location. In some embodiments, after repositioning the main menu user interface at the second main menu location (e.g., while the main menu user interface remains associated with the second main menu location, regardless of whether the main menu user interface is still displayed (e.g., after the main menu user interface at the second main menu location is closed, and before the main menu user interface is repositioned to a different main menu location)), the computer system displays the first user interface at the second location, including a first user interface displayed on the first user interface at a corresponding location outside a third sub-region of the first user interface, having a fourth simulated lighting effect. In some implementations, after the main menu user interface is repositioned to and displayed in the second main menu position, the main menu user interface is closed, and the first user interface is redisplayed or regains focus (e.g., if the first user interface remains displayed without focus while the main menu user interface is being displayed in the second main menu position). For example, see reference... Figure 7K As described, in some implementations, user 7002 redisplays the main user interface (e.g., in...). Figure 7K The user interface 7032 is repositioned (as shown in the new location), and light sources 7042 and 7044 are repositioned (e.g., using the main user interface as a reference, and / or at a predefined location relative to the main user interface). The simulated lighting effect that moves the user interface according to the positioning changes of the main menu user interface in the 3D environment (e.g., even without considering the movement of the user interface itself) allows the computer system to automatically use a consistent reference point when providing visual feedback about the spatial relationship between the user interface and the 3D environment. This further improves the user's contextual awareness of the 3D environment and enhances user physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible 3D environment).

[0281] In some embodiments, when the main menu user interface is associated with a first main menu location in the three-dimensional environment (e.g., before one or more inputs corresponding to a request to reposition the main menu user interface to a second main menu location are detected), the computer system displays a second user interface at a corresponding location in the three-dimensional environment. In some embodiments, the second user interface differs from the first user interface, and the corresponding location of the second user interface differs from a second location of the first user interface. In some embodiments, the second user interface displays a first corresponding simulated lighting effect at a first corresponding location on the second user interface outside a corresponding sub-region of the second user interface, wherein the corresponding sub-region of the second user interface is an area where simulated lighting effects are not displayed (e.g., blocked from display) when the second user interface is stationary relative to the three-dimensional environment. In some embodiments, depending on whether the relative location of the first user interface to the first main menu location differs from the relative location of the second user interface to the first main menu location, when the first user interface is in a second location in the three-dimensional environment and the main menu user interface is in the first main menu location, the first corresponding simulated lighting effect displayed for the second user interface differs from the simulated lighting effect displayed for the first user interface (e.g., a second or third simulated lighting effect).

[0282] In some embodiments, after the main menu user interface is repositioned to a second main menu location (e.g., while the main menu user interface remains associated with the second main menu location), the computer system displays (e.g., redisplays or refocuses) the second user interface at the corresponding location, including a second user interface displayed on the second user interface at a second corresponding position outside the corresponding sub-area of ​​the second user interface, having a second corresponding simulated lighting effect. In some embodiments, depending on the relative positioning of the first user interface and the second main menu location being different from the relative positioning of the second user interface and the second main menu location, when the first user interface is in a second location in the three-dimensional environment and the main menu user interface is in the second main menu location, the second corresponding simulated lighting effect displayed for the second user interface is different from the fourth simulated lighting effect displayed for the first user interface.

[0283] For example, in Figure 7L In the process, after repositioning light sources 7042 and 7044 (e.g., due to the redisplay of user interface 7032 and / or the main user interface), user interface 7050 is repositioned (e.g., in front of user interface 7032), and visual effects 7046 and 7054 are displayed in the upper left and lower right corners respectively (e.g., in the...). Figures 7G to 7JOutside of regions 7060, 7062, 7064, and 7066 (where no visual effects are displayed), and when one or more other user interfaces are displayed, the visual effects on those other user interfaces exhibit similar behavior to those on user interface 7050. The corresponding simulated lighting effects that move multiple user interfaces in the 3D environment according to the positioning changes of the main menu user interface (e.g., even without considering the movement of the user interfaces themselves) enable the computer system to automatically use a consistent reference point when providing visual feedback about the spatial relationships between the 3D environment and the different user interfaces within it. This further improves the user's contextual awareness of the 3D environment and enhances user physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible 3D environment).

[0284] In some embodiments, a first user interface displaying a first simulated lighting effect at a first location on a first user interface (e.g., when the first user interface is at a first user interface in a three-dimensional environment) includes: displaying a first portion (e.g., a first range) of the first simulated lighting effect when the user's viewpoint has a first spatial relationship with the first user interface at the first location in the three-dimensional environment; detecting movement of the user's viewpoint relative to the first location in the three-dimensional environment; and, in response to detecting such movement of the user's viewpoint, displaying a second portion (e.g., a second range) of the first simulated lighting effect when the user's viewpoint has a second spatial relationship with the first user interface at the first location in the three-dimensional environment, wherein the second spatial relationship is different from the first spatial relationship, wherein the second portion is different from the first portion. In some embodiments, similarly, when the first user interface is displayed at a second location in the three-dimensional environment, if the spatial relationship between the user's viewpoint and the first user interface at the second location changes, a different portion of the second simulated lighting effect is displayed. In some embodiments, when the first user interface does not move relative to the three-dimensional environment, the first simulated lighting effect remains within a first sub-region of the first user interface regardless of the amount of movement of the user's viewpoint.

[0285] For example, in Figure 7I In the process, the viewpoint of user 7002 moves (e.g., user 7002 moves to a different location), and a second part of visual effect 7054 (e.g., extending to the boundary of region 7062) is displayed (e.g., it was previously in...). Figure 7H(Not shown, i.e., when user 7002 is in its original position before the user's viewpoint moves). As described with reference to Figure 7E, in some embodiments, even if user 7002 moves significantly and / or moves to an extreme viewing angle, the movement of user 7002 (e.g., and / or the change of user 7002's viewpoint) does not cause light sources 7042 and 7044 or the corresponding visual effects 7052 and 7054 to "flip". Instead of moving the simulated lighting effects based on the change of the user's viewpoint relative to the user interface, the computer system automatically provides visual feedback about the user's current viewpoint relative to the three-dimensional environment and, in particular, the user interface. This visual feedback differs from the visual feedback provided about the spatial relationship between the user interface and the three-dimensional environment. This improves the user's contextual awareness of the three-dimensional environment and enhances the user's physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible three-dimensional environment).

[0286] In some implementations, a first simulated lighting effect, a second simulated lighting effect, and a third simulated lighting effect are displayed on the first user interface based on a first set of one or more simulated light sources in a three-dimensional environment. In some implementations, the first simulated lighting effect, the second simulated lighting effect, and / or the third simulated lighting effect are displayed based on one or more physical light sources in a physical environment corresponding to the three-dimensional environment. For example, such as... Figure 7D As shown, visual effect 7052 corresponds to light source 7042 (e.g., having an appearance that depends on the (e.g., spatial) relationship between visual effect 7052 and the light source), and visual effect 7054 corresponds to light source 7044 (e.g., having an appearance that depends on the (e.g., spatial) relationship between visual effect 7054 and the light source). Making different simulated lighting effects of the user interface based on the same simulated light source in the three-dimensional environment enables the computer system to automatically provide consistent visual feedback regarding the spatial relationship between the user interface and the three-dimensional environment. This improves the user's contextual awareness of the three-dimensional environment and enhances user physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible three-dimensional environment).

[0287] In some implementations, a first simulated lighting effect, a second simulated lighting effect, and a third simulated lighting effect are displayed based on two or more simulated light sources in a three-dimensional environment. In some implementations, the first simulated lighting effect, the second simulated lighting effect, and the third simulated lighting effect are displayed based on the same two or more simulated light sources in the three-dimensional environment or different groups of simulated light sources in the three-dimensional environment. For example, in... Figure 7DIn this context, visual effect 7052 corresponds to light source 7042 and / or light source 7044 (e.g., having an appearance dependent on the (e.g., spatial) relationship between visual effect 7052 and these light sources), and visual effect 7054 corresponds to light source 7044 and / or light source 7042 (e.g., having an appearance dependent on the (e.g., spatial) relationship between visual effect 7054 and these light sources). By basing different simulated lighting effects on the user interface on the same multiple simulated light sources in the 3D environment, the computer system automatically provides consistent visual feedback regarding the spatial relationship between the user interface and the 3D environment. This improves the user's contextual perception of the 3D environment and enhances user physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible 3D environment).

[0288] In some embodiments, the computer system displays a third user interface in a three-dimensional environment. In some embodiments, the third user interface differs from the first user interface. In some embodiments, the corresponding positioning of the third user interface differs from the first and second positioning of the first user interface. In some embodiments, the third user interface displays corresponding simulated lighting effects based on a second set of one or more simulated light sources in the three-dimensional environment, wherein the second set of one or more simulated light sources differs from the first set of one or more simulated light sources. In some embodiments, the corresponding simulated lighting effects are displayed on the third user interface at a location outside a corresponding sub-region of the third user interface, wherein the corresponding sub-region of the third user interface is an area where the simulated lighting effects are not displayed (e.g., blocked from display) when the third user interface is stationary relative to the three-dimensional environment. For example, as referenced... Figure 7D As described, in some embodiments, visual effect 7052 corresponds to a first light source (e.g., other than light sources 7042 and 7044) and / or a second light source (e.g., other than the first light source, light source 7042, and light source 7044), and visual effect 7054 corresponds to the second light source and / or the first light source. The first and second light sources are similar to light sources 7042 and 7044, respectively, but are positioned at predefined locations relative to the user interface 7050 (e.g., instead of user interface 7032, as in the case of light sources 7042 and 7044). Making the simulated lighting effects for different user interfaces based on different sets of simulated light sources in a three-dimensional environment allows the computer system to automatically provide customized and more context-sensitive visual feedback regarding the spatial relationship between the corresponding user interface and the three-dimensional environment. This improves the user's contextual perception of the three-dimensional environment and enhances user physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible three-dimensional environment).

[0289] In some embodiments, moving a first user interface from a first location in a three-dimensional environment to a second location in the three-dimensional environment causes the first user interface to move by a corresponding amount (e.g., a corresponding distance). In some embodiments, (e.g., based on determining that the distance between the first and second locations is less than a threshold distance) a first user interface displaying a second simulated lighting effect at a second location on the first user interface includes moving the simulated lighting effect from the first location on the first user interface to the second location on the first user interface by a first amount, which is less than the corresponding amount of movement of the first user interface. In some embodiments, (e.g., based on determining that the distance between the first and second locations is at least a threshold distance) a first user interface displaying a third simulated lighting effect at a third location on the first user interface includes moving the simulated lighting effect from the first location on the first user interface to the third location on the first user interface by a second amount, which is greater than the corresponding amount of movement of the first user interface (e.g., as referenced). Figure 7G (As described). When the simulated lighting effect moves within the same area of ​​the user interface (e.g., the same edge or corner) due to user interface movement less than a threshold amount, the simulated lighting effect moves less than the corresponding movement of the user interface. And when the simulated lighting effect moves beyond the intermediate limit area of ​​the user interface to a different area of ​​the user interface (e.g., relative edge or corner) due to user interface movement exceeding a threshold amount, the simulated lighting effect moves more than the corresponding movement of the user interface. This allows the computer system to automatically provide visual feedback on the movement and corresponding changes in the spatial relationship between the user interface and the 3D environment. This improves the user's contextual awareness of the 3D environment and enhances the user's physiological comfort by avoiding physiological discomfort associated with the user's body movements (which do not match the environmental response in the user's visible 3D environment).

[0290] In some implementations, a first sub-region of the first user interface includes a first corner region of the first user interface (e.g., a vertex or corner, optionally with roundness, or other forms of corner region), and a second sub-region of the first user interface includes a second corner region of the first user interface, wherein the second corner region differs from the first corner region (e.g., is separated from the first corner region) (e.g., the first corner region includes a first corner of the first user interface, and the second corner region includes a different second corner of the first user interface, which is separated from the first corner by an edge, and the third sub-region includes at least a portion of that edge). For example, in Figure 7F In this context, visual effect 7052 is displayed in the first sub-region (e.g., the upper left corner of user interface 7050), and region 7060 (the third sub-region) includes a portion of the top edge of user interface 7050. Figure 7HIn this configuration, visual effect 7052 is displayed in a second sub-region (e.g., the upper right corner of user interface 7050), and region 7060 is located between the upper left and upper right corners of user interface 7050. Based on the movement of the user interface relative to the 3D environment, when the user interface movement is less than a threshold amount, the simulated lighting effect of the user ...

Claims

1. A method comprising: at a computer system in communication with a display generation component and one or more sensors: displaying, via the display generation component, a first view of a three-dimensional environment, the first view of the three-dimensional environment including a first user interface at a first location in the three-dimensional environment, wherein the first user interface displays a first simulated lighting effect at a first position on the first user interface, wherein the first position is a position within a first sub-region of the first user interface; detecting an event corresponding to a movement of the first user interface; and in response to detecting the event, moving the first user interface from the first location in the three-dimensional environment to a second location in the three-dimensional environment, wherein moving the first user interface includes: in accordance with a determination that a distance between the first location and the second location is less than a threshold distance, displaying the first user interface with a second simulated lighting effect at a second position on the first user interface, wherein the second position is different from the first position on the first user interface, and wherein the second position is within the first sub-region of the first user interface; and in accordance with a determination that the distance between the first location and the second location is at least the threshold distance, displaying the first user interface with a third simulated lighting effect at a third position on the first user interface, wherein: the third position on the first user interface is different from the first position on the first user interface and the second position on the first user interface; the third position is within a second sub-region of the first user interface; the second sub-region of the first user interface is different from the first sub-region of the first user interface; and the first user interface includes a third sub-region between the first sub-region and the second sub-region of the first user interface, wherein the third sub-region is a region that does not display a simulated lighting effect when the first user interface is stationary with respect to the three-dimensional environment.

2. The method of claim 1, wherein moving the first user interface from the first location in the three-dimensional environment to the second location in the three-dimensional environment moves the first user interface relative to a reference point in the three-dimensional environment.

3. The method of claim 2, wherein the reference point corresponds to a first home menu location of a home menu user interface in the three-dimensional environment.

4. The method of claim 3, the method comprising: detecting one or more inputs corresponding to a request to reposition the home menu user interface at a second home menu location in the three-dimensional environment; in response to detecting the one or more inputs, repositioning the home menu user interface at the second home menu location; and ​ after repositioning the main menu user interface at the second main menu position, displaying the first user interface at the second position, including displaying the first user interface on the first user interface with a fourth simulated lighting effect at a fourth location on the first user interface that is outside of the third sub-region of the first user interface.

5. The method of claim 4, the method comprising: while the main menu user interface is associated with the first main menu position in the three-dimensional environment, displaying a second user interface at a respective position in the three-dimensional environment, wherein: the second user interface is different from the first user interface, and the respective position of the second user interface is different from the second position of the first user interface; and the second user interface is displayed with a first respective simulated lighting effect at a first respective location on the second user interface that is outside of a respective sub-region of the second user interface, wherein the respective sub-region of the second user interface is a region that does not display a simulated lighting effect when the second user interface is stationary relative to the three-dimensional environment; and after repositioning the main menu user interface at the second main menu position, displaying the second user interface at the respective position, including displaying the second user interface on the second user interface with a second respective simulated lighting effect at a second respective location on the second user interface that is outside of the respective sub-region of the second user interface.

6. The method of any one of claims 1-5, wherein displaying the first user interface on the first user interface with the first simulated lighting effect at the first location comprises: displaying a first portion of the first simulated lighting effect while a user’s point of view has a first spatial relationship with the first user interface at the first position in the three-dimensional environment; detecting movement of the user’s point of view relative to the first position in the three-dimensional environment; and in response to detecting the movement of the user’s point of view, displaying a second portion of the first simulated lighting effect while the user’s point of view has a second spatial relationship with the first user interface at the first position in the three-dimensional environment, wherein the second spatial relationship is different from the first spatial relationship, wherein the second portion is different from the first portion.

7. The method of any one of claims 1-6, wherein the first simulated lighting effect, the second simulated lighting effect, and the third simulated lighting effect of the first user interface are displayed in accordance with a first set of one or more simulated light sources in the three-dimensional environment.

8. The method of claim 7, wherein the first simulated lighting effect, the second simulated lighting effect, and the third simulated lighting effect are displayed in accordance with two or more simulated light sources in the three-dimensional environment.

9. The method of any one of claims 7-8, the method comprising displaying a third user interface in the three-dimensional environment, wherein: the third user interface is different from the first user interface; and the third user interface is displayed with a third simulated lighting effect at a third location on the third user interface that is outside of a third sub-region of the third user interface, wherein the third sub-region of the third user interface is a region that does not display a simulated lighting effect when the third user interface is stationary relative to the three-dimensional environment; and after repositioning the main menu user interface at the second main menu position, displaying the third user interface at the respective position, including displaying the third user interface on the third user interface with a fourth simulated lighting effect at a fourth location on the third user interface that is outside of the third sub-region of the third user interface. the third user interface is displayed with a respective simulated lighting effect in accordance with a second set of one or more simulated light sources in the three-dimensional environment, wherein the second set of one or more simulated light sources is different from the first set of one or more simulated light sources.

10. The method of any of claims 1-9, wherein: moving the first user interface from the first position in the three-dimensional environment to the second position in the three-dimensional environment moves the first user interface by a respective amount; displaying the first user interface with the second simulated lighting effect at the second location on the first user interface includes moving simulated lighting effect from the first location on the first user interface by a first amount to the second location on the first user interface, the first amount being less than the respective amount of movement of the first user interface; and displaying the first user interface with the third simulated lighting effect at the third location on the first user interface includes moving simulated lighting effect from the first location on the first user interface by a second amount to the third location on the first user interface, the second amount being greater than the respective amount of movement of the first user interface.

11. The method of any of claims 1-10, wherein the first sub-region of the first user interface includes a first corner region of the first user interface, and the second sub-region of the first user interface includes a second corner region of the first user interface, wherein the second corner region is different from the first corner region.

12. The method of any of claims 1-11, wherein moving the first user interface in response to detecting the event includes: displaying the first user interface moving through a plurality of intermediate positions in the three-dimensional environment prior to displaying the first user interface at the second position in the three-dimensional environment and with the second simulated lighting effect; and visually deemphasizing the first simulated lighting effect relative to the first user interface while displaying the first user interface moving through the plurality of intermediate positions in the three-dimensional environment.

13. The method of claim 12, wherein, while displaying the first user interface moving through the plurality of intermediate positions in the three-dimensional environment: the first simulated lighting effect is visually deemphasized by a first amount relative to the first simulated lighting effect prior to detecting the event; the first user interface is visually deemphasized by a second amount relative to the first user interface prior to detecting the event; and the first amount is greater than the second amount.

14. The method of any of claims 12-13, wherein: detecting the event corresponding to the movement of the first user interface includes detecting input directed to a movement affordance of the first user interface; and the method includes: visually deemphasizing the first user interface relative to the three-dimensional environment upon detecting the input directed to the movement affordance of the first user interface.

15. The method of any of claims 1-14, wherein the first simulated lighting effect, the second simulated lighting effect, and the third simulated lighting effect correspond to a respective type of lighting effect on the first user interface, and the method includes: detecting movement of a viewpoint of a user relative to the first user interface; and in response to detecting the movement of the viewpoint of the user relative to the first user interface: while the viewpoint of the user is at an angle relative to the first user interface that is within a threshold angular range of the first user interface, displaying the first user interface with a respective simulated lighting effect that is the respective type of simulated lighting effect; and while the viewpoint of the user is at an angle relative to the first user interface that is outside the threshold angular range, displaying the first user interface without the respective type of simulated lighting effect.

16. The method of any of claims 1-15, wherein: the first user interface includes a first user interface element and a second user interface element within a region of the first user interface element; and moving the first user interface in response to detecting the event includes: changing a simulated lighting effect corresponding to the first user interface element in a first manner; and changing a simulated lighting effect corresponding to the second user interface element in a second manner that is different from the first manner.

17. The method of any of claims 1-16, the method including: while displaying the first view of the three-dimensional environment, the first view of the three-dimensional environment including the first user interface at the first location in the three-dimensional environment, displaying the first user interface with a first simulated shadow at a first shadow location corresponding to the first user interface, wherein the first shadow location corresponds to a fourth sub-region of the first user interface; wherein moving the first user interface from the first location in the three-dimensional environment to the second location in the three-dimensional environment in response to detecting the event includes: in accordance with a determination that the distance between the first location and the second location is less than the threshold distance, displaying the first user interface with a second simulated shadow at a second shadow location corresponding to the first user interface, wherein the second shadow location is different from the first shadow location, and wherein the second shadow location corresponds to the fourth sub-region of the first user interface; and in accordance with a determination that the distance between the first location and the second location is at least the threshold distance, displaying the first user interface with a third simulated shadow at a third shadow location corresponding to the first user interface, wherein: the third shadow location corresponding to the first user interface is different from the first shadow location and the second shadow location; the third location is within a fifth sub-region of the first user interface; the fifth sub-region of the first user interface is different from the fourth sub-region of the first user interface; and The first user interface includes a sixth sub-region between the fourth and fifth sub-regions of the first user interface, wherein the sixth sub-region is a region that does not display simulated shadows when the first user interface is stationary relative to the three-dimensional environment.

18. The method of claim 17, wherein moving the first user interface in response to detecting the event includes: displaying the first user interface moving through a plurality of intermediate positions in the three-dimensional environment prior to displaying the first user interface at the second position in the three-dimensional environment; and continuing to display a simulated shadow corresponding to the first user interface while displaying the first user interface moving through the plurality of intermediate positions in the three-dimensional environment.

19. The method of claim 17, wherein moving the first user interface in response to detecting the event includes: displaying the first user interface moving through a plurality of intermediate positions in the three-dimensional environment prior to displaying the first user interface at the second position in the three-dimensional environment and with the second simulated shadow; and visually de-emphasizing the first simulated shadow relative to the first user interface while displaying the first user interface moving through the plurality of intermediate positions in the three-dimensional environment.

20. The method of any of claims 17-19, wherein displaying the first user interface with the first simulated shadow at the first shadow location corresponding to the first user interface comprises: maintaining display of the first simulated shadow at the first shadow position as a user’s point of view moves relative to the first user interface while displaying the first user interface at the first position in the three-dimensional environment.

21. The method of any of claims 17-20, wherein a respective simulated shadow at a respective shadow position simulates an occlusion of one or more light sources in the three-dimensional environment by a respective object at a respective object position, and displaying the respective simulated shadow at the respective shadow position includes displaying the respective simulated shadow moving through a plurality of intermediate appearance transitions corresponding to the respective object occluding the one or more light sources as the respective object moves into position at the respective object position.

22. The method of any of claims 1-21, wherein: the first user interface includes a first user interface element and a second user interface element; and displaying the first user interface with respective simulated lighting effects includes displaying the first user interface element with a first portion of the respective simulated lighting effects and displaying the second user interface element with a second portion of the respective simulated lighting effects.

23. The method of any of claims 1-22, wherein moving the first user interface in response to detecting the event includes: displaying the first user interface moving into position at the second position in the three-dimensional environment; and in accordance with a determination that a distance between the first position and the second position is less than the threshold distance: displaying an animation of the second simulated lighting effects moving into position at the second location on the first user interface; and in accordance with a determination that a distance between the first position and the second position is at least the threshold distance: displaying an animation of the third simulated lighting effect moving into place at the third location on the first user interface includes displaying an animation of a simulated lighting effect moving through the third sub-region of the first user interface as the first user interface moves into place at the second positioning of the three- dimensional environment.

24. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more sensors, the one or more programs including instructions for performing the method of any of claims 1-23.

25. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any of claims 1-23.

26. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: means for performing the method of any of claims 1-23.

27. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more sensors, the one or more programs including instructions for: displaying, via the display generation component, a first view of a three- dimensional environment, the first view of the three-dimensional environment including a first user interface at a first positioning in the three-dimensional environment, wherein the first user interface displays a first simulated lighting effect at a first location on the first user interface, wherein the first location is a location within a first sub-region of the first user interface; detecting an event corresponding to a movement of the first user interface; and in response to detecting the event, moving the first user interface from the first positioning in the three-dimensional environment to a second positioning in the three- dimensional environment, wherein moving the first user interface includes: in accordance with a determination that a distance between the first positioning and the second positioning is less than a threshold distance, displaying the first user interface with a second simulated lighting effect at a second location on the first user interface, wherein the second location is different from the first location on the first user interface, and wherein the second location is within the first sub-region of the first user interface; and in accordance with a determination that the distance between the first positioning and the second positioning is at least the threshold distance, displaying the first user interface with a third simulated lighting effect at a third location on the first user interface, wherein: the third location on the first user interface is different from the first location on the first user interface and the second location on the first user interface; the third position is within a second sub-region of the first user interface; the second sub-region of the first user interface is different from the first sub-region of the first user interface; and the first user interface includes a third sub-region between the first sub-region and the second sub-region of the first user interface, wherein the third sub-region is a region in which simulated lighting effects are not displayed when the first user interface is stationary relative to the three-dimensional environment.

28. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, via the display generation component, a first view of a three- dimensional environment, the first view of the three-dimensional environment including a first user interface at a first position in the three-dimensional environment, wherein the first user interface displays a first simulated lighting effect at a first position on the first user interface, wherein the first position is a position within a first sub-region of the first user interface; detecting an event corresponding to movement of the first user interface; and in response to detecting the event, moving the first user interface from the first position in the three-dimensional environment to a second position in the three- dimensional environment, wherein moving the first user interface includes: in accordance with a determination that a distance between the first position and the second position is less than a threshold distance, displaying the first user interface with a second simulated lighting effect at a second position on the first user interface, wherein the second position is different from the first position on the first user interface, and wherein the second position is within the first sub-region of the first user interface; and in accordance with a determination that the distance between the first position and the second position is at least the threshold distance, displaying the first user interface with a third simulated lighting effect at a third position on the first user interface, wherein: the third position on the first user interface is different from the first position on the first user interface and the second position on the first user interface; the third position is within a second sub-region of the first user interface; the second sub-region of the first user interface is different from the first sub-region of the first user interface; and the first user interface includes a third sub-region between the first sub-region and the second sub-region of the first user interface, wherein the third sub-region is a region in which simulated lighting effects are not displayed when the first user interface is stationary relative to the three-dimensional environment.

29. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: means for displaying, via the display generation component, a first view of a three-dimensional environment, the first view of the three-dimensional environment including a first user interface at a first position in the three-dimensional environment, wherein the first user interface displays a first simulated lighting effect at a first location on the first user interface, wherein the first location is a location within a first sub-region of the first user interface; means for detecting an event corresponding to movement of the first user interface; and means enabled in response to detecting the event for moving the first user interface from the first position in the three-dimensional environment to a second position in the three-dimensional environment, wherein moving the first user interface includes: in accordance with a determination that a distance between the first position and the second position is less than a threshold distance, displaying the first user interface with a second simulated lighting effect at a second location on the first user interface, wherein the second location is different from the first location on the first user interface, and wherein the second location is within the first sub-region of the first user interface; and in accordance with a determination that the distance between the first position and the second position is at least the threshold distance, displaying the first user interface with a third simulated lighting effect at a third location on the first user interface, wherein: the third location on the first user interface is different from the first location on the first user interface and the second location on the first user interface; the third location is within a second sub-region of the first user interface; the second sub-region of the first user interface is different from the first sub-region of the first user interface; and the first user interface includes a third sub-region between the first sub-region and the second sub-region of the first user interface, wherein the third sub-region is a region that does not display a simulated lighting effect when the first user interface is stationary relative to the three-dimensional environment.

30. A method, the method comprising: at a computer system in communication with a display generation component and one or more sensors: displaying, via the display generation component, a first user interface in a first view of a three-dimensional environment, wherein the first user interface has a simulated three-dimensional effect when the first user interface is visible in the first view of the three-dimensional environment, and the first user interface has a first spatial relationship to a viewpoint of a user; detecting an input corresponding to a request to change the viewpoint of the user to have a second spatial relationship to the first user interface, wherein the second spatial relationship is different from the first spatial relationship, and a first portion of the input corresponds to a respective amount of change to the viewpoint, and a second portion of the input after the first portion corresponds to the respective amount of change to the viewpoint; in response to detecting the first portion of the input: displaying a second view of the three-dimensional environment different from the first view of the three-dimensional environment; and changing the simulated three-dimensional effect by a first amount; and in response to detecting the second portion of the input: displaying a third view of the three-dimensional environment different from the first view of the three-dimensional environment and the second view of the three-dimensional environment; and changing the simulated three-dimensional effect by a second amount, wherein the second amount is different from the first amount.

31. The method of claim 30, wherein: the respective amount of change in the viewpoint includes a respective amount of movement in a first direction and the respective amount of movement in a second direction perpendicular to the first direction; and changing the simulated three-dimensional effect by a respective amount includes changing the simulated three-dimensional effect in the first direction by a greater amount than the amount of change in the simulated three-dimensional effect in the second direction.

32. The method of claim 31, wherein the amount of change in the simulated three-dimensional effect in the second direction is zero.

33. The method of any of claims 30-32, wherein displaying the simulated three-dimensional effect of the first user interface includes displaying a simulated shadow associated with the first user interface.

34. The method of any of claims 30-33, wherein displaying the simulated three-dimensional effect of the first user interface includes displaying a simulated lighting effect associated with the first user interface.

35. The method of any of claims 30-34, wherein: displaying the simulated three-dimensional effect of the first user interface includes displaying a first simulated visual effect and displaying a second simulated visual effect; changing the simulated three-dimensional effect by the first amount includes changing the first simulated visual effect based on a first amount of progression along a respective animation curve and changing the second simulated visual effect based on the first amount of progression along the respective animation curve; and changing the simulated three-dimensional effect by the second amount includes changing the first simulated visual effect based on a second amount of progression along the respective animation curve and changing the second simulated visual effect based on the second amount of progression along the respective animation curve.

36. The method of any of claims 30-35, wherein the first user interface includes a first activatable control and the simulated three-dimensional effect is a first simulated three-dimensional effect corresponding to the first activatable control.

37. The method of claim 36, wherein the first user interface includes a second activatable control that is displayed with a second simulated three-dimensional effect corresponding to the second activatable control when the first view of the three-dimensional environment is visible, and the method includes: in response to detecting the first portion of the input: changing the second simulated three-dimensional effect by a third amount; and in response to detecting the second portion of the input: changing the second simulated three-dimensional effect by a fourth amount, wherein the fourth amount is different from the third amount.

38. The method of any of claims 36-37, wherein: changing the first simulated three-dimensional effect corresponding to the first activatable control in a first manner changes the first simulated three-dimensional effect; and the method includes: displaying a respective simulated three-dimensional effect corresponding to the first user interface while the first view of the three-dimensional environment is visible, wherein the respective simulated three-dimensional effect is different from the first simulated three-dimensional effect corresponding to the first activatable control; in response to detecting the first portion of the input: changing the respective simulated three-dimensional effect corresponding to the first user interface in a second manner different from the first manner; and in response to detecting the second portion of the input: changing the respective simulated three-dimensional effect corresponding to the first user interface in the second manner.

39. The method of claim 38, the method including: while displaying the first user interface having a respective spatial relationship to the viewpoint of the user at a first location in the three-dimensional environment, detecting one or more inputs; and in response to detecting the one or more inputs: in accordance with a determination that the one or more inputs include input corresponding to a request to change the viewpoint of the user to have a changed spatial relationship to the first user interface that is different from the respective spatial relationship: displaying the first user interface in accordance with the changed spatial relationship to the viewpoint of the user; changing the first simulated three-dimensional effect corresponding to the first activatable control in accordance with the changed spatial relationship to the viewpoint of the user; changing the respective simulated three-dimensional effect corresponding to the first user interface in accordance with the changed spatial relationship to the viewpoint of the user; and in accordance with a determination that the one or more inputs include input corresponding to a request to reposition the first user interface to a second location in the three-dimensional environment: displaying the first user interface at the second location in the three-dimensional environment; and changing the respective simulated three-dimensional effect corresponding to the first user interface in accordance with repositioning the first user interface, but not changing the first simulated three-dimensional effect corresponding to the first activatable control in accordance with the repositioning of the first user interface.

40. The method of any of claims 38-39, wherein: changing the first simulated three-dimensional effect corresponding to the first activatable control in the first manner includes changing the first simulated three-dimensional effect in accordance with movement of the viewpoint of the user in a first direction relative to the first user interface without consideration of movement of the viewpoint of the user in a second direction relative to the first user interface, wherein the second direction is different from the first direction; and changing the respective simulated three-dimensional effect in the second manner includes changing the respective simulated three-dimensional effect according to movement of the user’s viewpoint relative to the first user interface in the first direction and according to movement of the user’s viewpoint relative to the first user interface in the second direction.

41. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more sensors, the one or more programs including instructions for performing the method of any of claims 30-40.

42. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any of claims 30-40.

43. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: means for performing the method of any of claims 30-40.

44. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more sensors, the one or more programs including instructions for: displaying, via the display generation component, a first user interface in a first view of a three-dimensional environment, wherein the first user interface has a simulated three-dimensional effect when the first user interface is visible in the first view of the three-dimensional environment, and the first user interface has a first spatial relationship to a viewpoint of a user; detecting an input corresponding to a request to change the viewpoint of the user to have a second spatial relationship to the first user interface, wherein the second spatial relationship is different from the first spatial relationship, and a first portion of the input corresponds to a respective amount of change to the viewpoint, and a second portion of the input after the first portion corresponds to the respective amount of change to the viewpoint; in response to detecting the first portion of the input: displaying a second view of the three-dimensional environment different from the first view of the three-dimensional environment; and changing the simulated three-dimensional effect by a first amount; and in response to detecting the second portion of the input: displaying a third view of the three-dimensional environment different from the first view of the three-dimensional environment and the second view of the three-dimensional environment; and changing the simulated three-dimensional effect by a second amount, wherein the second amount is different from the first amount.

45. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, via the display generation component, a first user interface in a first view of a three-dimensional environment, wherein the first user interface has a simulated three-dimensional effect when visible in the first view of the three-dimensional environment, and the first user interface has a first spatial relationship to a viewpoint of a user; detecting an input corresponding to a request to change the viewpoint of the user to have a second spatial relationship to the first user interface, wherein the second spatial relationship is different from the first spatial relationship, and a first portion of the input corresponds to a respective amount of change of the viewpoint, and a second portion after the first portion corresponds to the respective amount of change of the viewpoint; in response to detecting the first portion of the input: displaying a second view of the three-dimensional environment different from the first view of the three-dimensional environment; and changing the simulated three-dimensional effect by a first amount; and in response to detecting the second portion of the input: displaying a third view of the three-dimensional environment different from the first view of the three-dimensional environment and the second view of the three-dimensional environment; and changing the simulated three-dimensional effect by a second amount, wherein the second amount is different from the first amount.

46. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: means for displaying, via the display generation component, a first user interface in a first view of a three-dimensional environment, wherein the first user interface has a simulated three-dimensional effect when visible in the first view of the three-dimensional environment, and the first user interface has a first spatial relationship to a viewpoint of a user; means for detecting an input corresponding to a request to change the viewpoint of the user to have a second spatial relationship to the first user interface, wherein the second spatial relationship is different from the first spatial relationship, and a first portion of the input corresponds to a respective amount of change of the viewpoint, and a second portion after the first portion corresponds to the respective amount of change of the viewpoint; means enabled in response to detecting the first portion of the input, comprising: means for displaying a second view of the three-dimensional environment different from the first view of the three-dimensional environment; and means for changing the simulated three-dimensional effect by a first amount; and means enabled in response to detecting the second portion of the input, comprising: means for displaying a third view of the three-dimensional environment different from the first view of the three-dimensional environment and the second view of the three-dimensional environment; and means for changing the simulated three-dimensional effect by a second amount, wherein the second amount is different from the first amount.

47. A method, the method comprising: at a computer system in communication with a display generation component and one or more sensors: displaying a first user interface object at a first orientation and at a first object location in the three-dimensional environment, and displaying a first simulated shadow corresponding to the first user interface object at a first shadow location in the three-dimensional environment, while the first view of the three-dimensional environment is visible via the display generation component, wherein the first simulated shadow at the first shadow location has a first spatial relationship with the first user interface object; detecting a user input directed to the first user interface object; and in response to detecting the user input, changing an orientation of the first user interface object from the first orientation to a second orientation, including: displaying the first user interface object at the second orientation; and displaying the first simulated shadow at a second shadow location in the three-dimensional environment, wherein: the second shadow location in the first view of the three-dimensional environment is different from the first shadow location in the first view of the three-dimensional environment; the first simulated shadow at the second shadow location has a second spatial relationship with the first user interface object; and the second spatial relationship is different from the first spatial relationship.

48. The method of claim 47, including displaying the first simulated shadow at the second shadow location in the three-dimensional environment without changing a size of the first simulated shadow.

49. The method of any of claims 47-48, including: in response to detecting the user input: displaying the first user interface object at a second object location in the first view of the three-dimensional environment; wherein: an extent of change between the first object location and the second object location of the first user interface object is different from an extent of change between the first shadow location and the second shadow location of the first simulated shadow.

50. The method of any of claims 47-49, wherein displaying the first simulated shadow at a second shadow location in the three-dimensional environment includes: when the change in the orientation of the first user interface object from the first orientation to the second orientation includes a rotation of the first user interface object in a first direction, repositioning the first simulated shadow in a third direction corresponding to the first direction; and when the change in the orientation of the first user interface object from the first orientation to the second orientation includes a rotation of the first user interface object in a second direction different from the first direction, repositioning the first simulated shadow in a fourth direction corresponding to the second direction, wherein the fourth direction is different from the third direction.

51. The method of any of claims 47-50, wherein displaying the first simulated shadow at a second shadow location in the three-dimensional environment includes: ​ a first amount of change in spatial relationship between the first simulated shadow and the first user interface object when the change in the orientation of the first user interface object from the first orientation to the second orientation includes a first amount of rotation of the first user interface object; and a second amount of change in spatial relationship between the first simulated shadow and the first user interface object when the change in the orientation of the first user interface object from the first orientation to the second orientation includes a second amount of rotation of the first user interface object, wherein the second amount of rotation is different from the first amount of rotation, and the second amount of change in spatial relationship is different from the first amount of change in spatial relationship.

52. The method of claim 51, wherein changing the orientation of the first user interface object from the first orientation to the second orientation includes transitioning the first user interface object through a plurality of intermediate orientations between the first orientation and the second orientation, and the method includes: moving the first simulated shadow through a plurality of intermediate shadow positions between the first shadow position associated with the first spatial relationship and the second shadow position associated with the second spatial relationship as the first user interface object is transitioned through the plurality of intermediate orientations between the first orientation and the second orientation.

53. The method of any of claims 47-52, the method including: in response to detecting the user input: moving the first user interface object from the first object position to a second object position different from the first object position.

54. The method of claim 53, wherein changing the orientation of the first user interface object from the first orientation to the second orientation is performed automatically in accordance with the movement of the first user interface object from the first object position to the second object position.

55. The method of any of claims 47-54, the method including: displaying the first user interface object in the first orientation in accordance with displaying the first user interface object at the first object position at a first distance from a viewpoint of a user; and changing the orientation of the first user interface object from the first orientation to the second orientation in accordance with displaying the first user interface object at a second distance from the viewpoint of the user, wherein the second distance is different from the first distance.

56. The method of any of claims 47-55, wherein: the first simulated shadow is displayed at a respective position relative to the first user interface object in the first orientation when the first simulated shadow has the first spatial relationship with the first user interface object; and the first simulated shadow is displayed in front of the respective position relative to the first user interface object when the first simulated shadow has the second spatial relationship with the first user interface object in the second orientation.

57. The method of any of claims 47-56, wherein the first simulated shadow displayed at the first shadow location has a respective thickness, and the first simulated shadow displayed at the second shadow location has the respective thickness.

58. The method of any one of claims 47-57, comprising: in response to detecting the user input, changing an appearance of the first simulated shadow while maintaining display of the first user interface object at the first orientation and at the first object location.

59. The method of claim 58, the method comprising: detecting an end of the user input; and in response to detecting the end of the user input, at least partially reversing the change in the appearance of the first simulated shadow while maintaining display of the first user interface object at a respective orientation and at a respective object location.

60. The method of any of claims 47-59, wherein the user input directed to the first user interface object comprises an air gesture.

61. The method of any of claims 47-60, wherein: displaying the first simulated shadow at the first shadow location prior to detecting the user input directed to the first user interface object comprises: displaying the first simulated shadow with a first appearance when a user’s attention is not directed to the first user interface object; and displaying the first simulated shadow with a second appearance different from the first appearance when the user’s attention is directed to the first user interface object.

62. The method of any of claims 47-61, the method comprising: displaying the first simulated shadow at the first shadow location prior to detecting the user input directed to the first user interface object comprises displaying the first simulated shadow with a respective appearance; and upon detecting the user input directed to the first user interface object, changing one or more visual properties of the respective appearance of the first simulated shadow, including displaying the first simulated shadow with the one or more visual properties of the respective appearance changed as the first simulated shadow moves from the first shadow location to the second shadow location.

63. The method of any of claims 47-62, the method comprising: detecting an input corresponding to a request to move the first user interface object relative to the three-dimensional environment; and in response to detecting the input corresponding to the request to move the first user interface object relative to the three-dimensional environment: moving the first user interface object relative to the three-dimensional environment; and visually deemphasizing the first simulated shadow corresponding to the first user interface object as a distance between the first user interface object and a respective plane in the three-dimensional environment changes, wherein visually deemphasizing the first simulated shadow comprises changing one or more visual properties of the first simulated shadow.

64. The method of claim 63, wherein visually deemphasizing the first simulated shadow includes, in accordance with a determination that the distance between the first user interface object and the respective plane is greater than a threshold distance, ceasing to display the first simulated shadow.

65. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more sensors, the one or more programs including instructions for performing the method of any of claims 47-64.

66. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any of claims 47-64.

67. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: means for performing the method of any of claims 47-64.

68. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more sensors, the one or more programs including instructions for: displaying, while a first view of a three-dimensional environment is visible via the display generation component, a first user interface object at a first orientation and at a first object location in the three-dimensional environment, and a first simulated shadow corresponding to the first user interface object at a first shadow location in the three-dimensional environment, wherein the first simulated shadow at the first shadow location has a first spatial relationship with the first user interface object; detecting a user input directed to the first user interface object; and in response to detecting the user input, changing an orientation of the first user interface object from the first orientation to a second orientation, including: displaying the first user interface object at the second orientation; and displaying the first simulated shadow at a second shadow location in the three-dimensional environment, wherein: the second shadow location in the first view of the three-dimensional environment is different from the first shadow location in the first view of the three-dimensional environment; the first simulated shadow at the second shadow location has a second spatial relationship with the first user interface object; and the second spatial relationship is different from the first spatial relationship.

69. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: ​ while a first view of a three-dimensional environment is visible via the display generation component, displaying a first user interface object at a first orientation and at a first object location in the three-dimensional environment, and displaying a first simulated shadow corresponding to the first user interface object at a first shadow location in the three-dimensional environment, wherein the first simulated shadow at the first shadow location has a first spatial relationship with the first user interface object; detecting a user input directed to the first user interface object; and in response to detecting the user input, changing an orientation of the first user interface object from the first orientation to a second orientation, including: displaying the first user interface object at the second orientation; and displaying the first simulated shadow at a second shadow location in the three-dimensional environment, wherein: the second shadow location in the first view of the three-dimensional environment is different from the first shadow location in the first view of the three-dimensional environment; the first simulated shadow at the second shadow location has a second spatial relationship with the first user interface object; and the second spatial relationship is different from the first spatial relationship.

70. A computer system in communication with a display generation component and one or more sensors, the computer system comprising: means enabled while a first view of a three-dimensional environment is visible via the display generation component to display a first user interface object at a first orientation and at a first object location in the three-dimensional environment, and to display a first simulated shadow corresponding to the first user interface object at a first shadow location in the three-dimensional environment, wherein the first simulated shadow at the first shadow location has a first spatial relationship with the first user interface object; means for detecting a user input directed to the first user interface object; and means enabled in response to detecting the user input to change an orientation of the first user interface object from the first orientation to a second orientation, including: displaying the first user interface object at the second orientation; and displaying the first simulated shadow at a second shadow location in the three-dimensional environment, wherein: the second shadow location in the first view of the three-dimensional environment is different from the first shadow location in the first view of the three-dimensional environment; the first simulated shadow at the second shadow location has a second spatial relationship with the first user interface object; and the second spatial relationship is different from the first spatial relationship.

Images