Devices, methods, and graphical user interfaces for displaying shadows and light effects in three-dimensional environment

Abstract

A computer system displays a computer-generated object in a three-dimensional environment; detecting that the attention of the user points to the object; and in response, displaying a virtual shadow of the object at a first appearance, including displaying the shadow at a first value for a first visual attribute, while maintaining a pose of the object relative to the three-dimensional environment. While continuing to display the object in the three-dimensional environment, the computer system detects that the attention of the user has stopped pointing to the object; and in response, displaying the shadow of the object in a second appearance different from the first appearance while maintaining the pose of the object relative to the three-dimensional environment. Displaying the shadow of the object in the second appearance includes displaying the shadow in a second value for the first visual attribute. The second value is different from the first value.

Description

[0001] Related patent applications

[0002] This application is a continuation of U.S. patent application serial number 17 / 950,795 filed on September 22, 2022, which claims priority to U.S. provisional application serial number 63 / 408,810 filed on September 21, 2022, each of which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to computer systems that provide computer-generated experiences in communication with display generation components and one or more input devices, including but not limited to electronic devices that provide virtual reality experiences and mixed reality experiences via displays. Background Art

[0004] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays) for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Example virtual elements include virtual objects such as digital images, videos, text, icons, and control elements (such as buttons and other graphics). Summary of the invention

[0005] Some methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) including at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient or overly complex feedback about where objects are located in a three-dimensional environment and about the spatial relationship between objects, systems that provide insufficient feedback about which (which) objects have a focus for interaction or are currently the target of interaction or about which type of interaction is occurring, and systems in which the complexity of visual effects and the corresponding computational burden caused are inconsistent with the type of environment being displayed are complex, tedious, and error-prone, causing significant cognitive burdens to users, and detracting from the experience of virtual / augmented reality environments. In addition, these methods take longer than necessary, thereby wasting the energy of computer systems. This latter consideration is particularly important in battery-driven devices.

[0006] Therefore, there is a need for a computer system with improved methods and interfaces for communicating spatial relationships between objects in a three-dimensional environment and indicating the state of a user's interaction with the objects, so that interaction with the computer system is more efficient and intuitive for the user. Such methods and interfaces optionally supplement or replace conventional methods for providing an extended reality experience to a user. Such methods and interfaces reduce the amount, degree, and / or nature of inputs from a user by helping the user understand the connection between the inputs provided and the device's response to those inputs, thereby forming a more effective human-computer interface.

[0007] The above-mentioned defects and other problems associated with the user interface of the computer system are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or a head-mounted device). In some embodiments, the computer system has a touch pad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, in addition to the display generation component, the computer system also has one or more output devices, which include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI through contact and gestures of a stylus and / or finger on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body (as captured by a camera and other mobile sensors), and / or voice input (as captured by one or more audio input devices). In some embodiments, the functions performed by interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, testing support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a transient and / or non-transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0008] There is a need for electronic devices with improved methods and interfaces for communicating spatial relationships between objects in a three-dimensional environment and indicating the status of a user's interaction with the objects. Such methods and interfaces can supplement or replace conventional methods for communicating spatial relationships between objects in a three-dimensional environment and indicating the status of a user's interaction with the objects. Such methods and interfaces reduce the amount, degree, and / or nature of input from a user and produce a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges.

[0009] According to some embodiments, a method is performed at a computer system in communication with a display generation component and one or more input devices. The method includes displaying a first computer-generated user interface object in a three-dimensional environment when a view of the three-dimensional environment is visible via the display generation component. The method includes detecting that a user's attention is directed to the first user interface object. The method includes, in response to detecting that the user's attention is directed to the first user interface object, displaying a first virtual shadow of the first user interface object in a first appearance while maintaining a posture of the first user interface object relative to the three-dimensional environment. Displaying the first virtual shadow of the first user interface object in the first appearance includes displaying the first virtual shadow at a first value for a first visual attribute. The method includes, while continuing to display the first user interface object in the three-dimensional environment, detecting that the user's attention has stopped being directed to the first user interface object. The method includes, in response to detecting that the user's attention has stopped being directed to the first user interface object, displaying the first virtual shadow of the first user interface object in a second appearance different from the first appearance while maintaining the posture of the first user interface object relative to the three-dimensional environment. Displaying the first virtual shadow of the first user interface object in the second appearance includes displaying the first virtual shadow at a second value for the first visual attribute. The second value is different from the first value.

[0010] According to some embodiments, a method is performed at a computer system that communicates with a display generation component and one or more input devices. The method includes displaying a first user interface object at a first position in the environment and a virtual shadow corresponding to the first user interface object in the environment when a view of the environment is visible via the display generation component, including displaying the virtual shadow with a first value for a first visual attribute. The method includes detecting an input corresponding to a request to reposition the first user interface object from the first position in the environment to a second position in the environment that is different from the first position; and in response to detecting the input: repositioning the first user interface object from the first position in the environment to the second position in the environment according to movement of the input relative to the environment; and when the first user interface object is repositioned from the first position in the environment to the second position in the environment, displaying the virtual shadow corresponding to the first user interface object with a second value for the first visual attribute.

[0011] According to some embodiments, a method is performed at a computer system in communication with a display generation component and one or more input devices. The method includes displaying in the environment, when a view of the environment is visible via the display generation component: a group of user interface objects separated from each other in the environment, the group including the first user interface object and the second user interface object according to a corresponding spatial relationship between the first user interface object at a first position in the environment and the second user interface object at a second position in the environment; and a virtual shadow corresponding to the group of user interface objects. Displaying the virtual shadow corresponding to the group of user interface objects includes displaying a first virtual shadow portion corresponding to the first user interface object at the first position in the environment, without displaying a virtual shadow portion corresponding to the second user interface object in the environment. The method includes detecting a first input corresponding to a request to reposition the group of user interface objects in the environment; and in response to detecting the first input: repositioning the group of user interface objects in the environment, including repositioning the first user interface object from the first position to a third position in the environment and repositioning the second user interface object from the second position to a fourth position in the environment, while maintaining the corresponding spatial relationship between the first user interface object and the second user interface object; and repositioning the virtual shadow corresponding to the group of user interface objects, including displaying the first virtual shadow portion corresponding to the first user interface object at the third position in the environment without displaying the virtual shadow portion corresponding to the second user interface object in the environment.

[0012] According to some embodiments, a method is performed at a computer system in communication with a display generation component and one or more input devices. The method includes detecting an input corresponding to a request to display content in the environment when a view of the environment is visible via the display generation component. The method includes, in response to detecting the input: displaying the content in a user interface object in the environment; and displaying a visual effect corresponding to simulated light emitted from the content in one or more portions of the environment that are located outside the user interface object. Displaying the visual effect includes: displaying the visual effect with a first degree of fidelity to the content based on determining that the environment is a first type of environment; and displaying the visual effect with a second degree of fidelity to the content based on determining that the environment is a second type of environment that is different from the first type of environment. The second degree of fidelity is different from the first degree of fidelity.

[0013] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not comprehensive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art based on the drawings, the specification, and the claims. In addition, it should be noted that the language used in this specification is selected in principle for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0015] Figure 1 is a block diagram illustrating an operating environment of a computer system for providing an extended reality (XR) experience according to some embodiments.

[0016] Figure 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience according to some embodiments.

[0017] Figure 3 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of an XR experience to a user according to some embodiments.

[0018] Figure 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture input from a user according to some embodiments.

[0019] Figure 5 is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input of a user according to some embodiments.

[0020] Figure 6 is a flow chart illustrating a flash-assisted gaze tracking pipeline according to some embodiments.

[0021] 7A to 7H Example techniques for displaying attention-based shadows of user interface objects in an environment according to some embodiments are illustrated.

[0022] FIG. 8A to FIG. 8J Example techniques for displaying motion-based shadows of user interface objects in an environment according to some embodiments are illustrated.

[0023] 9A to 9E Example techniques for displaying simplified shadows of user interface objects in an environment according to some embodiments are illustrated.

[0024] FIG. 10A to FIG. 10E Example techniques for simulating light emanating from user interface objects displayed in an environment are illustrated according to some embodiments.

[0025] Fig.11 is a flow chart of a method for displaying attention-based shadows of user interface objects in an environment according to various embodiments.

[0026] Fig.12 is a flow chart of a method for displaying motion-based shadows of user interface objects in an environment according to various embodiments.

[0027] Fig.13 is a flow chart of a method of displaying simplified shadows of user interface objects in an environment according to various embodiments.

[0028] Fig.14 is a flow chart of a method of simulating light emanating from a user interface object displayed in an environment, according to various embodiments. DETAILED DESCRIPTION

[0029] According to some embodiments, the present disclosure relates to a user interface for providing an extended reality (XR) experience to a user.

[0030] The systems, methods, and GUIs described herein improve user interface interactions with virtual / augmented reality environments in a number of ways.

[0031] In some embodiments, a computer system displays a virtual shadow of a computer-generated user interface object in a three-dimensional environment (e.g., a virtual or mixed reality environment). When it is detected that the user is not directing his attention to the object, the computer system displays the shadow of the object in a first appearance (such as with a first opacity). In response to detecting that the user is directing his attention to the object and when it is detected that the user is directing his attention to the object, the computer system changes the appearance of the shadow of the object to a different second appearance, such as with a different second opacity. Displaying the virtual shadow of the computer-generated user interface object helps the user understand where the object is located in the three-dimensional environment, and changing the appearance of the corresponding shadow helps the user understand which object has focus for interaction.

[0032] In some embodiments, a computer system displays a virtual shadow of a computer-generated user interface object in a three-dimensional environment (e.g., a virtual or mixed reality environment). When input for moving an object in the three-dimensional environment is not detected (e.g., before or after detecting such input), the computer system displays the shadow of the object in a first appearance (such as with a first opacity). When input for moving an object in the three-dimensional environment is detected, the computer system displays the shadow of the object in a different second appearance (such as with a different second opacity) even before the object is actually moved. Displaying the virtual shadow of a computer-generated user interface object helps the user understand where the object is located in the three-dimensional environment, and changing the appearance of the corresponding shadow during input for moving the object helps the user understand which object is the current interaction target and is in the process of being moved.

[0033] In some embodiments, a computer system displays a virtual shadow of a computer-generated user interface object, which is a set of multiple component objects that are spatially separated from each other in a three-dimensional environment (e.g., a virtual or mixed reality environment). The virtual shadow of the set of objects is a simplified shadow that includes the shadow of at least one of the component objects and does not include the shadow of at least one other of the component objects. When the set of component objects moves in the three-dimensional environment, the spatial relationship between the component objects is maintained, and the virtual shadow follows the set of component objects and continues to include the shadow of at least one of the component objects without including the shadow of at least one other of the component objects. Displaying simplified virtual shadows of computer-generated user interface objects helps users understand where objects are located in the three-dimensional environment while reducing visual clutter and reducing the computational load imposed on the computer system by the task of generating shadows.

[0034] In some embodiments, when content in a three-dimensional environment (such as content displayed in a particular user interface object) begins to play, the computer system stops displaying a virtual shadow of a computer-generated user interface object in a three-dimensional environment (e.g., a virtual or mixed reality environment). While the content is playing, the computer system displays a light overflow visual effect that simulates light emanating from the content into a surrounding portion of the three-dimensional environment. The light overflow visual effect has different degrees of fidelity to the content based on the type of surrounding environment, which provides a more immersive or less immersive experience based on the type of surrounding environment. For example, the light overflow visual effect optionally has a higher degree of fidelity to the content played in a computer-generated virtual environment (e.g., by including the display of light projected from the content onto other objects in the environment, and thus providing a more immersive experience) than the content played in a mixed reality environment (e.g., by only displaying a glow extending from the content, and thus providing a less immersive experience). Displaying the light overflow visual effect from the content displayed in the object in the three-dimensional environment with different degrees of fidelity to the content based on the type of surrounding environment helps the user understand where the content and objects are located in the three-dimensional environment, while achieving different levels of immersion and different corresponding computational burdens on the computer system.

[0035] Figures 1 to 6 A description of an example computer system for providing an XR experience to a user is provided. 7A to 7H Example techniques for displaying attention-based shadows of user interface objects in an environment according to some embodiments are illustrated. FIG. 8A to FIG. 8J Example techniques for displaying motion-based shadows of user interface objects in an environment according to some embodiments are illustrated. 9A to 9E Example techniques for displaying simplified shadows of user interface objects in an environment according to some embodiments are illustrated. FIG. 10A to FIG. 10E Example techniques for simulating light emanating from user interface objects displayed in an environment are illustrated according to some embodiments. Fig.11 is a flow chart of a method for displaying attention-based shadows of user interface objects in an environment according to various embodiments.

[0036] Fig.12 is a flow chart of a method for displaying motion-based shadows of user interface objects in an environment according to various embodiments. Fig.13 is a flow chart of a method of displaying simplified shadows of user interface objects in an environment according to various embodiments. Fig.14 is a flow chart of a method of simulating light emanating from a user interface object displayed in an environment, according to various embodiments. 7A to 7H , FIG. 8A to FIG. 8J , 9A to 9E and FIG. 10A to FIG. 10E The user interface in Fig.11 , Fig.12 , Fig.13 and Fig.14 process.

[0037] The processes described below enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various techniques, including by providing improved visual, audio, and / or tactile feedback to the user, improving user safety and reducing motion sickness typically associated with XR experiences, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation without further user input when a set of conditions have been met, reducing the computational burden associated with displaying the user interface, improving privacy and / or security, providing a more diverse, detailed, and / or realistic user experience while saving storage space, and / or additional techniques. These techniques also reduce power usage and extend the battery life of the device by enabling the user to use the device faster and more efficiently. Saving battery power, and therefore weight, improves the ergonomics of the device. These techniques also enable real-time communication, allow the use of fewer and / or less precise sensors, resulting in a more compact, lighter, and cheaper device, and enable the device to be used in a variety of lighting conditions. These techniques reduce energy usage and thereby reduce the heat emitted by the device, which is particularly important for wearable devices where if the device generates too much heat well within the operating parameters of the device components, it may become uncomfortable for the user to wear the device.

[0038] In addition, in the method described herein where one or more steps depend on one or more conditions being met, it should be understood that the described method can be repeated in multiple repetitions so that in the process of repetition, all conditions for determining the steps in the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the ordinary technician will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on one or more conditions being met can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether the possible situation has been met without explicitly repeating the steps of the method until all conditions for determining the steps in the method have been met. It will also be understood by ordinary technicians in the art that, similar to the method with the contingent step, the system or computer-readable storage medium can repeat the steps of the method as many times as needed to ensure that all the contingent steps have been performed.

[0039] In some embodiments, such as Figure 1 As shown, an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a household appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head mounted device or a handheld device).

[0040] In describing an XR experience, various terms are used to distinguishably refer to several related but distinct environments that a user can sense and / or with which the user can interact (e.g., using inputs detected by the computer system 101 generating the XR experience, which inputs cause the computer system generating the XR experience to generate audio, visual, and / or tactile feedback corresponding to the various inputs provided to the computer system 101). The following is a subset of these terms:

[0041] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0042] Extended Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In XR, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. For example, an XR system may detect a person's head turn, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to the characteristics of virtual objects in the XR environment may be made in response to a representation of physical movement (e.g., a voice command). People can sense and / or interact with XR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, people can sense and / or interact with audio objects, which create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some XR environments, a person can sense and / or interact only with an audio object.

[0043] Examples of XR include virtual reality and mixed reality.

[0044] Virtual Reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0045] Mixed Reality: In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between, but not including, a fully physical environment at one end and a virtual reality environment at the other end. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render MR environments can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment, or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0046] Examples of mixed reality include augmented reality and augmented virtuality.

[0047] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. A person uses the system to indirectly view the physical environment via an image or video of the physical environment and perceives virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, such as as a hologram or on a physical surface, so that a person using the system perceives virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not real version of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating a portion thereof or blurring a portion thereof.

[0048] Augmented Virtual: An augmented virtual (AV) environment refers to a simulated environment in which a virtual environment or a computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0049] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same position and / or location in a user's viewpoint, the virtual object is viewpoint-locked even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the forward direction of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); thus, without moving the user's head, the user's viewpoint remains fixed even when the user's gaze shifts. In embodiments where the computer system has a display generation component (e.g., a display screen) that is repositionable relative to the user's head, the user's viewpoint is an augmented reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object that is displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or location of the viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object."

[0050] Environment-locked visual objects: A virtual object is environment-locked (alternatively, "world-locked") when a computer system displays a virtual object at a position and / or location in a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) a location and / or object in a three-dimensional environment (e.g., a physical environment or a virtual environment). As the user's viewpoint moves, the location and / or objects in the environment change relative to the user's viewpoint, which causes the environment-locked virtual object to be displayed at a different location and / or location in the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) such that the tree is now to the left of center in the user's viewpoint (e.g., the tree location in the user's viewpoint shifts), the environment-locked virtual object locked to the tree is displayed to the left of center in the user's viewpoint. In other words, the position and / or orientation of the virtual object that is locked to the environment in the user's viewpoint depends on the position and / or orientation of the object in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference system (e.g., a coordinate system anchored to a fixed position and / or object in the physical environment) to determine the position of the virtual object that is locked to the user's viewpoint. The virtual object that is locked to the environment can be locked to a stationary part of the environment (e.g., a floor, wall, table or other stationary object), or can be locked to a movable part of the environment (e.g., a vehicle, animal, person or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm or foot of the user) so that the virtual object moves as the viewpoint or the part of the environment moves to maintain a fixed relationship between the virtual object and the part of the environment.

[0051] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits an inertial following behavior that reduces or delays the movement of the environment-locked or viewpoint-locked virtual object relative to the movement of the reference point followed by the virtual object. In some embodiments, when exhibiting inertial following behavior, when detecting the movement of a reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5cm and 300cm from the viewpoint) that the virtual object is following, the computer system intentionally delays the movement of the virtual object. For example, when the reference point (e.g., the portion of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits inertial following behavior, the device ignores small amounts of movement of the reference point (e.g., ignoring movements of the reference point below a threshold movement amount, such as moving 0 degrees to 5 degrees or moving 0cm to 50cm). For example, when a reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and when the reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., a “lazy follow” threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some embodiments, maintaining a substantially fixed position of the virtual object relative to the reference point includes displaying the virtual object within a threshold distance (e.g., 1 cm, 2 cm, 3 cm, 5 cm, 15 cm, 20 cm, 50 cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the position of the reference point).

[0052] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet devices, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection technology that projects graphic images onto a person's retina. The projection system may also be configured to project virtual objects into a physical environment, such as as a hologram or on a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. The following description with respect to Figure 2Controller 110 is described in more detail. In some embodiments, controller 110 is a computing device that is located locally or remotely relative to scene 105 (e.g., physical environment). For example, controller 110 is a local server located within scene 105. As another example, controller 110 is a remote server (e.g., cloud server, central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., HMD, display, projector, touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). In another example, controller 110 is included within a housing (e.g., a physical housing) of display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors, etc.), one or more input devices of input devices 125, one or more output devices of output devices 155, one or more sensors of sensors 190, and / or one or more peripheral devices of peripheral devices 195, or shares the same physical housing or support structure with one or more of the above devices.

[0053] In some embodiments, the display generation component 120 is configured to provide an XR experience (e.g., at least the visual component of the XR experience) to the user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Display generation component 120 is described in further detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0054] According to some embodiments, the display generation component 120 provides an XR experience to the user when the user is virtually and / or physically present within the scene 105.

[0055] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head, on his / her hand, etc.). In this way, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smart phone or tablet device) configured to present XR content, and the user holds a device with a display facing the user's field of view and a camera facing the scene 105. In some embodiments, the handheld device is optionally placed in a shell worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is an XR room, shell, or room configured to present XR content, in which the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in the space in front of a handheld device or a tripod-mounted device may similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and responses to the XR content are displayed via the HMD. Similarly, a user interface showing interactions with XR content triggered based on movement of a handheld device or a tripod-mounted device relative to a physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)) may similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)).

[0056] Despite Figure 1 Relevant features of operating environment 100 are shown in , but those of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the example embodiments disclosed herein.

[0057] Figure 2is a block diagram of an example of a controller 110 according to some embodiments. While some specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0058] In some embodiments, the one or more communication buses 204 include circuits that interconnect and control communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0059] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 230 and an XR experience module 240.

[0060] The operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data sending unit 248.

[0061] In some embodiments, the data acquisition unit 242 is configured to obtain Figure 1 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, data acquisition unit 242 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0062] In some embodiments, the tracking unit 244 is configured to map the scene 105 and track at least the display generation component 120 relative to the scene 105. Figure 1 The tracking unit 244 may include instructions and / or logic for instructions and heuristics and metadata for the heuristics. In some embodiments, the tracking unit 244 includes a hand tracking unit 245 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 245 is configured to track the position / location of one or more parts of the user's hand, and / or the position of one or more parts of the user's hand relative to the user's hand. Figure 1 The movement of the scene 105 relative to the display generation component 120 and / or relative to a coordinate system (the coordinate system is defined relative to the user's hand). Figure 4 The hand tracking unit 245 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the position or movement of the user's gaze (or more generally, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hands)) or relative to the XR content displayed via the display generation component 120. Figure 5 The eye tracking unit 243 is described in more detail.

[0063] In some embodiments, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripherals 195. To this end, in various embodiments, coordination unit 246 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0064] In some embodiments, the data sending unit 248 is configured to send data (e.g., presentation data, position data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For this purpose, in various embodiments, the data sending unit 248 includes instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0065] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 245), the coordination unit 246, and the data sending unit 248 are illustrated as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 245), the coordination unit 246, and the data sending unit 248 may be located in separate computing devices.

[0066] also, Figure 2 It is more of a functional description of various features that may be present in a particular implementation, as opposed to a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0067] Figure 3is a block diagram of an example of a display generation component 120 according to some embodiments. While some specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the display generation component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0068] In some embodiments, one or more communication buses 304 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0069] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to the user. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more XR displays 312 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. As another example, the display generation component 120 includes an XR display for each eye of the user. In some embodiments, one or more XR displays 312 can present MR and VR content. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0070] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to the user's hands and, optionally, at least a portion of the user's arms (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to a scene that the user would see in the absence of the display generation component 120 (e.g., an HMD) (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0071] The memory 320 includes a high-speed random access memory, such as a DRAM, SRAM, DDR RAM, or other random access solid-state memory device. In some embodiments, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 330 and an XR rendering module 340.

[0072] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR rendering module 340 is configured to present XR content to a user via one or more XR displays 312. For this purpose, in various embodiments, the XR rendering module 340 includes a data acquisition unit 342, an XR rendering unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0073] In some embodiments, the data acquisition unit 342 is configured to at least Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For this purpose, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0074] In some embodiments, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. To this end, in various embodiments, the XR rendering unit 344 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0075] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which a computer-generated object can be placed to generate an extended reality) based on the media content data. For this purpose, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0076] In some embodiments, the data sending unit 348 is configured to send data (e.g., presentation data, position data, etc.) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For this purpose, in various embodiments, the data sending unit 348 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0077] Although the data acquisition unit 342, the XR rendering unit 344, the XR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., Figure 1 , but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data sending unit 348 may be located in a separate computing device.

[0078] also, Figure 3 It is more of a functional description of various features that may be present in a particular implementation, as opposed to a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0079] Figure 4 is a schematic illustration of an example implementation of the hand tracking device 140. In some implementations, the hand tracking device 140 ( Figure 1 ) is controlled by the hand tracking unit 245 ( Figure 2 ) to track the position / location of one or more parts of a user's hand, and / or one or more parts of a user's hand relative to Figure 1The hand tracking device 140 can be used to monitor movement of the scene 105 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system (which is defined relative to the user's hands)). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to the head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0080] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a human user's hand 406. The image sensor 404 captures hand images with sufficient resolution so that fingers and their corresponding positioning can be distinguished. The image sensor 404 typically captures images of other parts of the user's body, or may capture images of all parts of the body, and may have zoom capabilities or a dedicated sensor with increased magnification to capture images of the hand with a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or is used as an image sensor to capture the physical environment of the scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a manner that the field of view of the image sensor 404 or a portion thereof is used to define an interaction space, in which the hand movements captured by the image sensor are considered as input to the controller 110.

[0081] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D image data (and, in addition, possible color image data) to the controller 110, which extracts high-level information from the image data. The high-level information is typically provided to an application running on the controller via an application program interface (API), which drives the display generation component 120 accordingly. For example, a user can interact with software running on the controller 110 by moving their hands 406 and / or changing their hand gestures.

[0082] In some embodiments, the image sensor 404 projects a speckled pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral offset of the spots in the pattern. This method is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. The method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In the present disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis, so that the depth coordinates of points in the scene correspond to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods based on a single or multiple cameras or other types of sensors, such as stereo imaging or time-of-flight measurement.

[0083] In some embodiments, the hand tracking device 140 captures and processes a time series of depth maps containing the user's hand as the user moves their hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract image patch descriptors of the hand in these depth maps. The software can match these descriptors with image patch descriptors stored in the database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D location of the user's hand joints and finger tips.

[0084] The software can also analyze the trajectory of the hand and / or fingers over multiple frames in the sequence to identify gestures. The pose estimation function described herein can be alternated with the motion tracking function so that the image block-based pose estimation is performed only once every two (or more) frames, and the tracking is used to find the changes in pose that occur on the remaining frames. The pose, motion, and gesture information is provided to the application running on the controller 110 via the above-mentioned API. The program can, for example, move and modify the image presented on the display generation component 120 in response to the pose and / or gesture information, or perform other functions.

[0085] In some embodiments, gestures include air gestures. An air gesture is a gesture detected without the user touching an input element that is part of a device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) (or independent of an input element that is part of a device) and based on detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of a user's finger relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0086] In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with an XR environment (e.g., a virtual or mixed reality environment) performed by movement of a user's fingers relative to other fingers or parts of the user's hand. In some embodiments, an air gesture is a gesture detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and based on detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0087] In some embodiments where the input gesture is an in-air gesture (e.g., in the absence of physical contact with an input device that provides information to the computer system about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Thus, in specific implementations involving in-air gestures, for example, the input gesture is combined with (e.g., simultaneously with) movement of the user's fingers and / or hand to detect attention (e.g., gaze) toward a user interface element to perform a pinch and / or tap input, as described in more detail below.

[0088] In some embodiments, an input gesture pointing to a user interface object is performed directly or indirectly with reference to the user interface object. For example, the user input is performed directly on the user interface object according to performing the input with the user's hand at a location corresponding to the location of the user interface object in the three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some embodiments, when the user's attention (e.g., gaze) to the user interface object is detected, the input gesture is performed indirectly on the user interface object according to the user's hand being located not at the location corresponding to the location of the user interface object in the three-dimensional environment while the user performs the input gesture. For example, for a direct input gesture, the user can direct the user's input to the user interface object by initiating a gesture at or near a location corresponding to the display location of the user interface object (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge of the option or the center portion of the option). For an indirect input gesture, the user can direct the user's input to the user interface object by focusing on the user interface object (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location that does not correspond to the display location of the user interface object).

[0089] In some embodiments, according to some embodiments, the input gestures (e.g., air gestures) used in the various examples and embodiments described herein include pinch inputs and tap inputs for interacting with a virtual or mixed reality environment. For example, the pinch inputs and tap inputs described below are performed as air gestures.

[0090] In some embodiments, the pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes movement of two or more fingers of a hand to contact each other, that is, optionally followed by an immediate (e.g., within 0 seconds to 1 second) interruption of contact with each other. A long pinch gesture as an air gesture includes movement of two or more fingers of a hand in contact with each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact with each other is detected. For example, a long pinch gesture includes the user maintaining a pinch gesture (e.g., in which two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture as an air gesture includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected consecutively immediately (e.g., within a predefined time period) with each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts contact between two or more fingers), and performs a second pinch input within a predefined time period after releasing the first pinch input (e.g., within 1 second or within 2 seconds).

[0091] In some embodiments, the pinch and drag gesture as an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., following) a drag input that changes the positioning of the user's hand from a first positioning (e.g., a starting positioning for dragging) to a second positioning (e.g., an ending positioning for dragging). In some embodiments, the user maintains the pinch gesture while performing the drag input, and releases the pinch gesture (e.g., opens their two or more fingers) to end the drag gesture (e.g., at the second positioning). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers to contact each other and moves the same hand to the second positioning in the air using the drag gesture). In some embodiments, a pinch input is performed by a first hand of a user, and a drag input is performed by a second hand of the user (e.g., the second hand of the user moves from a first position to a second position in the air while the user continues the pinch input with the first hand of the user. In some embodiments, the input gesture as an air gesture includes an input performed using both hands of the user (e.g., a pinch and / or tap input). For example, the input gesture includes two (e.g., or more) pinch inputs performed in conjunction with each other (e.g., concurrently or within a predefined time period). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) is performed using the first hand of the user, and in conjunction with the pinch input performed using the first hand, a second pinch input is performed using another hand (e.g., a second hand of the user's two hands). In some embodiments, movement between the two hands of the user (e.g., increasing and / or decreasing the distance or relative orientation between the two hands of the user).

[0092] In some embodiments, a tap input performed as an air gesture (e.g., pointing to a user interface element) includes movement of a user's finger toward the user interface element, movement of the user's hand toward the user interface element (optionally, extension of the user's finger toward the user interface element), downward movement of the user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture movement of the finger or hand, which is a movement of the finger or hand away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by an end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of the movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of movement of the finger or hand, and / or a reversal of the acceleration direction of the movement of the finger or hand).

[0093] In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment (optionally, no other conditions are required). In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment using one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., a dwell duration) and / or requiring the gaze to be directed to the portion of the three-dimensional environment when the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment, so that the device determines that the user's attention is directed to the portion of the three-dimensional environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0094] In some embodiments, the detection of a ready state configuration of a user or a portion of a user is detected by a computer system. The detection of the ready state configuration of a hand is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein) performed by the hand. For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart to prepare for a pinch or grab gesture, or a pre-tap with one or more fingers extended and the palm facing away from the user), based on whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or based on whether the hand has moved in a particular manner (e.g., toward an area in front of the user above the user's waist and below the user's head or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of a user interface responds to attention (e.g., gaze) input.

[0095] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or may alternatively be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is also stored in a memory associated with the controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although in Figure 4Controller 110 is shown in FIG. 1 , but some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand tracking device) or other device associated with the image sensor 404, for example, as a separate unit from the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing functions of the image sensor 404 may also be integrated into a computer or other computerized device to be controlled by the sensor output.

[0096] Figure 4 Also included is a schematic diagram of a depth map 410 captured by the image sensor 404 according to some embodiments. As explained above, the depth map includes a matrix of pixels with corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the figure. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z distance from the image sensor 404), where gray shades become darker as depth increases. The controller 110 processes these depth values ​​in order to identify and segment components of the image that have human hand characteristics (i.e., a group of adjacent pixels). These characteristics may include, for example, overall size, shape, and motion from frame to frame in the depth map sequence.

[0097] Figure 4 Also schematically illustrated is a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406 according to some embodiments. Figure 4 In the hand skeleton 414, a hand background 416 that has been segmented from the original depth map is superimposed. In some embodiments, key feature points of the hand and optionally on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, finger tips, center of the palm, end of the hand connected to the wrist, etc.) are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points over multiple image frames to determine the gesture performed by the hand or the current state of the hand according to some embodiments.

[0098] Figure 5 The eye tracking device 130 ( Figure 1 ). In some embodiments, the eye tracking device 130 is composed of an eye tracking unit 243 ( Figure 2) controls to track the position and movement of the user's gaze relative to the scene 105 or relative to the XR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as headphones, helmets, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for viewing by the user and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye tracking device 130 is optionally a device separate from the handheld device or the XR room. In some embodiments, the eye tracking device 130 is a head-mounted device or a part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or a display generation component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally used in conjunction with a head mounted display generation component. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally part of a non-head mounted display generation component.

[0099] In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display a frame including a left image and a right image in front of the user's eyes, thereby providing a 3D virtual view to the user. For example, a head-mounted display generation component may include a left optical lens and a right optical lens (referred to herein as an eye lens) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture a video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display, and display a virtual object on the transparent or translucent display, and the user can directly view the physical environment through the transparent or translucent display. In some embodiments, the display generation component projects the virtual object into the physical environment. The virtual object may, for example, be projected on a physical surface or projected as a hologram so that an individual uses the system to observe a virtual object superimposed on the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

[0100] like Figure 5As shown, in some embodiments, the eye tracking device 130 (e.g., gaze tracking device) includes at least one eye tracking camera (e.g., an infrared (IR) or near infrared (NIR) camera), and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive the IR or NIR light reflected directly from the eyes by the light source, or alternatively can be pointed at "hot" mirrors located between the user's eyes and the display panel, which reflect the IR or NIR light from the eyes to the eye tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and transmits the gaze tracking information to the controller 110. In some embodiments, the user's two eyes are tracked separately by corresponding eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by the corresponding eye tracking camera and illumination source.

[0101] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for a specific operating environment 100, such as the 3D geometric relationships and parameters of the LED, camera, thermal mirror (if present), eye lens, and display screen. The device-specific calibration process can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration process can be an automatic calibration process or a manual calibration process. According to some embodiments, the user-specific calibration process may include an estimate of the eye parameters of a specific user, such as pupil position, foveal position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific parameters and user-specific parameters are determined for the eye tracking device 130, a flash-assisted method can be used to process the images captured by the eye tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0102] like Figure 5As shown, the eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system, which includes at least one eye tracking camera 540 (e.g., an infrared (IR) or near infrared (NIR) camera) positioned on the side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye tracking camera 540 can be directed toward a mirror 550 (which reflects IR or NIR light from the eye 592 while allowing visible light to pass) located between the user's eye 592 and a display 510 (e.g., a left display panel or a right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) (e.g., as Figure 5 ), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion of Figure 5 ).

[0103] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0104] Several possible use cases for the user's current gaze direction are described below and are not intended to be limiting. As an example use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content at a higher resolution in the foveal area determined according to the user's current gaze direction than in the peripheral area. As another example, the controller may position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller may display specific virtual content in a view based at least in part on the user's current gaze direction. As another example use case in an AR application, the controller 110 may guide an external camera for capturing the physical environment of the XR experience to focus in the determined direction. The autofocus mechanism of the external camera may then focus on an object or surface in the environment that the user is currently looking at on the display 510. As another example use case, the eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of the eye lens 520 so that the virtual object that the user is currently looking at has an appropriate degree of convergence to match the convergence of the user's eye 592. The controller 110 can use the gaze tracking information to guide the eye lens 520 to adjust the focus so that nearby objects that the user is looking at appear at the correct distance.

[0105] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lenses 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)). The light source emits light (e.g., IR or NIR light) toward the user's eye 592. In some embodiments, the light sources may be arranged in a ring or circle around each of the lenses, such as Figure 5 In some embodiments, for example, eight light sources 530 (eg, LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0106] In some embodiments, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. Note that the positions and angles of the eye tracking cameras 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0107] like Figure 5 Embodiments of the gaze tracking system illustrated in the drawings may be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience.

[0108] Figure 6 A flash-assisted gaze tracking pipeline according to some embodiments is illustrated. In some embodiments, the gaze tracking pipeline is implemented by a flash-assisted gaze tracking system (e.g., Figure 1 and Figure 5 The flash-assisted gaze tracking system may maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and the flash in the current frame. When not in the tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and the flash in the current frame, and if successful, initializes the tracking state to "yes" and continues with the next frame in the tracking state.

[0109] like Figure 6 As shown, the gaze tracking camera can capture left and right images of the user's left and right eyes. The captured images are then input to the gaze tracking pipeline for processing starting at 610. As indicated by the arrow returning to element 600, the gaze tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each group of captured images can be input to the pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are processed by the pipeline.

[0110] At 610, for the currently captured image, if the tracking status is yes, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as indicated at 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0111] At 640, if advancing from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from previous frames. At 640, if advancing from element 630, the tracking state is initialized based on the pupils and glints detected in the current frame. The processing results at element 640 are checked to verify that the results of the tracking or detection can be credible. For example, the results can be checked to determine whether the pupil and a sufficient number of glints for performing gaze estimation are successfully tracked or detected in the current frame. At 650, if the results are not likely to be credible, at element 660, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the results are credible, the method advances to element 670. At 670, the tracking state is set to yes (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0112] Figure 6 It is intended to be used as an example of an eye tracking technology that can be used for a particular implementation. As recognized by one of ordinary skill in the art, according to various embodiments, other eye tracking technologies currently existing or developed in the future can be used in place of or in combination with the flash-assisted eye tracking technology described herein in a computer system 101 for providing an XR experience to a user.

[0113] In the present disclosure, various input methods are described with respect to interaction with a computer system. When an input device or input method is used to provide an example, and another input device or input method is used to provide another example, it should be understood that each example is compatible with the input device or input method described with respect to another example and optionally utilizes the input device or input method. Similarly, various output methods are described with respect to interaction with a computer system. When an output device or output method is used to provide an example, and another output device or output method is used to provide another example, it should be understood that each example is compatible with the output device or output method described with respect to another example and optionally utilizes the output device or output method. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment by a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with the method described with respect to another example and optionally utilizes these methods. Therefore, the present disclosure discloses an embodiment as a combination of features of multiple examples, without exhaustively listing all features of the embodiment in the description of each example embodiment.

[0114] User interface and associated processes

[0115] Attention is now directed to embodiments of a user interface ("UI") and associated processes that may be implemented on a computer system, such as a portable multifunction device or a head mounted device, in communication with a display generation component and one or more input devices.

[0116] 7A to 7H , FIG. 8A to FIG. 8J , 9A to 9E and FIG. 10A to FIG. 10EA three-dimensional environment visible via a display generation component (e.g., display generation component 7100 or display generation component 120) of a computer system (e.g., computer system 101) and interactions occurring in the three-dimensional environment caused by user input directed to the three-dimensional environment and / or input received from other computer systems and / or sensors are illustrated. In some embodiments, input is directed to a virtual object within the three-dimensional environment by a user gaze detected in an area occupied by the virtual object or by a gesture performed at a location in the physical environment corresponding to the area of ​​the virtual object. In some embodiments, input is directed to a virtual object within the three-dimensional environment by a gesture performed (e.g., optionally, at a location in the physical environment that is unrelated to the area of ​​the virtual object in the three-dimensional environment) when the virtual object has input focus (e.g., when the virtual object has been selected by concurrently and / or previously detected gaze input, by concurrently or previously detected pointer input, and / or by concurrently and / or previously detected gesture input). In some embodiments, input is directed to a virtual object within the three-dimensional environment by an input device that has positioned a focus selector object (e.g., a pointer object or a selector object) at the location of the virtual object. In some embodiments, input is directed to a virtual object within a three-dimensional environment via other means (e.g., voice and / or control buttons). In some embodiments, input is directed to a physical object or a representation of a virtual object corresponding to a physical object through user hand movement (e.g., whole hand movement, whole hand movement in a corresponding posture, movement of one part of a user's hand relative to another part of the hand, and / or relative movement between both hands) and / or manipulation relative to a physical object (e.g., touch, swipe, tap, open, move toward, and / or move relative to). In some embodiments, a computer system displays some changes to a three-dimensional environment (e.g., displaying additional virtual content, stopping displaying existing virtual content, and / or transitioning between different immersion levels of displayed visual content) based on input from sensors (e.g., image sensors, temperature sensors, biometric sensors, motion sensors, and / or proximity sensors) and contextual conditions (e.g., location, time, and / or the presence of other people in the environment). In some embodiments, a computer system displays some changes in a three-dimensional environment (e.g., displaying additional virtual content, ceasing to display existing virtual content, and / or transitioning between different immersive levels of displayed visual content) based on input from other computers used by other users sharing a computer-generated environment with a user of the computer system (e.g., in a shared computer-generated experience, in a shared virtual environment, and / or in a shared virtual or augmented reality environment of a communication session).In some embodiments, a computer system displays some changes in a three-dimensional environment (e.g., displaying movement, deformation, and / or changes in visual characteristics of a user interface, virtual surfaces, user interface objects, and / or virtual scenery) based on input from a sensor that detects the movement of other people and objects as well as movement of a user that may not meet the criteria for being a recognized gesture input that triggers an associated operation of the computer system.

[0117] In some embodiments, the three-dimensional environment visible via the display generation component described herein is a virtual three-dimensional environment including virtual objects and content at different virtual locations in the three-dimensional environment without a representation of the physical environment. In some embodiments, the three-dimensional environment is a mixed reality environment that displays virtual objects at different virtual locations in the three-dimensional environment that are constrained by one or more physical aspects of the physical environment (e.g., walls, floors, positioning and orientation of surfaces, gravity direction, time of day, and / or spatial relationships between physical objects). In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of the physical environment. In some embodiments, the representation of the physical environment includes corresponding representations of physical objects and surfaces at different locations in the three-dimensional environment, so that the spatial relationship between different physical objects and surfaces in the physical environment is reflected by the spatial relationship between the representations of the physical objects and surfaces in the three-dimensional environment. In some embodiments, when a virtual object is placed relative to the positioning of the representations of the physical objects and surfaces in the three-dimensional environment, the virtual object appears to have a corresponding spatial relationship with the physical objects and surfaces in the physical environment. In some embodiments, a computer system transitions between displaying different types of environments based on user input and / or situational conditions (e.g., transitioning between presenting computer-generated environments or experiences with different levels of immersion, adjusting the relative prominence of audio / visual sensory input from virtual content and from representations of the physical environment).

[0118] In some embodiments, the display generation component includes a see-through portion in which a representation of the physical environment is displayed or visible. In some embodiments, the see-through portion of the display generation component is a transparent or translucent (e.g., see-through) portion of the display generation component that shows at least a portion of the physical environment around the user or within the user's field of view (sometimes referred to as "optical see-through"). For example, the see-through portion is a portion of a head-mounted display or a head-up display that is made translucent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10% or 5% opacity) or transparent, so that the user can see through it to view the real world around the user without removing the head-mounted display or moving away from the head-up display. In some embodiments, when a virtual or mixed reality environment is displayed, the see-through portion gradually transitions from translucent or transparent to completely opaque. In some embodiments, the see-through portion of the display generation component displays a real-time feed of an image or video of at least a portion of the physical environment captured by one or more cameras (e.g., a rear-facing camera of a mobile device or associated with a head-mounted display, or other cameras that feed image data to a computer system) (sometimes referred to as "optical see-through"). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is directly in front of the user's eyes (e.g., behind the display generation component relative to the user of the display generation component). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is not directly in front of the user's eyes (e.g., in a different physical environment, or to the side or behind the user).

[0119] In some embodiments, when virtual objects are displayed at a location corresponding to the location of one or more physical objects in a physical environment (e.g., at a location in a virtual reality environment, a mixed reality environment, or an augmented reality environment), at least some of the virtual objects are displayed to replace (e.g., replace the display of) a portion of the camera's real-time view (e.g., a portion of the physical environment captured in the real-time view). In some embodiments, at least some of the virtual objects and content are projected onto a physical surface or empty space in the physical environment and are visible through a see-through portion of a display generation component (e.g., visible as part of the camera's view of the physical environment, or visible through a transparent or translucent portion of the display generation component). In some embodiments, at least some of the virtual objects and content are displayed to cover a portion of the display and obstruct at least a portion of the view of the physical environment visible through a transparent or translucent portion of the display generation component.

[0120] In some embodiments, the display generation component displays different views of the three-dimensional environment based on user input or movement that changes the virtual positioning of the viewpoint of the currently displayed view of the three-dimensional environment relative to the three-dimensional environment. In some embodiments, when the three-dimensional environment is a virtual environment, the viewpoint moves based on navigation or motion requests (e.g., aerial gestures and / or gestures performed by movement of one part of the hand relative to another part of the hand) without requiring movement of the user's head, torso, and / or display generation component in the physical environment. In some embodiments, movement of the user's head and / or torso and / or movement of other positioning sensing elements of the display generation component or computer system (e.g., due to the user holding the display generation component or wearing an HMD) relative to the physical environment causes a corresponding movement of the viewpoint relative to the three-dimensional environment (e.g., with a corresponding movement direction, movement distance, movement speed, and / or orientation change), thereby causing a corresponding change in the current displayed view of the three-dimensional environment. In some embodiments, when a virtual object has a preset spatial relationship relative to a viewpoint (e.g., anchored or fixed to the viewpoint), movement of the viewpoint relative to the three-dimensional environment will cause movement of the virtual object relative to the three-dimensional environment while maintaining the position of the virtual object in the field of view (e.g., the virtual object is said to be head-locked). In some embodiments, the virtual object is body-locked to the user and moves relative to the three-dimensional environment as the user as a whole moves in the physical environment (e.g., carrying or wearing the display generation components and / or other position sensing components of the computer system), but will not move in the three-dimensional environment in response to individual user head movements (e.g., the display generation components and / or other position sensing components of the computer system rotate around a fixed position of the user in the physical environment). In some embodiments, the virtual object is optionally locked to another part of the user, such as the user's hand or the user's wrist, and moves in the three-dimensional environment in accordance with the movement of that part of the user in the physical environment to maintain a preset spatial relationship between the position of the virtual object and the virtual position of that part of the user in the three-dimensional environment. In some embodiments, a virtual object is locked to a preset portion of a field of view provided by a display generation component and moves in a three-dimensional environment based on movement of the field of view, regardless of movement of the user that does not cause changes in the field of view.

[0121] In some embodiments, such as 7A to 7H , FIG. 8A to FIG. 8J , 9A to 9E and FIG. 10A to FIG. 10EAs shown, the view of the three-dimensional environment sometimes does not include the representation of the user's hand, arm and / or wrist. In some embodiments, the representation of the user's hand, arm and / or wrist is included in the view of the three-dimensional environment. In some embodiments, the representation of the user's hand, arm and / or wrist is included in the view of the three-dimensional environment as a part of the representation of the physical environment provided via the display generation component. In some embodiments, these representations are not part of the representation of the physical environment, and are captured separately (e.g., by one or more cameras pointing to the user's hand, arm and wrist) and are displayed in the three-dimensional environment independently of the view currently displayed by the three-dimensional environment. In some embodiments, these representations include camera images captured by one or more cameras of the computer system or stylized versions of arms, wrists and / or hands based on information captured by various sensors. In some embodiments, these representations replace the display of a part of the representation of the physical environment, overlay on this part of the representation of the physical environment or block the view of this part of the representation of the physical environment. In some embodiments, when the display generation component does not provide a view of the physical environment and provides a completely virtual environment (e.g., no camera view and no transparent pass-through portion), a real-time visual representation of one or both arms, wrists, and / or hands of the user is optionally still displayed in the virtual environment (e.g., a stylized representation or a segmented camera image). In some embodiments, if no representation of the user's hands is provided in the view of the three-dimensional environment, the position corresponding to the user's hands is optionally indicated in the three-dimensional environment, for example, by changing the appearance of the virtual content at a position in the three-dimensional environment corresponding to the position of the user's hands in the physical environment (e.g., by a change in translucency and / or simulated reflectivity). In some embodiments, the representation of the user's hand or wrist is outside the currently displayed view of the three-dimensional environment, and the virtual position corresponding to the position of the user's hand or wrist in the three-dimensional environment is outside the current field of view provided by the display generation component; and in response to the virtual position corresponding to the position of the user's hand or wrist moving within the current field of view due to the movement of the display generation component, the user's hand or wrist, the user's head, and / or the user as a whole, the representation of the user's hand or wrist is made visible in the view of the three-dimensional environment.

[0122] 7A to 7H An example of displaying attention-based shadows of user interface objects in an environment is illustrated. 7A to 7H The user interface in is used to illustrate the process described below, including Fig.11 process.

[0123] Fig. 7AAn example physical environment 7000 is illustrated including a user 7002 interacting with a computer system 101. The physical environment 7000 includes physical walls 7004 and 7006 and a floor 7008. A physical box 7014 having a textured (e.g., rough or irregular) surface is positioned on the floor 7008. 7A to 7H As shown in the example in FIG. 7 , the display generation component 7100 of the computer system 101 is a touch screen held by the user 7002 using the hand 7020 and / or the hand 7022. In some embodiments, the display generation component of the computer system 101 is a head mounted display worn on the head of the user 7002' (e.g., in FIG. 7 ). 7A to 7H 7002 when wearing a head-mounted display). In some embodiments, the display generation component is a stand-alone display, a projector, or another type of display. In some embodiments, the computer system communicates with one or more input devices, which include cameras or other sensors and input devices that detect the movement of the user's hands, the movement of the user's entire body, and / or the movement of the user's head in the physical environment. In some embodiments, the one or more input devices detect the movement and current posture, orientation, and position of the user's hands, face, and / or entire body. In some embodiments, user input is detected via a touch-sensitive surface or a touch screen. In some embodiments, one or more input devices include an eye tracking component that detects the location and movement of the user's gaze. In some embodiments, the display generation component and optionally one or more input devices and the computer system are part of a head-mounted device that moves and rotates with the user's head in the physical environment and changes the user's viewpoint in the three-dimensional environment provided by the display generation component. In some embodiments, the display generating component is a heads-up display that does not move or rotate with the user's head or the user's entire body, but optionally changes the user's viewpoint in the three-dimensional environment based on the movement of the user's head or body relative to the display generating component. In some embodiments, the display generating component (e.g., a touch screen) is optionally moved and rotated by the user's hands relative to the physical environment or relative to the user's head, and changes the user's viewpoint in the three-dimensional environment based on the movement of the display generating component relative to the user's head or face or relative to the physical environment. In some embodiments, one or more portions of the view of the three-dimensional environment visible to the user 7002 via the display generating component 7100 are digital pass-through portions that include the physical environment 7000 ( Fig. 7AIn some embodiments, one or more portions of the view of the three-dimensional environment visible to the user 7002 via the display generation component 7100 are optically see-through portions, because the user 7002 can see one or more portions of the physical environment 7000 through one or more transparent or semi-transparent portions of the display generation component 7100.

[0124] Figure 7B A view of a three-dimensional environment visible to a user 7002 via a display generation component 7100 of the computer system 101 is illustrated. Figure 7B The three-dimensional environment optionally includes representations of objects in a physical environment such as physical environment 7000 (e.g., as captured by one or more cameras of computer system 101) or optical views of objects in the physical environment (e.g., as visible through one or more transparent or semi-transparent portions of display generation component 7100). For example, in Figure 7B , the three-dimensional environment includes a representation (or optical view) 7004' of a wall 7004, a representation (or optical view) 7008' of a physical floor 7008, and a representation (or optical view) 7014' of a physical box 7014. In addition, the three-dimensional environment includes one or more computer-generated objects (also referred to as virtual objects) displayed via the display generation component 7100, such as a table 7024 (e.g., which is not a representation or optical view of a physical table in the physical environment 7000) and windows 7010 and 7012 (e.g., which are not representations or optical views of physical elements in the physical environment 7000). In some embodiments, window 7010 corresponds to a user interface of a software application (e.g., an email application, a web browser, a communication application, a map application, or other software application) executed on the computer system 101. Similarly, in some embodiments, window 7012 corresponds to a user interface of a software application (e.g., an email application, a web browser, a communications application, a mapping application, or other software application) executed on the computer system 101 (optionally the same application as the application in window 7010 or a different application).

[0125] exist Figure 7B , window 7010 is displayed with a computer generated shadow 7016, and window 7012 is displayed with a computer generated shadow 7018. 7B to 7HDescribed in more detail, the appearance of the shadow associated with a given object in a three-dimensional environment optionally changes based on whether the user's 7002 attention is directed toward the object. In some embodiments, the user's 7002 attention is directed toward an object if the user is looking at the object, optionally has a hand engaged (e.g., in a ready state for performing an air gesture or contacting a touch-sensitive surface but not necessarily pressing the touch-sensitive surface), and / or has moved a focus indicator (e.g., a cursor) over the object or is hovering a hand or finger over the object. Figure 7B , user 7002's attention is not directed to window 7010 or window 7012 (e.g., user 7002 is not looking at either window, has not moved a cursor over either window, or has not selected either window for interaction). Therefore, windows 7010 and 7012 are displayed in a first (e.g., neutral or default) appearance without visual emphasis. Figure 7B The shadow 7016 of the window 7010 in represents a shadow displayed at a first opacity, and Figure 7B The shadow 7018 of window 7012 in the three-dimensional environment also represents a shadow displayed at the first opacity. In some embodiments, the shadow of an object in the three-dimensional environment is not positioned based on the lighting in the three-dimensional environment. For example, if an object is below and to the left of a source of (e.g., real or simulated) light in the three-dimensional environment, the shadow of the object is displayed directly below and centered on the object (e.g., even if the object is moved and / or resized), rather than offset to the left (e.g., unlike what would be expected for a physical object casting a physical shadow opposite a physical light source).

[0126] exist Figure 7C 7026 (e.g., a top-down view) shows the relative positioning of different elements of the three-dimensional environment visible from the viewpoint of the user 7002. The top view 7026 shows that the window 7012 has been moved from its original position. Figure 7B The positioning in moves forward and is therefore closer to the viewpoint 7028-1 of the user 7002. Figure 7C The window 7012 in FIG. 7014 partially overhangs the left and front edges of the box 7014′, rather than Figure 7B14'). As such, the shadow 7018 of the window 7012 is moved forward along the top surface of the box 7014' so as to be at least partially projected on the left and front edges of the box 7014' as seen via the display generation component 7100. In some embodiments, the display of the shadow 7018 along the top and sides of the box 7014' is based on a three-dimensional model of the box 7014 determined by the computer system 101 using depth data about the box 7014 captured (e.g., via one or more cameras, structured light scanners, time-of-flight sensors, etc.), a time-of-flight sensor, or other depth sensor of the computer system). In some embodiments, the three-dimensional model of box 7014 is a simplified model that smooths out or ignores irregularities in the physical surface of box 7014 (e.g., by modeling box 7014 with a smooth surface even if box 7014 is carpeted and / or by modeling box 7014 with a flat surface even if a low profile object such as a pen or magazine is placed on box 7014).

[0127] Fig.7D 7012 (e.g., after moving from Figure 7B 7002 is directed to window 7012. Fig.7D7012 has focus for further interaction). Those skilled in the art will recognize that, in lieu of or in addition to displaying a selection outline, such as using increased opacity, increased brightness, and / or other forms of visual emphasis, one or more of the various visual attributes of the windows described herein may be changed to give visual emphasis to the window. However, in some embodiments, one or more visual attributes of a window or other computer-generated object described herein remain the same even when the user's attention is directed to the window and the window otherwise has a visually emphasized appearance. For example, in some embodiments, the size (e.g., length, width, and / or thickness), positioning, and / or orientation of a window or object is the same when the user's attention is directed to the window as when the user's attention is not (e.g., has stopped or has not yet) directed to the window. In some embodiments, the attention of user 7002 must be directed to an object such as window 7012 for at least a threshold amount of time T. th , as indicated by timer 7032, so that the object is given focus and displayed with visual emphasis.

[0128] In response to user 7002's attention being directed to window 7012, the appearance of shadow 7018 corresponding to window 7012 also changes (eg, changes to visually emphasize shadow 7018 in conjunction with visually emphasizing corresponding window 7012). Fig.7D The shaded 7018 in the figure indicates that the Figure 7B The shadow of the first opacity and the second opacity are displayed (for example, Fig.7D With Figure 7B7018). In some embodiments, the change in the appearance of the shadow 7018 (or any other shadow described herein) occurs gradually, for example, by transitioning from a first opacity through multiple intermediate opacities to a second opacity. In some embodiments, the change in the appearance of the shadow 7018 occurs after a delay so that the appearance of the shadow 7018 does not flicker when the user's 7002 attention moves across the border of the window 7012. In some embodiments, the delay is 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 3 seconds, or other time period; in some embodiments, the delay is long enough to introduce lag to reduce flicker in the user interface, but short enough to be imperceptible or nearly imperceptible to the unassisted eye. One of ordinary skill in the art will readily appreciate that, in lieu of or in addition to opacity, one or more of the various visual attributes of the shadow described herein may be changed in conjunction with giving visual emphasis to a window or other object corresponding to the shadow. However, in some embodiments, one or more visual attributes of the shadow described herein remain the same even when the user's attention is directed toward the corresponding window (e.g., in the absence of movement of the corresponding window) and the shadow otherwise has a changed intensity (e.g., opacity). For example, in some embodiments, the size (e.g., length, width, and / or thickness), position, orientation, and / or spread (e.g., degree of blur) of the window or object is the same when the user's attention is directed toward the window as when the user's attention is not (e.g., has stopped or has not yet) directed toward the window. In accordance with the user's 7002 attention remaining unchanged relative to window 7010 (e.g., continuing to turn away from the window), window 7010 and shadow 7016 remain the same. Figure 7B Same appearance in .

[0129] Fig. 7E The user 7002's attention is directed to the window 7010 instead of to the Fig.7D 7010 (e.g., the gaze, finger, or cursor of user 7002 has moved from window 7012 to window 7010), as indicated by attention indicator 7030-1. In response to user 7002's attention being directed to window 7010, Fig. 7E Window 7010 in is displayed with a selection outline (and / or other indication that window 7010 has focus) while maintaining the same size and orientation in the three-dimensional environment. Fig. 7E The shadow 7016 in FIG. 7014 indicates that the appearance of the shadow corresponding to window 7010 has also changed, optionally gradually and / or after a delay (e.g., as described herein with reference to Fig.7D ), and is displayed with an increased opacity (e.g., a second opacity or another opacity greater than the first opacity) (e.g., as described by Fig. 7EWith Figure 7B 7012) while maintaining the same size and orientation in the three-dimensional environment. In addition, in response to user 7002's attention no longer being directed toward window 7012, a previous change in the appearance of window 7012 (e.g., in response to when user 7002's attention was directed toward window 7012) to a second appearance is at least partially reversed, and optionally, so is a previous change in the appearance of shadow 7018 corresponding to window 7012. For example, in the absence of Fig.7D The selected contour is displayed Fig. 7E Window 7012 in, and Fig. 7E The shadow 7018 in FIG. 7014 indicates that the opacity of the shadow corresponding to the window 7012 has been reduced (e.g., from Fig.7D The second opacity is reduced to Figure 7B The first opacity is reduced to a first opacity, or in some embodiments is reduced to a third opacity that is also lower than the second opacity but different from the first opacity), optionally gradually reduced and / or reduced after a delay (e.g., as described herein with reference to Fig.7D described).

[0130] Figure 7F Different views of the three-dimensional environment visible via display generation component 7100 according to user 7002 viewing the three-dimensional environment from viewpoint 7028-2 are illustrated, as shown by top view 7026. Figure 7F , windows 7010 and 7012 remain at the same simulated positioning relative to the three-dimensional environment (e.g., relative to table 7024 and box 7014', respectively), as also shown in top view 7026. Window 7010 and a portion of window 7012 are visible from viewpoint 7028-2. In addition, a computer-generated speaker 7034 on table 7024 (e.g., the speaker is virtual and is not a representation of a physical speaker in physical environment 7000) is visible. Speaker 7034 is displayed with a computer-generated shadow 7036. Because the attention of user 7002 continues to be directed to window 7010, attention indicator 7030-1 continues to be displayed relative to window 7010, and window 7010 continues to be displayed with a selection outline. In addition, shadow 7016 corresponding to window 7010 continues to be displayed at a second opacity. Because the user 7002's attention is not directed to the window 7012, the shadow 7018 corresponding to the window 7012 continues to be displayed at the first opacity (or other opacities less than the second opacity). Also, because the user 7002's attention is not directed to the speaker 7034, the shadow 7036 corresponding to the speaker 7034 is displayed at the first opacity (or other opacities less than the second opacity).

[0131] Figure 7GThe user 7002's attention is directed to the speaker 7034 rather than Figure 7F In response to user 7002's attention being directed toward speaker 7034, Figure 7G Speaker 7034 in the three-dimensional environment can be used to select an outline (and / or other indication that speaker 7034 has focus) while maintaining the same size and pose in the three-dimensional environment. Figure 7G The shadow 7036 in FIG. 7034 indicates that the appearance of the shadow corresponding to the speaker 7034 has also changed, optionally gradually and / or after a delay (e.g., as described herein with reference to Fig.7D ), and is displayed with an increased opacity (e.g., a second opacity) (e.g., as described by Figure 7G With Figure 7F 7036 of a different fill pattern in the 3D environment), while maintaining the same size and pose in the 3D environment. In addition, in response to user 7002's attention no longer being directed toward window 7010, a previous change in the appearance of window 7010 (e.g., in response to when user 7002's attention was directed toward window 7010) is at least partially reversed, and optionally, so is a previous change in the appearance of shadow 7016 corresponding to window 7010. For example, in the absence of Figure 7F The selected contour is displayed Figure 7G 7010 in the window, and Figure 7G The shadow 7016 in FIG. 7010 indicates that the opacity of the shadow corresponding to the window 7010 has been reduced (e.g., from Figure 7F The second opacity to Fig.7D ), optionally gradually decreasing and / or decreasing after a delay (e.g., as described herein with reference to Fig.7D described). Figure 7G The appearance of window 7012 and shadow 7018 in Figure 7F Compared to the same window 7012, the user 7002's attention has not changed relative to the window 7012 (e.g., the user 7002's attention is not directed to the window 7012). Figure 7F or Figure 7G 7012 in the window).

[0132] exist Figure 7H 7028 - 3 , the view of the three-dimensional environment visible via display generation component 7100 is from viewpoint 7028 - 3 , such that window 7010 , window 7012 , and speaker 7034 are being viewed from behind. Figure 7HEmbodiments are illustrated in which computer-generated shadows are not displayed for computer-generated objects that are being viewed from outside a predefined range of angles (e.g., from behind or from the side, more than 90 degrees from a front view) (e.g., the displayed shadows cease to be displayed when the user's viewpoint moves outside the predefined range of angles). Figure 7H When viewed from behind (e.g., with Figure 7G When viewed at 180 degrees (the front view shown), no shadows are displayed for window 7010, window 7012, and speaker 7034.

[0133] FIG. 8A to FIG. 8J An example of displaying motion-based shadows of user interface objects in an environment is illustrated. FIG. 8A to FIG. 8J The user interface in is used to illustrate the process described below, including Fig.12 process.

[0134] Fig. 8A 1 illustrates a view of a three-dimensional environment visible to a user 7002 via a display generation component 7100 of a computer system 101. In some embodiments, one or more portions of the view of the three-dimensional environment are digital pass-through portions that include the physical environment 7000 ( Fig. 7A In some embodiments, one or more portions of the view of the three-dimensional environment are optically transparent because the user 7002 can see one or more portions of the physical environment 7000 through one or more transparent or semi-transparent portions of the display generation component 7100. Fig. 8A , the view of the visible three-dimensional environment includes a representation (or optical view) 7004' of a physical wall 7004 and a representation (or optical view) 7008' of a physical floor 7008 (e.g., as captured by one or more cameras of the computer system 101 or visible through one or more transparent or semi-transparent portions of the display generation component 7100). FIG. 8A to FIG. 8J As shown in the example in , the content visible via the display generation component 7100 of the computer system 101 is displayed on a touch screen optionally held by the user 7002. In some embodiments, the display generation component 7100 of the computer system 101 is a head-mounted display worn on the head of the user 7002 (e.g., in FIG. 8A to FIG. 8J The content shown as visible via the display generation component 7100 of the computer system 101 corresponds to the field of view of the user 7002 when wearing the head mounted display).

[0135] exist Fig. 8A7024 and window 7010 are displayed (e.g., they are not representations or optical views of physical objects in physical environment 7000). Window 7010 has a focus for interaction (e.g., user 7002's attention is directed toward window 7010), as indicated by attention indicator 7030-1, and accordingly, window 7010 is shown with a visually emphasized appearance (such as Fig. 8A ) (e.g., the same as when window 7010 does not have focus). Figure 7B 7010 having focus). Thus, the increased opacity associated with the window 7010 having focus (e.g., Fig. 7E A shadow 7016 corresponding to window 7016 (eg, a shadow 7016 corresponding to window 7010 when window 7010 does not have focus) is displayed at a second opacity Figure 7B Those of ordinary skill in the art will recognize that one or more of the various visual attributes of the window described herein may be altered to give visual emphasis to the window, instead of or in addition to displaying a selection outline, and that one or more of the various visual attributes of the shadow described herein may be altered in conjunction with the shadow's corresponding window or other object having focus, instead of or in addition to opacity.

[0136] Fig. 8A Light sources 7038 and 7040 are also illustrated. Light source 7038 represents a physical light source in physical environment 7000 or a simulated light source that projects (e.g., computer-generated) light in a three-dimensional environment. Similarly, light source 7040 represents a physical light source or a simulated light source. In some embodiments, light sources 7038 and 7040 are both physical light sources that project light in physical environment 7000, and in some cases, the light is at least partially visible in three dimensions as seen via display generation component 7100. In some embodiments, light sources 7038 and 7040 are both simulated light sources. In some embodiments, one of light sources 7038 and 7040 is a physical light source in physical environment 7000, and the other is a simulated light source. Fig. 8A As shown, window 7010 is displayed as exhibiting specular reflections, with a visual effect simulating window 7010 reflecting light from light sources 7038 and 7040 from the edges of window 7010. For example, reflection 7042-1 represents a specular reflection of physical light or virtual light from physical light source or virtual light source 7038, and reflection 7042-2 represents a specular reflection of physical light or virtual light from light source 7040. In some embodiments, based on determining (e.g., based on physical light and / or computer-generated simulated light) that there is at least a threshold amount of light in the three-dimensional environment, such as Fig. 8AAs illustrated in FIG. 7042 , computer-generated specular reflections (e.g., reflections 7042-1 and 7042-2) from one or more objects in the three-dimensional environment are displayed, and / or computer-generated shadows (e.g., shadow 7016) of one or more objects in the three-dimensional environment are displayed.

[0137] Figure 8B A view of a three-dimensional environment is illustrated in which light sources 7038 and 7040 are dimmed or turned off (e.g., due to a reduction in physical light and / or simulated light). In some embodiments, for simulated light sources, computer system 101 is configured to reduce the amount of simulated light displayed from the light sources or to completely stop displaying simulated light from the light sources. In some embodiments, for physical light sources, computer system 101 is configured to apply a darkening filter to the view of the three-dimensional environment visible via display generation component 7100 (e.g., by displaying a partially transparent darkening layer overlying the see-through portion of the view of the three-dimensional environment). Thus, Figure 8B The three-dimensional environment in appears darker, and specular reflections 7042-1 and 7042-2 are not displayed (e.g., consistent with light sources 7038 and 7040 being dimmed or turned off). In some embodiments, shadows are not displayed for objects in the three-dimensional environment in accordance with the darker three-dimensional environment. Figure 8B , the shadow 7016 corresponding to window 7010 has ceased to be displayed (eg, but window 7010 remains in focus, as indicated by attention indicator 7030-1, and thus continues to be displayed with a selection outline).

[0138] Figure 8C A view of a three-dimensional environment is illustrated in response to input performed by user 7002 using hand 7022 to initiate movement of window 7010 in the three-dimensional environment, as indicated by movement indicator 7030-2 (e.g., from Fig. 8A In some embodiments, the input is a touch input (e.g., long press drag), an air gesture (e.g., air pinch drag), a mouse input (e.g., click drag), or other repositioning input via a touch-sensitive surface. In response to the input initiating the movement of window 7010, the appearance of shadow 7016 corresponding to window 7010 changes. For example, Figure 8C The shadow 7016 in FIG. 7010 indicates that the opacity of the shadow corresponding to the window 7010 has been reduced (e.g., from Fig. 8A The opacity of window 7010 is reduced to a lower opacity. The shadow corresponding to window 7010 is reduced to a lower opacity during the movement input ( Figure 8C ) is optionally the same or different than the opacity of the shadow corresponding to window 7010 in the absence of user 7002's attention ( Figure 7B )(For example, as in Figure 8C With Figure 7B In addition, window 7010 is visually weakened (e.g., for the duration of the input used to move window 7010), and in particular, the opacity is reduced (e.g., the same or different amount as shadow 7016) to reduce the degree to which window 7010 obscures underlying portions of the three-dimensional environment so that user 7002 can better perceive where and how window 7010 is moving relative to the three-dimensional environment. Figure 8C In the window 7010, Figure 8C The light sources 7038 and 7040 have the same Fig. 8A The same positioning of reflections 7042-1 and 7042-2 relative to window 7010 is maintained at the same Fig. 8A The same positioning in .

[0139] In some embodiments, the change in appearance of shadow 7016 occurs in response to input initiating movement of window 7010, and optionally continues to be displayed during the input initiating movement of window 7010, before actual movement of window 7010, or even without that actual movement. For example, during hand 7022 input, shadow 7016 may appear Figure 8C The same as before the hand 7022 input Fig. 8A Different appearance, even if Windows 7010 in Figure 8C In between Fig. 8A In some embodiments, the change in the appearance of shadow 7016 occurs in response to an input that initiates movement of window 7010, regardless of the lighting conditions in the three-dimensional environment (e.g., as long as there is a threshold amount of light in the three-dimensional environment). For example, if Fig. 8A and Figure 8C The three-dimensional environment is illuminated by light source 7038 (e.g., as a real or simulated light source), shadow 7016 changes from a first appearance to a second appearance in response to detecting input that initiates movement of window 7010; and if Fig. 8A and Figure 8C If the three-dimensional environment of the window 7010 is illuminated by light source 7040 (e.g., as a real or simulated light source), or even by both light sources 7038 and 7040, shadow 7016 still changes from the first appearance to the second appearance in response to detecting an input initiating movement of window 7010. Those skilled in the art will readily appreciate that various other lighting conditions may exist. Fig. 8A and Figure 8C , and shadow 7016 will, optionally, still change from the first appearance to the second appearance in response to detecting input that initiates movement of window 7010.

[0140] Fig.8DThe movement of window 7010 in a three-dimensional environment according to the movement of the input of hand 7022 (e.g., touch input, air gesture, mouse input, or other input of dragging or other movement-related portion) is illustrated. In conjunction with the movement of window 7010 to the left in the three-dimensional environment, shadow 7016 moves to the left by a corresponding amount in the three-dimensional environment. In some embodiments, the intensity of shadow 7016 is the same during the input for moving window 7010, regardless of the direction or magnitude of the movement of the input. For example, Fig.8D The shadow 7016 in the figure indicates that the opacity of the shadow corresponding to the window 7010 is Fig.8D In and Figure 8C in the same way (for example, as in Fig.8D In Figure 8C ), regardless of the direction (e.g., left or right) or magnitude of the movement of window 7010 in response to the input of hand 7022. In another example described in more detail herein, Figure 8G The shadow 7016 in the figure indicates that the opacity of the shadow corresponding to the window 7010 is Figure 8G In and Figure 8F in the same way (for example, as in Figure 8G In Figure 8F ), regardless of the direction (e.g., backward or forward, further or closer) or magnitude of the movement of window 7010 in response to the input of hand 7022 (e.g., as long as window 7010 is within a threshold distance of the user's viewpoint and / or corresponding surface, as described herein with reference to FIG. 8F to FIG. 8J described in more detail).

[0141] In some embodiments, the spatial relationship between a window and its corresponding shadow is not affected by the positioning of the light source in the three-dimensional environment. Figure 8C Compared with Windows 7010, Fig.8D 7010 has moved relative to the three-dimensional environment and relative to light sources 7038 and 7040, which remain fixed relative to the three-dimensional environment. Although the spatial relationship between a physical window and its corresponding physical shadow will generally change (e.g., in terms of distance, relative positioning, and / or relative size) in response to movement of the physical window relative to one or more light sources (e.g., which remain fixed), Fig.8D The spatial relationship between the center 7010 and the shadow 7016 is Figure 8C Same as in Windows 7010, although Windows 7010 from FIG. 8C to FIG. 8D Move relative to light sources 7038 and 7040. For example, Figure 8C and Fig.8DIn both, shadow 7016 is displayed directly below window 7010 on the top surface of table 7024 and has the same appearance (e.g., size, feathering, and opacity). In some embodiments, other elements in the three-dimensional environment are affected by the positioning of the light source in the three-dimensional environment, including some elements associated with objects whose spatial relationship to the corresponding shadow is not affected by the light source. For example, when window 7010 is Figure 8C and Fig.8D When moving to the left in the three-dimensional environment, the reflection 7042-1 corresponding to the light source 7038 and the reflection 7042-2 corresponding to the light source 7040 are both offset accordingly (eg, to the right) along the edge of the window 7010.

[0142] In some embodiments, the shadow corresponding to the object moves with the object independently of the movement of the user's gaze. Fig.8D The gaze indicator 7044 in indicates that the gaze of the user 7002 is directed toward the right side of the table 7024 and optionally moves toward the right side of the table, while the window 7010 moves to the left in the three-dimensional environment. Fig.8D The shadow 7016 is illustrated as moving to the left with the window 7010, regardless of the location and / or direction of movement of the user's 7002 gaze (e.g., the shadow 7016 moves to the left with the window 7010, even if the user's 7002 gaze is moving to the right and / or directed elsewhere in the three-dimensional environment). In some embodiments, during input for moving an object, the shadow corresponding to the object has an appearance that is independent of whether the user's gaze is directed toward the object. For example, if Fig.8D 7044 in the image is changed to be above the window 7010, the shadow 7016 will still have the Fig.8D The same appearance as shown (eg, but shadow 7016 may appear when input for moving window 7010 is not in progress) as described herein. 7B to 7H attention-based behaviors described).

[0143] exist Fig. 8E In , the input of hand 7022 has ended. Fig. 8E In the example of , window 7010 retains focus after the input of hand 7022 ends (e.g., before a subsequent input gives focus to a different object in the three-dimensional environment). Attention indicator 7030-1 is shown corresponding to window 7010 to indicate that window 7010 has focus. In response to the end of the input of hand 7022, and in accordance with window 7010 having focus without input directed to window 7010 to initiate movement, the appearance of window 7010 and shadow 7016 is restored to what they were before. Fig. 8Aor at least partially restored, but according to the window 7010 relative to Fig. 8E The light sources 7038 and 7040 have the same Fig.8D 7042-1 and 7042-2 remain at the same position relative to window 7010. For example, window 7010 is visually emphasized with a selection outline, and shadow 7016 indicates that the opacity of the shadow corresponding to window 7010 has been increased (e.g., from Fig.8D to a higher opacity, such as Fig. 8A opacity) (e.g., as determined by Fig. 8E In Fig.8D 7016).

[0144] FIG. 8F to FIG. 8I It is illustrated that the appearance of a shadow corresponding to an object changes as the distance between the object and the user's viewpoint increases (eg, causing the object to appear smaller based on perspective). Figure 8F The second input initiating movement of window 7010 in the three-dimensional environment is illustrated as being performed by user 7002 using hand 7022 (e.g., with FIG. 8C to FIG. 8D or, alternatively, if Fig. 8E The end of the input illustrated in is not detected, then FIG. 8C to FIG. 8D (Continue with input). Figure 8F The top view 7026 in FIG. 7 illustrates a simulated positioning of the window 7010 in the three-dimensional environment less than a threshold distance D from the viewpoint 7046 of the user 7002. th1 Thus, in response to the second input initiating movement of window 7010, shadow 7016 indicates that the opacity of the shadow corresponding to window 7010 has been reduced to the same level as the opacity of window 7010. Fig. 8E The first reduced opacity of the shadow 7016 in the image is different from the first reduced opacity, and the window 7010 includes being visually weakened by the reduced opacity (e.g., for the duration of the input used to move the window 7010), but according to the window 7010 relative to the image Figure 8F The light sources 7038 and 7040 have the same Fig. 8E In the same position in the image, reflections 7042-1 and 7042-2 remain at the same position relative to window 7010.

[0145] Figure 8G 7010 is illustrated as having moved further back from the viewpoint of the user 7002 in the three-dimensional environment in response to movement of the second input of the hand 7022, as indicated by arrow 7048 (which is included in the illustration for illustrative purposes). Figure 8G7002) and indicated in top view 7026. Due to perspective, window 7010 is therefore shown as being larger in size due to the increased distance between window 7010 and the viewpoint of user 7002. Figure 8F According to the reduced size of the window 7010 (the reduced size includes a reduced width), the width of the shadow 7016 corresponding to the window 7010 is also reduced relative to Figure 8F The shadow 7016 in the Figure 8G 7010 is kept optionally with Figure 8F The same reduced opacity in the Figure 8F In comparison, according to Figure 8G Window 7010 in has been repositioned relative to light sources 7038 and 7040 to move further away from viewpoint 7046 of user 7002, and reflections 7042-1 and 7042-2 are offset relative to window 7010 and slightly reduced in size. Figure 8G The top view 7026 in FIG. 7 illustrates a simulated positioning of the window 7010 in the three-dimensional environment to maintain a distance less than a threshold distance D from the viewpoint 7046 of the user 7002. th1 Therefore, the shadow 7016 indicates that the shadow corresponding to the window 7010 remains the same as Figure 8F The same opacity as Fig. 8E The opacity of the shadow 7016 in the first reduced opacity is different).

[0146] Figure 8H 7010 has been moved further back from the viewpoint of the user 7002 in the three-dimensional environment in response to further movement of the second input of the hand 7022, as indicated by arrow 7050 (which is included in the figure for illustrative purposes). Figure 8H 7002) and indicated in top view 7026. Due to perspective, window 7010 is therefore shown as being larger in size due to the increased distance between window 7010 and the viewpoint of user 7002. Figure 8G According to the reduced width of window 7010, the width of shadow 7016 corresponding to window 7010 is also relatively Figure 8G The shadow 7016 in the Figure 8H 7010 is kept optionally with Figure 8G The same opacity is visually weakened. Figure 8G In comparison, according to Figure 8HWindow 7010 in has been repositioned relative to light sources 7038 and 7040 to move further away from viewpoint 7046 of user 7002, and reflections 7042-1 and 7042-2 are again offset relative to window 7010 and slightly further reduced in size. Figure 8H The top view 7026 in FIG. 7 illustrates the simulated positioning of the window 7010 in the three-dimensional environment, which is now greater than the threshold distance D from the viewpoint 7046 of the user 7002. th1 , but still less than the threshold distance D th2 In some embodiments, when window 7010 moves across a predefined distance range from the viewpoint of user 7002, the shadow corresponding to window 7010 gradually fades out (e.g., the shadow fades more as the distance from the window to the user's viewpoint increases). Figure 8H The opacity of the shadow corresponding to window 7010 in Figure 8G Reduce to FIG. 8F to FIG. 8G In some embodiments, fading of a shadow corresponding to an object as the object moves further away from the user's viewpoint occurs in conjunction with (and optionally at a faster rate than) a reduction in shadow width due to perspective-induced reduction in object width (e.g., the shadow fades out completely before the shadow shrinks to zero width).

[0147] Fig.8I 7010 has been moved still further back from the viewpoint of the user 7002 in the three-dimensional environment in response to still further movement of the second input of the hand 7022, as indicated by arrow 7052 (which is included in the drawing for illustrative purposes). Fig.8I 7002) and indicated in top view 7026. Due to perspective, window 7010 is therefore shown as being larger in size due to the increased distance between window 7010 and the viewpoint of user 7002. Figure 8H Window 7010 is optionally available with Figure 8H The same opacity is visually weakened. Figure 8H In comparison, according to Fig.8I Window 7010 in FIG. 7 has been repositioned relative to light sources 7038 and 7040 to move further away from viewpoint 7046 of user 7002, and reflections 7042-1 and 7042-2 are again offset relative to window 7010 and further reduced in size. Figure 8H The top view 7026 in FIG. 7 illustrates that the simulated positioning of the window 7010 in the three-dimensional environment is now even greater than the threshold distance D from the viewpoint 7046 of the user 7002. th2In some embodiments, when the window 7010 is beyond a predefined distance range from the user's 7002 viewpoint (e.g., the distance between the simulated position of the window 7010 and the user's viewpoint is at least D th2 ), the shadow corresponding to window 7010 stops being displayed completely (e.g., it gradually fades out when window 7010 moves across the predefined distance range). Fig.8I The shadow 7016 is not displayed in the three-dimensional environment (or optionally, the shadow 7016 exists but is completely transparent).

[0148] Figure 8J 1 illustrates a change in the appearance of a shadow corresponding to an object (e.g., from a shadow corresponding to an object to a surface that is displayed on the object (e.g., a surface below the object)) as the distance between the object and a surface corresponding to the object increases (e.g., the object moves away from the surface). Figure 8F of transformation). Figure 8J The third input for moving the window 7010 in the three-dimensional environment is illustrated as being performed by the user 7002 using the hand 7022 (e.g., from Fig.8D or Figure 8F In response to the movement of the third input, window 7010 has moved further upward from the top surface of table 7024 in the three-dimensional environment, as indicated by arrow 7054 (which is included in the drawing for illustrative purposes). Figure 8J and optionally not displayed by display generation component 7100). Therefore, Figure 8J 7016 in the figure represents an increase in the degree of blurring and / or feathering of the shadow corresponding to window 7010 (e.g., rather than a change in opacity as would occur when the user's 7002 attention relative to window 7010 changes or when an input initiating movement of window 7010 is detected). Window 7010 is optionally blurred with respect to the third input during the third input. Fig.8D or Figure 8F The same reduced opacity in the Figure 8F In comparison, according to Figure 8J Window 7010 in has been repositioned closer to light sources 7038 and 7040, and reflections 7042-1 and 7042-2 are slightly offset relative to window 7010 and increased in size.

[0149] In some embodiments, although certain aspects of the three-dimensional environment (such as responding to initiating FIG. 8A to FIG. 8J The type of change in the appearance of the shadow 7016 of the input of the movement of the window 7010 in the 3D environment) is independent of the lighting conditions in the 3D environment (e.g., as long as there is a threshold amount of light in the 3D environment), but other aspects of the 3D environment (such as specular reflections) do change based on different lighting conditions in the 3D environment. For example, if FIG. 8A to FIG. 8J If the three-dimensional environment is illuminated by light source 7038 (e.g., as a real or simulated light source) but not by light source 7040, then reflection 7042-1 of light source 7038 is displayed, while reflection 7042-2 of light source 7040 is not displayed. In another example, if FIG. 8A to FIG. 8J If the three-dimensional environment is illuminated by light source 7040 (eg, as a real or simulated light source) but not by light source 7038, then reflection 7042-2 of light source 7040 is displayed, while reflection 7042-1 of light source 7038 is not displayed.

[0150] In some embodiments, a shadow corresponding to an object exhibits behavior similar to that in response to changes in the distance between the object and the surface on which the shadow corresponding to the object is displayed, in response to changes in the distance between the object and the user's viewpoint. For example, as the degree of blur and / or feathering of the shadow 7016 changes as the window 7010 moves further upward and away from the top surface of the table 7024, the shadow 7016 gradually fades out (e.g., the shadow fades out more as the height of the window 7010 above the table 7024 increases). In some embodiments, the fading of the shadow 7016 occurs within a predefined height (e.g., distance) range of the window 7010 above the top surface of the table 7024, starting at a first threshold height of the window 7010 above the top surface of the table 7024, and causing the shadow 7016 to completely fade out at a larger second threshold height of the window 7010 above the top surface of the table 7024. In some embodiments, when the window 7010 is below a first threshold height above the top surface of the table 7024, the intensity (e.g., opacity) of the shadow 7016 corresponding to the window 7010 remains the same, regardless of the direction or magnitude of the movement of the window 7010. In some embodiments, fading of the shadow corresponding to the object occurs in conjunction with (and optionally at a faster rate than) an increase in the degree of blurring and / or feathering of the shadow as the object moves further away from the corresponding surface (e.g., the shadow fades out completely before the shadow blur and / or feathering radius reaches a maximum value). In some embodiments, the distance range relative to the corresponding surface is different from the distance range relative to the user's viewpoint. For example, the shadow 7016 optionally fades when the window 7010 moves through a first distance range (e.g., between 1 meter, 1.5 meters, 2 meters, 3 meters, or 4 meters and 4.5 meters, 5 meters, 6 meters, or 7 meters, or other range) from the perspective of the user 7002, but fades when the window 7010 moves through a different (e.g., higher or lower and / or wider or narrower) second distance range (e.g., between 1 meter, 1.5 meters, or 2 meters and 2.5 meters, 3 meters, or 4 meters, or other range) from the top surface of the table 7024.

[0151] 9A to 9EIllustrate an example of displaying simplified shadows of user interface objects in an environment. 9A to 9E The user interface in is used to illustrate the process described below, including Fig.13 process.

[0152] Fig. 9A 1 illustrates a view of a three-dimensional environment visible to a user 7002 via a display generation component 7100 of a computer system 101. In some embodiments, one or more portions of the view of the three-dimensional environment are digital pass-through portions that include the physical environment 7000 ( Fig. 7A In some embodiments, one or more portions of the view of the three-dimensional environment are optically transparent because the user 7002 can see one or more portions of the physical environment 7000 through one or more transparent or semi-transparent portions of the display generation component 7100. Fig. 9A , user interface 7060 includes multiple constituent elements, including, for example: element E1, which is a first area of ​​user interface 7060 (e.g., a main area or window); elements E2 and E3, which are together in a second area of ​​user interface 7060 (e.g., elements in a toolbar, menu bar, or navigation bar, such as tabs, forward and / or back buttons, or other menus or navigation controls); element E4, which is a third area of ​​user interface 7060 (e.g., another toolbar, menu bar, navigation bar, or other control collection, including, for example, tabs, forward and / or back buttons, scroll bars, or other menus or navigation controls); element E5 (e.g., three-dimensional content displayed in user interface 7060); a close button 7062 (e.g., for dismissing user interface 7060); a move button 7064 (e.g., for repositioning user interface 7060 in a three-dimensional environment); and a resize button 7066 (e.g., for resizing user interface 7060). User interface 7060 is optionally a user interface of an application executed on computer system 101. In some cases, user interface 7060 is a 7B to 7H and FIG. 8A to FIG. 8J A more detailed view of an application user interface (e.g., user interface 7060 exhibits the same functionality as described herein) more simply as window 7010 or window 7012. 7B to 7H and FIG. 8A to FIG. 8J 7010 and / or similar behavior of window 7012 as described above). 9A to 9E As shown in the example in , the content visible via the display generation component 7100 of the computer system 101 is displayed on a touch screen optionally held by the user 7002. In some embodiments, the display generation component 7100 of the computer system 101 is a head-mounted display worn on the head of the user 7002 (e.g., in 9A to 9EThe content shown as visible via the display generation component 7100 of the computer system 101 corresponds to the field of view of the user 7002 when wearing the head mounted display).

[0153] At least some of the multiple elements that make up user interface 7060 are separated from each other in space in a three-dimensional environment. For example, as shown in top view 7026, element E1 is separated from elements E2 and E3, element E4, and buttons 7062, 7064, and 7066 (e.g., not adjacent). In the same plane as element E1 (or in some embodiments, at the same distance from the viewpoint 7068 of user 7002), the close button is positioned to be separated from element E1 and on the right side of the element. Move button 7064 and resize button 7066 are in front of element E1 (e.g., forward from element E1 in the z direction or radial direction), closer to the viewpoint 7068 of user 7002, and optionally in the same plane as each other (or at the same distance from the viewpoint 7068 of user 7002). The user interface area containing elements E2 and E3 is positioned in front of element E1 and in front of move button 7064 and resize button 7066 (e.g., forward in the z-direction or radial direction), closer to viewpoint 7068 of user 7002. Element E4 is also positioned in front of element E1 and in front of move button 7064 and resize button 7066, closer to viewpoint 7068 of user 7002, and optionally in the same plane as the user interface area containing elements E2 and E3, or at the same distance from viewpoint 7068 as the user interface area containing elements E2 and E3. Element E5 (the three-dimensional content of user interface 7060) extends forward from element E1 and toward viewpoint 7068 of user 7002 beyond the other elements.

[0154] Fig. 9A Also shown is a shadow 7070 corresponding to the user interface 7060 being displayed. The shadow 7070 is a computer-generated shadow (e.g., rendered by the computer system 101). The appearance (such as size and shape) of the shadow 7070 is based on at least some of the multiple elements that make up the user interface 7060. Specifically, the shadow 7070 includes a portion of the shadow corresponding to the element E1 of the user interface 7060. Fig. 9AIn the example of FIG. 7002 , shadow 7070 does not include portions that are shadows corresponding to other elements of user interface 7060. For example, shadow 7070 does not include a shadow corresponding to close button 7062, nor does it include a shadow corresponding to resize button 7066 (which would cause shadow 7070 to extend further to the right). Shadow 7070 also does not include a shadow corresponding to the user interface area containing elements E2 and E3, nor does it include a shadow corresponding to elements E4 and E5, nor does it include a shadow corresponding to move button 7064 and resize button 7066 (any of which would cause shadow 7070 to extend further forward toward viewpoint 7068 of user 7002).

[0155] In some embodiments, the computer system 101 is configured to display shadows on a computer-generated surface and / or on a see-through view of a physical surface. Fig. 9A The shadow 7070 in is partially displayed on the top surface of the computer-generated table 7024 and partially displayed on the top surface of the box 7014', which is a see-through view of the physical box 7014 in the physical environment 7000. Fig. 9B The shadow 7070 is illustrated as being displayed entirely on a computer-generated surface, such as the top surface of a computer-generated table 7024. Those skilled in the art will readily appreciate that the shadow 7070 may also be displayed entirely on a see-through view of a physical surface, such as being displayed entirely on the top surface of a box 7014', which is a physical box 7014 ( Fig. 7A )’s transparent view. Fig. 9B Example from Fig. 9A , in which, in response to detecting an input for repositioning user interface 7060 (e.g., activating move button 7064 and optionally including an input such as a drag gesture to move to the left), user interface 7060 is repositioned in the three-dimensional environment (e.g., to the left). The spatial relationship between the constituent elements of user interface 7060 remains the same as when user interface 7060 was moved from the previous position to the next position. Fig. 9A The first position shown (eg, having a first set of positions for the constituent elements) is relocated to Fig. 9B The second positions shown (eg, having a second set of positions for constituent elements) are the same.

[0156] In some embodiments, the appearance of shadow 7070 is independent of the lighting conditions in the three-dimensional environment (e.g., as long as there is a threshold amount of light in the three-dimensional environment). FIG. 9A to FIG. 9B The three-dimensional environment is illuminated by a light source 7038 (e.g., as a real or simulated light source), or if FIG. 9A to FIG. 9BShadow 7070 will have the same appearance if the three-dimensional environment is illuminated by light source 7040 (e.g., as a real or simulated light source) or even by both light sources 7038 and 7040, or if the three-dimensional environment changes from being illuminated by one combination of light sources 7038 and 7040 to being illuminated by a different combination of light sources 7038 and 7040. Those skilled in the art will readily appreciate that various other lighting conditions may exist. FIG. 9A to FIG. 9B three-dimensional environment, and shadow 7070 will optionally be the same under various lighting conditions.

[0157] Fig. 9C The user interface 7060 having a changed appearance is illustrated. Fig. 9C In FIG. 7 , user interface 7060 does not include elements E4 and E5. Although the appearance of user interface 7060 has changed, the appearance of shadow 7070 corresponding to user interface 7060 is unchanged. Fig. 9C In particular, because FIG. 9A to FIG. 9B The shadow 7070 in does not include the parts corresponding to the shadows of elements E4 and E5, so the shadows from Fig. 9C The absence of elements E4 and E5 in the user interface 7060 does not affect Fig. 9C The look of the shade 7070.

[0158] Fig.9D The user interface 7060 is illustrated as having been resized (e.g., in response to an input activating the resize button 7066 and optionally including an input such as a drag gesture moving to the right). In particular, the element E1 has been resized to be larger than Fig. 9C In some embodiments, as the width of element E1 increases, close button 7062 and resize button 7066 are repositioned to the right, optionally so as to remain beyond the right edge of element E1 and separate from element E1. In some embodiments, as the width of element E1 increases, move button 7064 is repositioned to the right, optionally so as to remain centered relative to element E1. Fig.9D In the example of FIG. 1 , when element E1 is resized, the positioning of element E4 and the user interface area containing elements E2 and E3 in the three-dimensional environment is maintained to the left and above element E1, respectively. As element E1 of user interface 7060 is resized, shadow 7070 corresponding to user interface 7060 is also resized. In particular, the width of shadow 7070 increases by an amount corresponding to the increase in the width of element E1. Because FIG. 9A to FIG. 9B The shadow 7070 in FIG. 7 does not include the portions of the shadow corresponding to the elements of the user interface 7060 other than the element E1, so only the change to the element E1 affects Fig.9D The look of the shade 7070.

[0159] Fig.9E The user interface 7060 is illustrated as having been tilted relative to the vertical direction in a three-dimensional environment (e.g., in Fig. 9C ). Side views 7056 and 7058 show views of user interface 7060 from the right. Side view 7056 shows a view of user interface 7060 in a previous orientation (e.g., as in Fig. 9C 7060), wherein the user interface 7060 and specifically the element E1 of the user interface 7060 are oriented vertically above the table 7024. The side view 7058 shows Fig.9E , wherein the user interface 7060 and specifically the element E1 are tilted toward the viewpoint of the user 7002 such that the top portion of the user interface 7060 (e.g., the top portion of the element E1) is closer to the viewpoint of the user 7002 than the bottom portion of the user interface 7060 (e.g., the bottom portion of the element E1). Fig. 9C 7058 ). However, in some embodiments, the increase in depth of the shadow 7070 is different from (e.g., less than or greater than) the increase in depth that would occur for a physical shadow of a physical object. For example, in the side view 7056, the depth of the shadow 7070 is the same as the top-down thickness of the user interface 7060 (or in particular the element E1) (e.g., the thickness of the user interface 7060 or the element E1 when viewed from directly above), while the depth of the shadow 7070 is less than the top-down thickness of the user interface 7060 in the side view 7058. Optionally, the shadow 7070 has a first depth when the user interface 7060 is vertical and has a first top-down thickness; the shadow 7070 has a second depth when the user interface 7060 is tilted and has a second top-down thickness; and the degree of change between the first depth and the second depth of the shadow 7070 is less than (or greater than) the degree of change between the first top-down thickness and the second top-down thickness of the user interface 7060.

[0160] FIG. 10A to FIG. 10E An example of simulating light emanating from a user interface object displayed in an environment is illustrated. FIG. 10A to FIG. 10E The user interface in is used to illustrate the process described below, including Fig.14 process.

[0161] Fig. 10A1 illustrates a view of a scene in a three-dimensional environment visible to a user 7002 via a display generation component 7100 of a computer system 101. In some embodiments, one or more portions of the view of the three-dimensional environment are digital pass-through portions that include the physical environment 7000 ( Fig. 7A ) of the corresponding portion of the three-dimensional environment. In some embodiments, one or more portions of the view of the three-dimensional environment are optically see-through portions because the user 7002 can see one or more portions of the physical environment 7000 through one or more transparent or translucent portions of the display generation component 7100. The view includes a representation (or optical view) 7004' (also referred to as wall 7004' for ease of reference), a representation (or optical view) 7008' (also referred to as floor 7008' for ease of reference) of the physical floor 7008, and a representation (or optical view) 7014' (also referred to as box 7014' for ease of reference) of the physical object 7014 (e.g., as captured by one or more cameras of the computer system 101 or visible through one or more transparent or translucent portions of the display generation component 7100). FIG. 10A to FIG. 10E As shown in the example in , the content visible via the display generation component 7100 of the computer system 101 is displayed on a touch screen optionally held by the user 7002. In some embodiments, the display generation component 7100 of the computer system 101 is a head-mounted display worn on the head of the user 7002 (e.g., in FIG. 8A to FIG. 8J The content shown as visible via the display generation component 7100 of the computer system 101 corresponds to the field of view of the user 7002 when wearing the head mounted display).

[0162] Fig. 10A The scene in also includes a view of speakers 7034 on table 7024 (e.g., a computer-generated object as described), as well as a view of television 7076 (e.g., a physical television or a computer-generated representation or optical view of a television), a view of television cabinet 7074 (e.g., a physical television cabinet or a computer-generated representation or optical view of a television cabinet), and a view of lamp 7072 (e.g., a physical lamp or a computer-generated representation or optical view of a lamp). Fig. 10A , the three-dimensional environment is well lit (e.g., there is at least a threshold amount of light in the environment) based on physical lighting in the corresponding physical environment 7000 and / or simulated lighting from one or more computer-generated light sources. Thus, television 7076 is shown with computer-generated shadows 7078. Speakers 7034 are optionally shown without computer-generated shadows, as shown in FIG. Fig. 10A , or optionally displayed with computer-generated shadows (e.g., similar to shadow 7036 ( Figure 7F )). Additionally, the content (e.g., video) is not playing on the television 7076, as indicated by the play-pause button 7080 which is displayed as a play button.

[0163] FIG. 10B to FIG. 10C Illustrated is ongoing playback of content in a three-dimensional environment when the three-dimensional environment is a mixed reality environment, as indicated by a play-pause button displayed as a pause button. FIG. 10B to FIG. 10C The view of the three-dimensional environment in includes walls 7004', floor 7008', and box 7014' (eg, physical environment 7000 ( Fig. 7A ) in FIG. 7006 , and the corresponding transparent views of the physical wall 7004, physical floor 7008 and physical box 7014 in FIG. 7006 . FIG. 10B to FIG. 10C It is also illustrated that one or more of the television 7076, the television cabinet 7074, and the lamp 7072 are physical counterparts (for example, these physical counterparts will also be in Fig. 7A An example of a transparent view of the physical environment 7000 is shown.

[0164] Fig. 10B 7076 for playing content (e.g., video content) via television 7076. Fig. 10A A view of the three-dimensional environment visible by activating the input of the play button 7080 in the scene illustrated in FIG. FIG. 10B to FIG. 10C As shown, in response to detecting an input for playing content (and while playing content), the computer system 101 dims or darkens the view of the three-dimensional environment, such as by reducing the amount of simulated light from one or more computer-generated light sources and / or applying a darkening filter to one or more computer-generated elements in the three-dimensional environment and / or to one or more transparent portions of the view of the three-dimensional environment (e.g., by overlaying a partially transparent darkening layer). In addition, the computer system 101 stops displaying computer-generated shadows corresponding to objects in the three-dimensional environment, such as shadow 7078 corresponding to television 7076.

[0165] When the content is played in a three-dimensional environment (e.g., on television 7076 or in another user interface in a three-dimensional environment), computer system 101 displays a light bloom visual effect that simulates light emanating from the content. Fig. 10B In, according to Fig. 10BThe three-dimensional environment in is a mixed reality environment, and the light overflow 7082-1 ​​appears as a glow with a first degree of fidelity (e.g., beyond the boundaries of the TV 7076) for the content being played. In some embodiments, the current playing scene A of the content being played is sampled to generate the light overflow 7082-1 ​​using a first number of samples (such as 12, 18, 32, 45, or any other first number of samples). For example, the colors sampled from various areas of the content (such as edge areas and / or internal areas) (e.g., samples from scene A) are blurred beyond the edge of the TV 7076 to form the light overflow 7082-1. In some embodiments, the light overflow 7082-1 ​​has a hue based on a second number of samples (e.g., 1, 2, 3, 8, or any other second number of samples) taken from the content (e.g., scene A). The sampled color is optionally the color of a specific point in the content, the median color of the color in the content (or in a specific area of ​​the content), the average of the color in the content (or in a specific area of ​​the content), or a color determined from the content using another means.

[0166] In some embodiments, the light spill 7082-1 ​​is displayed in the same plane as the content (or in some embodiments is displayed along a surface that is the same distance from the viewpoint of the user 7002 as the content). For example, based on the plane of the television 7076, the light spill 7082-1 ​​is displayed in front of the wall 7004' and in front of at least a portion of the TV cabinet 7074, while being at least partially obscured by the table 7024 and the speaker 7034. In some embodiments, the intensity of the visual effect decreases with distance from the content. For example, portions of the light spill 7082-1 ​​that are closer to the television 7076 are more intense (e.g., as indicated by a denser grid fill pattern), while portions of the light spill 7082-1 ​​that are farther away from the television 7076 are less intense (e.g., as indicated by a sparser horizontal regular fill pattern).

[0167] By illustrating the change in appearance of light overflow 7082 when the appearance of the content changes, Fig. 10C illustrating the appearance of light spill 7082 (e.g., Fig. 10B The glow visual effect of the scene A in which the appearance of the light overflow 7082-1 ​​is based on the current appearance of the content being played on the TV 7076. Fig. 10B In comparison, Fig. 10C In the example, the content played on TV 7076 has progressed from scene A to scene B, and therefore Fig. 10C The appearance of the light bloom 7082-2 in the scene B is based on the content (e.g., as described by Fig. 10B The horizontal rules and grid pattern fills in the light overflow 7082-1 ​​are different Fig. 10CIn some embodiments, one or more other aspects of bloom 7082 remain the same as the change in appearance of the bloom due to the change in appearance of the corresponding content. For example, like bloom 7082-1, bloom 7082-2 is optionally generated based on a first number of samples taken from scene B, and / or based on a first number of samples taken from scene B (rather than as in Fig. 10B In some embodiments, the light bloom 7082-2 optionally continues to be displayed in the same plane as the content (or along the same surface whose points are at the same distance from the viewpoint of the user 7002 as the content) and / or continues to decrease in intensity as the distance from the content increases (e.g., as indicated by the progression of the light bloom 7082-2 from a denser cross-hatched fill pattern to a sparser hatched fill pattern as the distance from the television 7076 increases).

[0168] FIG. 10D to FIG. 10E The ongoing playback of content in a three-dimensional environment is illustrated when the three-dimensional environment is a computer-generated virtual reality environment, as indicated by the play-pause button being displayed as a pause button. FIG. 10D to FIG. 10E The view of the three-dimensional environment in includes computer-generated walls 7090 instead of transparent walls 7004', includes computer-generated floors 7092 instead of transparent floors 7008', and does not include transparent boxes 7014'. FIG. 10D to FIG. 10E In the example of FIG. 7070 , television 7076 , television cabinet 7074 , and lamp 7072 are computer-generated objects.

[0169] Fig. 10D A view of a three-dimensional environment that is visible in response to detecting an input for playing content (e.g., video content) via a television 7076 is illustrated, and the view is similar to Fig. 10B The view shown in the example is different from Fig. 10D The three-dimensional environment visible in the virtual reality environment is Fig. 10B The three-dimensional environment visible in the image is a mixed reality environment. In some embodiments, the input for playing the content is directed to a location such as Fig. 10A The computer system 101 transforms the three-dimensional environment into a mixed reality environment such as Fig. 10D 7004', and / or floor 7008') and / or by displaying one or more additional computer-generated elements (e.g., floor 7092, walls 7090, and / or tablet 7096). In some embodiments, input for playing content is directed to the virtual environment (e.g., such as if Fig. 10D is from Fig. 10A The example shows that when the light is turned on Fig. 10D different versions of the virtual environment). FIG. 10D to FIG. 10E As shown, in response to detecting input for playing content (e.g., whether directed to a mixed reality or virtual environment) (and while playing the content), the computer system 101 dims or darkens the view of the three-dimensional environment, such as by reducing the amount of simulated light from one or more computer-generated light sources and / or applying a darkening filter to various computer-generated elements in the three-dimensional environment and / or to one or more transparent portions of the view of the three-dimensional environment (e.g., if the input for playing the content is directed to the mixed reality environment). In addition, the computer system 101 stops displaying computer-generated shadows corresponding to objects in the three-dimensional environment, such as shadows 7078 corresponding to television 7076.

[0170] When scene A of the content is played in a three-dimensional environment (e.g., on television 7076 or in another user interface in the three-dimensional environment), computer system 101 displays a light overflow visual effect that simulates light emanating from the content. Fig. 10D In, according to Fig. 10D The three-dimensional environment in the three-dimensional virtual environment is a virtual reality environment, and the light overflow 7088-1 appears as light projected into the three-dimensional environment (e.g., beyond the boundaries of the television 7076) at a second fidelity level higher than the first fidelity level for the content being played. In some embodiments, the light overflow 7088 (e.g., including a three-dimensional model (e.g., a mesh) of the three-dimensional virtual environment is generated. Fig. 10D The light overflow in 7088-1 and Fig.10E 7088-2) in the three-dimensional environment so that light projected onto other objects in the three-dimensional environment and / or reflected by these objects is displayed more realistically. In some embodiments, scene A of the content is sampled to generate light bloom 7088-1, optionally using a third number of samples that is greater than the first number of samples used for the lower first fidelity level of light bloom. For example, colors sampled from various areas of the content (such as edge areas and / or interior areas) (e.g., sampled from scene A) that exceed the edge of the television 7076 are blurred to form a portion of light bloom 7088-1.

[0171] In some embodiments, as part of light spill 7088-1, colors sampled from various areas of scene A are used to generate reflections that create the appearance of other objects in the three-dimensional environment reflecting light from the content, as represented by reflection 7084-1 from speaker 7034 and reflection 7086-1 from table 7024 and from objects on table 7024 (such as tablet computer 7096) (e.g., reflection 7086-1 represents the visual effect of light from content on television 7076 being reflected by tablet computer 7096 (e.g., if the display of tablet computer 7096 is turned off), rather than the visual effect of light generated by tablet computer 7096). In some embodiments, as part of light spill 7088-1, computer system 101 displays shadows beyond one or more objects in the three-dimensional environment to simulate one or more objects blocking at least some light emitted from the content, as represented by shadow 7094 that appears to be cast by speaker 7034. Thus, for a virtual environment, light overflow visual effects such as light overflow 7088-1 are not constrained to the same plane as the content (or a surface whose points are at the same distance from the viewpoint of the user 7002 as the content), because such light overflow visual effects are optionally used in a mixed reality environment. For example, light overflow 7088-1 (e.g., including reflections 7084-1 and 7086-1 and shadows 7094) is displayed beyond the plane of the television 7076. In some embodiments, the intensity of the light overflow visual effect decreases with distance from the content (e.g., for a virtual environment as well as for a mixed reality environment). For example, portions of the light overflow 7088-1 that are closer to the television 7076 are more intense, while portions of the light overflow 7088-1 that are farther away from the television 7076 are less intense (e.g., as indicated by the feathering of the fill pattern of the light overflow 7088-1). In another example, portions of the reflections 7084-1 and 7086-1 that are closer to the TV 7076 (e.g., portions that are closer to the speaker 7034 and the tablet computer 7096 that face the TV 7076, respectively) are stronger, while portions of the reflections 7084-1 and 7086-1 that are farther away from the TV 7076 (e.g., portions that are closer to the speaker 7034 and the tablet computer 7096 that face away from the TV 7076, respectively) are less intense.

[0172] By illustrating that the appearance of light overflow 7088 changes when the appearance of the content changes, Fig.10E Illustrate the appearance of light overflow 7088-2 (e.g., Fig. 10D light projection visual effects) based on the current appearance of the content being played (e.g. FIG. 10D to FIG. 10E together illustrates how the bloom visual effect in a virtual environment can change based on the content currently playing in the virtual environment, similar to FIG. 10B to FIG. 10C1 and 2 illustrate how the light bloom visual effect in a mixed reality environment can change based on the content currently being played in the mixed reality environment. Fig.10E Similar to Fig. 10C , the difference is that in Fig.10E The three-dimensional environment visible in the virtual reality environment is Fig. 10C The three-dimensional environment visible in is a mixed reality environment. Scene A in which the appearance of light overflow 7088-1 is based on the content played on TV 7076 Fig. 10D In comparison, Fig.10E In the example, the content played on TV 7076 has progressed from scene A to scene B, and therefore Fig.10E The appearance of the light bloom 7088-2 in scene B is based on the content (e.g., as determined by Fig. 10D The grid pattern fills of overflow 7088-1, reflection 7084-1 and reflection 7086-1 are different Fig.10E In some embodiments, one or more other aspects of bloom 7088 remain the same as the change in appearance of the bloom due to a change in the appearance of the corresponding content. For example, like bloom 7082-1, bloom 7088-2 is optionally based on the change in appearance of the scene B (rather than the change in appearance of the corresponding content). Fig. 10D In some embodiments, light spill 7088-2 is optionally displayed in the same or similar area of ​​the three-dimensional environment as light spill 7088-1, such as by extending into the same space beyond the boundaries of television 7076 and / or by reflecting off the same objects (e.g., speaker 7034, table 7024, and / or tablet computer 7096). In some embodiments, like light spill 7082-2, the intensity of light spill 7088-2 decreases with increasing distance from the content and / or casts the same or similar shadow (e.g., speaker 7034 casts shadow 7094 with both light spill 7088-1 and 7088-2).

[0173] A person of ordinary skill will readily appreciate that, alternatively, Fig.10E and Fig. 10B together and similarly Fig. 10D and Fig. 10C Together, the invention illustrates how a bloom visual effect in a 3D environment can be displayed based on the content currently playing in the 3D environment (e.g., based on scene B being a later scene than scene A) and based on a change in the immersion level of the 3D environment (e.g., from Fig. 10B A less immersive mixed reality environment to Fig.10E Increased immersion in more immersive virtual environments, and from Fig. 10D A more immersive virtual environment Fig. 10C to a less immersive mixed reality environment).

[0174] In some embodiments, the light bloom visual effect has a size (e.g., length and / or width) based on the size (e.g., length and / or width) of the corresponding content. FIG. 10B to FIG. 10C The light overflow 7082 in has a width and height based on the size of the corresponding content being played on the television 7076 (e.g., based on the size of the television 7076). In another example, FIG. 10D to FIG. 10E The light spill 7088 in 7082 has a width and height based on the size of the corresponding content being played on the television 7076 (e.g., based on the size of the television 7076); furthermore, the extent to which the light is simulated as being projected into the three-dimensional environment (such as in the form of reflections 7084 and 7086) is based on the size of the corresponding content being played on the television 7076 (e.g., based on the size of the television 7076). If the television 7076 is wider and taller, the light spill 7082 and the light spill 7088 will also be wider and taller, and optionally, the light will be projected further into the three-dimensional environment, such that the reflections 7084 and 7086 will be brighter and the reflection 7086 will be wider and extend further from the television 7076.

[0175] Reference below about 7A to 7H , FIG. 8A to FIG. 8J , 9A to 9E and FIG. 10A to FIG. 10E The described methods 1100, 1200, 1300 and 1400 provide information about 7A to 7H , FIG. 8A to FIG. 8J , 9A to 9E and FIG. 10A to FIG. 10E Additional description of .

[0176] Fig.11 is a flow chart of an exemplary method 1100 for displaying an attention-based shadow of a user interface object in an environment according to some embodiments. In some embodiments, the method 1100 is performed on a computer system (e.g., Figure 1 101) in the computer system, which is executed by the display generation component (e.g., Figure 1 , Figure 3 and Figure 4100) (e.g., hardware elements, including one or more display devices, such as a display, a touch screen, a projector, a head-up display, a head-mounted display, etc.) and one or more input devices (e.g., one or more cameras (e.g., color sensors, infrared sensors, structured light scanners, and / or other depth sensing cameras) pointing forward and downward from the user's head to the user's hands and / or facing the user; an eye tracking device; a controller held by the user and / or worn by the user; a touch-sensitive surface and / or other input hardware). In some embodiments, method 1100 is stored in a non-transitory (or transient) computer-readable storage medium and is executed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 1100 are optionally combined, and / or the order of some operations is optionally changed.

[0177] When a view of the three-dimensional environment is visible via a display generation component (e.g., including one or more computer-generated portions and optionally one or more pass-through portions), the computer system displays (1102) a first computer-generated user interface object (e.g., a computer-generated object that is not a view of a physical object in a field of view of one or more cameras of the computer system) in the three-dimensional environment (e.g., window 7010 ( Figure 7B ), Window 7012( Figure 7B ) or speaker 7034( Figure 7F )).

[0178] The computer system detects (1104) that a user's attention (e.g., gaze) is directed toward a first user interface object. In response to detecting that the user's attention is directed toward the first user interface object (e.g., and while continuing to detect that the user's attention is directed toward the first user interface object), the computer system displays (1106) a first virtual shadow (e.g., a computer-generated shadow) of the first user interface object in a first appearance (e.g., appears to be cast by the first user interface object onto a portion of an environment) while maintaining a pose (e.g., location and / or orientation) of the first user interface object relative to the three-dimensional environment (e.g., and optionally, independent of a pose of a light source in the environment). Displaying the first virtual shadow of the first user interface object in the first appearance includes displaying the first virtual shadow (e.g., shadow 7016 of window 7010 ( Figure 7B ), the shadow 7018 of the window 7012 ( Figure 7B ) or the shadow 7036 of the speaker 7034 ( Figure 7F )).

[0179] While continuing to display the first user interface object in the three-dimensional environment (optionally, at the same location relative to other real and / or virtual objects in the three-dimensional environment), the computer system detects (1108) that the user's attention has ceased to be directed toward the first user interface object (e.g., while displaying the first user interface object; in other words, the first user interface object remains in view but no longer has focus). In response to detecting that the user's attention has ceased to be directed toward the first user interface object, the computer system displays (1110) a first virtual shadow (e.g., a computer-generated shadow) of the first user interface object in a second appearance that is different from the first appearance, while maintaining the position (e.g., location and / or orientation) of the first user interface object relative to the three-dimensional environment (e.g., and optionally, independent of the position of the light source in the environment). Displaying the first virtual shadow of the first user interface object in the second appearance includes (1112) displaying the first virtual shadow in a second value for the first visual attribute. The second value (1114) is different from the first value. In some embodiments, before detecting that the user's attention is directed toward the first user interface object, the first virtual shadow of the first user interface object is displayed in the second appearance. For example, as described herein, when the gaze of the user 7002 is directed toward the window 7010 ( Fig. 7E ) and when the user 7002's gaze is not (e.g., has stopped) directed to the window 7010 ( Figure 7B or Figure 7G ), the shadow 7016 of window 7010 is displayed with a different opacity. Similarly, when the user 7002's gaze is directed toward window 7012 ( Fig.7D ) and when the user 7002's gaze is not (e.g., has stopped) directed to the window 7012 ( Figure 7B or Fig. 7E ), the shadow 7018 of the window 7012 is displayed with a different opacity. Similarly, when the user 7002's gaze is directed toward the speaker 7034 ( Figure 7G ) and when the user 7002's gaze is not (e.g., has stopped) directed toward the speaker 7034 ( Figure 7F ), the shadow 7036 of the speaker 7034 is displayed with a different opacity.

[0180] When a computer-generated shadow of a user interface object is displayed in a three-dimensional environment (which provides improved visual feedback about the spatial relationship between the user interface object and the three-dimensional environment), changing the value of a visual attribute of the shadow based on whether the user's attention is directed toward the user interface object indicates which part of the three-dimensional environment currently has focus for further interaction, which provides improved visual feedback about the state of the computer system.

[0181] In some embodiments, in response to detecting that the user's attention is directed to the first user interface object, the computer system displays the first virtual shadow in a first appearance while maintaining the size of the first user interface object (e.g., in one or more dimensions such as height, width, and / or depth); and in response to detecting that the user's attention has stopped being directed to the first user interface object, the computer system displays the first virtual shadow in a second appearance while maintaining the size of the first user interface object. Maintaining the size of the user interface object while changing the appearance of the computer-generated shadow of the user interface object balances providing improved visual feedback about the state of the computer system with reducing the likelihood that the user will experience motion sickness while using the computer system by not cluttering the user interface with too many changes.

[0182] In some embodiments, displaying a first virtual shadow of a first user interface object with a first appearance (e.g., in response to detecting that a user's attention is directed toward the first user interface object) is performed based on determining that the user's attention has been directed toward the first user interface object for at least a first threshold amount of time (e.g., the computer system abandons displaying the first virtual shadow with the first appearance until it detects that the user's attention has been directed toward the first user interface object for at least a first threshold amount of time) (e.g., as described herein with reference to Fig.7D In some embodiments, the first virtual shadow is not displayed in the first appearance (e.g., the first virtual shadow continues to be displayed in the previous appearance) until the user's attention is directed to the first user interface object for at least a first threshold amount of time. Requiring the user's attention to be directed to a user interface object for at least a threshold amount of time in order for the computer system to indicate that the user interface object has focus for further interaction enables the computer system to more accurately automatically discern user intent and thereby reduce input errors and unexpected results.

[0183] In some embodiments, detecting that the user's attention is directed to the first user interface object includes detecting that the user's gaze is directed to the first user interface object, and detecting that the user's attention has ceased to be directed to the first user interface object includes ceasing to detect that the user's gaze is directed to the first user interface object (e.g., by detecting that the user's gaze has moved away from the first user interface object or by ceasing to detect the user's gaze). Requiring that the user's gaze is directed to the user interface object in order for the computer system to indicate that the user interface object has focus for further interaction based on the user's attention being directed to the user interface object enables the computer system to more accurately automatically discern user intent and thereby reduce input errors and unexpected results.

[0184] In some embodiments, the first visual attribute is visual intensity (e.g., opacity, darkness, and / or other visual attributes). In some embodiments, when the first virtual shadow is displayed with a first appearance (e.g., in response to detecting that the user's attention is directed toward the first user interface object), the first virtual shadow is displayed with a first value for visual intensity (e.g., opacity, blur, feathering, darkness, and / or other visual attributes), and when the first virtual shadow is displayed with a second appearance (e.g., in response to detecting that the user's attention has stopped being directed toward the first user interface object), the first virtual shadow is displayed with a second value for visual intensity that is different from the first value for visual intensity (e.g., stronger or weaker than the first value for visual intensity). Changing the visual intensity of the shadow (e.g., by changing the value of a visual attribute such as opacity or darkness) based on whether the user's attention is directed toward the user interface object indicates which portion of the three-dimensional environment currently has focus for further interaction, which provides improved visual feedback about the state of the computer system.

[0185] In some embodiments, the computer system displays a first virtual shadow at least partially on a computer-generated surface in a three-dimensional environment (e.g., the first virtual shadow has a spatial relationship with the first user interface object, such as below the first user interface object). For example, shadow 7016 is displayed FIG. 7B to FIG. 7G Displaying the shadow of the user interface object on the computer-generated surface in the three-dimensional environment provides improved visual feedback about the spatial relationship between the user interface object and the three-dimensional environment.

[0186] In some embodiments, the computer system displays the first virtual shadow at least in part over a see-through view of a physical surface in the physical environment that corresponds to the three-dimensional environment (e.g., the physical surface corresponds to a simulated location in the three-dimensional environment below the simulated location of the first user interface object). For example, shadow 7018 is displayed FIG. 7B to FIG. 7G Displaying shadows of user interface objects in a three-dimensional environment on a see-through view of a physical surface in a corresponding physical environment provides improved visual feedback about the spatial relationship between the user interface objects, the three-dimensional environment, and the corresponding physical environment.

[0187] In some embodiments, the first virtual shadow displayed at least partially on the see-through view of the physical surface follows the three-dimensional model of the physical surface. Figure 7C As described, based on the physical box 7014 ( Fig. 7A), shadows 7018 are displayed on the top surface and down the sides of the see-through box 7014'. In some embodiments, the three-dimensional model of the physical surface is determined and / or generated based on depth data about the physical surface and / or the physical environment captured (e.g., via one or more cameras, time-of-flight sensors, or other depth sensors of the computer system). Displaying the shadow of the user interface object in the three-dimensional environment according to the three-dimensional model or mesh of the physical surface in the corresponding physical environment provides improved visual feedback about the spatial relationship between the user interface object, the three-dimensional environment, and the corresponding physical environment.

[0188] In some embodiments, the pass-through view of the physical surface has a first degree of fidelity to the physical surface; the three-dimensional model of the physical surface has a second degree of fidelity to the physical surface; and the second degree is less than the first degree. In some embodiments, the three-dimensional model of the physical surface has a degree of fidelity to the physical surface that smoothes the physical surface (e.g., so as to ignore small bumps and / or thin objects on the physical surface). For example, as described herein with reference to Figure 7C As described, displaying shadows 7018 along the top and sides of the transparent box 7014' is optionally based on the physical box 7014 ( Fig. 7A ) that smooths or ignores irregularities in the physical surface of box 7014. Displaying shading of user interface objects in a three-dimensional environment based on a three-dimensional model or mesh having lower fidelity to the physical surface than an optical view of the physical surface (e.g., a simplified three-dimensional model that smooths or ignores smaller irregularities in the physical surface and / or thin objects on the physical surface) reduces the computational load imposed on the computer system while providing improved visual feedback about the spatial relationship between the user interface objects, the three-dimensional environment, and the corresponding physical environment.

[0189] In some embodiments, the computer system displays a second computer-generated user interface object and a second virtual shadow of the second user interface object (e.g., window 7010 and shadow 7016 ( Figure 7B ), window 7012 and shadow 7018 ( Figure 7B ) or speaker 7034 and shadow 7036 ( Figure 7F )). Displaying computer-generated shadows of multiple computer-generated user interface objects in a three-dimensional environment provides improved visual feedback about the spatial relationship between the various user interface objects and the three-dimensional environment.

[0190] In some embodiments, while displaying a first virtual shadow of a first user interface object in a first appearance (e.g., in response to detecting that a user's attention is directed toward the first user interface object), a computer system displays a second virtual shadow of a second user interface object in a third appearance, including displaying the second virtual shadow at a third value for a first visual attribute, wherein the third value is different from the first value. In some embodiments, while continuing to display the second user interface object in a three-dimensional environment, the computer system detects that the user's attention is directed toward the second user interface object after ceasing to be directed toward the first user interface object (e.g., the user's attention moves from the first user interface object to the second user interface object). In some embodiments, in response to detecting that the user's attention is directed toward the second user interface object, the computer system displays the second virtual shadow of the second user interface object in a fourth appearance that is different from the third appearance, while maintaining the position of the second user interface object relative to the three-dimensional environment (e.g., and optionally, independent of the position of the light source in the environment). In some embodiments, displaying the second virtual shadow of the second user interface object in a fourth appearance includes displaying the second virtual shadow at a fourth value for the first visual attribute; and the fourth value is different from the third value.

[0191] In some embodiments, the third value is the same as the second value (e.g., with respect to the first visual attribute, the appearance of the first virtual shadow when the user is not directing attention to the first user interface object is the same as the appearance of the second virtual shadow when the user is not directing attention to the second user interface object). In some embodiments, the fourth value is the same as the first value (e.g., with respect to the first visual attribute, the appearance of the first virtual shadow when the user is directing attention to the first user interface object is the same as the appearance of the second virtual shadow when the user is directing attention to the second user interface object). In some embodiments, when the user's attention has stopped being directed to the second user interface object, the second virtual shadow of the second user interface object exhibits similar behavior to the first virtual shadow of the first user interface object (e.g., the value of the first visual attribute of the second virtual shadow changes). For example, as described herein with reference FIG. 7D to FIG. 7E As described, when user 7002's attention moves from window 7012 to window 7010, the appearance of shadow 7018 of window 7012 changes in one way, while the appearance of shadow 7016 of window 7010 changes in the opposite way (optionally, shadow 7018 and shadow 7016 swap appearances). In another example, as described herein with reference to FIG. 7F to FIG. 7GAs described, when user 7002's attention moves from window 7010 to speaker 7034, the appearance of shadow 7016 of window 7010 changes in one way, while the appearance of shadow 7036 of speaker 7034 changes in the opposite way (optionally, shadow 7016 and shadow 7036 swap appearances). The values ​​of the visual properties of the shadows of different user interface objects in three dimensions also change (e.g., in the opposite way) based on which user interface object the user's attention is directed to (e.g., particularly when the user's attention moves from one user interface object to another user interface object), while the values ​​of the visual properties of the shadow of one user interface object change (e.g., in one way), indicating which part of the three-dimensional environment currently has (and which part does not have) the focus for further interaction, which provides improved visual feedback about the state of the computer system.

[0192] In some embodiments, when the first virtual shadow is displayed at the second value for the first visual attribute, the first virtual shadow has a lower visual intensity (e.g., lower opacity, less dark, more blurry, more feathered, or other difference in the visual attribute) than when the first virtual shadow is displayed at the first value for the first visual attribute (e.g., the visual intensity of the first virtual shadow decreases when the user's attention moves away from the first user interface object). In some embodiments, when the second virtual shadow is displayed at the fourth value for the first visual attribute, the second virtual shadow has a greater visual intensity than when the second virtual shadow is displayed at the third value for the first visual attribute (e.g., the visual intensity of the second virtual shadow increases when the user's attention moves to the second user interface object). For example, as described herein with reference to FIG. 7D to FIG. 7E As described, when the user 7002's attention moves from window 7012 to window 7010, the opacity of the shadow 7018 of window 7012 decreases, while the opacity of the shadow 7016 of window 7010 increases. In another example, as described herein with reference to FIG. 7F to FIG. 7G As described, when the user 7002's attention moves from window 7010 to speaker 7034, the opacity of the shadow 7016 of window 7010 decreases, while the opacity of the shadow 7036 of speaker 7034 increases. Changing the value of the visual attribute of the shadow of the user interface object to which the user's attention is directed so as to increase the visual intensity of the shadow (e.g., while the shadows of other user interface objects to which the user's attention is not directed have a reduced visual intensity in comparison) indicates which part of the three-dimensional environment currently has (and which part does not have) focus for further interaction, which provides improved visual feedback about the state of the computer system.

[0193] In some embodiments, a computer system displays a third computer-generated user interface object and a third virtual shadow of the third user interface object, including: when the user's attention is directed to the first user interface object (e.g., not directed to the third user interface object), displaying the third virtual shadow with a fifth value for the first visual attribute (e.g., while maintaining the position of the third user interface object relative to the three-dimensional environment); and when the user's attention is directed to the second user interface object after stopping pointing to the first user interface object (e.g., the user's attention is not directed to the third user interface object), displaying the third virtual shadow with a fifth value for the first visual attribute (e.g., while maintaining the position of the third user interface object relative to the three-dimensional environment). For example, as described herein with reference to FIG. 7F to FIG. 7G As described, the appearance (e.g., opacity) of shadow 7018 of window 7012 remains unchanged as user 7002's attention moves from window 7010 to speaker 7034. Maintaining the value of the visual attribute of the shadow of a user interface object that is distinct from the two user interface objects as the user's attention moves between the two user interface objects indicates which portion of the three-dimensional environment does not have focus for further interaction (and has neither received nor lost focus), which provides improved visual feedback about the state of the computer system.

[0194] In some embodiments, when the user's attention is not directed to the first user interface object, the computer system displays the first user interface object with a first value for the corresponding visual attribute of the first user interface object (optionally the same as the first visual attribute of the first virtual shadow). In some embodiments, in response to detecting that the user's attention is directed to the first user interface object, the computer system displays the first user interface object with a second value for the corresponding visual attribute of the first user interface object (e.g., by increasing the opacity of the first user interface object when the user's attention is directed to the first user interface object), while maintaining the posture of the first user interface object relative to the three-dimensional environment, wherein the second value for the corresponding visual attribute is different from the first value for the corresponding visual attribute. For example, as described herein with reference to Fig.7D As described, in response to user 7002 directing attention to window 7012, window 7012 is displayed with a visually emphasized appearance that optionally adds a selection outline and / or increases the opacity of window 7012. Window 7010 ( Fig. 7E ) and / or speaker 7034( Figure 7G) optionally exhibits similar behavior when being the target of user 7002's attention. In some embodiments, other user interface objects exhibit similar behavior to the first user interface object. For example, when the user's attention is not directed to the second user interface object, the second user interface object is displayed with a value for the corresponding visual attribute of the second user interface object that is different from when the user's attention is directed to the second user interface object. In another example, when the user's attention is directed to the first user interface object and when the user's attention is directed to the second user interface object after ceasing to be directed to the first user interface object, the third user interface object is displayed with the same value for the corresponding visual attribute of the third user interface object. In combination with changing the value of the visual attribute of the shadow of the user interface object based on whether the user's attention is directed to the user interface object, changing the value of the visual attribute of the user interface object itself (e.g., a visual attribute that is the same as or different from the shadow) further indicates which part of the three-dimensional environment currently has focus for further interaction, which provides improved visual feedback about the state of the computer system.

[0195] In some embodiments, in response to detecting that the user's attention has stopped being directed toward the first user interface object, the computer system causes the first visual attribute of the first virtual shadow to gradually progress through a plurality of intermediate values ​​between the first value and the second value (e.g., causes the first virtual shadow to progress through a plurality of intermediate display states corresponding to the changing values ​​of the first visual attribute) (e.g., as described herein with reference to Fig.7D Shadows in 7018, Fig. 7E The shadows in 7016 and Figure 7G For shadows whose appearance changes (e.g., in response to a user directing their attention toward or away from a corresponding object), changing the appearance of the shadow gradually balances providing improved visual feedback about the state of the computer system with reducing the likelihood that the user will experience motion sickness while using the computer system by not changing the user interface too abruptly.

[0196] In some embodiments, in response to detecting that the user's attention is directed toward the first user interface object, the computer system delays displaying the first virtual shadow of the first user interface object in the first appearance until a second threshold amount of time has passed since detecting that the user's attention is directed toward the first user interface object (e.g., a non-zero delay period of 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 3 seconds, or other time period) (e.g., as described herein with reference to Fig.7D Shadows in 7018, Fig. 7E The shadows in 7016 and Figure 7GIn some embodiments, the first virtual shadow of the first user interface object is displayed in the first appearance based on determining that a second threshold amount of time has passed since detecting that the user's attention is directed toward the first user interface object. In some embodiments, the first virtual shadow is displayed in the first appearance based on determining that a second threshold amount of time has passed since one or more other preconditions for displaying the first virtual shadow in the first appearance are satisfied (for example, the user's attention remains directed toward the first user interface object for at least the first threshold amount of time).

[0197] In some embodiments, in response to detecting that the user's attention has stopped being directed toward the first user interface object, the computer system delays displaying the first virtual shadow of the first user interface object in the second appearance until a second threshold amount of time has passed since detecting that the user's attention has stopped being directed toward the first user interface object (e.g., as described herein with reference to Fig. 7E The shadows in 7018 and Figure 7G In some embodiments, the first virtual shadow of the first user interface object is displayed with the second appearance based on determining that a second threshold amount of time has passed since detecting that the user's attention has stopped being directed toward the first user interface object (optionally at least the threshold amount of time). Delaying changing the appearance of the shadow until the threshold amount of time has passed in response to the user directing their attention toward or away from the corresponding object enables the computer system to more accurately automatically discern user intent and adds lag that reduces flicker in the user interface that would increase the likelihood that the user will experience motion sickness while using the computer system.

[0198] In some embodiments, displaying the first virtual shadow of the first user interface object is performed based on determining that the user's viewpoint is within a viewpoint range relative to the first user interface object. In some embodiments, the computer system detects that the user's viewpoint has moved beyond the viewpoint range relative to the first user interface object, and in response to detecting that the user's viewpoint has moved beyond the viewpoint range relative to the first user interface object, stops displaying the first virtual shadow of the first user interface object (e.g., while displaying the first user interface object based on the user's current viewpoint that is outside the viewpoint range) (e.g., as described herein with reference to Figure 7H Stopping displaying shadows of user interface objects that are being viewed from outside a particular viewpoint range (e.g., outside a viewing angle range) in a three-dimensional environment (such as from the side or from behind) visually deemphasizes objects that are not being viewed directly, which provides improved visual feedback about the spatial relationship between the user interface objects and the three-dimensional environment.

[0199] In some embodiments, the computer system displays the first virtual shadow of the first user interface object without displaying a visual effect corresponding to simulated light emitted from content displayed in the first user interface object, and stops displaying the first virtual shadow of the first user interface object while displaying the visual effect corresponding to simulated light emitted from the content. In some embodiments, the computer system stops displaying the first virtual shadow of the first user interface object based on determining that the visual effect corresponding to simulated light emitted from content displayed in the first user interface object is displayed or in response to the same input or event that caused the display of the visual effect. For example, as described herein with reference to Fig. 10B and Fig. 10D As described, when the computer system 101 begins to display the bloom visual effect 7082 or 7088, the shadow 7078 of the television 7076 and optionally Fig. 10A Other computer-generated shadows in the scene of the user interface object cease to be displayed. Displaying a visual effect corresponding to simulated light emanating from content displayed in a user interface object is described in greater detail herein with reference to method 1400. Ceasing to display a shadow of a user interface object in a three-dimensional environment when beginning to display a light spill visual effect simulating light emanating from content displayed in the user interface object (both of which provide visual feedback regarding the spatial relationship between the user interface object and the three-dimensional environment) reduces the computational burden of providing some visual feedback when other visual feedback serving a similar purpose is available.

[0200] In some embodiments, the computer system displays a visual effect corresponding to simulated light emitted from the content while playback of the content is ongoing (and optionally based on determining that playback of the content is ongoing), and forgoes displaying a visual effect corresponding to simulated light emitted from the content when playback of the content is not ongoing (e.g., paused or terminated) (and optionally based on determining that playback of the content is not ongoing). Fig. 10B and Fig. 10D As described, the light bloom visual effects 7082 and 7088 are respectively displayed when the content on the television 7076 is being played, and are not displayed when the content on the television 7076 is not being played, as described herein with reference to Fig. 10A Displaying a light bloom visual effect that simulates light emanating from content displayed in a user interface object while content (e.g., video content) is playing and not displaying the light bloom visual effect when the content is not playing provides improved visual feedback about the spatial relationship between the user interface object displaying the content and a three-dimensional environment while the content is visually salient and more likely to have the user's attention, and reduces the computational burden of generating the light bloom visual effect at other times.

[0201] In some embodiments, the first user interface object is (or includes) a user interface of an application (e.g., window 7010 ( Figure 7B ) or window 7012( Figure 7B )). Changing the value of a visual attribute of a shadow of an application user interface displayed in a three-dimensional environment based on whether the user's attention is directed toward the application user interface indicates which part of the three-dimensional environment currently has focus for further interaction, which provides improved visual feedback about the status of the computer system.

[0202] In some embodiments, the first user interface object is a three-dimensional object in a three-dimensional environment (e.g., speaker 7034 ( Figure 7F ), which optionally also represents a software application or a user interface of the software application). Changing the value of a visual attribute of a shadow of a three-dimensional user interface object displayed in a three-dimensional environment based on whether the user's attention is directed toward the three-dimensional object indicates which part of the three-dimensional environment currently has focus for further interaction, which provides improved visual feedback about the state of the computer system.

[0203] In some embodiments, displaying a first virtual shadow of a first user interface object with a first appearance includes displaying the first virtual shadow with a first value for a second visual attribute; and maintaining a pose of the first user interface object relative to the three-dimensional environment includes displaying the first user interface object at a first position in the three-dimensional environment and at a first distance from a surface in the three-dimensional environment that corresponds to the first position (e.g., a surface that has a spatial relationship with the first user interface object at the first position (such as below the first user interface object)). In some embodiments, the computer system detects (e.g., optionally when a user's attention is directed toward the first user interface object) input corresponding to a request to reposition the first user interface object. In some embodiments, input corresponding to a request to reposition a first user interface object includes input directed to the first user interface object, such as having a first portion that enables and / or initiates repositioning of the first user interface object (e.g., a press or long press input via a touch-sensitive surface while a cursor or other focus indicator corresponds to the first user interface object, input that activates a control for repositioning the first user interface object (e.g., as described herein with reference to method 1300), or other means of initiating repositioning), and a second portion that includes movement of the input for repositioning the first user interface object (e.g., movement of a contact along the touch-sensitive surface, movement of a hand performing an air gesture, movement of the field of view of the computer system while continuing to detect input, or other input movement).

[0204] In some embodiments, in response to detecting input corresponding to a request to reposition a first user interface object, a computer system: repositions the first user interface object to a second position in the three-dimensional environment that is a second distance from a surface in the three-dimensional environment that corresponds to the second position (e.g., a surface that has a spatial relationship with the first user interface object at the second position (such as below the first user interface object), and optionally is the same or a different surface as before the first user interface object was repositioned from the first position); and, based on determining that the second distance is different from the first distance (and optionally is less than a threshold distance, as described herein with reference to method 1300), displays a first virtual shadow of the first user interface object for a second value of a second visual attribute, where the second value is different from the first value. For example, as described herein with reference to Figure 8J As described, as the window 7010 moves further away from the top surface of the table 7024 below the window 7010, the blur and / or feathering of the shadow 7016 of the window 7010 changes. Figure 8J When the table 7024 is shown in a higher position (e.g., whether it has an opacity corresponding to motion input directed to the window 7010, an opacity corresponding to the attention of the user 7002 directed to the window 7010, or an opacity corresponding to the user 7002 not directing attention to the window 7010 and not interacting with the window, provided that the window 7010 is within a threshold distance above the table 7024 at which the shadow 7016 begins to fade), the shadow 7016 has an opacity corresponding to the Figure 8F In some embodiments, based on determining that the second distance is the same as the first distance (and optionally less than a threshold distance), the computer system displays a first virtual shadow of the first user interface object with a first value for a second visual attribute. Changing the values ​​of different visual attributes of the shadow of a user interface object, such as by changing one visual attribute based on whether the user's attention is directed toward the user interface object and changing different visual attributes based on the distance between the user interface object and a corresponding surface (e.g., below the user interface object), enables providing more complex visual feedback that indicates which portion of the three-dimensional environment currently has focus for further interaction and indicates the spatial relationship between the user interface object and the three-dimensional environment.

[0205] In some embodiments, the first visual attribute is the opacity of the first virtual shadow, and the second visual attribute is the size (e.g., length and / or area) and / or blurriness (e.g., as described herein with reference to Figure 8JChanging the opacity of a shadow of a user interface object based on whether the user's attention is directed toward the user interface object and changing the size and / or blurriness of the shadow based on the distance between the user interface object and a corresponding surface (e.g., beneath the user interface object) enables providing more complex visual feedback that indicates which portion of the three-dimensional environment currently has focus for further interaction and indicates the spatial relationship between the user interface object and the three-dimensional environment.

[0206] In some embodiments, aspects / operations of methods 1200, 1300, and 1400 may be interchanged, replaced, and / or added between these methods. For example, the computer-generated objects and virtual shadows of method 1100 optionally also exhibit the movement-based behaviors of method 1200, and / or the virtual shadows of method 1100 are simplified shadows as described for method 1300. For the sake of brevity, these details are not repeated here.

[0207] Fig.12 is a flow chart of an exemplary method 1200 for displaying a motion-based shadow of a user interface object in an environment according to some embodiments. In some embodiments, the method 1200 is performed on a computer system (e.g., Figure 1 101) in the computer system, which is executed by the display generation component (e.g., Figure 1 , Figure 3 and Figure 4 ) and one or more input devices (e.g., one or more cameras (e.g., color sensors, infrared sensors, structured light scanners, and / or other depth sensing cameras) pointing forward and downward from the user's head to the user's hands and / or facing the user; eye tracking devices; controllers held by the user and / or worn by the user; touch-sensitive surfaces and / or other input hardware). In some embodiments, method 1200 is performed by storing in a non-transitory (or transient) computer-readable storage medium and by one or more processors of a computer system (e.g., one or more processors 202 of computer system 101) (e.g., Figure 1 Some operations in method 1200 are optionally combined, and / or the order of some operations is optionally changed.

[0208] When a view of an environment is visible via a display generating component (e.g., the environment is a two-dimensional or three-dimensional environment that includes one or more computer-generated portions and, optionally, one or more transparent portions), the computer system displays (1202) a first user interface object at a first location in the environment and a virtual shadow (e.g., a computer-generated shadow) corresponding to the first user interface object (e.g., that appears to be cast by the first user interface object onto a portion of the environment), including displaying the virtual shadow at a first value for a first visual attribute (e.g., shadow intensity or opacity). For example, shadow 7016 of window 7010 at Fig. 8A and Fig. 8E Displayed at first opacity.

[0209] The computer system detects (1204) input corresponding to a request to reposition a first user interface object from a first location in an environment to a second location in the environment that is different from the first location. In response to detecting the input (1206): the computer system repositions (1208) the first user interface object from the first location in the environment to the second location in the environment based on movement of the input relative to the environment (e.g., moves the first user interface object through a plurality of intermediate locations between the first location and the second location); and when repositioning the first user interface object from the first location in the environment to the second location in the environment, the computer system displays (1210) a virtual shadow corresponding to the first user interface object with a second value for a first visual attribute (e.g., changes the virtual shadow from having a first value for the first visual attribute to having a second value for the first visual attribute). For example, when window 7010 moves in the three-dimensional environment, in Fig.8D and Figure 8G In some embodiments, a shadow 7016 of window 7010 is displayed at a second opacity relative to the first opacity when the input is detected and corresponding to the first user interface object (e.g., regardless of whether the first user interface object is repositioned). For example, when an input for repositioning window 7010 is detected before window 7010 is moved in a three-dimensional environment, a shadow 7016 of window 7010 is displayed at a second opacity relative to the first opacity. In some embodiments, a virtual shadow corresponding to the first user interface object is displayed at a second value for the first visual attribute when the input is detected (e.g., regardless of whether the first user interface object is repositioned). Figure 8C and Figure 8F Shadow 7016 of window 7010 is displayed at the second opacity.

[0210] When a computer-generated shadow of a user interface object is displayed in a three-dimensional environment (which provides improved visual feedback about the spatial relationship between the user interface object and the three-dimensional environment), changing the values ​​of visual attributes of the shadow based on whether input for repositioning the user interface object is detected (e.g., whether the user interface object is actually repositioned) indicates which portion of the three-dimensional environment is currently the target of user interaction and the type of interaction that is occurring, which provides improved visual feedback about the state of the computer system.

[0211] In some embodiments, in response to detecting the input, the computer system repositions the virtual shadow corresponding to the first user interface object (e.g., in the image above) according to repositioning the first user interface object from a first location to a second location (e.g., according to movement of the input relative to the environment). FIG. 8A to FIG. 8J , as window 7010 is repositioned, shadow 7016 is repositioned accordingly). When a user interface object is repositioned in a three-dimensional environment, the corresponding repositioning of the user interface object's shadow provides improved visual feedback regarding the spatial relationship between the user interface object and the three-dimensional environment.

[0212] In some embodiments, in response to detecting an input: based on determining that the input includes a first amount of movement in a first direction relative to the environment, the computer system: repositions the first user interface object (e.g., and the corresponding virtual shadow) by a second amount corresponding to the first amount and in a second direction corresponding to the first direction, and when the first user interface object is repositioned by the second amount in the second direction, displays the virtual shadow corresponding to the first user interface object with a second value for the first visual attribute. For example, as described herein with reference to Fig.8D As described, when the window 7010 and the shadow 7016 are FIG. 8C to FIG. 8D When the hand 7022 moves to the left in the three-dimensional environment, FIG. 8C to FIG. 8D 7016 of window 7010 is displayed at a second opacity. In some embodiments, in response to detecting an input: based on determining that the input includes a third amount of movement in a third direction relative to the environment, wherein the third amount is different from the first amount and / or the third direction is different from the first direction, the computer system: repositions the first user interface object (e.g., and a corresponding virtual shadow) by a fourth amount corresponding to the third amount and in a fourth direction corresponding to the third direction, wherein when the third amount is different from the first amount, the fourth amount is different from the second amount, and when the third direction is different from the first direction, the fourth direction is different from the second direction, and when the first user interface object is repositioned by the fourth amount in the fourth direction, displays a virtual shadow corresponding to the first user interface object at the second value for the first visual attribute. For example, as described herein with reference to Fig.8D As described, when the window 7010 and the shadow 7016 are FIG. 8F to FIG. 8G When the magnitude and direction of the movement of the hand 7022 input in the three-dimensional environment moves further away from the viewpoint 7046 of the user 7002, FIG. 8F to FIG. 8G China-Israel FIG. 8C to FIG. 8D 7016 of the same second opacity display window 7010 in the three-dimensional environment. When a user interface object and the corresponding shadow are repositioned in the three-dimensional environment based on the direction and / or magnitude of movement of an input used to reposition the user interface object, changing the value of the visual attribute of the corresponding shadow based on the fact that the repositioning input is detected without regard to the direction and / or magnitude of movement of the repositioning input provides improved visual feedback regarding the spatial relationship between the user interface object and the three-dimensional environment, which visual feedback indicates in a consistent manner which portion of the three-dimensional environment is currently the target of user interaction and the type of interaction that is occurring.

[0213] In some embodiments, in response to detecting the input, before repositioning the first user interface object, the computer system displays a virtual shadow corresponding to the first user interface object with a third value for the first visual attribute, where the third value is different from the first value (e.g., changing the virtual shadow from having the first value for the first visual attribute to having the third value for the first visual attribute even before repositioning the first user interface object). For example, as described herein with reference to Figure 8C As described above, in response to an input for initiating movement of window 7010 indicated by movement indicator 7030-2 (eg, even if Figure 8C The window 7010 in has not yet been moved in the three-dimensional environment (e.g., Figure 8C The windows 7010 is still working with Fig. 8A In some embodiments, the third value is the same as the second value. In some embodiments, the third value is different from the second value. Changing the value of the visual attribute of the shadow based on whether an input for repositioning a user interface object is detected even before the user interface object is actually repositioned indicates which part of the three-dimensional environment is currently the target of user interaction and the type of interaction that is occurring, which provides improved visual feedback about the state of the computer system.

[0214] In some embodiments, the computer system stops detecting the input (e.g., detects the end of the input); and in response to stopping detecting the input, the computer system displays a virtual shadow corresponding to the first user interface object with a fourth value for the first visual attribute, wherein the fourth value is different from the second value. Fig. 8E As described, in response to detecting an input for repositioning window 7010 ( FIG. 8C to FIG. 8D ), the appearance of the shadow 7016 of the window 7010 is changed (e.g., from FIG. 8C to FIG. 8DThe appearance of the shadow in 7016 is restored to Fig. 8A In some embodiments, the fourth value is the same as the first value (e.g., the change in the first visual attribute of the virtual shadow from the first value to the second value is reversed). In some embodiments, the first value and the fourth value are both greater than the second value or both less than the second value. In the case of changing the value of the visual attribute of the shadow when detecting input for repositioning the user interface object, at least partially reversing the change when the input ends indicates which portion of the three-dimensional environment has ceased to be the target of user interaction and, optionally, indicates the type of interaction that has occurred, which provides improved visual feedback about the state of the computer system.

[0215] In some embodiments, detecting input corresponding to a request to reposition the first user interface object includes detecting an air gesture (e.g., such as an air pinch gesture, an air pinch-drag gesture, or other air gesture performed with one or both of the user's hands) (e.g., as described herein with reference to FIG. 8C to FIG. 8D Changing the value of the visual attribute of the shadow based on whether an in-air gesture for repositioning a user interface object is detected indicates which portion of the three-dimensional environment is currently the target of a user interaction and the type of interaction that is occurring (this is particularly helpful for in-air gestures that are often performed spatially separated from their target user interface objects, making it more difficult to tell which object is being manipulated (e.g., compared to touch input that directly manipulates a user interface object displayed on a touch screen)), which provides improved visual feedback about the state of the computer system and reduces the amount of time required to perform interactions using the computer system.

[0216] In some embodiments, before displaying the first user interface object and the virtual shadow with the first value for the first visual attribute: the computer system displays the first user interface object; when the user's attention is not directed to the first user interface object, the computer system displays the virtual shadow corresponding to the first user interface object with a fifth value for the first visual attribute, wherein the fifth value is different from the first value; and the computer system detects that the user's attention is directed to the first user interface object. In some embodiments, displaying the virtual shadow with the first value for the first visual attribute is performed in response to detecting that the user's attention is directed to the user interface object. For example, as described herein with reference to FIG. 8A to FIG. 8J As described, shadow 7016 optionally exhibits any combination of the properties and behaviors of shadows described herein with reference to method 1100, including, for example, Fig. 8A and Fig. 8EAn attention-based appearance is displayed. In some embodiments, the fifth value and the second value are both less than the first value or both greater than the second value. In some embodiments, the fifth value is different from the second value. In some embodiments, the fifth value is the same as the second value. Before changing the value of the visual attribute of the shadow based on whether input for repositioning the user interface object is detected, changing the value of the visual attribute of the shadow based on whether the user's attention is directed to the user interface object indicates which part of the three-dimensional environment currently has the focus for subsequent repositioning interaction, which provides improved visual feedback about the state of the computer system.

[0217] In some embodiments, when a first user interface object is relocated from a first location to a second location in the environment, the computer system maintains the appearance of the virtual shadow corresponding to the first user interface object regardless of whether the user's attention is directed toward the first user interface object (e.g., even if the computer system detects that the user's attention has moved to the first user interface object or the user's attention has moved away from the first user interface object, the computer system maintains the appearance of the virtual shadow corresponding to the first user interface object). FIG. 8A to FIG. 8J The shadow 7016 in the embodiment optionally exhibits any combination of the properties and behaviors of the shadow described herein with reference to method 1100, including, for example, Fig. 8A and Fig. 8E The attention-based appearance of the shadow 7016 is displayed, but the appearance of the shadow 7016 is optionally not changed based on whether the user's attention is directed to the shadow 7016 while the input for moving the corresponding window 7010 is in progress, as described herein with reference to Fig.8D Changing the value of a visual attribute of a shadow based on factors other than whether a user's attention is directed toward the corresponding user interface object when detecting input for repositioning the user interface object (e.g., whether the user interface object is actually repositioned) gives priority to indicating which portion of the three-dimensional environment is currently the target of a user interaction and the type of interaction that is occurring (e.g., over indicating which portion of the three-dimensional environment has focus for a subsequent interaction), which provides improved visual feedback about the state of a computer system.

[0218] In some embodiments, before detecting the input, the computer system displays the first user interface object with a first value for the corresponding visual attribute of the first user interface object. In some embodiments, when the first user interface object is relocated from the first location in the environment to the second location in the environment, the computer system displays the first user interface object with a second value for the corresponding visual attribute of the first user interface object, wherein the second value for the corresponding visual attribute is different from the first value for the corresponding visual attribute. For example, as described herein with reference to Figure 8C and Figure 8FAs described, the opacity of window 7010 is reduced during input for moving window 7010 to help user 7002 see how window 7010 appears relative to the three-dimensional environment when window 7010 is moved. In some embodiments, the visual attribute of the first user interface object is the same as the first visual attribute or the second visual attribute of the virtual shadow (for example, the visual attribute of the first user interface object is opacity, size and / or blurriness). In some embodiments, after stopping detecting the input, the first user interface object is displayed with a third value for the corresponding visual attribute of the first user interface object, which is optionally the same as the first value. In combination with changing the value of the visual attribute of the shadow of the user interface object when detecting input for repositioning the user interface object, changing the value of the visual attribute of the user interface object itself (for example, a visual attribute that is the same or different from the shadow) further indicates which part of the three-dimensional environment is currently the target of user interaction and the type of interaction that is occurring, and (for example, if the change in the value of the visual attribute of the user interface object increases the transparency of the user interface object) makes it easier for the user to see where the user interface object is moving in the three-dimensional environment, which provides improved visual feedback about the state of the computer system.

[0219] In some embodiments, a virtual shadow corresponding to a first user interface object is displayed on a surface in an environment (e.g., a surface below the first user interface object, as viewed from a user's viewpoint); repositioning the first user interface object from a first position to a second position in the environment changes the distance between the first user interface object and the surface; and before detecting the input, the virtual shadow displayed at the first value for the first visual attribute is displayed at the first value for the second visual attribute. In some embodiments, in response to detecting the input, the computer system displays the virtual shadow corresponding to the first user interface object at the second value for the second visual attribute based on the change in the distance between the first user interface object and the surface (e.g., changing the second visual attribute of the shadow through multiple intermediate values ​​as the first user interface object moves through multiple intermediate positions between the first position and the second position), while continuing to display the virtual shadow at the second value for the first visual attribute. For example, as described herein with reference to Figure 8J As described, as the window 7010 moves further away from the top surface of the desk 7024 below the window 7010, the degree of blur and / or feathering of the shadow 7016 of the window 7010 changes (e.g., rather than a change in opacity).

[0220] In some embodiments, displaying the virtual shadow at the second value for the second visual attribute is independent of lighting conditions in the environment (e.g., a change from a first value for the second visual attribute to a second value for the second visual attribute is different from a change that would occur based on one or more light sources in the environment (such as a change in the size or length of a shadow as an object casting a shadow moves toward or away from a light source)). In some embodiments, after ceasing to detect input (e.g., in response to ceasing to detect input), the computer system continues to display the virtual shadow at the second value for the second visual attribute based on the object having been repositioned from the first position to the second position (e.g., compared to changing the value of the first visual attribute from the second value to a different value (such as back to the first value) when ceasing to detect input). Changing the values ​​of different visual attributes of a shadow corresponding to a user interface object, such as by changing one visual attribute based on whether input for repositioning the user interface object is detected and changing different visual attributes based on a distance between the user interface object and a corresponding surface (e.g., beneath the user interface object), enables providing more complex visual feedback that indicates which portion of the three-dimensional environment is currently the target of user interaction and indicates the spatial relationship between the user interface object and the three-dimensional environment.

[0221] In some embodiments, the first visual attribute is the opacity of the virtual shadow, and the second visual attribute is the size (e.g., length and / or area) and / or blurriness of the virtual shadow (e.g., as described herein with reference to Figure 8J Changing the opacity of a shadow corresponding to a user interface object based on whether input for repositioning the user interface object is detected and changing the size and / or blurriness of the shadow based on the distance between the user interface object and a corresponding surface (e.g., beneath the user interface object) enables providing more complex visual feedback that indicates which portion of the three-dimensional environment is currently the target of user interaction and indicates the spatial relationship between the user interface object and the three-dimensional environment.

[0222] In some embodiments, displaying a virtual shadow at a first value for a first visual attribute and displaying a virtual shadow at a second value for the first visual attribute are performed when the environment has a first set of lighting conditions (e.g., based on real and / or virtual lighting). In some embodiments, when the environment has a second set of lighting conditions that are different from the first set of lighting conditions, the computer system: displays a first user interface object and a virtual shadow corresponding to the first user interface object in the environment, including displaying the virtual shadow at the first value for the first visual attribute; detects a second input corresponding to a request to reposition the first user interface object from a first position in the environment to a second position in the environment; and in response to detecting the second input: repositions the first user interface object from the first position in the environment to the second position in the environment based on movement of the second input relative to the environment; and when the first user interface object is repositioned from the first position in the environment to the second position in the environment, displays the virtual shadow corresponding to the first user interface object at the second value for the first visual attribute. For example, as referenced herein: Figure 8C As described, shadow 7016, optionally changes from a first appearance to a second appearance in response to detecting an input that initiates movement of window 7010, regardless of whether the three-dimensional environment is illuminated by light sources 7038, 7040, or both.

[0223] More generally, a change in a first visual property of a virtual shadow corresponding to a first user interface object from a first value to a second value is the same regardless of whether the environment has the first set of lighting conditions or the second set of lighting conditions, optionally so long as both the first set of lighting conditions and the second set of lighting conditions provide a threshold amount of light in the environment. In the case where the value of a visual property of a shadow of a user interface object in a three-dimensional environment is changed based on whether input for repositioning the user interface object is detected, displaying the same change in the visual property of the shadow regardless of what lighting conditions are present in the three-dimensional environment indicates in a consistent manner which portion of the three-dimensional environment is currently the target of a user interaction and the type of interaction that is occurring, which provides improved visual feedback about the state of the computer system.

[0224] In some embodiments, aspects / operations of methods 1100, 1300, and 1400 may be interchanged, replaced, and / or added between these methods. For example, the computer-generated objects and virtual shadows of method 1200 optionally also exhibit the attention-based behavior of method 1100, and / or the virtual shadows of method 1200 are simplified shadows as described for method 1300. For the sake of brevity, these details are not repeated here.

[0225] Fig.13is a flow chart of an exemplary method 1300 for displaying simplified shadows of user interface objects in an environment according to some embodiments. In some embodiments, method 1300 is performed on a computer system (e.g., Figure 1 101) in the computer system, which is executed by the display generation component (e.g., Figure 1 , Figure 3 and Figure 4 100) (e.g., hardware elements, including one or more display devices, such as a display, a touch screen, a projector, a head-up display, a head-mounted display, etc.) and one or more input devices (e.g., one or more cameras (e.g., color sensors, infrared sensors, structured light scanners, and / or other depth sensing cameras) pointing forward and downward from the user's head to the user's hands and / or facing the user; an eye tracking device; a controller held by the user and / or worn by the user; a touch-sensitive surface and / or other input hardware). In some embodiments, method 1300 is stored in a non-transitory (or transient) computer-readable storage medium and is executed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 1300 are optionally combined, and / or the order of some operations is optionally changed.

[0226] When a view of an environment is visible via a display generation component (e.g., the environment is a two-dimensional or three-dimensional environment that includes one or more computer-generated portions and, optionally, one or more transparent portions), the computer system displays (1302) in the environment a set of user interface objects (1304) that are separated from one another in the environment (e.g., separated in simulated space), the set including a first user interface object and a second user interface object based on a corresponding spatial relationship between a first user interface object at a first location in the environment and a second user interface object at a second location in the environment. In some embodiments, the set of user interface objects, although separated in simulated space, are associated with the same application. In some embodiments, the first user interface object is a primary user interface or window of the application, and the second user interface object is a secondary user interface or window of the application (e.g., a menu, toolbar, or other peripheral user interface feature).

[0227] In addition, the computer system displays (1302) virtual shadows (1306) corresponding to the set of user interface objects in the environment. Displaying the virtual shadows corresponding to the set of user interface objects includes displaying a first virtual shadow portion corresponding to a first user interface object at a first location in the environment (e.g., appearing to be cast by the first user interface object onto a portion of the environment) without displaying a virtual shadow portion corresponding to a second user interface object in the environment. For example, as described herein with reference to Fig. 9A As described, user interface 7060 is displayed together with shadow 7070, and shadow 7070 includes a portion of the shadow of element E1 of user interface 7060, but does not include portions of shadows of other elements other than element E1 of user interface 7060. In some embodiments, the displayed shadow corresponding to the first user interface object simulates a shadow cast by the first user interface object according to a simulated light source (e.g., having a corresponding spatial relationship with the set of user interface objects), and the computer system abandons displaying the shadow corresponding to the second user interface object, even if the shadow would have been cast by the second user interface object according to the same simulated light source.

[0228] The computer system detects (1308) a first input corresponding to a request to reposition the set of user interface objects in the environment. In response to detecting the first input (1310): the computer system repositions (1312) the set of user interface objects in the environment, including repositioning the first user interface object from the first position to a third position in the environment and repositioning the second user interface object from the second position to a fourth position in the environment, while maintaining a corresponding spatial relationship between the first user interface object and the second user interface object (e.g., and maintaining other spatial relationships between other pairs of user interface objects in the set); and the computer system repositions (1314) a virtual shadow corresponding to the set of user interface objects, including displaying a first virtual shadow portion corresponding to the first user interface object at the third position in the environment without displaying a virtual shadow portion corresponding to the second user interface object in the environment. For example, as described herein with reference to Fig. 9B As described, in response to detecting a signal for repositioning Fig. 9AIn response to input from user interface 7060, user interface 7060 and shadow 7070 are repositioned in the three-dimensional environment while the constituent elements of user interface 7060 maintain the same spatial relationship relative to each other, and the repositioned shadow 7070 includes a portion of the shadow of element E1 that is user interface 7060, but does not include portions of shadows of other elements other than element E1 that are user interface 7060. In some embodiments, the second shadow displayed corresponding to the first user interface object simulates a shadow cast by the first user interface object according to a simulated light source (e.g., having a corresponding spatial relationship with the set of user interface objects), and the computer system abandons displaying the shadow corresponding to the second user interface object, even if the shadow would have been cast by the second user interface object according to the same simulated light source.

[0229] Displaying simplified shadows for a set of user interface objects (e.g., that make up a given user interface) in a three-dimensional environment by displaying shadows corresponding to some of the set without displaying shadows corresponding to other user interface objects reduces the computational load imposed on a computer system while providing improved visual feedback regarding the spatial relationship between the set of user interface objects and the three-dimensional environment.

[0230] In some embodiments, the computer system displays virtual shadows corresponding to the set of user interface objects at least partially on a computer-generated surface in the three-dimensional environment (e.g., below the set of user interface objects). Fig. 9A and Fig. 9B As depicted, shadow 7070 of user interface 7060 is at least partially displayed on a top surface of computer-generated table 7024. Displaying the shadows of a set of user interface objects on a computer-generated surface in a three-dimensional environment provides improved visual feedback regarding the spatial relationship between the set of user interface objects and the three-dimensional environment.

[0231] In some embodiments, the computer system displays virtual shadows corresponding to the set of user interface objects at least in part on a see-through view of a physical surface in the physical environment that corresponds to the three-dimensional environment (e.g., the physical surface corresponds to a simulated location in the three-dimensional environment below the simulated location of the first user interface object). Fig. 9A As depicted, a shadow 7070 of user interface 7060 is at least partially displayed on the top surface of see-through box 7014'. Displaying the shadows of a set of user interface objects in a three-dimensional environment on a see-through view of a physical surface in a corresponding physical environment provides improved visual feedback regarding the spatial relationship between the set of user interface objects, the three-dimensional environment, and the corresponding physical environment.

[0232] In some embodiments, the first user interface object (for which the first virtual shadow portion is displayed) is a main area of ​​the application user interface (e.g., user interface 7060 ( Fig. 9A ) of element E1), and the second user interface object (for which the virtual shadow portion is not displayed) is a secondary area of ​​the primary area of ​​the application user interface (eg, user interface 7060 ( Fig. 9A ) of the user interface area including elements E2 and E3, element E4, element E5, close button 7062, move button 7064 and / or resize button 7066). In some embodiments, the group of user interface objects constitutes the application user interface. In some embodiments, the group of user interface objects includes one or more user interface objects other than the first user interface object (e.g., the second user interface object, and optionally includes the third user interface object, the fourth user interface object, the fifth user interface object or any number of other user interface objects). In some embodiments, the one or more user interface objects other than the first user interface object of the group are the auxiliary area of ​​the main area of ​​the application user interface. In some embodiments, the virtual shadow portion is not displayed for one or more user interface objects other than the first user interface object of the group. In some embodiments, the virtual shadow portion is displayed for one or more user interface objects other than the first user interface object of the group. In some embodiments, the second user interface object includes a plurality of constituent objects, such as controls and / or content as described below. When simplified shadows of a set of user interface objects that make up an application user interface are displayed in a three-dimensional environment, shadows corresponding to a primary area of ​​the application user interface are displayed without displaying shadows corresponding to one or more secondary areas of the application user interface, giving visual prominence to the primary area of ​​the application user interface and reducing the computational load imposed on the computer system while providing improved visual feedback about the spatial relationship between the primary area of ​​the application user interface and the three-dimensional environment.

[0233] In some embodiments, the second user interface object includes a first control (e.g., move button 7064 ( Fig. 9A )). In some embodiments, the computer system detects input corresponding to activation of the first control, and in response to detecting the input corresponding to activation of the first control, initiates (e.g., in response to movement of the input corresponding to activation of the first control, the movement of the input optionally including movement in the x-direction, y-direction, and / or z-direction or in polar, azimuthal, and / or radial directions in three-dimensional space) repositioning of the first user interface object (e.g., enabling a mode for repositioning the first user interface object) (e.g., move button 7064 ( Fig. 9A) is configured to be repositioned in response to further input (e.g., movement of a user's hand performing an air gesture, movement of a touch input via a touch-sensitive surface, or other input), as shown in reference to FIG. 8A to FIG. 8J ). In some embodiments, the appearance (e.g., position, size, distance relative to the main area of ​​the application user interface, opacity, blur, and / or other aspects) of a corresponding control or corresponding content (such as a first control) described herein changes in response to a user interacting with the corresponding control. For example, the corresponding control moves forward in the z direction (e.g., closer to the user's viewpoint) in response to the user directing their gaze toward the corresponding control. In another example, in response to the user activating the corresponding control (e.g., via an air gesture and / or via a press input on a touch-sensitive surface), the corresponding control moves backward in the z direction (e.g., further away from the user's viewpoint, and optionally to or beyond the position of the corresponding control just before the user directed their gaze toward the corresponding control). Abandoning displaying a shadow of a control used to initiate repositioning of the application user interface (or the main area of ​​the application user interface) as part of displaying a simplified shadow of the application user interface reduces the computational load imposed on the computer system while providing improved visual feedback about the spatial relationship between the application user interface and the three-dimensional environment.

[0234] In some embodiments, the second user interface object includes a second control (e.g., resize button 7066 ( Fig. 9A )). In some embodiments, the computer system detects input corresponding to activation of the second control, and in response to detecting the input corresponding to activation of the second control, initiates (e.g., in response to movement of the input corresponding to activation of the second control) resizing of the first user interface object (e.g., enabling a mode for resizing the first user interface object) (e.g., resize button 7066 ( Fig. 9A ) is configured to be resized in response to further input (e.g., movement of a user's hand performing an air gesture, movement of a touch input via a touch-sensitive surface, or other input), as shown in reference to Fig.9D In some embodiments, in response to a user interacting with the second control, the second control exhibits behavior similar to that described herein with reference to the first control. Forgoing displaying a shadow of a control for initiating resizing of the application user interface (or a main area of ​​the application user interface) as part of displaying a simplified shadow of the application user interface reduces the computational load imposed on the computer system while providing improved visual feedback about the spatial relationship between the application user interface and the three-dimensional environment.

[0235] In some embodiments, the second user interface object includes one or more controls that can be activated to navigate within the application user interface (e.g., forward and / or back buttons, scroll bars, tabs and / or other paging indicators or other navigation controls, such as those represented by elements E2, E3, and / or E4 ( Fig. 9A ) represented). In some embodiments, the computer system detects input corresponding to activation of one or more controls (e.g., corresponding controls in the one or more controls), and in response to detecting input corresponding to activation of the one or more controls, performs a navigation operation with respect to the content displayed in the first user interface object (e.g., navigating to the next content (e.g., using element E3) or the previous content (e.g., using element E2), viewing the content of different tabs or pages (e.g., where elements E2 and E3 represent different tabs or pages) or scrolling the content). In some embodiments, in response to user interaction with the one or more controls, the one or more controls exhibit behavior similar to that described herein with reference to the first control. Abandoning the display of a shadow of one or more controls for navigating the content of the application user interface (e.g., displayed in the main area of ​​the application user interface) as part of a simplified shadow for displaying the application user interface reduces the computational load imposed on the computer system while providing improved visual feedback about the spatial relationship between the application user interface and the three-dimensional environment.

[0236] In some embodiments, the second user interface object includes a third control (e.g., close button 7062 ( )) that can be activated to dismiss (e.g., close or hide) the application user interface (e.g., including the primary area and / or one or more or all secondary areas of the application user interface). Fig. 9A )). In some embodiments, the computer system detects input corresponding to activation of the third control, and in response to detecting input corresponding to activation of the third control, stops displaying the set of user interface objects (e.g., activation of close button 7062 closes user interface 7060). In some embodiments, in response to user interaction with the third control, the third control exhibits behavior similar to that described herein with reference to the first control. In some embodiments, in response to activation of the third control, the computer system stops displaying a window, tab, pop-up window, or other portion of the application user interface represented by the set of user interface objects without stopping displaying the entire set of user interface objects. Abandoning displaying a shadow of a control for dismissing the application user interface as part of displaying a simplified shadow of the application user interface reduces the computational load imposed on the computer system while providing improved visual feedback about the spatial relationship between the application user interface and the three-dimensional environment.

[0237] In some embodiments, the second user interface object includes three-dimensional content (e.g., element E5 ( Fig. 9A )). In some embodiments, in response to a user interacting with the content of the application, the content of the application exhibits behavior similar to that described herein with reference to the first control. Forgoing displaying a shadow of the three-dimensional (e.g., volumetric) content of the application user interface as part of displaying a simplified shadow of the application user interface reduces the computational load imposed on the computer system while providing improved visual feedback about the spatial relationship between the application user interface and the three-dimensional environment.

[0238] In some embodiments, in response to detecting the first input, repositioning the set of user interface objects repositions the set of user interface objects within a threshold distance of a reference location in the environment (e.g., a user's viewpoint, a surface on which a virtual shadow corresponding to the set of user interface objects is displayed, or other reference location). In some embodiments, the computer system detects a second input corresponding to a request to reposition the set of user interface objects in the environment. In some embodiments, in response to detecting the second input, the computer system repositions the set of user interface objects in the environment, including repositioning the first user interface object from a third location to a fifth location in the environment and repositioning the second user interface object from a fourth location to a sixth location in the environment, while maintaining a corresponding spatial relationship between the first user interface object and the second user interface object. In some embodiments, in response to detecting the second input, based on determining that repositioning the set of user interface objects in response to detecting the second input repositions the set of user interface objects within a threshold distance of a reference location, the computer system repositions the virtual shadow corresponding to the set of user interface objects, including displaying a first virtual shadow portion corresponding to the first user interface object at a fifth location in the environment without displaying a virtual shadow portion corresponding to the second user interface object in the environment. In some embodiments, in response to detecting the second input, based on determining that repositioning the group of user interface objects in response to detecting the second input repositions the group of user interface objects to a threshold distance beyond the reference position, the computer system abandons displaying the virtual shadows corresponding to the group of user interface objects (e.g., including abandoning displaying the first virtual shadow portion corresponding to the first user interface object at the fifth position in the environment while continuing not to display the virtual shadow portion corresponding to the second user interface object). For example, user interface 7060 ( 9A to 9E ) The shadow 7070 optionally shows a reference FIG. 8A to FIG. 8J At least some of the behaviors of the shadow 7016 of the window 7010 described herein. FIG. 8F to FIG. 8J As described in the shadow 7016, 9A to 9E Shadow 7070 optionally ceases to be displayed when user interface 7060 moves beyond a threshold distance from a reference location, such as a top surface of table 7024 or a viewpoint of user 7002. Ceasing to display a shadow of a user interface object (e.g., a group of user interface objects) in a three-dimensional environment when the user interface object moves greater than a threshold distance from a reference location in the three-dimensional environment provides improved visual feedback regarding the spatial relationship between the user interface object and the reference location in the three-dimensional environment.

[0239] In some embodiments, the reference location is the user's viewpoint (e.g., as described herein with reference to FIG. 8F to FIG. 8I 7016 in the figure, the shadow 7016 is optionally moved beyond a threshold distance D from the viewpoint 7046 of the user 7002 in the window 7010. th2 Stop displaying when ; similarly, 9A to 9E The shadow 7070 is optionally in the user interface 7060 from 9A to 9E In some embodiments, when the group of user interface objects is greater than a threshold distance of 0.5 meters, 1 meter, 2 meters, 3 meters, 4 meters, 5 meters, 10 meters, or other threshold distance from the user's viewpoint, the virtual shadows corresponding to the group of user interface objects cease to be displayed. Stopping display of shadows of user interface objects (e.g., a group of user interface objects) in a three-dimensional environment when the user interface objects move greater than a threshold distance from the user's viewpoint provides improved visual feedback about the spatial relationship between the user interface objects and the user's viewpoint.

[0240] In some embodiments, the reference location is a surface in the three-dimensional environment (e.g., a surface on which a virtual shadow corresponding to the set of user interface objects is displayed) (a corresponding point on the surface) (e.g., as described herein with reference to Figure 8J , the shadow 7016 optionally stops being displayed when the window 7010 moves beyond a threshold height (e.g., a second threshold height) above the top surface of the table 7024; similarly, 9A to 9E The shadow 7070 is optionally moved beyond the user interface 7060 9A to 9EIn some embodiments, the virtual shadows corresponding to the set of user interface objects cease to be displayed when the set of user interface objects is greater than a threshold distance of 0.5 meters, 1 meter, 2 meters, 2.5 meters, 3 meters, 5 meters, or other threshold distance from the reference surface. Cease displaying the shadows of user interface objects (e.g., a set of user interface objects) in a three-dimensional environment when the user interface objects move greater than a threshold distance from a surface corresponding to the user interface objects (particularly a surface on which the shadows are displayed (e.g., below the user interface objects)) provides improved visual feedback about the spatial relationship between the user interface objects, the corresponding surface, and the three-dimensional environment.

[0241] In some embodiments, in response to detecting a second input, as the set of user interface objects is repositioned toward a threshold distance from a reference location (e.g., while remaining within the threshold distance from the reference location), the computer system displays a virtual shadow corresponding to the set of user interface objects that progresses through a plurality of intermediate states, wherein visual attenuation of the virtual shadow increases (e.g., before ceasing to fully display the virtual shadow when the set of user interface objects moves to or beyond the threshold distance from the reference location). For example, as described herein with reference to FIG. 8F to FIG. 8I , which optionally moves with the window 7010 and approaches a threshold distance D from the viewpoint 7046 of the user 7002. th2 The fade-out (e.g., where the fade-in and fade-out of the shadow 7016 is gradual and occurs when the window 7010 reaches a threshold distance D closer to the viewpoint 7046 of the user 7002) is th1 In another example, as referenced herein, Figure 8J , shadow 7016 optionally fades out as window 7010 moves and approaches a threshold height (e.g., a second threshold height) above the top surface of table 7024, at which threshold height shadow 7016 stops being displayed (e.g., where the fading of shadow 7016 is gradual and begins when window 7010 reaches a first threshold height that is closer above table 7024). Similarly, 9A to 9E The shadow 7070 of the user interface 7060 optionally moves as the user interface 7060 approaches a threshold distance from the viewpoint of the user 7002 and / or 9A to 9EIn some embodiments, the virtual shadow is progressively visually weakened by causing the corresponding visual attribute of the virtual shadow to progress through multiple intermediate values ​​(e.g., from non-zero opacity to 0% opacity). When the shadow of a user interface object (e.g., a group of user interface objects) is stopped from being displayed in the three-dimensional environment when the user interface object moves greater than a threshold distance from a reference position in the three-dimensional environment, gradually ceasing to display the shadow provides improved visual feedback about the spatial relationship between the user interface object and the reference position in the three-dimensional environment, while reducing the likelihood that the user will suffer from motion sickness when using the computer system by not changing the user interface too abruptly.

[0242] In some embodiments, displaying virtual shadows corresponding to the group of user interface objects is performed when the environment has a first set of lighting conditions (e.g., based on real and / or virtual lighting), including displaying a first virtual shadow portion corresponding to the first user interface object, without displaying a virtual shadow portion corresponding to the second user interface object. In some embodiments, when the environment changes from having a first set of lighting conditions to having a second set of lighting conditions (e.g., based on real and / or virtual lighting) that is different from the first set of lighting conditions (e.g., and in the absence of a change to the group of user interface objects), the computer system continues to display virtual shadows corresponding to the group of user interface objects, including displaying a first virtual shadow portion corresponding to the first user interface object, without displaying a virtual shadow portion corresponding to the second user interface object. For example, as described herein with reference FIG. 9A to FIG. 9B As described, shadows 7070 optionally maintain the same appearance even if the three-dimensional environment is to transition from being illuminated by one combination of light sources 7038 and 7040 to being illuminated by a different combination of light sources 7038 and 7040. When displaying simplified shadows of a set of user interface objects (e.g., user interface objects that make up a given user interface) in a three-dimensional environment, displaying the same shadows for the set of user interface objects regardless of the lighting conditions present in the three-dimensional environment reduces the computational load imposed on the computer system while providing consistent, improved visual feedback regarding the spatial relationship between the set of user interface objects and the three-dimensional environment.

[0243] In some embodiments, when the environment has a first set of lighting conditions (e.g., based on real and / or virtual lighting), the computer system: displays a third (e.g., computer-generated) user interface object independent of the set of user interface objects; and displays the third user interface object with a first visual effect (e.g., specular reflection and / or shadow) according to the first set of lighting conditions. In some embodiments, when the environment changes from having a first set of lighting conditions to having a second set of lighting conditions (e.g., based on real and / or virtual lighting), the computer system: continues to display the third user interface object; and displays the third user interface object with a second visual effect (e.g., specular reflection and / or shadow) according to the second set of lighting conditions, wherein the second visual effect is different from the first visual effect. For example, as described herein with reference to FIG. 8A to FIG. 8J As described, different specular reflections are displayed based on different lighting conditions in a three-dimensional environment. Changing the appearance of some aspects of a three-dimensional environment based on what lighting conditions exist in the three-dimensional environment while not changing the appearance of other aspects (such as displayed shadows) provides improved visual feedback about the spatial relationships between user interface objects, the three-dimensional environment, and real and / or simulated light sources in the three-dimensional environment, regardless of what lighting conditions exist in the three-dimensional environment, while reducing the computational load imposed on a computer system.

[0244] In some embodiments, when a first user interface object is displayed at a first size and a first virtual shadow portion corresponding to the first user interface object is displayed at a second size corresponding to the first size, a computer system detects an input corresponding to a request to adjust the size of a first user interface object in the group of user interface objects. In some embodiments, the input corresponds to a request to collectively adjust the size of the group of user interface objects, including adjusting the size of the first user interface object and optionally adjusting the size of one or more other user interface objects in the group and / or repositioning the one or more other user interface objects. In some embodiments, in response to detecting the input corresponding to the request to adjust the size of the first user interface object, the computer system: displays the first user interface object at a third size different from the first size; and displays the first virtual shadow portion at a fourth size corresponding to the third size and different from the second size (e.g., changing the length of the first virtual shadow portion, such as in the x-direction or the y-direction or the azimuth direction relative to the user's viewpoint). For example, as described herein with reference to Fig.9DAs described, resizing element E1 changes the width of shadow 7070 of user interface 7060 by a corresponding amount. Changing the size (e.g., width in the x-direction, y-direction, polar direction, or azimuth direction) of shadows of a group of user interface objects in a three-dimensional environment in accordance with a change in the size (e.g., width in the x-direction, y-direction, polar direction, or azimuth direction) of the group of user interface objects provides improved visual feedback about the spatial relationship between the group of user interface objects and the three-dimensional environment.

[0245] In some embodiments, when a first user interface object is at a first position in the environment and a second user interface object is at a second position in the environment: the group of user interface objects has a first angle relative to the vertical direction in the environment (e.g., a zero angle because the group of user interface objects is oriented vertically, or a first non-zero angle because the group of user interface objects is tilted relative to the vertical direction); and the virtual shadows corresponding to the group of user interface objects have a first thickness (e.g., a width in the z-direction or radial direction relative to the user's viewpoint). In some embodiments, when the first user interface object is at a third position in the environment and the second user interface object is at a fourth position in the environment: the group of user interface objects has a second angle relative to the vertical direction in the environment (e.g., a non-zero angle because the group of user interface objects is tilted relative to the vertical direction, the non-zero angle being optionally a second non-zero angle different from the first non-zero angle); and the virtual shadows corresponding to the group of user interface objects have a second thickness (e.g., width) different from the first thickness. For example, when user interface 7060 is as Fig. 9C When oriented vertically as in FIG. 70, shadow 7070 of user interface 7060 has a side view 7056 ( Fig.9E ) shown in the first thickness, and when the user interface 7060 is as Fig.9E 58, the shadow 7070 of the user interface 7060 has a second, larger thickness as shown in the side view 7058. In some embodiments, a change in the angle of the group of user interface objects (e.g., relative to the vertical direction) causes a change in one visual attribute (thickness) of the virtual shadow corresponding to the group of user interface objects without causing a change in one or more other visual attributes of the virtual shadow (such as blurriness, length (e.g., in the x-direction, y-direction, or azimuth direction relative to the user's viewpoint), and / or opacity. Changing the size (e.g., depth in the z-direction or radial direction) of the shadow of a group of user interface objects in a three-dimensional environment in accordance with a change in the orientation (e.g., the amount of tilt relative to the vertical direction) of the group of user interface objects provides improved visual feedback about the spatial relationship between the group of user interface objects and the three-dimensional environment.

[0246] In some embodiments, when a group of physical objects corresponding to the group of user interface objects (e.g., elements having the same size and shape as the elements of the group of user interface objects for which virtual shadows are displayed) has a first angle relative to vertical in a physical environment with a corresponding set of lighting conditions, the group of physical objects casts a physical shadow having a first thickness. In some embodiments, when the group of physical objects has a second angle relative to vertical in a physical environment with a corresponding set of lighting conditions, the group of physical objects casts a physical shadow having a third thickness; and the second thickness of the virtual shadow (of the group of user interface objects at the second angle relative to the vertical in the environment visible via the display generation component) is less than the third thickness of the physical shadow (of the group of physical objects at the same second angle relative to the vertical in the physical environment). For example, as referenced herein Fig.9E described, although (for example, in FIG. 9C to FIG. 9E 7058) changes the top-down thickness of user interface 7060 by a certain amount, but the thickness of shadow 7070 changes by a smaller amount as seen in the transition from side view 7056 to side view 7058. Changing the size (e.g., depth in the z-direction or radial direction) of the shadows of a set of user interface objects in a three-dimensional environment in acco...

Claims

1. A method comprising: At a computer system in communication with a display generation component and one or more input devices. displaying a first computer-generated user interface object in the three-dimensional environment when a view of the three-dimensional environment is visible via the display generation component; Detecting that the user's attention is directed toward the first user interface object; In response to detecting that the user's attention is directed toward the first user interface object, displaying a first virtual shadow of the first user interface object in a first appearance, while maintaining the position of the first user interface object relative to the three-dimensional environment, wherein displaying the first virtual shadow of the first user interface object in the first appearance comprises displaying the first virtual shadow at a first value for a first visual attribute; While continuing to display the first user interface object in the three-dimensional environment, detecting that the user's attention has stopped being directed toward the first user interface object; as well as In response to detecting that the user's attention has ceased to be directed toward the first user interface object, displaying the first virtual shadow of the first user interface object in a second appearance different from the first appearance while maintaining the pose of the first user interface object relative to the three-dimensional environment, wherein: Displaying the first virtual shadow of the first user interface object in the second appearance includes displaying the first virtual shadow at a second value for the first visual attribute; and The second value is different from the first value.

2. The method according to claim 1, comprising: In response to detecting that the user's attention is directed toward the first user interface object, displaying the first virtual shadow in the first appearance while maintaining the size of the first user interface object; as well as In response to detecting that the user's attention has ceased to be directed toward the first user interface object, the first virtual shadow is displayed in the second appearance while maintaining the size of the first user interface object.

3. A method according to any one of claims 1 to 2, wherein displaying the first virtual shadow of the first user interface object in the first appearance is performed based on determining that the user's attention has been directed to the first user interface object for at least a first threshold amount of time.

4. A method according to any one of claims 1 to 3, wherein detecting that the user's attention is directed to the first user interface object includes detecting that the user's gaze is directed to the first user interface object, and detecting that the user's attention has stopped being directed to the first user interface object includes stopping detecting that the user's gaze is directed to the first user interface object.

5. The method according to any one of claims 1 to 4, wherein the first visual attribute is visual intensity.

6. The method of any one of claims 1 to 5, comprising displaying the first virtual shadow at least partially on a computer-generated surface in the three-dimensional environment.

7. The method of any one of claims 1 to 6, comprising displaying the first virtual shadow at least partially on a see-through view of a physical surface in a physical environment corresponding to the three-dimensional environment. 8 . The method of claim 7 , wherein the first virtual shadow displayed at least partially on the see-through view of the physical surface follows a three-dimensional model of the physical surface.

9. The method according to claim 8, wherein: The pass-through view of the physical surface has a first degree of fidelity to the physical surface; The three-dimensional model of the physical surface has a second degree of fidelity to the physical surface; and The second extent is smaller than the first extent.

10. The method of any one of claims 1 to 9, comprising displaying a second computer-generated user interface object and a second virtual shadow of the second user interface object.

11. The method according to claim 10, comprising: while displaying the first virtual shadow of the first user interface object in the first appearance, displaying the second virtual shadow of the second user interface object in a third appearance, comprising displaying the second virtual shadow in a third value for the first visual attribute, wherein the third value is different from the first value; while continuing to display the second user interface object in the three-dimensional environment, detecting that the user's attention is directed toward the second user interface object after stopping being directed toward the first user interface object; as well as In response to detecting that the user's attention is directed toward the second user interface object, displaying the second virtual shadow of the second user interface object in a fourth appearance different from the third appearance while maintaining a pose of the second user interface object relative to the three-dimensional environment, wherein: Displaying the second virtual shadow of the second user interface object in the fourth appearance includes displaying the second virtual shadow at a fourth value for the first visual attribute; and The fourth value is different from the third value.

12. The method according to claim 11, wherein: when the first virtual shadow is displayed at the second value for the first visual attribute, the first virtual shadow has a lower visual intensity than when the first virtual shadow is displayed at the first value for the first visual attribute; and When the second virtual shadow is displayed at the fourth value for the first visual attribute, the second virtual shadow has a greater visual intensity than when the second virtual shadow is displayed at the third value for the first visual attribute.

13. The method according to any one of claims 11 to 12, comprising: Displaying a third computer-generated user interface object and a third virtual shadow of the third user interface object, comprising: When the user's attention is directed toward the first user interface object, displaying the third virtual shadow with a fifth value for the first visual attribute; and When the user's attention is directed to the second user interface object after ceasing to be directed to the first user interface object, the third virtual shadow is displayed with the fifth value for the first visual attribute.

14. The method according to any one of claims 11 to 13, comprising: When the user's attention is not directed toward the first user interface object, displaying the first user interface object with a first value for a corresponding visual attribute of the first user interface object; as well as In response to detecting that the user's attention is directed toward the first user interface object, the first user interface object is displayed with a second value for the corresponding visual attribute of the first user interface object while maintaining the position of the first user interface object relative to the three-dimensional environment, wherein the second value for the corresponding visual attribute is different from the first value for the corresponding visual attribute.

15. The method according to any one of claims 1 to 14 includes gradually progressing the first visual attribute of the first virtual shadow through a plurality of intermediate values ​​between the first value and the second value in response to detecting that the user's attention has stopped being directed toward the first user interface object.

16. The method according to any one of claims 1 to 15, comprising: in response to detecting that the user's attention is directed toward the first user interface object, delaying displaying the first virtual shadow of the first user interface object in the first appearance until a second threshold amount of time has elapsed since detecting that the user's attention is directed toward the first user interface object; as well as In response to detecting that the user's attention has ceased being directed toward the first user interface object, delaying displaying the first virtual shadow of the first user interface object in the second appearance until a second threshold amount of time has passed since detecting that the user's attention has ceased being directed toward the first user interface object.

17. The method of any one of claims 1 to 16, wherein displaying the first virtual shadow of the first user interface object is performed based on determining that a viewpoint of the user is within a viewpoint range relative to the first user interface object, and the method comprises: detecting that the viewpoint of the user has moved beyond the viewpoint range relative to the first user interface object; as well as In response to detecting that the viewpoint of the user has moved beyond the viewpoint range relative to the first user interface object, ceasing to display the first virtual shadow of the first user interface object.

18. The method according to any one of claims 1 to 17, comprising displaying the first virtual shadow of the first user interface object without displaying a visual effect corresponding to simulated light emitted from content displayed in the first user interface object, and stopping displaying the first virtual shadow of the first user interface object when displaying the visual effect corresponding to simulated light emitted from the content.

19. The method of claim 18, comprising displaying the visual effect corresponding to the simulated light emitted from the content when playback of the content is ongoing, and forgoing displaying the visual effect corresponding to the simulated light emitted from the content when playback of the content is not ongoing.

20. The method of any one of claims 1 to 19, wherein the first user interface object is a user interface of an application.

21. A method according to any one of claims 1 to 20, wherein the first user interface object is a three-dimensional object in the three-dimensional environment.

22. The method according to any one of claims 1 to 21, wherein: Displaying the first virtual shadow of the first user interface object in the first appearance includes displaying the first virtual shadow at a first value for a second visual attribute; Maintaining the pose of the first user interface object relative to the three-dimensional environment includes: displaying the first user interface object at a first position in the three-dimensional environment and at a first distance from a surface in the three-dimensional environment corresponding to the first position; and The method comprises: detecting an input corresponding to a request to reposition the first user interface object; In response to detecting the input corresponding to the request to reposition the first user interface object: repositioning the first user interface object to a second location in the three-dimensional environment, the second location being a second distance from a surface in the three-dimensional environment corresponding to the second location; and Based on determining that the second distance is different from the first distance, the first virtual shadow of the first user interface object is displayed for a second value of the second visual property, wherein the second value is different from the first value.

23. The method of claim 22, wherein the first visual attribute is an opacity of the first virtual shadow, and the second visual attribute is a size and / or blurriness of the first virtual shadow.

24. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs being configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 23.

25. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 23.

26. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: Components for carrying out the method according to any one of claims 1 to 23.

27. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for: displaying a first computer-generated user interface object in the three-dimensional environment when a view of the three-dimensional environment is visible via the display generation component; Detecting that the user's attention is directed toward the first user interface object; In response to detecting that the user's attention is directed toward the first user interface object, displaying a first virtual shadow of the first user interface object with a first appearance while maintaining a pose of the first user interface object relative to the three-dimensional environment, wherein displaying the first virtual shadow of the first user interface object with the first appearance includes displaying the first virtual shadow with a first value for a first visual attribute; while continuing to display the first user interface object in the three-dimensional environment, detecting that the user's attention has stopped being directed toward the first user interface object; as well as In response to detecting that the user's attention has ceased to be directed toward the first user interface object, displaying the first virtual shadow of the first user interface object in a second appearance different from the first appearance while maintaining the pose of the first user interface object relative to the three-dimensional environment, wherein: Displaying the first virtual shadow of the first user interface object in the second appearance includes displaying the first virtual shadow at a second value for the first visual attribute; and The second value is different from the first value.

28. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a first computer-generated user interface object in the three-dimensional environment when a view of the three-dimensional environment is visible via the display generation component; Detecting that the user's attention is directed toward the first user interface object; In response to detecting that the user's attention is directed toward the first user interface object, displaying a first virtual shadow of the first user interface object with a first appearance while maintaining a pose of the first user interface object relative to the three-dimensional environment, wherein displaying the first virtual shadow of the first user interface object with the first appearance includes displaying the first virtual shadow with a first value for a first visual attribute; while continuing to display the first user interface object in the three-dimensional environment, detecting that the user's attention has stopped being directed toward the first user interface object; as well as In response to detecting that the user's attention has ceased to be directed toward the first user interface object, displaying the first virtual shadow of the first user interface object in a second appearance different from the first appearance while maintaining the pose of the first user interface object relative to the three-dimensional environment, wherein: Displaying the first virtual shadow of the first user interface object in the second appearance includes displaying the first virtual shadow at a second value for the first visual attribute; and The second value is different from the first value.

29. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: means for displaying a first computer-generated user interface object in the three-dimensional environment enabled when a view of the three-dimensional environment is visible via the display generation component; a component for detecting that the user's attention is directed toward the first user interface object; means for displaying a first virtual shadow of the first user interface object with a first appearance while maintaining a pose of the first user interface object relative to the three-dimensional environment, enabled in response to detecting that the user's attention is directed toward the first user interface object, wherein displaying the first virtual shadow of the first user interface object with the first appearance comprises displaying the first virtual shadow with a first value for a first visual attribute; means for detecting that the user's attention has ceased to be directed toward the first user interface object while continuing to display the first user interface object in the three-dimensional environment; as well as means for displaying the first virtual shadow of the first user interface object in a second appearance different from the first appearance while maintaining the pose of the first user interface object relative to the three-dimensional environment, enabled in response to detecting that the user's attention has ceased to be directed toward the first user interface object, wherein: Displaying the first virtual shadow of the first user interface object in the second appearance includes displaying the first virtual shadow at a second value for the first visual attribute; and The second value is different from the first value.

30. A method comprising: At a computer system in communication with a display generation component and one or more input devices. displaying in the environment a first user interface object at a first location in the environment and a virtual shadow corresponding to the first user interface object, including displaying the virtual shadow at a first value for a first visual property, when a view of the environment is visible via the display generation component; detecting an input corresponding to a request to reposition the first user interface object from the first location in the environment to a second location in the environment that is different from the first location; as well as In response to detecting the input: repositioning the first user interface object from the first location in the environment to the second location in the environment based on movement of the input relative to the environment; as well as When the first user interface object is repositioned from the first location in the environment to the second location in the environment, the virtual shadow corresponding to the first user interface object is displayed with a second value for the first visual attribute.

31. The method of claim 30, comprising, in response to detecting the input, repositioning the virtual shadow corresponding to the first user interface object in accordance with repositioning the first user interface object from the first position to the second position.

32. A method according to any one of claims 30 to 31, comprising, in response to detecting the input: Based on determining that the input includes a first amount of movement in a first direction relative to the environment: repositioning the first user interface object by a second amount corresponding to the first amount and in a second direction corresponding to the first direction; as well as displaying the virtual shadow corresponding to the first user interface object at the second value for the first visual attribute when the first user interface object is repositioned in the second direction by the second amount; as well as Based on determining that the input comprises a third amount of movement in a third direction relative to the environment, wherein the third amount is different from the first amount and / or the third direction is different from the first direction: repositioning the first user interface object by a fourth amount corresponding to the third amount and in a fourth direction corresponding to the third direction, wherein the fourth amount is different from the second amount when the third amount is different from the first amount, and the fourth direction is different from the second direction when the third direction is different from the first direction; as well as When the first user interface object is repositioned in the fourth direction by the fourth amount, the virtual shadow corresponding to the first user interface object is displayed with the second value for the first visual property.

33. The method according to any one of claims 30 to 32 includes, in response to detecting the input, displaying the virtual shadow corresponding to the first user interface object with a third value for the first visual attribute before repositioning the first user interface object, wherein the third value is different from the first value.

34. A method according to any one of claims 30 to 33, comprising: ceasing to detect said input; as well as In response to ceasing to detect the input: The virtual shadow corresponding to the first user interface object is displayed with a fourth value for the first visual attribute, wherein the fourth value is different from the second value.

35. The method of any one of claims 30 to 34, wherein detecting the input corresponding to the request to reposition the first user interface object comprises detecting an air gesture.

36. A method according to any one of claims 30 to 35, comprising: Prior to displaying the first user interface object and the virtual shadow at the first value for the first visual property: displaying the first user interface object; as well as when the user's attention is not directed toward the first user interface object, displaying the virtual shadow corresponding to the first user interface object with a fifth value for the first visual attribute, wherein the fifth value is different from the first value; as well as Detecting that the user's attention is directed toward the first user interface object; Wherein displaying the virtual shadow at the first value for the first visual attribute is performed in response to detecting that the user's attention is directed toward the user interface object.

37. The method of claim 36, comprising maintaining the appearance of the virtual shadow corresponding to the first user interface object when the first user interface object is repositioned from the first position to the second position in the environment, regardless of whether the user's attention is directed toward the first user interface object.

38. A method according to any one of claims 30 to 37, comprising: prior to detecting the input, displaying the first user interface object with a first value for a corresponding visual attribute of the first user interface object; as well as When the first user interface object is repositioned from the first position in the environment to the second position in the environment, the first user interface object is displayed with a second value for the corresponding visual attribute of the first user interface object, wherein the second value for the corresponding visual attribute is different from the first value for the corresponding visual attribute.

39. A method according to any one of claims 30 to 38, wherein: The virtual shadow corresponding to the first user interface object is displayed on a surface in the environment; repositioning the first user interface object from the first location to the second location in the environment changes the distance between the first user interface object and the surface; Prior to detecting the input, the virtual shadow displayed at the first value for the first visual attribute is displayed at a first value for a second visual attribute; and The method comprises, in response to detecting the input: Based on the change in the distance between the first user interface object and the surface, the virtual shadow corresponding to the first user interface object is displayed at a second value for the second visual attribute, while continuing to display the virtual shadow at the second value for the first visual attribute.

40. The method of claim 39, wherein the first visual attribute is an opacity of the virtual shadow and the second visual attribute is a size and / or blurriness of the virtual shadow.

41. The method of any one of claims 30 to 40, wherein displaying the virtual shadow at the first value for the first visual attribute and displaying the virtual shadow at the second value for the first visual attribute is performed when the environment has a first set of lighting conditions, and the method comprises: When the environment has a second set of lighting conditions different from the first set of lighting conditions: displaying the first user interface object and the virtual shadow corresponding to the first user interface object in the environment, including displaying the virtual shadow at the first value for the first visual attribute; detecting a second input corresponding to a request to reposition the first user interface object from the first location in the environment to the second location in the environment; as well as In response to detecting the second input: repositioning the first user interface object from the first location in the environment to the second location in the environment based on movement of the second input relative to the environment; as well as When the first user interface object is repositioned from the first location in the environment to the second location in the environment, the virtual shadow corresponding to the first user interface object is displayed with the second value for the first visual attribute.

42. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs being configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for executing a method according to any one of claims 30 to 41.

43. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 30 to 41.

44. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: Means for carrying out the method according to any one of claims 30 to 41.

45. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for: displaying in the environment a first user interface object at a first location in the environment and a virtual shadow corresponding to the first user interface object, including displaying the virtual shadow at a first value for a first visual property, when a view of the environment is visible via the display generation component; detecting an input corresponding to a request to reposition the first user interface object from the first location in the environment to a second location in the environment that is different from the first location; as well as In response to detecting the input: repositioning the first user interface object from the first location in the environment to the second location in the environment based on movement of the input relative to the environment; as well as When the first user interface object is repositioned from the first location in the environment to the second location in the environment, the virtual shadow corresponding to the first user interface object is displayed with a second value for the first visual attribute.

46. ​​A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying in the environment a first user interface object at a first location in the environment and a virtual shadow corresponding to the first user interface object, including displaying the virtual shadow at a first value for a first visual property, when a view of the environment is visible via the display generation component; detecting an input corresponding to a request to reposition the first user interface object from the first location in the environment to a second location in the environment that is different from the first location; as well as In response to detecting the input: repositioning the first user interface object from the first location in the environment to the second location in the environment based on movement of the input relative to the environment; as well as When the first user interface object is repositioned from the first location in the environment to the second location in the environment, the virtual shadow corresponding to the first user interface object is displayed with a second value for the first visual attribute.

47. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: means for displaying in the environment a first user interface object at a first location in the environment and a virtual shadow corresponding to the first user interface object, enabled when a view of the environment is visible via the display generating component, including means for displaying the virtual shadow at a first value for a first visual attribute; means for detecting an input corresponding to a request to reposition the first user interface object from the first location in the environment to a second location in the environment that is different from the first location; as well as The component enabled in response to detecting the input includes: means for repositioning the first user interface object from the first location in the environment to the second location in the environment based on movement of the input relative to the environment; as well as Means for displaying the virtual shadow corresponding to the first user interface object at a second value for the first visual attribute are enabled when the first user interface object is relocated from the first location in the environment to the second location in the environment.

48. A method comprising: At a computer system in communication with a display generation component and one or more input devices: When a view of an environment is visible via the display generation component, displaying in the environment: a set of user interface objects separated from one another in the environment, the set of user interface objects comprising the first user interface object and the second user interface object according to a respective spatial relationship between a first user interface object at a first location in the environment and a second user interface object at a second location in the environment; and a virtual shadow corresponding to the set of user interface objects, wherein displaying the virtual shadow corresponding to the set of user interface objects comprises displaying a first virtual shadow portion corresponding to the first user interface object at the first location in the environment without displaying a virtual shadow portion corresponding to the second user interface object in the environment; detecting a first input corresponding to a request to reposition the set of user interface objects in the environment; as well as In response to detecting the first input: repositioning the set of user interface objects in the environment, comprising repositioning the first user interface object from the first location to a third location in the environment and repositioning the second user interface object from the second location to a fourth location in the environment while maintaining the respective spatial relationships between the first user interface object and the second user interface object; as well as Repositioning the virtual shadow corresponding to the set of user interface objects includes displaying the first virtual shadow portion corresponding to the first user interface object at the third position in the environment without displaying the virtual shadow portion corresponding to the second user interface object in the environment.

49. The method of claim 48, comprising displaying the virtual shadows corresponding to the set of user interface objects at least in part on a computer-generated surface in the three-dimensional environment.

50. The method of any one of claims 48 to 49, comprising displaying the virtual shadows corresponding to the set of user interface objects at least in part on a see-through view of a physical surface in a physical environment corresponding to the three-dimensional environment.

51. A method according to any one of claims 48 to 50, wherein the first user interface object is a primary area of ​​an application user interface, and the second user interface object is a secondary area of ​​the primary area of ​​the application user interface.

52. A method according to claim 51, wherein the second user interface object comprises a first control activatable to select and / or reposition the first user interface object, and the method comprises: detecting an input corresponding to activation of the first control; as well as In response to detecting the input corresponding to the activation of the first control, repositioning of the first user interface object is initiated.

53. A method according to any one of claims 51 to 52, wherein the second user interface object comprises a second control activatable to resize the first user interface object, and the method comprises: detecting an input corresponding to activation of the second control; as well as In response to detecting the input corresponding to the activation of the second control, resizing of the first user interface object is initiated.

54. A method according to any one of claims 51 to 53, wherein the second user interface object comprises one or more controls activatable to navigate within the application user interface, and the method comprises: detecting input corresponding to activation of the one or more controls; as well as In response to detecting the input corresponding to the activation of the one or more controls, a navigation operation is performed with respect to content displayed in the first user interface object.

55. A method according to any one of claims 51 to 54, wherein the second user interface object comprises a third control activatable to dismiss the application user interface, and the method comprises: detecting an input corresponding to activation of the third control; as well as In response to detecting the input corresponding to the activation of the third control, ceasing to display the set of user interface objects.

56. A method according to any one of claims 51 to 55, wherein the second user interface object comprises three-dimensional content of the application.

57. A method according to any one of claims 48 to 56, wherein repositioning the set of user interface objects in response to detecting the first input repositions the set of user interface objects within a threshold distance of a reference location in the environment, and the method comprises: detecting a second input corresponding to a request to reposition the set of user interface objects in the environment; as well as In response to detecting the second input: repositioning the set of user interface objects in the environment, including repositioning the first user interface object from the third location to a fifth location in the environment and repositioning the second user interface object from the fourth location to a sixth location in the environment while maintaining the respective spatial relationships between the first user interface object and the second user interface object; as well as repositioning the set of user interface objects within the threshold distance of the reference location in accordance with determining to reposition the set of user interface objects in response to detecting the second input: repositioning the virtual shadow corresponding to the set of user interface objects, including displaying a portion of the first virtual shadow corresponding to the first user interface object at the fifth location in the environment without displaying a portion of the virtual shadow corresponding to the second user interface object in the environment; as well as repositioning the set of user interface objects to the threshold distance beyond the reference position in accordance with determining to reposition the set of user interface objects in response to detecting the second input: Displaying the virtual shadow corresponding to the set of user interface objects is abandoned.

58. The method of claim 57, wherein the reference location is a viewpoint of the user.

59. The method of claim 57, wherein the reference location is a surface in the three-dimensional environment.

60. The method according to any one of claims 57 to 59 includes, in response to detecting the second input, displaying the virtual shadow corresponding to the group of user interface objects that progresses through a plurality of intermediate states when the group of user interface objects is repositioned toward the threshold distance from the reference position, wherein the visual weakening of the virtual shadow increases.

61. The method of any one of claims 48 to 60, wherein displaying the virtual shadows corresponding to the set of user interface objects is performed when the environment has a first set of lighting conditions, comprises displaying the first virtual shadow portion corresponding to the first user interface object without displaying the virtual shadow portion corresponding to the second user interface object, and the method comprises: When the environment changes from the first set of lighting conditions to a second set of lighting conditions different from the first set of lighting conditions, continue to display the virtual shadows corresponding to the group of user interface objects, including displaying the first virtual shadow portion corresponding to the first user interface object without displaying the virtual shadow portion corresponding to the second user interface object.

62. The method of claim 61, comprising: When the environment has the first set of lighting conditions: displaying a third user interface object independent of the set of user interface objects; as well as displaying the third user interface object with a first visual effect according to the first set of lighting conditions; as well as When the environment changes from the first set of lighting conditions to the second set of lighting conditions: continuing to display the third user interface object; and The third user interface object is displayed with a second visual effect based on the second set of lighting conditions, wherein the second visual effect is different from the first visual effect.

63. A method according to any one of claims 48 to 62, comprising: detecting an input corresponding to a request to resize the first user interface object in the set of user interface objects while the first user interface object is displayed at a first size and the first virtual shadow portion corresponding to the first user interface object is displayed at a second size corresponding to the first size; as well as In response to detecting the input corresponding to the request to resize the first user interface object: displaying the first user interface object at a third size different from the first size; and The first virtual shadow portion is displayed in a fourth size corresponding to the third size and different from the second size.

64. A method according to any one of claims 48 to 63, wherein: When the first user interface object is at the first location in the environment and the second user interface object is at the second location in the environment: The set of user interface objects has a first angle relative to vertical in the environment; and The virtual shadow corresponding to the set of user interface objects has a first thickness; and When the first user interface object is at the third location in the environment and the second user interface object is at the fourth location in the environment: The set of user interface objects has a second angle relative to the vertical direction in the environment; and The virtual shadow corresponding to the set of user interface objects has a second thickness different from the first thickness.

65. The method of claim 64, wherein: when a set of physical objects corresponding to the set of user interface objects has the first angle relative to vertical in a physical environment having a corresponding set of lighting conditions, the set of physical objects casts a physical shadow having the first thickness; When the set of physical objects has the second angle relative to the vertical direction in the physical environment having the corresponding set of lighting conditions, the set of physical objects casts a physical shadow having a third thickness; and The second thickness of the virtual shadow is less than the third thickness of the physical shadow.

66. A method according to any one of claims 48 to 65, wherein: The virtual shadows corresponding to the set of user interface objects are displayed on a surface in the environment; the first input corresponding to the request to reposition the set of user interface objects in the environment is detected when the set of user interface objects are a first distance from the surface and when the virtual shadows corresponding to the set of user interface objects are displayed at a first value for a first visual property based on the first distance; and The method includes, in response to detecting the first input corresponding to the request to reposition the set of user interface objects in the environment: repositioning the set of user interface objects in the environment to a second distance from the surface; as well as When the group of user interface objects are at the second distance from the surface, the virtual shadow corresponding to the group of user interface objects is displayed with a second value for the first visual attribute according to the second distance, wherein the second value is different from the first value.

67. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for executing a method according to any one of claims 48 to 66.

68. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 48 to 66.

69. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: Means for carrying out the method according to any one of claims 48 to 66.

70. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for: When a view of an environment is visible via the display generation component, displaying in the environment: a set of user interface objects separated from one another in the environment, the set of user interface objects comprising the first user interface object and the second user interface object according to a respective spatial relationship between a first user interface object at a first location in the environment and a second user interface object at a second location in the environment; and a virtual shadow corresponding to the set of user interface objects, wherein displaying the virtual shadow corresponding to the set of user interface objects comprises displaying a first virtual shadow portion corresponding to the first user interface object at the first location in the environment without displaying a virtual shadow portion corresponding to the second user interface object in the environment; detecting a first input corresponding to a request to reposition the set of user interface objects in the environment; as well as In response to detecting the first input: repositioning the set of user interface objects in the environment, comprising repositioning the first user interface object from the first location to a third location in the environment and repositioning the second user interface object from the second location to a fourth location in the environment while maintaining the respective spatial relationships between the first user interface object and the second user interface object; as well as Repositioning the virtual shadow corresponding to the set of user interface objects includes displaying the first virtual shadow portion corresponding to the first user interface object at the third position in the environment without displaying the virtual shadow portion corresponding to the second user interface object in the environment.

71. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: When a view of an environment is visible via the display generation component, displaying in the environment: a set of user interface objects separated from one another in the environment, the set of user interface objects comprising the first user interface object and the second user interface object according to a respective spatial relationship between a first user interface object at a first location in the environment and a second user interface object at a second location in the environment; and a virtual shadow corresponding to the set of user interface objects, wherein displaying the virtual shadow corresponding to the set of user interface objects comprises displaying a first virtual shadow portion corresponding to the first user interface object at the first location in the environment without displaying a virtual shadow portion corresponding to the second user interface object in the environment; detecting a first input corresponding to a request to reposition the set of user interface objects in the environment; as well as In response to detecting the first input: repositioning the set of user interface objects in the environment, comprising repositioning the first user interface object from the first location to a third location in the environment and repositioning the second user interface object from the second location to a fourth location in the environment while maintaining the respective spatial relationships between the first user interface object and the second user interface object; as well as Repositioning the virtual shadow corresponding to the set of user interface objects includes displaying the first virtual shadow portion corresponding to the first user interface object at the third position in the environment without displaying the virtual shadow portion corresponding to the second user interface object in the environment.

72. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: Means for displaying in the environment the following, enabled when a view of the environment is visible via the display generating component: a set of user interface objects separated from one another in the environment, the set of user interface objects comprising the first user interface object and the second user interface object according to a respective spatial relationship between a first user interface object at a first location in the environment and a second user interface object at a second location in the environment; and a virtual shadow corresponding to the set of user interface objects, wherein displaying the virtual shadow corresponding to the set of user interface objects comprises displaying a first virtual shadow portion corresponding to the first user interface object at the first location in the environment without displaying a virtual shadow portion corresponding to the second user interface object in the environment; means for detecting a first input corresponding to a request to reposition the set of user interface objects in the environment; as well as The component enabled in response to detecting the first input comprises: means for repositioning the set of user interface objects in the environment, including repositioning the first user interface object from the first position to a third position in the environment and repositioning the second user interface object from the second position to a fourth position in the environment while maintaining the respective spatial relationships between the first user interface object and the second user interface object; as well as A component for repositioning the virtual shadow corresponding to the set of user interface objects includes displaying the first virtual shadow portion corresponding to the first user interface object at the third position in the environment, without displaying the virtual shadow portion corresponding to the second user interface object in the environment.

73. A method comprising: At a computer system in communication with a display generation component and one or more input devices. detecting input corresponding to a request to display content in the environment while a view of the environment is visible via the display generating component; In response to detecting the input: displaying the content in a user interface object in the environment; as well as Displaying a visual effect corresponding to simulated light emitted from the content in one or more portions of the environment that are outside of the user interface object, wherein displaying the visual effect comprises: Based on determining that the environment is a first type of environment, displaying the visual effect at a first level of fidelity to the content; as well as Based on determining that the environment is a second type of environment that is different from the first type of environment, the visual effect is displayed at a second level of fidelity to the content, wherein the second level of fidelity is different from the first level of fidelity.

74. The method of claim 73, wherein: The first type of environment includes one or more transparent parts and one or more computer-generated parts; The second type of environment includes one or more computer-generated portions and does not include one or more transparent portions; and The first degree of fidelity is less than the second degree of fidelity.

75. The method of any one of claims 73 to 74, wherein: Displaying the visual effect at the first level of fidelity to the content includes displaying light extending from one or more edges of the content to the one or more portions of the environment outside of the user interface objects, independent of the objects in the one or more portions of the environment; and Displaying the visual effect at the second level of fidelity to the content includes displaying light projected by the content onto one or more computer-generated objects in the one or more portions of the environment that are outside of the user interface object.

76. The method of claim 75, comprising displaying the light extending from the one or more edges of the content into the one or more portions of the environment based on a representative set of visual attributes sampled from a first number of portions of the content.

77. The method of any one of claims 75 to 76, wherein the content corresponds to a corresponding plane in the environment, and the light extending from the one or more edges of the content corresponds to the corresponding plane.

78. The method of any one of claims 73 to 77, wherein displaying the visual effect at the first level of fidelity to the content comprises applying a tint based on the content to one or more pass-through portions of the environment.

79. The method of claim 78, comprising applying the tint using a representative set of colors sampled from a second number of portions of the content.

80. A method according to any one of claims 78 to 79 comprising applying the tint using a median color of the content.

81. A method according to any one of claims 73 to 80, wherein displaying the visual effect with the second level of fidelity to the content includes displaying light projected into the one or more portions of the environment located outside the user interface object based on a three-dimensional model of the environment.

82. The method of claim 81, wherein displaying the light projected into the one or more portions of the environment simulates one or more computer-generated objects reflecting the simulated light emitted from the content.

83. The method of any one of claims 81 to 82, wherein: displaying the visual effect at the first degree of fidelity to the content is based on a representative set of visual attributes sampled from a third number of portions of the content; displaying the visual effect at the second level of fidelity to the content based on a representative set of visual attributes sampled from a fourth number of portions of the content, including displaying the light projected into the one or more portions of the environment that are outside of the user interface object based on the three-dimensional model; and The fourth number is greater than the third number.

84. The method of any one of claims 73 to 83, comprising visually weakening a background of the environment when displaying the visual effect corresponding to simulated light emitted from the content in the one or more portions of the environment located outside of the user interface object.

85. The method of any one of claims 73 to 84, comprising visually weakening one or more transparent portions of the environment when displaying the visual effect corresponding to the simulated light emitted from the content in the one or more portions of the environment located outside the user interface object.

86. The method of any one of claims 73 to 85, comprising visually weakening one or more computer-generated portions of the environment other than the user interface object when displaying the visual effect corresponding to simulated light emitted from the content in the one or more portions of the environment located outside the user interface object.

87. The method of any one of claims 73 to 86, wherein displaying the content in the user interface object in the environment comprises playing video content, and displaying the visual effect comprises: while playing a first portion of the video content, displaying the visual effect with a first set of values ​​for one or more visual attributes based on the first portion of the video content; as well as When a second portion of the video content different from the first portion is played, the visual effect is displayed based on the second portion of the video content with a second set of values ​​for the one or more visual attributes, wherein the second set of values ​​is different from the first set of values.

88. The method of any one of claims 73 to 87, wherein displaying the content in the user interface object in the environment comprises playing video content, and the method comprises: detecting an input corresponding to a request to stop playing the video content; as well as In response to detecting the input corresponding to the request to stop playing the video content: Stop playing the video content; as well as Displaying the visual effect corresponding to simulated light emanating from the content in one or more portions of the environment outside of the user interface object is ceased.

89. A method according to any one of claims 73 to 88, wherein displaying the visual effect with the second level of fidelity to the content includes displaying shadows that simulate one or more objects in the one or more portions of the environment located outside the user interface object that block the simulated light emitted from the content.

90. The method of any one of claims 73 to 89, comprising: when displaying the content at the first width, displaying the visual effect in a first corresponding portion of the environment that is outside of the user interface object, the first corresponding portion having a second width that corresponds to the first width of the content; as well as When the content is displayed at a third width different from the first width, the visual effect is displayed in a second corresponding portion of the environment located outside the user interface object, and the second corresponding portion has a fourth width corresponding to the third width of the content and different from the second width.

91. A method according to any one of claims 73 to 90, wherein the visual effect extends outward from one or more edges of the content into the one or more portions of the environment that are located outside the user interface object.

92. The method of any one of claims 73 to 91, comprising: displaying a shadow corresponding to the user interface object when the user interface object is displayed in the environment without displaying the visual effect corresponding to the simulated light emitted from the content in the user interface object; as well as When the visual effect corresponding to the simulated light emitted from the content is displayed in the user interface object, displaying a shadow corresponding to the user interface object is abandoned.

93. The method of any one of claims 73 to 92, comprising: displaying the user interface object in the environment with a shadow corresponding to the user interface object when the set of lighting conditions in the environment satisfies a corresponding criterion; as well as The user interface object is displayed in the environment without a shadow corresponding to the user interface object when the set of lighting conditions in the environment does not satisfy the corresponding criteria.

94. The method of claim 93, wherein the set of lighting conditions comprises physical light from a physical environment corresponding to the environment visible via the display generating component.

95. A method according to any one of claims 93 to 94, wherein the set of lighting conditions comprises computer-generated lighting of the environment.

96. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs are configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs including instructions for executing a method according to any one of claims 73 to 95.

97. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 73 to 95.

98. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: Means for performing a method according to any one of claims 73 to 95.

99. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, the one or more programs comprising instructions for: detecting input corresponding to a request to display content in the environment while a view of the environment is visible via the display generating component; In response to detecting the input: displaying the content in a user interface object in the environment; as well as Displaying a visual effect corresponding to simulated light emitted from the content in one or more portions of the environment that are outside of the user interface object, wherein displaying the visual effect comprises: Based on determining that the environment is a first type of environment, displaying the visual effect at a first level of fidelity to the content; as well as Based on determining that the environment is a second type of environment that is different from the first type of environment, the visual effect is displayed at a second level of fidelity to the content, wherein the second level of fidelity is different from the first level of fidelity.

100. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: detecting input corresponding to a request to display content in the environment while a view of the environment is visible via the display generating component; In response to detecting the input: displaying the content in a user interface object in the environment; as well as Displaying a visual effect corresponding to simulated light emitted from the content in one or more portions of the environment that are outside of the user interface object, wherein displaying the visual effect comprises: Based on determining that the environment is a first type of environment, displaying the visual effect at a first level of fidelity to the content; as well as Based on determining that the environment is a second type of environment that is different from the first type of environment, the visual effect is displayed at a second level of fidelity to the content, wherein the second level of fidelity is different from the first level of fidelity.

101. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: means for detecting input corresponding to a request to display content in the environment, enabled when a view of the environment is visible via the display generating component; The component enabled in response to detecting the input includes: means for displaying said content in a user interface object in said environment; and Means for displaying a visual effect corresponding to simulated light emitted from the content in one or more portions of the environment that are outside of the user interface object, wherein displaying the visual effect comprises: Based on determining that the environment is a first type of environment, displaying the visual effect at a first level of fidelity to the content; and Based on determining that the environment is a second type of environment that is different from the first type of environment, the visual effect is displayed at a second level of fidelity to the content, wherein the second level of fidelity is different from the first level of fidelity.

Images