Method for sharing content and interacting with physical devices in three-dimensional environment

Abstract

The invention relates to a method for sharing content and interacting with a physical device in a three-dimensional environment. In some embodiments, a computer system displays user interface elements corresponding to content shared with the computer system, the user interface elements for interacting with the shared content in a three-dimensional environment. In some embodiments, a computer system displays one or more virtual control elements in a three-dimensional environment, the one or more virtual control elements being selectable to cause execution of one or more corresponding operations involving a physical device.

Description

[0001] This application is a divisional application based on the Chinese invention patent application with application date of December 23, 2022, application number 2022800847966, and invention name “Method for sharing content and interacting with physical devices in a three-dimensional environment”. Technical Field

[0002] The present invention generally relates to computer systems that provide computer-generated experiences, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via displays. Background Art

[0003] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays) for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Exemplary virtual elements include virtual objects such as digital images, videos, text, icons, and control elements (such as buttons and other graphics). Summary of the invention

[0004] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve desired results in augmented reality environments, and systems that make virtual object manipulation complex, cumbersome, and error-prone can impose a huge cognitive burden on users and detract from the experience of virtual / augmented reality environments. In addition, these methods take longer than necessary, wasting energy on the computer system. This latter consideration is particularly important in battery-powered devices.

[0005] Therefore, there is a need for computer systems with improved methods and interfaces to provide computer-generated experiences to users, so that the user's interaction with the computer system is more efficient and more intuitive to the user. Such methods and interfaces optionally supplement or replace conventional methods for providing extended reality experiences to users. Such methods and interfaces reduce the amount, degree and / or nature of inputs from the user by helping the user understand the connection between the inputs provided and the device's response to those inputs, thereby forming a more effective human-computer interface.

[0006] The above-mentioned defects and other problems associated with the user interface of the computer system are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or a head-mounted device). In some embodiments, the computer system has a touch pad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, in addition to displaying a generating component, the computer system also has one or more output devices, which include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI through contact and gestures of a stylus and / or finger on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body (as captured by a camera and other motion sensors), and / or voice input (as captured by one or more audio input devices). In some embodiments, the functions performed by interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, testing support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a transient and / or non-transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0007] There is a need for electronic devices with improved methods and interfaces for interacting with content in a three-dimensional environment. Such methods and interfaces can supplement or replace conventional methods for interacting with content in a three-dimensional environment. Such methods and interfaces reduce the amount, degree, and / or nature of input from a user and produce a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges.

[0008] In some embodiments, the computer system displays a user interface element corresponding to the content shared with the computer system, the user interface element being used to interact with the shared content in a three-dimensional environment. In some embodiments, the computer system displays one or more virtual control elements in the three-dimensional environment, the one or more virtual control elements being selectable to cause execution of one or more corresponding operations involving a physical device.

[0009] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not comprehensive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art based on the drawings, the specification, and the claims. In addition, it should be noted that the language used in this specification is selected in principle for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0011] Figure 1 is a block diagram illustrating an operating environment of a computer system for providing an XR experience according to some embodiments.

[0012] Figure 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience according to some embodiments.

[0013] Figure 3 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of an XR experience to a user according to some embodiments.

[0014] Figure 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture input from a user according to some embodiments.

[0015] Figure 5 is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input of a user according to some embodiments.

[0016] Figure 6 is a flow chart illustrating a flash-assisted gaze tracking pipeline according to some embodiments.

[0017] 7A to 7F An example of a computer system receiving content shared with the computer system in a three-dimensional environment is shown according to some embodiments.

[0018] FIG. 8A to FIG. 8L is a flow chart illustrating a method of receiving content shared with a computer system in a three-dimensional environment according to some embodiments.

[0019] 9A to 9E An example of a computer system that facilitates interaction with virtual controls associated with physical devices, the virtual controls displayed in a three-dimensional environment, is shown according to some embodiments.

[0020] FIG. 10A to FIG. 10F is a flow diagram illustrating a method for facilitating interaction with virtual controls associated with a physical device, the virtual controls being displayed in a three-dimensional environment, according to some embodiments. DETAILED DESCRIPTION

[0021] According to some embodiments, the present disclosure relates to a user interface for providing a computer-generated (CGR) experience to a user.

[0022] The systems, methods, and GUIs described herein provide improved ways for electronic devices to facilitate interaction with and manipulation of objects in a three-dimensional environment.

[0023] In some embodiments, the computer system displays a three-dimensional environment with one or more virtual objects. In some embodiments, in response to detecting an indication of a request to share content with the computer system, the computer system displays a user interface element corresponding to the content in the three-dimensional environment. In some embodiments, the user interface element is viewpoint locked in the three-dimensional environment. In some embodiments, the computer system displays an object corresponding to the content based on determining that an operation related to the content has been completed. In some embodiments, the object corresponding to the content is world locked in the three-dimensional environment.

[0024] In some embodiments, a computer system displays a three-dimensional environment including one or more representations of one or more physical devices in a physical environment. In some embodiments, in response to detecting a user's attention directed to the one or more representations of the one or more physical devices, the computer system displays one or more virtual control elements associated with the one or more physical devices in the three-dimensional environment. In some embodiments, the one or more virtual control elements are world-locked relative to the one or more representations of the one or more physical devices in the three-dimensional environment. In some embodiments, the one or more virtual control elements can be selected to cause execution of one or more corresponding actions involving the one or more physical devices.

[0025] Figures 1 to 6 A description of an example computer system for providing an XR experience to a user (such as described below with respect to methods 800 and 1000 ) is provided. 7A to 7FExample techniques for interacting with notifications of shared content in a three-dimensional environment are shown according to some embodiments. FIG. 8A to FIG. 8L is a flow chart illustrating a method of interacting with notifications of shared content in a three-dimensional environment according to various embodiments. 7A to 7F The user interface in FIG. 8A to FIG. 8L process. 9A to 9E Exemplary techniques for interacting with one or more virtual control elements associated with a physical device in a three-dimensional environment are shown according to some embodiments. FIG. 10A to FIG. 10F is a flow chart of a method of interacting with one or more virtual control elements associated with a physical device in a three-dimensional environment according to various embodiments. 9A to 9E The user interface in FIG. 10A to FIG. 10F process.

[0026] The processes described below enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation without further user input when a set of conditions have been met, improving privacy and / or security, and / or additional techniques. These techniques also reduce power usage and extend the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0027] In addition, in the method described herein where one or more steps depend on one or more conditions being met, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions for determining the steps in the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the ordinary technician will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on one or more conditions being met can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether the possible situation has been met without explicitly repeating the steps of the method until all conditions for determining the steps in the method have been met. It will also be understood by ordinary technicians in the art that, similar to the method with the contingent step, the system or computer-readable storage medium can repeat the steps of the method as many times as needed to ensure that all the contingent steps have been performed.

[0028] In some embodiments, such as Figure 1 As shown in , an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a household appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head mounted device or a handheld device).

[0029] In describing an XR experience, various terms are used to distinguishably refer to several related but distinct environments that a user can sense and / or with which the user can interact (e.g., using inputs detected by the computer system 101 generating the XR experience, which inputs cause the computer system generating the XR experience to generate audio, visual, and / or tactile feedback corresponding to the various inputs provided to the computer system 101). The following is a subset of these terms:

[0030] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0031] Extended Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In XR, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. For example, an XR system may detect a person's head turn, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to the features of virtual objects in the XR environment may be made in response to a representation of physical movement (e.g., a voice command). People can sense and / or interact with XR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, people can sense and / or interact with audio objects, which create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some XR environments, a person can sense and / or interact only with an audio object.

[0032] Examples of XR include virtual reality and mixed reality.

[0033] Virtual Reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0034] Mixed Reality: In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between, but not including, a fully physical environment at one end and a virtual reality environment at the other end. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render MR environments can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment, or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0035] Examples of mixed reality include augmented reality and augmented virtuality.

[0036] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. A person uses the system to indirectly view the physical environment via an image or video of the physical environment and perceives virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, such as as a hologram or on a physical surface, so that a person using the system perceives virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not real version of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating a portion thereof or blurring a portion thereof.

[0037] Augmented Virtual: An augmented virtual (AV) environment refers to a simulated environment in which a virtual environment or a computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0038] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same position and / or location in a user's viewpoint, the virtual object is viewpoint-locked even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the forward direction of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); thus, without moving the user's head, the user's viewpoint remains fixed even when the user's gaze shifts. In embodiments where the computer system has a display generation component (e.g., a display screen) that is repositionable relative to the user's head, the user's viewpoint is an augmented reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object that is displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or location of the viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object."

[0039] Environment-locked visual objects: A virtual object is environment-locked (alternatively, "world-locked") when a computer system displays a virtual object at a position and / or location in a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) a location and / or object in a three-dimensional environment (e.g., a physical environment or a virtual environment). As the user's viewpoint moves, the location and / or objects in the environment change relative to the user's viewpoint, which causes the environment-locked virtual object to be displayed at a different location and / or location in the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) such that the tree is now to the left of center in the user's viewpoint (e.g., the tree location in the user's viewpoint shifts), the environment-locked virtual object locked to the tree is displayed to the left of center in the user's viewpoint. In other words, the position and / or orientation of the virtual object that is locked to the environment in the user's viewpoint depends on the position and / or orientation of the object and / or the position to which the virtual object is locked in the environment. In some embodiments, the computer system uses a stationary reference system (e.g., a coordinate system anchored to a fixed position and / or object in the physical environment) to determine the positioning of the virtual object that is locked to the user's viewpoint. The virtual object that is locked to the environment can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object), or can be locked to a movable part of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot of the user) so that the virtual object moves as the viewpoint or the part of the environment moves to maintain a fixed relationship between the virtual object and the part of the environment.

[0040] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits an inertial following behavior that reduces or delays the movement of the environment-locked or viewpoint-locked virtual object relative to the movement of the reference point followed by the virtual object. In some embodiments, when exhibiting inertial following behavior, when detecting the movement of a reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 cm and 300 cm from the viewpoint) that the virtual object is following, the computer system intentionally delays the movement of the virtual object. For example, when the reference point (e.g., the portion of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits inertial following behavior, the device ignores small amounts of movement of the reference point (e.g., ignoring movements of the reference point below a threshold movement amount, such as moving 0 to 5 degrees or moving 0 to 50 cm). For example, when a reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and when the reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., a “lazy follow” threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some embodiments, maintaining a substantially fixed position of the virtual object relative to the reference point includes displaying the virtual object within a threshold distance (e.g., 1 cm, 2 cm, 3 cm, 5 cm, 15 cm, 20 cm, 50 cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the position of the reference point).

[0041] Hardware: There are many different types of electronic systems that enable a person to sense various XR environments and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to become selectively opaque. The projection-based system can employ retinal projection technology that projects graphic images onto a person's retina. The projection system can also be configured to project virtual objects into a physical environment, such as as a hologram or on a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Reference is made below to Figure 2The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device that is located locally or remotely relative to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. As another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the display generation component 120 (e.g., HMD, display, projector, touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE802.3x, etc.). In another example, the controller 110 is included within a housing (e.g., a physical housing) of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors, etc.), one or more input devices of the input devices 125, one or more output devices of the output devices 155, one or more sensors of the sensors 190, and / or one or more peripheral devices of the peripheral devices 195, or shares the same physical housing or support structure with one or more of the above devices.

[0042] In some embodiments, the display generation component 120 is configured to provide an XR experience (e.g., at least the visual component of the XR experience) to the user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Display generation component 120 is described in further detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0043] According to some embodiments, display generation component 120 provides an XR experience to the user when the user is virtually and / or physically present within scene 105.

[0044] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head, on his / her hand, etc.). In this way, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smart phone or tablet) configured to present XR content, and the user holds a device with a display facing the user's field of view and a camera facing the scene 105. In some embodiments, the handheld device is optionally placed in a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is an XR room, housing, or room configured to present XR content, wherein the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in the space in front of a handheld device or a tripod-mounted device may similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and responses to the XR content are displayed via the HMD. Similarly, a user interface showing interactions with XR content triggered based on movement of a handheld device or a tripod-mounted device relative to a physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)) may similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)).

[0045] Despite Figure 1 Relevant features of operating environment 100 are shown, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein.

[0046] Figure 2is a block diagram of an example of a controller 110 according to some embodiments. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), IEEE802.3x, IEEE 802.11x, IEEE802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0047] In some embodiments, the one or more communication buses 204 include circuits that interconnect and control communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0048] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 230 and an XR experience module 240.

[0049] The operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0050] In some embodiments, the data acquisition unit 241 is configured to obtain Figure 1 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data acquisition unit 241 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0051] In some embodiments, tracking unit 242 is configured to map scene 105 and track at least display generation component 120 relative to Figure 1 105, and optionally relative to the position / location of one or more of the tracked input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand, and / or the position / location of one or more parts of the user's hand relative to the user's hand. Figure 1 The movement of the scene 105 relative to the display generation component 120 and / or relative to a coordinate system (the coordinate system is defined relative to the user's hand). Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the position or movement of the user's gaze (or more generally, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hands)) or relative to the XR content displayed via the display generation component 120. Figure 5 The eye tracking unit 243 is described in more detail.

[0052] In some embodiments, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripherals 195. To this end, in various embodiments, coordination unit 246 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0053] In some embodiments, data transfer unit 248 is configured to transfer data (e.g., presentation data, position data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data transfer unit 248 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0054] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are illustrated as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may be located in separate computing devices.

[0055] also, Figure 2 It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0056] Figure 31 is a block diagram of an example of a display generation component 120 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the display generation component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 8023x, IEEE80211x, IEEE 80216x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0057] In some embodiments, one or more communication buses 304 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0058] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to the user. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more XR displays 312 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. In another example, the display generation component 120 includes an XR display for each eye of the user. In some embodiments, one or more XR displays 312 can present MR and VR content. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0059] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to the user's hands and, optionally, at least a portion of the user's arms (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to a scene that the user would see in the absence of the display generating component 120 (e.g., an HMD) (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0060] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDRRAM or other random access solid-state memory devices. In some embodiments, the memory 320 includes a non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or their subsets, including an optional operating system 330 and an XR rendering module 340.

[0061] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR rendering module 340 is configured to present XR content to a user via one or more XR displays 312. To this end, in various embodiments, the XR rendering module 340 includes a data acquisition unit 342, an XR rendering unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0062] In some embodiments, the data acquisition unit 342 is configured to at least Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For the purposes described, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0063] In some embodiments, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. For such purposes, in various embodiments, the XR rendering unit 344 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0064] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which a computer-generated object can be placed to generate an extended reality) based on the media content data. For such purposes, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0065] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, position data, etc.) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For such purposes, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0066] Although the data acquisition unit 342, the XR rendering unit 344, the XR map generation unit 346, and the data transfer unit 348 are shown as residing on a single device (e.g., Figure 1 , but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0067] also, Figure 3 More as a functional description of various features that may be present in a particular embodiment, rather than a schematic diagram of the structures of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0068] Figure 4 is a schematic illustration of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1 ) is controlled by the hand tracking unit 244 ( Figure 2 ) to track the position / location of one or more parts of a user's hand and / or one or more parts of a user's hand relative to Figure 1The hand tracking device 140 can be used to monitor movement of the scene 105 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system (which is defined relative to the user's hands)). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to the head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0069] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a human user's hand 406. The image sensor 404 captures hand images with sufficient resolution so that fingers and their corresponding positions can be distinguished. The image sensor 404 typically captures images of other parts of the user's body, and also or possibly all parts of the body, and may have zoom capabilities or a dedicated sensor with increased magnification to capture images of the hand with a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or is used as an image sensor to capture the physical environment of the scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a manner that the field of view of the image sensor 404 or a portion thereof is used to define an interaction space, in which the hand movements captured by the image sensor are considered as input to the controller 110.

[0070] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D mapping data (and in addition, possible color image data) to the controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided to an application running on the controller via an application program interface (API), which drives the display generation component 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving his hand 406 and changing his hand posture.

[0071] In some embodiments, the image sensor 404 projects a speckled pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral offset of the spots in the pattern. This method is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. The method gives the depth coordinates of points in the scene at a specific distance from the image sensor 404 relative to a predetermined reference plane. In the present disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis, so that the depth coordinates of points in the scene correspond to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods based on a single or multiple cameras or other types of sensors, such as stereo imaging or time-of-flight measurement.

[0072] In some embodiments, the hand tracking device 140 captures and processes a time series of depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or the processor in the controller 110 processes the 3D mapping data to extract image patch descriptors of the hand in these depth maps. The software can match these descriptors with image patch descriptors stored in the database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and finger tips.

[0073] The software can also analyze the trajectory of the hand and / or finger over multiple frames in the sequence to recognize gestures. The pose estimation function described herein can be alternated with the motion tracking function so that the image block-based pose estimation is performed only once every two (or more) frames, and the tracking is used to find the changes in pose that occur on the remaining frames. The pose, motion, and gesture information is provided to the application running on the controller 110 via the above-mentioned API. The program can, for example, move and modify the image presented on the display generation component 120 in response to the pose and / or gesture information, or perform other functions.

[0074] In some embodiments, gestures include air gestures. An air gesture is a gesture detected without the user touching an input element that is part of a device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) (or independent of an input element that is part of a device) and based on detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of a user's finger relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0075] In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with an XR environment (e.g., a virtual or mixed reality environment) performed by movement of a user's fingers relative to other fingers or parts of the user's hand. In some embodiments, an air gesture is a gesture detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and based on detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture including a predetermined speed or rotation amount of a part of the user's body)).

[0076] In some embodiments where the input gesture is an in-air gesture (e.g., in the absence of physical contact with an input device that provides information to the computer system about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Thus, in embodiments involving in-air gestures, for example, the input gesture is combined (e.g., simultaneously) with movement of the user's fingers and / or hand to detect attention (e.g., gaze) toward a user interface element to perform a pinch and / or tap input, as described below.

[0077] In some embodiments, an input gesture pointing to a user interface object is performed directly or indirectly with reference to the user interface object. For example, the user input is performed directly on the user interface object according to performing input with the user's hand at a location corresponding to the location of the user interface object in the three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some embodiments, when the user's attention (e.g., gaze) to the user interface object is detected, the input gesture is performed indirectly on the user interface object according to the location of the user's hand not being at the location corresponding to the location of the user interface object in the three-dimensional environment while the user performs the input gesture. For example, for a direct input gesture, the user can guide the user's input to the user interface object by initiating a gesture at or near a location corresponding to the display location of the user interface object (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge of the option or the center portion of the option). For an indirect input gesture, the user can guide the user's input to the user interface object by focusing on the user interface object (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location that does not correspond to the display location of the user interface object).

[0078] In some embodiments, according to some embodiments, the input gestures (e.g., air gestures) used in various examples and embodiments described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0079] In some embodiments, the pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes movement of two or more fingers of a hand to contact each other, that is, optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact with each other. A long pinch gesture as an air gesture includes movement of two or more fingers of a hand in contact with each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact with each other is detected. For example, a long pinch gesture includes the user maintaining a pinch gesture (e.g., in which two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture as an air gesture includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected consecutively immediately (e.g., within a predefined time period) with each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts contact between two or more fingers), and performs a second pinch input within a predefined time period after releasing the first pinch input (e.g., within 1 second or within 2 seconds).

[0080] In some embodiments, the pinch and drag gesture as an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., following) a drag input that changes the position of the user's hand from a first position (e.g., the start position of the drag) to a second position (e.g., the end position of the drag). In some embodiments, the user maintains the pinch gesture while performing the drag input, and releases the pinch gesture (e.g., opens their two or more fingers) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers to contact each other and moves the same hand to a second position in the air using the drag gesture). In some embodiments, a pinch input is performed by a first hand of a user, and a drag input is performed by a second hand of the user (e.g., while the user continues the pinch input with the user's first hand, the user's second hand moves in the air from a first position to a second position. In some embodiments, the input gesture as an air gesture includes an input performed using both hands of the user (e.g., a pinch and / or tap input). For example, the input gesture includes two (e.g., or more) pinch inputs performed in conjunction with each other (e.g., simultaneously or within a predefined time period). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) is performed using the first hand of the user, and in conjunction with the pinch input performed using the first hand, a second pinch input is performed using another hand (e.g., a second hand of the user's two hands). In some embodiments, movement between the two hands of the user (e.g., increasing and / or decreasing the distance or relative orientation between the two hands of the user)

[0081] In some embodiments, a tap input performed as an air gesture (e.g., pointing to a user interface element) includes movement of a user's finger toward the user interface element, movement of a user's hand toward the user interface element (optionally, extension of the user's finger toward the user interface element), downward movement of a user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of a user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture movement of the finger or hand, which is a movement of the finger or hand away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by an end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of the movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of movement of the finger or hand, and / or a reversal of the acceleration direction of the movement of the finger or hand).

[0082] In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment (optionally, no other conditions are required). In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment using one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., a dwell duration) and / or requiring the gaze to be directed to the portion of the three-dimensional environment when the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment, so that the device determines that the user's attention is directed to the portion of the three-dimensional environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0083] In some embodiments, detection of a ready state configuration of a user or a portion of a user is detected by a computer system. Detection of a ready state configuration of a hand is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein) performed by the hand. For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart to prepare for a pinch or grab gesture, or a pre-tap with one or more fingers extended and the palm facing away from the user), based on whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or based on whether the hand has moved in a particular manner (e.g., toward an area in front of the user above the user's waist and below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of a user interface responds to attention (e.g., gaze) input.

[0084] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or may alternatively be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is also stored in a memory associated with the controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although in Figure 4Controller 110 is shown in FIG. 1 , but some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand tracking device) or other device associated with the image sensor 404, for example, as a separate unit from the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing functions of the image sensor 404 may also be integrated into a computer or other computerized device to be controlled by the sensor output.

[0085] Figure 4 Also included is a schematic diagram of a depth map 410 captured by the image sensor 404 according to some embodiments. As described above, the depth map includes a matrix of pixels with corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z distance from the image sensor 404), where gray shades become darker as depth increases. The controller 110 processes these depth values ​​in order to identify and segment components of the image that have human hand features (i.e., a group of adjacent pixels). These features may include, for example, overall size, shape, and motion from frame to frame in the depth map sequence.

[0086] Figure 4 Also schematically shown is a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406 according to some embodiments. Figure 4 In the hand skeleton 414, a hand background 416 that has been segmented from the original depth map is superimposed. In some embodiments, key feature points of the hand and optionally on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, finger tips, palm center, end of the hand connected to the wrist, etc.) are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points over multiple image frames to determine the gesture performed by the hand or the current state of the hand according to some embodiments.

[0087] Figure 5 The eye tracking device 130 ( Figure 1 ). In some embodiments, the eye tracking device 130 is composed of an eye tracking unit 243 ( Figure 2) controls to track the position and movement of the user's gaze relative to the scene 105 or relative to the XR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as headphones, helmets, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for viewing by the user and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye tracking device 130 is optionally a device separate from the handheld device or the XR room. In some embodiments, the eye tracking device 130 is a head-mounted device or a part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or a display generation component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally used in conjunction with a head mounted display generation component. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally part of a non-head mounted display generation component.

[0088] In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display a frame including a left image and a right image in front of the user's eyes, thereby providing a 3D virtual view to the user. For example, the head-mounted display generation component may include a left optical lens and a right optical lens (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture a video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display, and display a virtual object on the transparent or translucent display, and the user can directly view the physical environment through the transparent or translucent display. In some embodiments, the display generation component projects the virtual object into the physical environment. The virtual object may, for example, be projected on a physical surface or projected as a hologram so that an individual using the system observes the virtual object superimposed on the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

[0089] like Figure 5As shown in , in some embodiments, the eye tracking device 130 (e.g., gaze tracking device) includes at least one eye tracking camera (e.g., an infrared (IR) or near infrared (NIR) camera), and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive the IR or NIR light reflected directly from the eyes by the light source, or alternatively can be pointed at "hot" mirrors located between the user's eyes and the display panel, which reflect the IR or NIR light from the eyes to the eye tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and transmits the gaze tracking information to the controller 110. In some embodiments, the user's two eyes are tracked separately by corresponding eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by the corresponding eye tracking camera and illumination source.

[0090] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for a specific operating environment 100, such as the 3D geometric relationships and parameters of the LED, camera, thermal mirror (if present), eye lens, and display screen. The device-specific calibration process can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration process can be an automatic calibration process or a manual calibration process. According to some embodiments, the user-specific calibration process may include an estimate of the eye parameters of a specific user, such as pupil position, foveal position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific parameters and user-specific parameters are determined for the eye tracking device 130, a flash-assisted method can be used to process the images captured by the eye tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0091] like Figure 5As shown in , the eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system, which includes at least one eye tracking camera 540 (e.g., an infrared (IR) or near infrared (NIR) camera) positioned on the side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye tracking camera 540 can be directed toward a mirror 550 (which reflects IR or NIR light from the eye 592 while allowing visible light to pass) located between the user's eye 592 and a display 510 (e.g., a left display panel or a right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) Figure 5 ), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion of Figure 5 ).

[0092] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0093] Several possible use cases for the user's current gaze direction are described below and are not intended to be limiting. As an exemplary use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content at a higher resolution in the foveal region determined according to the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller may display specific virtual content in a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 may direct an external camera used to capture the physical environment of the XR experience to focus in the determined direction. The autofocus mechanism of the external camera may then focus on an object or surface in the environment that the user is currently looking at on the display 510. As another exemplary use case, the eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of the eye lens 520 so that the virtual object that the user is currently looking at has an appropriate degree of convergence to match the convergence of the user's eye 592. The controller 110 can use the gaze tracking information to guide the eye lens 520 to adjust the focus so that nearby objects that the user is looking at appear at the correct distance.

[0094] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lenses 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)). The light source emits light (e.g., IR or NIR light) toward the user's eye 592. In some embodiments, the light sources may be arranged in a ring or circle around each of the lenses, such as Figure 5 In some embodiments, for example, eight light sources 530 (eg, LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0095] In some embodiments, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. Note that the positions and angles of the eye tracking cameras 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0096] like Figure 5 Embodiments of the gaze tracking system shown in may, for example, be used in computer generated reality, virtual reality, and / or mixed reality applications to provide a computer generated reality, virtual reality, augmented reality, and / or augmented virtual experience to a user.

[0097] Figure 6 FIG. 1 shows a flash-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a flash-assisted gaze tracking system (e.g., Figure 1 and Figure 5 The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and the flash in the current frame. When not in the tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and the flash in the current frame, and if successful, initializes the tracking state to "yes" and continues with the next frame in the tracking state.

[0098] like Figure 6 As shown in , the gaze tracking camera can capture left and right images of the user's left and right eyes. The captured images are then input to the gaze tracking pipeline for processing starting at 610. As indicated by the arrow returning to element 600, the gaze tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each group of captured images can be input to the pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are processed by the pipeline.

[0099] At 610, for the currently captured image, if the tracking status is yes, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as indicated at 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0100] At 640, if advancing from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from previous frames. At 640, if advancing from element 630, the tracking state is initialized based on the pupils and glints detected in the current frame. The processing results at element 640 are checked to verify that the results of tracking or detection can be credible. For example, the results can be checked to determine whether the pupil and a sufficient number of glints for performing gaze estimation are successfully tracked or detected in the current frame. At 650, if the results are not likely to be credible, at element 660, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the results are credible, the method advances to element 670. At 670, the tracking state is set to YES (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0101] Figure 6 It is intended to be used as an example of an eye tracking technology that can be used for a particular implementation. As recognized by one of ordinary skill in the art, according to various embodiments, other eye tracking technologies currently existing or developed in the future can be used in place of or in combination with the flash-assisted eye tracking technology described herein in a computer system 101 for providing an XR experience to a user.

[0102] In some embodiments, the captured portion of the real-world environment 602 is used to provide an XR experience to the user, such as a mixed reality environment in which one or more virtual objects are superimposed on top of a representation of the real-world environment 602 .

[0103] Thus, the description herein describes some embodiments of a three-dimensional environment (e.g., an XR environment) including representations of real-world objects and representations of virtual objects. For example, the three-dimensional environment optionally includes a representation of a table present in a physical environment, which is captured and displayed in the three-dimensional environment (e.g., actively displayed via a camera and display of a computer system or passively displayed via a transparent or translucent display of a computer system). As previously described, the three-dimensional environment is optionally a mixed reality system, wherein the three-dimensional environment is based on a physical environment captured by one or more sensors of a computer system and displayed via a display generation component. As a mixed reality system, the computer system is optionally capable of selectively displaying parts and / or objects of the physical environment so that the corresponding parts and / or objects of the physical environment appear as if they exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system is optionally capable of displaying virtual objects in a three-dimensional environment by placing virtual objects at corresponding positions in the three-dimensional environment that have corresponding positions in the real world to appear as if the virtual objects exist in the real world (e.g., a physical environment). For example, the computer system optionally displays a vase so that the vase appears as if a real vase is placed on top of a table in the physical environment. In some embodiments, a corresponding position in the three-dimensional environment has a corresponding position in the physical environment. Thus, when a computer system is described as displaying a virtual object at a corresponding position relative to a physical object (e.g., such as a position at or near a user's hand or at or near a physical table), the computer system displays the virtual object at a specific position in the three-dimensional environment so that it appears as if the virtual object is at or near the physical object in the physical environment (e.g., the virtual object is displayed at a position in the three-dimensional environment that corresponds to the position in the physical environment where the virtual object would be displayed if it were a real object at that specific position).

[0104] In some embodiments, real-world objects present in the physical environment that are displayed in the three-dimensional environment (e.g., and / or visible via a display generation component) can interact with virtual objects that only exist in the three-dimensional environment. For example, the three-dimensional environment may include a table and a vase placed on top of the table, where the table is a view (or representation) of a physical table in the physical environment and the vase is a virtual object.

[0105] Similarly, the user is optionally able to use one or both hands to interact with virtual objects in the three-dimensional environment as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the computer system optionally capture one or more hands of the user and display representations of the user's hands in the three-dimensional environment (e.g., in a manner similar to displaying real-world objects in the three-dimensional environment described above), or in some embodiments, due to the transparency / translucency of a portion of the user interface being displayed by the display generation component, or due to the projection of the user interface onto a transparent / translucent surface or the projection of the user interface onto the user's eyes or into the field of view of the user's eyes, the user's hands can be seen via the display generation component, via the ability to see the physical environment through the user interface. Therefore, in some embodiments, the user's hands are displayed at corresponding locations in the three-dimensional environment and are treated as if they are objects in the three-dimensional environment, and these objects can interact with virtual objects in the three-dimensional environment as if these virtual objects are physical objects in the physical environment. In some embodiments, the computer system can update the display of the representation of the user's hands in the three-dimensional environment in conjunction with the movement of the user's hands in the physical environment.

[0106] In some of the embodiments described below, the computer system is optionally capable of determining an "effective" distance between a physical object in the physical world and a virtual object in a three-dimensional environment, for example, to determine whether the physical object is directly interacting with the virtual object (e.g., whether the hand is touching, grabbing, holding, etc. a virtual object or is within a threshold distance of the virtual object). For example, a hand that directly interacts with a virtual object optionally includes one or more of the following: a finger of a hand pressing a virtual button, a user's hand grabbing a virtual vase, a user's hand closing together and pinching / holding the user interface of an application, and two fingers that perform any other type of interaction described herein. For example, when determining whether a user is interacting with a virtual object and / or how a user is interacting with a virtual object, the computer system optionally determines the distance between the user's hand and the virtual object. In some embodiments, the computer system determines the distance between the user's hand and the virtual object by determining the distance between the position of the hand in the three-dimensional environment and the position of the virtual object of interest in the three-dimensional environment. For example, the one or more hands of the user are located at a specific location in the physical world, and the computer system optionally captures the one or more hands and displays the one or more hands at a specific corresponding location in the three-dimensional environment (e.g., if the hand is a virtual hand instead of a physical hand, the location where the hand will be displayed in the three-dimensional environment). The location of the hand in the three-dimensional environment is optionally compared with the location of the virtual object of interest in the three-dimensional environment to determine the distance between the one or more hands of the user and the virtual object. In some embodiments, the computer system optionally determines the distance between the physical object and the virtual object by comparing the location in the physical world (e.g., instead of comparing the location in the three-dimensional environment). For example, when determining the distance between the one or more hands of the user and the virtual object, the computer system optionally determines the corresponding position of the virtual object in the physical world (e.g., if the virtual object is a physical object instead of a virtual object, the location where the virtual object will be located in the physical world), and then determines the distance between the corresponding physical location and the one or more hands of the user. In some embodiments, the same technology is optionally used to determine the distance between any physical object and any virtual object. Therefore, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, a computer system optionally executes any of the techniques described above to map the position of the physical object to a three-dimensional environment and / or to map the position of the virtual object to the physical environment.

[0107] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed to, and / or where and what the physical stylus held by the user is pointed to. For example, if the user's gaze is directed to a particular location in the physical environment, the computer system optionally determines the corresponding location in the three-dimensional environment (e.g., the virtual location of the gaze), and if a virtual object is located at the corresponding virtual location, the computer system optionally determines that the user's gaze is directed to the virtual object. Similarly, the computer system is optionally able to determine the direction in which the stylus is pointing in the physical environment based on the orientation of the physical stylus. In some embodiments, based on this determination, the computer system determines a corresponding virtual location in the three-dimensional environment that corresponds to the location in the physical environment that the stylus is pointing to, and optionally determines that the stylus is pointing to the corresponding virtual location in the three-dimensional environment.

[0108] Similarly, the embodiments described herein may refer to the position of a user (e.g., a user of a computer system) in a three-dimensional environment and / or the position of a computer system in a three-dimensional environment. In some embodiments, the user of the computer system is holding, wearing, or otherwise located at or near the computer system. Therefore, in some embodiments, the position of the computer system is used as a proxy for the position of the user. In some embodiments, the position of the computer system and / or the user in the physical environment corresponds to the corresponding position in the three-dimensional environment. For example, the position of the computer system will be a position in the physical environment (and its corresponding position in the three-dimensional environment), and if the user stands at this position, facing the corresponding part of the physical environment visible via the display generation component, the user will see from this position in the physical environment in the same positioning, orientation and / or size (e.g., in an absolute sense and / or relative to each other) of the objects displayed in the three-dimensional environment by the display generation component of the computer system or visible in the three-dimensional environment via the display generation component. Similarly, if the virtual objects displayed in the three-dimensional environment are physical objects in the physical environment (e.g., physical objects placed at the same locations in the physical environment as the virtual objects are located in the three-dimensional environment, and physical objects that have the same size and orientation in the physical environment as they do in the three-dimensional environment), then the position of the computer system and / or user is the position from which the user would see the positions of the virtual objects in the physical environment at the same positions, orientations, and / or sizes (e.g., in an absolute sense and / or relative to each other and real-world objects) as the virtual objects displayed in the three-dimensional environment by the display generation components of the computer system.

[0109] In the present disclosure, various input methods are described with respect to interaction with a computer system. When an input device or input method is used to provide an example, and another input device or input method is used to provide another example, it should be understood that each example is compatible with the input device or input method described with respect to another example and optionally utilizes the input device or input method. Similarly, various output methods are described with respect to interaction with a computer system. When an output device or output method is used to provide an example, and another output device or output method is used to provide another example, it should be understood that each example is compatible with the output device or output method described with respect to another example and optionally utilizes the output device or output method. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment by a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with the method described with respect to another example and optionally utilizes these methods. Therefore, the present disclosure discloses an embodiment as a combination of features of multiple examples, without exhaustively listing all features of the embodiment in the description of each exemplary embodiment.

[0110] User interface and associated processes

[0111] Attention is now turned to embodiments of a user interface ("UI") and associated processes that may be implemented on a computer system (such as a portable multifunction device or a head mounted device) having display generating components, one or more input devices, and (optionally) one or more cameras.

[0112] 7A to 7F An example of a computer system receiving content shared with the computer system in a three-dimensional environment is shown according to some embodiments.

[0113] Fig. 7A The computer system (eg, electronic device) 101 is shown via a display generating component (eg, Figure 1 The display generation component 120 of the computer system 101 displays the three-dimensional environment 702 from the viewpoint of the user 726 shown in a top view (e.g., facing the back wall of the physical environment in which the computer system 101 is located). In some embodiments, the computer system 101 includes a display generation component (e.g., a touch screen) and multiple image sensors (e.g., Figure 3The image sensor 314). The image sensor optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the computer system 101 can use to capture one or more images of a user or a portion of a user (e.g., one or more hands of a user) when the user interacts with the computer system 101. In some embodiments, the user interface shown and described below can also be implemented on a head-mounted display, which includes a display generation component that displays a user interface or a three-dimensional environment to the user, and sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's attention (e.g., gaze) (e.g., internal sensors facing inward toward the user's face).

[0114] like Fig. 7A As shown, the computer system 101 captures one or more images of the physical environment (e.g., operating environment 100) surrounding the computer system 101 (including one or more objects in the physical environment surrounding the computer system 101). In some embodiments, the computer system 101 displays a representation of the physical environment in a three-dimensional environment 702. For example, the three-dimensional environment 702 includes a representation 722a of a coffee table (corresponding to the table 722b in the top view), which is optionally a representation of a physical coffee table in the physical environment, and the three-dimensional environment 702 includes a representation 724a of a sofa (corresponding to the sofa 724b in the top view), which is optionally a representation of a physical sofa in the physical environment.

[0115] exist Fig. 7A, the three-dimensional environment 702 also includes user interface elements 704a (corresponding to element 704b in the top view) and 706a (corresponding to element 706b in the top view) and a virtual object 708a (corresponding to object 708b in the top view). User interface elements 704a and 706a are optionally at the same distance from the viewpoint of user 726, and the distance of virtual object 708a from the viewpoint of user 726 is optionally greater than the distance of user interface elements 704a and 706a from the user's viewpoint. In some embodiments, user interface elements 704a and 706a correspond to notifications of content that have been shared with computer system 101. For example, user interface elements 704a and 706a are notifications of shared content displayed in the three-dimensional environment 702 in response to computer system 101 detecting an indication of a request to share content with computer system 101. In some embodiments, the content corresponding to user interface elements 704a and 706a is shared with computer system 101 by a second computer system (e.g., a second electronic device such as a mobile device, desktop computer, laptop computer, or tablet computer) configured to communicate with computer system 101. In some embodiments, virtual object 708a is one or more of the following: a user interface of an application containing content (e.g., a web browser application, a messaging application, or a quick view window that displays or contains content), a three-dimensional object (e.g., a virtual clock, a virtual ball, a virtual car, etc.), or any other element displayed by computer system 101 that is not included in the physical environment of computer system 101. It should be understood that although in 7A to 7F Multiple user interface elements 706a and 708a are shown, but such user interface elements need not be displayed simultaneously by computer system 101; rather, in some embodiments, computer system 101 independently displays the user interface elements shown and described in response to detecting separate indications corresponding to requests to independently share content with computer system 101.

[0116] In some embodiments, when computer system 101 identifies a corresponding sender of content corresponding to a request to share content with computer system 101, a user interface element corresponding to the content shared with computer system 101 is displayed in three-dimensional environment 702 along with a preview of the content. Fig. 7AAs shown, the content corresponding to user interface element 704a is optionally shared by a corresponding sender identified by computer system 101. For example, a second computer system configured to communicate with computer system 101 and known to computer system 101 (e.g., a user profile, phone number, and / or email address associated with the second computer system is stored and / or accessible at computer system 101 (e.g., as a known contact)). Therefore, user interface element 704a is displayed in three-dimensional environment 702 together with preview 710. In some embodiments, preview 710 of the content includes at least a portion of the content shared with computer system 101 (e.g., a representation thereof). The content corresponding to user interface element 706a is optionally shared by a corresponding sender not identified by computer system 101. For example, a second computer system configured to communicate with computer system 101 but unknown to computer system 101 (e.g., a user profile, phone number, and / or email address associated with the second computer system is not stored and / or accessible at computer system 101 (e.g., no known contact exists at computer system 101)). Therefore, user interface element 706a is not displayed with the preview in three-dimensional environment 702, as shown in FIG. Fig. 7A shown.

[0117] In some embodiments, the content shared with computer system 101 may include multiple content items (e.g., multiple files corresponding to separate content items) that are shared with computer system 101 at the same time. In some embodiments, the multiple content items include various types of content. For example, the content shared with computer system 101 optionally includes a first content item ("File 1") 711, a second content item ("File 2") 713, and a third content item ("File 3") 715. As described above, in some embodiments, the content corresponding to user interface element 704a is shared by the corresponding sender identified by computer system 101, and thus user interface element 704a is displayed with preview 710. In some embodiments, when the content shared with computer system 101 includes multiple content items, the preview 710 of the content includes multiple of the content items or at least a portion of each of the content items (e.g., a representation thereof). For example, as Fig. 7AAs shown, preview 710 includes a representation of a first content item 711, which is optionally a contact, a representation of a second content item 713, which is optionally a photo, and a representation of a third content item 715, which is optionally a song. In some embodiments, the representations of the content items included in preview 710 include information and / or detailed information corresponding to the content items. For example, the representation of first content item 711 optionally includes a contact identifier corresponding to a contact shared with computer system 101 (e.g., a portion of a photo or avatar of a contact, the initials of a contact's name, a portion of a contact's name, etc.). The representation of second content item 713 optionally includes a portion of a photo shared with computer system 101. The representation of third content item 715 optionally includes a song identifier corresponding to a song shared with computer system 101 (e.g., a portion of the artwork corresponding to the song, the title of the song, the album of which the song is a part, etc.).

[0118] In some embodiments, a user interface element corresponding to content shared with computer system 101 is displayed in three-dimensional environment 702 without stopping display of other virtual objects (e.g., virtual objects that were displayed before receiving an indication corresponding to a request to share content with computer system 101) in three-dimensional environment 702. As mentioned above, three-dimensional environment 702 optionally includes virtual object 708a. In some embodiments, virtual object 708a is displayed in three-dimensional environment 702 before computer system 101 detects an indication corresponding to a request to share content with computer system 101. Fig. 7A As shown, in some embodiments, the computer system 101 maintains display of the virtual object 708a while displaying the user interface elements 704a and / or 706a in the three-dimensional environment 702. For example, the user interface elements 704a and / or 706a are displayed in the three-dimensional environment 702 regardless of the state and / or position of the virtual object 708a in the three-dimensional environment 702. Thus, the user may (e.g., continue to) interact with the virtual object 708a despite the user interface elements 704a and / or 706a being displayed in the three-dimensional environment 702. In some embodiments, detection of one or more corresponding notification events (e.g., notification events corresponding to an application running at and / or stored at the computer system 101, notification events corresponding to an incoming message or email message, and / or notification events corresponding to an incoming phone call) other than an indication corresponding to a request to share content with the computer system 101 causes the computer system 101 to stop displaying the virtual object 708a in the three-dimensional environment 702 when it displays one or more corresponding indications for those corresponding notification events. In some embodiments, computer system 101 maintains display of user interface elements 704a and / or 706a in response to detecting one or more corresponding notification events.

[0119] In some embodiments, a user interface element corresponding to content shared with computer system 101 is displayed in three-dimensional environment 702 at a predetermined portion (e.g., center) of the field of view of the three-dimensional environment from the viewpoint of user 726 (e.g., before receiving input to interact with the user interface element in three-dimensional environment 702 (which will be described later)). Fig. 7A As shown, in response to detecting an indication corresponding to a request to share content with computer system 101, computer system 101 optionally displays user interface elements 704a and / or 706a, respectively, at the center of the user's field of view of three-dimensional environment 702. In some embodiments, user interface elements 704a and / or 706a are displayed in three-dimensional environment 702 based on a corresponding portion of user 726. For example, user interface elements 704a and / or 706a are displayed in three-dimensional environment 702 based on a position and / or orientation of the user's head (e.g., or a portion thereof), such as a particular distance (e.g., 1 cm, 3 cm, 5 cm, 10 cm, 20 cm, 50 cm, 100 cm, 500 cm, 1000 cm, or 5000 cm) from the center of the user's head (e.g., so at the center of the user's viewpoint, as discussed above).

[0120] In some embodiments, the position of a user interface element corresponding to content shared with the computer system 101 in the three-dimensional environment 702 changes in response to detecting a change in the user's viewpoint of the three-dimensional environment 702. For example, as discussed below, the change in the user's viewpoint of the three-dimensional environment 702 causes the computer system 101 to display the user interface element corresponding to the content at a new position in the three-dimensional environment 702, such as the center of the field of view of the three-dimensional environment 702 from the user's updated viewpoint. Thus, in some embodiments, the user interface elements 704a and / or 706a are viewpoint locked until a specific input, described in more detail below, is directed to the user interface elements 704a and / or 706a. In some embodiments, when the computer system 101 detects that the user's attention is directed to a user interface element in the three-dimensional environment 702, the computer system 101 displays the user interface element with a greater level of prominence, as discussed below.

[0121] exist Fig. 7A , hand 703a is providing movement input directed toward computer system 101 that corresponds to a change in the user's viewpoint of three-dimensional environment 702 (e.g., and / or a change in the viewpoint of a portion of the physical environment visible via display generation component 120 of computer system 101). FIG. 7A to FIG. 7B, hand 703a is optionally providing input for moving computer system 101 to the right in the physical environment surrounding computer system 101. In some embodiments, the input for changing the viewpoint of user 726 corresponds to movement of the head of user 726 in the physical environment (e.g., movement of a head mounted display worn by user 726 in the physical environment). In some embodiments, computer system 101 detects that hand 703a moves computer system 101 (e.g., and display generation component 120) to a new corresponding position in the physical environment.

[0122] In addition, Fig. 7A , attention (e.g., gaze of user 726) is directed to user interface elements 704a and / or 706a, respectively. FIG. 7A to FIG. 7B , the first gaze ("gaze 1") 721 is directed to the user interface element 704a, and the second gaze ("gaze 2") 723 is directed to the user interface element 706a. It should be understood that although 7A to 7F Multiple gaze points are shown in the figure, but such gaze points need not be detected simultaneously by the computer system 101; rather, in some embodiments, the computer system 101 independently responds to such gaze points in response to independently detecting the gaze points shown and described.

[0123] In response to Fig. 7A In response to the detected change in the user's viewpoint of the three-dimensional environment 702, the computer system 101 optionally updates the displayed portion of the three-dimensional environment 702 accordingly and moves the user interface elements 704a and 706a in the three-dimensional environment 702 accordingly, such as Figure 7B For example, user interface elements 704a and / or 706a are displayed at a new location in three-dimensional environment 702 that corresponds to the center of the field of view of three-dimensional environment 702 from the new viewpoint of user 726. In some embodiments, computer system 101 updates the display of user interface elements 704a and / or 706a as the viewpoint of user 726 changes. For example, in Fig. 7A , when the user's hand 703a moves to the right in the physical environment to change the viewpoint of the user 726, the computer system 101 optionally animates the movement of the user interface elements 704a and / or 706a so that the user interface elements 704a and / or 706a are maintained in the center of the field of view of the three-dimensional environment 702 from the changing viewpoint of the user 726.

[0124] Furthermore, in some embodiments, in response to Fig. 7A In the embodiment, the computer system 101 detects a gaze directed to the user interface elements 704a and / or 706a, respectively, and displays the user interface elements 704a and 706a with a greater level of prominence, such as Figure 7BFor example, the size of user interface elements 704a and / or 706a is increased in the three-dimensional environment 702, as shown in FIG. Figure 7B Additionally or alternatively, the distance between the user interface elements 704a and / or 706a and the viewpoint of the user 726 is optionally reduced, as shown in FIG. Figure 7B 704a and 706a from the viewpoint of user 726, to avoid objects consuming a large portion of the field of view of user 726 from their current viewpoint. Thus, in some embodiments, user interface elements 704a and 704b are associated with appropriate or optimal sizes for their current distance from the viewpoint of user 726, and computer system 101 automatically changes the sizes of user interface elements 704a and 706a to conform to their appropriate or optimal sizes.

[0125] In some embodiments, in response to detecting a gaze directed toward user interface elements 704a and / or 706a, respectively, computer system 101 displays one or more options for accepting or denying a request to share content with computer system 101. For example, Figure 7BAs shown, the one or more options include acceptance options 717a / 717b and rejection options 719a / 719ba. As shown, acceptance options 717a and rejection options 719a are optionally displayed together with a part of user interface element 704a (for example, above, below, on one side or inside the part), and / or acceptance options 717b and rejection options 719b are optionally displayed together with a part of user interface element 706a. In some embodiments, as discussed in more detail below, the selection of the acceptance option 717a of user interface element 704a makes the content (for example, the first content item 711, the second content item 713 and the third content item 715) corresponding to the user interface element 704a and shared with the computer system 101 is downloaded to the computer system 101 (for example, stored at the computer system and / or can be accessed at the computer system), and / or the selection of the acceptance option 717b of user interface element 706a makes the content corresponding to the user interface element 706a and shared with the computer system 101 be downloaded to the computer system 101. In some embodiments, selection of the reject option 719a for user interface element 704a causes the content corresponding to user interface element 704a and shared with computer system 101 to be abandoned from downloading to computer system 101, and / or selection of the reject option 719b for user interface element 706a causes the content corresponding to user interface element 706a and shared with computer system 101 to be abandoned from downloading to computer system 101. In some embodiments, selection of the reject option 719a causes computer system 101 to stop displaying user interface element 704a, and / or selection of the reject option 719b causes computer system 101 to stop displaying user interface element 706a in three-dimensional environment 702.

[0126] In some embodiments, the content shared with the computer system 101 can be interacted with (e.g., viewed, played, opened, etc.) in the three-dimensional environment 702 without downloading the content to the computer system 101. For example, in response to detecting a corresponding input directed to the content (e.g., corresponding to a selection of one of the representations of the first content item 711, the second content item 713, and / or the third content item 715), the computer system 101 performs a corresponding operation involving the content. For example, the computer system 101 displays the full representation of the selected content item, initiates playback of the video or audio associated with the selected content item, or initiates sending a message to a contact associated with the selected content item or placing an outgoing phone call to the contact. In some embodiments, after performing the corresponding operation involving the content, the computer system 101 closes the selected content item (e.g., the representation of the selected content item), and / or redisplays the one or more options (e.g., accept option 717a / 717b and / or reject option 719a / 719b) for accepting or rejecting the request to share the content with the computer system 101.

[0127] As mentioned above, in some embodiments, in response to selecting the one or more options to accept the request to share content in the three-dimensional environment 702, the computer system 101 downloads the content shared with the computer system 101 (e.g., stores data and / or files corresponding to the content). Figure 7B , hand 705a (e.g., in hand state A) is providing selection input directed to accept option 717a associated with user interface element 704a, and hand 707a (e.g., in hand state A) is providing selection input directed to accept option 717b associated with user interface element 706a. In some embodiments, the selection input from hands 705a and / or 707a is an air gesture input. For example, from FIG. 7B to FIG. 7C , computer system 101 detects hand 705a moving away from user 726's body and provides a pinch or tap pointing to accept option 717 associated with user interface element 704a, and computer system 101 optionally detects hand 707a moving away from user 726's body and provides a pinch or tap pointing to accept option 717 associated with user interface element 706a. It should be understood that although 7A to 7F Multiple hands and multiple corresponding inputs are shown, but such hands and inputs need not be detected simultaneously by the computer system 101; rather, in some embodiments, the computer system 101 independently responds to such hands and / or inputs in response to independently detecting the hands and / or inputs shown and described.

[0128] In response to Figure 7BUpon detecting a selection of accept option 717a and / or 717b in the example of FIG. 1 , computer system 101 optionally begins downloading the shared content corresponding to user interface elements 704a and / or 706a, respectively. Figure 7C In some embodiments, when the content corresponding to user interface elements 704a and / or 706a begins to download to computer system 101, computer system 101 displays one or more visual indicators of the download progress of the content in three-dimensional environment 702. For example, Figure 7C As shown, one or more download progress bars are displayed for the content corresponding to user interface elements 704a and / or 706a. In some embodiments, the one or more download progress bars are displayed with the representation of the content item (e.g., displayed below each representation). Figure 7C As shown, a representation of the first content item 711 is optionally displayed with a download progress bar 728a, a representation of the second content item 713 is optionally displayed with a download progress bar 728b, and a representation of the third content item 715 is optionally displayed with a download progress bar 728c. Figure 7C As shown, in response to Figure 7B In response to a selection of accept option 717b associated with user interface element 706a detected in the 3D environment 702, a representation of content 736 corresponding to user interface element 706a is optionally displayed in the 3D environment 702. The representation of content 736 is optionally displayed in user interface element 706a along with a download progress bar 728d.

[0129] In some embodiments, downloading of content corresponding to a corresponding user interface element can be paused in three-dimensional environment 702. For example, when content corresponding to a corresponding user interface element is being downloaded to computer system 101, in response to detecting attention (e.g., gaze) directed toward a portion of user interface element 706a, computer system 101 displays a selectable option for pausing downloading of the content. Figure 7C As shown, in response to detecting a gaze 723 directed to the download progress bar 728d, the computer system 101 optionally displays a pause option 727 in the three-dimensional environment 702. In some embodiments, the pause option 727 is displayed in a portion of the user interface element 706a (e.g., adjacent to the download progress bar 728d). In some embodiments, selection of the pause option 727 causes the computer system 101 to stop downloading content 736. In some embodiments, a subsequent selection of the pause option 727 causes the computer system 101 to resume downloading content 736.

[0130] In some embodiments, after user interface element 704a and / or 706a begins downloading content corresponding to user interface element 704a and / or 706a, computer system 101 causes user interface element 704a and / or 706a to become world-locked in three-dimensional environment 702. Figure 7C As shown, user interface elements 704a and / or 706a may be displayed with a handle or grab bar 738, respectively, which indicates that user interface elements 704a and / or 706a are displayed in three-dimensional environment 702 relative to three-dimensional environment 702 and are no longer displayed in the three-dimensional environment relative to the viewpoint of user 726. For example, in response to detecting a change in the user's viewpoint (e.g., as shown in FIG. 1 ), the user interface elements 704a and / or 706a may be displayed with a handle or grab bar 738, respectively, which indicates that the user interface elements 704a and / or 706a are displayed in three-dimensional environment 702 relative to three-dimensional environment 702 and are no longer displayed in the three-dimensional environment relative to the viewpoint of user 726. FIG. 7A to FIG. 7B ), the computer system 101 abandons moving the user interface elements 704a and / or 706a to a new location in the three-dimensional environment 702 (e.g., corresponding to the center of the field of view with the three-dimensional environment 702). In some embodiments, the handles or grab bars 738 of the user interface elements 704a and / or 706a, respectively, can be selected (e.g., via selection input provided by the hand of the user 726) to initiate movement of each of the user interface elements 704a and 706a in the three-dimensional environment 702.

[0131] exist Fig.7D , after the content corresponding to user interface elements 704a and / or 706a is downloaded to computer system 101, computer system 101 displays virtual objects 734a and / or 736a corresponding to the content in three-dimensional environment 702. In some embodiments, computer system 101 replaces the display of user interface elements 704a and 706a with virtual objects 734a (e.g., which corresponds to object 734b in the top view) and 736a (e.g., which corresponds to object 736b in the top view) in three-dimensional environment 702, respectively. In addition, in some embodiments, virtual objects 734a and / or 736a are respectively displayed as having a grabber or handle 738, which can be selected to initiate movement of virtual object 734a or virtual object 736a in three-dimensional environment 702. Fig.7D As shown, virtual object 734a optionally includes representations of first content item 711, second content item 713, and third content item 715 downloaded to computer system 101. Virtual object 736a optionally includes representations of content 736a downloaded to computer system 101. In some embodiments, the representations of the content displayed in virtual objects 734a and / or 736a include more and / or additional portions of the content than the representations of the content previously displayed in user interface elements 704a and / or 706a, respectively, in three-dimensional environment 702. For example, Fig.7DAs shown, the representation of the first content item 711 includes more parts of the contact information corresponding to the first content item (e.g., the name of the contact). The representation of the second content item 713 optionally includes more parts of the photo corresponding to the second content item, and the representation of the third content item 715 optionally includes more parts of the song information corresponding to the third content item (e.g., the singer of the song). Similarly, the representation of the content displayed in the virtual object 736a includes more parts (e.g., or all) of the photo corresponding to the content.

[0132] In some embodiments, multiple content items are displayed in a virtual object in a predetermined arrangement (e.g., which is predetermined by computer system 101). Fig.7D As shown, virtual object 734a includes a plurality of content items (e.g., representations of content items) arranged in a grid-like arrangement. The representation of the first content item 711 is optionally displayed in virtual object 734a at a first size, and the representation of the second content item 713 and the representation of the third content item 715 are optionally displayed in virtual object 734a at a second size that is smaller than the first size. Fig.7D , a representation of a second content item 713 is displayed adjacent to a first portion (e.g., first half) of a representation of a first content item 711, and a representation of a third content item is displayed adjacent to a second portion (e.g., second half) of a representation of a first content item 711, wherein the representation of the second content item 713 is displayed on top of a representation of the third content item 715. It should be understood that the arrangement of the representations of the content items 711-715 in the virtual object 734a is exemplary, and alternative arrangements (including alternative sizes of the representations of the content items) are possible (e.g., depending on the number of content items). For example, if the virtual object 734a is to contain two content items, the computer system 101 will display representations of the two content items within the virtual object 734a in a side-by-side arrangement. Additionally or alternatively, if the virtual object 734a is to contain four content items, the computer system 101 will optionally display representations of the four content items within the virtual object 734a in a 2x2 grid arrangement.

[0133] In some embodiments, after the content items are downloaded to the computer system 101, different types of content items are displayed in different objects in the three-dimensional environment 702. For example, as mentioned above, Figure 7C The content corresponding to the user interface element 706a in is or includes a photo, and is therefore a first type of content. Fig.7D, after the content corresponding to the user interface element 706a is downloaded to the computer system 101, the photo content is displayed in the virtual object 736a. In some embodiments, the virtual object 736a is a first type corresponding to the first type of content. For example, the virtual object 736a is a window of a photo editing application that displays photos in the three-dimensional environment 702. Similarly, as mentioned above, the second content item 713 is optionally or includes a photo, and therefore the second content item 713 will be displayed in a virtual object of the first type in the three-dimensional environment 702. As described in more detail below, in the case where multiple content items are displayed in a corresponding virtual object (e.g., such as virtual object 734a), individual content items in the multiple content items can be removed from the corresponding virtual object so that the computer system 101 displays the individual content item in a second virtual object of the corresponding type corresponding to the type of the individual content item. In addition, in some embodiments, individual content items in the multiple content items can be selected in the corresponding virtual object so that the computer system 101 visually emphasizes the individual content items in the corresponding virtual object, as described below.

[0134] In some embodiments, the physical environment surrounding computer system 101 includes a second computer system (e.g., a second electronic device) configured to communicate with computer system 101. In some such embodiments, a representation of the second computer system is displayed in three-dimensional environment 702 or otherwise visible via display generation component 120 of computer system 101. For example, in Fig.7D , a representation of a second computer system 730a (e.g., which corresponds to computer system 730b in the top view) is visible in the three-dimensional environment 702. The representation of the second computer system 730a is optionally visible on top of a representation 728a of a round table (which corresponds to table 728b in the top view), which is optionally a representation of a physical round table on which the second computer system is placed in the physical environment. In some embodiments, the representation of the second computer system 730a is associated with a corresponding content item 732a in the three-dimensional environment 702. For example, the representation of the corresponding content item 732a is displayed via a display device of the second computer system 730a. In some embodiments, as described below, content (e.g., data and / or files corresponding to the content) can be shared between the computer system 101 and the second computer system 730a in the physical environment by interacting with the representation of the content in the three-dimensional environment 702.

[0135] exist Fig.7D , hand 705b (e.g., in hand state B) is providing selection input pointing to a representation of first content item 711 in virtual object 734a. FIG. 7D to FIG. 7E, the computer system 101 detects that the hand 705b moves away from the body of the user 726 and provides a pinch or tap directed to the representation of the first content item 711 in the virtual object 734a while the gaze 721 of the user 726 is directed to the representation of the first content item 711. Fig.7D , hand 707b (e.g., in hand state B) is providing movement input directed to virtual object 736a (e.g., or grab bar 738 associated with virtual object 736a), and hand 709a (e.g., in hand state A) is providing movement input directed to a representation of a corresponding content item 732a. Hand 707b is optionally providing input for moving virtual object 736a in the three-dimensional environment 702 in the direction of a representation of a second computer system 730a, and hand 709a is optionally providing input for moving the representation of the corresponding content item 732a in the three-dimensional environment 702 closer to a viewpoint of a user 726. In some embodiments, such movement input includes movement of the user's hand relative to the body of the user 726 when the hand is in a pinching hand shape (e.g., when the tips of the thumb and index finger of the hand are touching) and when looking at a pointed object. For example, from FIG. 7D to FIG. 7E , the computer system 101 optionally detects that the hand 707b moves to the left relative to the body of the user 726 when in a pinched hand shape and when the gaze 723 is directed toward the virtual object 736a, and the computer system 101 optionally detects that the hand 709a moves toward the body of the user 726 when in a pinched hand shape and when the gaze 725 is directed toward the representation of the corresponding content item 732a.

[0136] exist Fig. 7E In the three-dimensional environment 702, in response to detecting a selection of the representation of the first content item 711 in the virtual object 734a, the computer system 101 visually emphasizes the representation of the first content item 711 in the virtual object 734a in the three-dimensional environment 702. For example, the representation of the first content item 711 is expanded and / or enlarged within the virtual object 734a to correspond to the size of the virtual object 734a, and the representations of the second content item 713 and the third content item 715 cease to be displayed in the virtual object 734a. In addition, the representation of the first content item 711 includes an additional portion of a contact corresponding to the first content item (e.g., additional contact information, such as a contact number, an email address, a home / business address, etc.). In some embodiments, in response to detecting a selection of the representation of the first content item 711 in the virtual object 734a, the computer system 101 displays one or more control options associated with the first content item in the three-dimensional environment 702. For example, in Fig. 7E, the one or more control options include a marking option 731 and a sharing option 733. In some embodiments, the marking option 731 may be selected to initiate a process of marking (e.g., editing, deleting, etc.) the first content item 711 (e.g., its representation). For example, the marking option 731 may be selected to initiate a process of modifying one or more details of a contact. In some embodiments, the sharing option 733 may be selected to initiate a process of sharing the first content item 711 (e.g., its representation) (e.g., sending a copy of the first content item, transmitting the first content item, etc.). For example, the sharing option 733 may be selected to initiate a process of sharing the contact with a second computer system (e.g., such as the second computer system 730a) or a second electronic device configured to receive data from the computer system 101.

[0137] In addition, Fig. 7E In response to detecting the movement of the virtual object 736a to the location of the representation of the second computer system 730a, the computer system 101 shares the content corresponding to the virtual object 736a with the second computer system 730a. For example, as shown, after the computer system 101 shares the content with the second computer system 730a, the virtual object 736a is displayed (e.g., visible via the display device) of the second computer system 730a that is visible in the three-dimensional environment 702. In some embodiments, the computer system 101 stops displaying the virtual object 736a in the three-dimensional environment 702. In some embodiments, the computer system 101 maintains the display of the virtual object 736a in the three-dimensional environment 702 (e.g., at a location near (e.g., above) the representation of the second computer system 730a). In some such embodiments, a copy of the virtual object 736a is displayed (e.g., visible via the display device) of the second computer system 730a that is visible in the three-dimensional environment 702.

[0138] In response to detecting movement of the representation of the corresponding content item 732a away from the representation of the second computer system 730a, the computer system 101 receives the corresponding content item from the second computer system 101. For example, as shown, after the corresponding content item is received by (e.g., shared with) the computer system 101, the computer system 101 displays the representation of the corresponding content item 732a as a new object in the three-dimensional environment 702. Accordingly, the representation of the corresponding content item 732a is optionally displayed with a grabber or handle 738 that can be selected to initiate movement of the representation of the corresponding content item 732a in the three-dimensional environment 702.

[0139] In some embodiments, movement of a representation of a content item displayed in a virtual object away from the virtual object in three-dimensional environment 702 causes computer system 101 to display the representation of the content item as a new object in three-dimensional environment 702. Fig. 7E , hand 705c (e.g., in hand state C) is providing movement input directed toward a representation of first content item 711. Hand 705c optionally provides input for moving the representation of first content item 711 toward the viewpoint of user 726 and away from virtual object 734a in three-dimensional environment 702. In some embodiments, such movement input includes the user's hand moving toward the body of user 726 when the hand is in a pinching hand shape (e.g., when the thumb and index finger of the hand are touching) and when the user's gaze is directed toward the object. For example, from FIG. 7E to FIG. 7F , the computer system 101 optionally detects that the hand 705c moves toward the body of the user 726 when in a pinching hand shape and when the gaze 721 is directed toward the representation of the first content item 711 .

[0140] exist Figure 7F In response to detecting movement of the representation of the first content item 711 away from the virtual object 734a in the three-dimensional environment 702, the computer system 101 optionally displays a new object 711a corresponding to the first content item in the three-dimensional environment 702. In some embodiments, after displaying the new object 711a corresponding to the first content item in the three-dimensional environment 702, the representation of the first content item (e.g., 711) is no longer displayed in the virtual object 734a. For example, Figure 7F As shown, the computer system 101 stops displaying the representation of the first content item in the virtual object 734a, and displays the representation of the second content item 715 and the representation of the third content item 713 in the virtual object 734a in a new predetermined arrangement (e.g., a new grid-like arrangement). As shown, the representation of the second content item 715 and the representation of the third content item 713 are optionally displayed in a larger (e.g., larger than the second size and / or equal to the size described above) format. Fig.7DIn some embodiments, the first content item is or includes a contact and is therefore a second type of content (e.g., different from the first type of content (e.g., photo) discussed above). Therefore, virtual object 711a (e.g., which corresponds to object 711b in the top view) is optionally a second type corresponding to the second type of content. For example, virtual object 711a is a contact card of a contact application or a phone call application that displays contact information in the three-dimensional environment 702. In some embodiments, computer system 101 displays virtual object 711a as having a grabber or handle that can be selected to initiate movement of virtual object 711a in the three-dimensional environment 702. In some embodiments, virtual objects 734a and 711a can be moved independently in the three-dimensional environment 702.

[0141] In some embodiments, in response to detecting movement of the representation of the first content item 711 away from the virtual object 734a in the three-dimensional environment 702, the computer system 101 displays the virtual object 711a corresponding to the first content item in the three-dimensional environment 702 at a size different from the size of the representation of the first content item 711 in the virtual object 734a. For example, referring back to Fig.7D , a representation of the first content item 711 is displayed in the virtual object 734a at a first size in the three-dimensional environment 702. Figure 7F In the embodiment of the present invention, the computer system 101 displays the virtual object 711a in the three-dimensional environment 702 at a third size, which is larger than the first size and different from the second size of each of the representation of the second content item 713 and the representation of the third content item 715. Additionally or alternatively, in some embodiments, based on the content type of the content item, the content item is displayed in the three-dimensional environment 702 at a corresponding size. For example, as mentioned above, the virtual object 711a corresponding to the first content item is displayed in the three-dimensional environment 702 at a third size. In some embodiments, if a movement input is provided to the representation of the third content item 715 that is or includes a song file to move the representation of the third content item 715 away from the virtual object 734a in the three-dimensional environment 702, the computer system 101 will display the third content item in the new object, which will optionally have a fourth size in the three-dimensional environment 702. In some such embodiments, the fourth size will be different from the third size of the virtual object 711a.

[0142] FIG. 8A to FIG. 8L 8 is a flow chart illustrating an exemplary method 800 of receiving content shared with a computer system in a three-dimensional environment according to some embodiments. In some embodiments, the method 800 is performed on a computer system (e.g., Figure 1The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, a smart phone, a wearable computer, or a head mounted device, the computer system including a display generating component (e.g., Figure 1 , Figure 3 and Figure 4 In some embodiments, method 800 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0143] In some embodiments, method 800 is performed at a computer system (e.g., 101) that communicates with a display generation component (e.g., 120) and one or more input devices (e.g., 314). For example, the computer system is or includes a mobile device (e.g., a tablet, a smart phone, a media player, or a wearable device) or a computer. In some embodiments, the display generation component is a display (optionally a touch screen display) integrated with the electronic device, an external display such as a monitor, a projector, a television, or a hardware component (optionally integrated or external) for projecting a user interface or making the user interface visible to one or more users. In some embodiments, the one or more input devices include a device capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information associated with the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a trackpad (optionally integrated or external), a touchpad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, an eye tracking device, and / or a motion sensor (e.g., a hand tracking device, a hand motion sensor), etc. In some embodiments, the computer system communicates with a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, trackpad)). In some embodiments, the hand tracking device is a wearable device, such as a smart glove. In some embodiments, the hand tracking device is a handheld input device, such as a remote control or a stylus.

[0144] In some embodiments, although a three-dimensional environment such as Fig. 7AA three-dimensional environment 702 in the computer system (e.g., an environment or virtual environment corresponding to the physical environment surrounding the display generation component and / or the computer system) is visible via the display generation component. In some embodiments, the three-dimensional environment is generated, displayed, or otherwise enabled to be viewed by the computer system (e.g., a computer generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment, etc.). In some embodiments, the physical environment is visible through a transparent portion of the display generation component (e.g., real or reality transmission). In some embodiments, a representation of the physical environment (e.g., virtual or video transmission) is displayed in the three-dimensional environment via the display generation component, and the computer system receives (802a) via the one or more input devices A first indication corresponding to a request to share content with the computer system. For example, a request to share one or more images (e.g., photos), videos (e.g., video / movie / television clips), contacts (e.g., names, phone numbers, emails, etc.), music (e.g., songs, playlists, etc.), applications, files, etc. with the computer system. In some embodiments, the request is optionally received from a second electronic device separate from the computer system. In some embodiments, the request is optionally received based on satisfying one or more criteria (e.g., the second electronic device is in communication with the electronic device, and / or the second electronic device is less than a threshold distance from the computer system, and / or the second electronic device is authorized to share content with the computer system, and / or the computer system is authorized to receive content from the second electronic device, etc.).

[0145] In some embodiments, in response to receiving the first indication and before completing an operation related to the content (e.g., before receiving at least a predefined portion or all of the content), the computer system displays (802b) via a display generation component at a first location in the three-dimensional environment a user interface element (e.g., a notification) corresponding to the content, such as Fig. 7AThe user interface elements 704a and / or 706a in (e.g., an icon corresponding to the content, a message window, an alert pop-up window, etc.) are displayed at a first position in the three-dimensional environment. In some embodiments, the notification includes a preview of the content shared with the electronic device (e.g., a portion of the shared image, a video portion of the shared video, a contact photo corresponding to the shared contact, etc.). In some embodiments, the preview is displayed based on satisfying one or more criteria (e.g., the second electronic device is a device known to the computer system, and / or the second electronic device is authorized to share content with the electronic device). In some embodiments, the notification optionally includes one or more selectable enable representations for accepting / rejecting the request to share content with the electronic device. For example, the notification includes an "Accept" button, which, when selected, accepts the content from the second electronic device (e.g., starts downloading the content shared by the second electronic device to the electronic device). In some embodiments, the notification is displayed until one or more criteria are satisfied (e.g., until the content is downloaded to the computer system, and / or until input of selecting, tapping, and / or moving the notification is received).), wherein the first position is based on the user of the computer system (e.g., Fig. 7A A first corresponding pose (e.g., a position and / or orientation in the physical environment) of a corresponding portion of a user 726 in the three-dimensional environment). For example, a portion of the field of view of the three-dimensional environment based on the user's head relative to the viewpoint of the user of the electronic device on the three-dimensional environment (e.g., regardless of the relative positioning of the user's gaze). For example, the notification is optionally displayed along a normal to the user's viewpoint, which is optionally defined by the pose of the user's head. In some embodiments, the first position in the three-dimensional environment is defined as a predefined position (e.g., center) of the user's field of view (e.g., the notification is displayed based on the center of the user's head, regardless of the current position of the user's gaze).

[0146] In some embodiments, while displaying a user interface element corresponding to content at a first location and before completing an operation related to the content (e.g., before receiving at least a predefined portion or all of the content), the computer system detects (802c) movement of a corresponding portion of the user from a first corresponding posture to a second corresponding posture different from the first corresponding posture, such as Fig. 7A . For example, movement of the user's head to a new position / orientation that causes the user's viewpoint of the three-dimensional environment to change. In some embodiments, detecting movement of the user's head does not necessarily correspond to detecting movement of the user's gaze in the three-dimensional environment. For example, the user's head moves to a new position / orientation, while the user's gaze remains at the same position in the three-dimensional environment as before the movement was detected. In some embodiments, movement of the user's head is detected before receiving the content.

[0147] In some embodiments, in response to detecting movement of a corresponding portion of the user and before completing an operation related to the content (e.g., before receiving at least a predefined portion or all of the content), the computer system displays 802d a user interface element corresponding to the content at a second location in the three-dimensional environment different from the first location via a display generation component, such as, for example, Figure 7B The user interface elements 704a and / or 706a shown are displayed (for example, an icon, a message window, an alert pop-up window, etc. corresponding to the content is moved to a second position in the three-dimensional environment and / or displayed at a second position in the three-dimensional environment. In some embodiments, a notification is displayed at a second position in the three-dimensional environment based on determining that the user's gaze is not directed at the notification. For example, the user's gaze is directed at a corresponding position different from the second position. In some embodiments, the notification corresponding to the content is displayed at the second position before the content is received.), wherein the second position is based on a second corresponding posture of the corresponding part of the user. For example, based on a portion of the field of view of the three-dimensional environment viewed from the user's new viewpoint relative to the user's head (for example, regardless of the relative positioning of the user's gaze). In some embodiments, the user interface element is displayed at the second position to maintain the orientation and / or position of the second corresponding posture of the corresponding part of the user. In some embodiments, the orientation and / or position of the user interface element at the first position and at the second position relative to the corresponding posture of the corresponding part of the user are the same. For example, the position and / or orientation of a notification in a three-dimensional environment changes based on movement and / or rotation of the user's head (e.g., movement and / or rotation of the electronic device and / or display generating components), but the position and / or orientation of the notification relative to the corresponding pose of the corresponding part of the user is the same (e.g., the notification is displayed in the center of the user's field of view of the three-dimensional environment in the first position and the second position).

[0148] In some embodiments, after completing the operation related to the content (802e) (e.g., after detecting input directed to accepting or downloading a notification of shared content, as described in more detail below), the computer system receives (802f) via the one or more input devices a signal related to moving a first object corresponding to the content to a third location in the three-dimensional environment (such as, for example, Fig.7DThe first input corresponding to the request to move the object 736a shown (for example, after detecting the input pointing to the notification of accepting or downloading the shared content, the content is downloaded, saved, or otherwise made available for viewing via the computer system. In some embodiments, the first object corresponding to the content is displayed at the location of the notification corresponding to the content (for example, the notification corresponding to the content stops being displayed in the three-dimensional environment, and the first object corresponding to the content is displayed in the three-dimensional environment).) For example, when the distance between the user's hand and the first object is greater than a threshold distance (for example, 0.2cm, 0.5cm, 1cm, 2cm, 3cm, 5cm, 10cm, 12cm, 24cm or 26cm), while the user's gaze is directed to the first object, a pinch gesture of the index finger and thumb of the user's hand (for example, the index finger and thumb are put together and touched to form a pinch hand shape while maintaining contact with each other) and subsequent movement of the hand in the pinch hand shape, or when the distance between the user's hand and the first object is less than the threshold distance, a pinch gesture of the index finger and thumb of the user's hand pointing to the first object and subsequent movement of the hand in the pinch hand shape, regardless of the user's gaze position. In some embodiments, the first input is or includes an air gesture. In some embodiments, the first object is optionally a representation of content shared by the second electronic device, and is optionally displayed by an application configured to support the content. For example, the first object is a photo displayed in a window, a video displayed in a playback window of a video player application, contact information displayed in a contact card of a contact application, and the like. In some embodiments, the content optionally includes multiple (e.g., separate) and / or multiple types of content, each of which is optionally displayed in one or more supporting application windows in a three-dimensional environment, as will be described in more detail below.

[0149] In some embodiments, in response to receiving the first input, the computer system displays (802g) the first object at a third location in the three-dimensional environment according to the first input, such as Fig. 7E The displayed object 736a is shown (eg, the first object is moved to a third position in the three-dimensional environment according to the movement of the pinch gesture of the user's hand).

[0150] In some embodiments, when the first object is displayed at a third location in the three-dimensional environment, the computer system detects (802h) a second movement of the corresponding portion of the user from a third corresponding pose (e.g., the pose of the corresponding portion of the user when the first input was received) to a fourth corresponding pose, such as, for example Fig. 7AMovement of the hand 703a shown (e.g., movement of the user's head to a new position / orientation (e.g., a fourth corresponding pose) that causes the user's viewpoint of the three-dimensional environment to change). In some embodiments, detecting movement of the user's head does not necessarily correspond to detecting movement of the user's gaze in the three-dimensional environment. For example, the user's head moves to a new position / orientation, while the user's gaze remains at the same position in the three-dimensional environment as before the second movement was detected. In some embodiments, detection of movement of the user's head occurs simultaneously with movement of the user's gaze to a new position in the three-dimensional environment (e.g., to a position away from the first object at a third position in the three-dimensional environment).

[0151] In some embodiments, in response to detecting a second movement of the corresponding portion of the user, the computer system displays (802i) the first object at a third location to maintain the first object relative to the three-dimensional environment, such as Fig.7D Display objects 734a and / or 736a are shown. For example, the first object moves to a new position (e.g., a fourth position) in the three-dimensional environment without responding to movement of the user's head and / or movement of the user's gaze. After the content has been accepted and / or received by the computer system, the position of the first object in the three-dimensional environment optionally no longer depends on the position of the user's head. For example, the movement of the first object requires a separate input (e.g., a pinch gesture of the user's hand pointing to the first object) corresponding to a request to move the first object to a new position in the three-dimensional environment. Displaying a notification of shared content at a location in the three-dimensional environment determined based on the position of the corresponding part of the user facilitates user input for interacting with the notification, and / or facilitates discovering that content is being shared with the computer system, thereby improving user-device interaction.

[0152] In some embodiments, the operation related to the content includes downloading the content to the computer system (804a) (e.g., saving and / or storing the content to a memory or storage device of the computer system. In some embodiments, as described below, the content can be used without downloading the content to the computer system.). In some embodiments, when the content is downloaded to the computer system and when a user interface element corresponding to the content is displayed at a second location in the three-dimensional environment, the computer system detects (804b) via the one or more input devices a corresponding movement of the corresponding portion of the user from the second corresponding pose to a fifth corresponding pose, such as Fig. 7A Movement of hand 703a in (e.g., movement of the user's head to a new position / orientation that causes the user's viewpoint of the three-dimensional environment to change, as discussed above).

[0153] In some embodiments, in response to detecting the corresponding movement of the corresponding part of the user, the computer system displays (804c) a user interface element (e.g., a notification) corresponding to the content at a second location to maintain the user interface element relative to the three-dimensional environment, such as Figure 7C , and / or 706a. For example, in response to the input of downloading content and when the content is being downloaded, the user interface element becomes world-locked. In some embodiments, the computer system actively maintains the user interface element as world-locked in response to detecting the movement of the user's head. In some embodiments, the display of the notification is no longer based on a portion of the field of view of the three-dimensional environment viewed from the user's new viewpoint of the user's head (e.g., regardless of the relative positioning of the user's gaze) (e.g., the notification is no longer viewpoint-locked). For example, the display of the notification in the three-dimensional environment is based on (e.g., bound to or anchored to) a second position in the three-dimensional environment (e.g., a physical environment or virtual environment visible via a display generation component). Displaying a notification of shared content at a position in the three-dimensional environment determined based on the position in the three-dimensional environment allows the user's viewpoint to not be blocked by the notification of the shared content, thereby improving user-device interaction.

[0154] In some embodiments, when the content is downloaded to the computer system, the user interface element is displayed with a first corresponding user interface element associated with the user interface element corresponding to the content, such as Figure 7C The grabber or handle 738 associated with user interface elements 704a and / or 706a is shown, wherein the first corresponding user interface element is used to move the user interface element (806) relative to the three-dimensional environment. For example, the first corresponding user interface element is a grabber or handle configured to be selected to move the user interface element (e.g., notification) to a corresponding position in the three-dimensional environment. In some embodiments, the grabber or handle is displayed below the notification. In some embodiments, the grabber or handle is displayed at the top / above or side of the notification. In some embodiments, the index finger and thumb of the user's hand pointing / towards the pinch gesture of the grabber or handle and the subsequent movement of the hand toward the corresponding position in the three-dimensional environment in the shape of the pinched hand optionally moves the notification toward the corresponding position in the three-dimensional environment. The grabber displayed for the notification of sharing content is conducive to the user input for moving the notification within the three-dimensional environment, and / or is conducive to finding that the notification can be moved within the three-dimensional environment, thereby improving user-device interaction.

[0155] In some embodiments, when a first indication corresponding to a request to share content with the computer system is received (808a), the three-dimensional environment includes a second object at a corresponding location in the three-dimensional environment, such as Fig. 7A708a in (e.g., the second object is optionally a user interface of an application running on the computer system. In some embodiments, when a request is received to share one or more images (e.g., photos), videos (e.g., video / movie / television clips), contacts (e.g., names, phone numbers, emails, etc.), music (e.g., songs, playlists, etc.), applications, files, etc. with the computer system, the user is interacting with the second object (e.g., viewing the second object or providing input to / towards the second object).). In some embodiments, in response to receiving the first indication and before completing the content-related operation, the computer system maintains (808b) display of the second object at a corresponding position in the three-dimensional environment, such as, Figure 7B 708a. For example, a request to share content with a computer system does not cause a second object to be stopped from being displayed in the three-dimensional environment. In some embodiments, a user interface element corresponding to the content (e.g., a notification) is displayed in the three-dimensional environment without causing the second object to be stopped from being displayed in the three-dimensional environment. For example, a notification corresponding to the content and the second object are displayed simultaneously in the three-dimensional environment (e.g., in multitasking mode). Maintaining the display of an object when a content sharing request is received allows an interaction involving the object to be performed when content corresponding to the sharing request is received, thereby improving user-device interaction.

[0156] In some embodiments, when the second object is displayed at the corresponding position in the three-dimensional environment, the computer system receives (810a) via the one or more input devices a second indication corresponding to a notification event other than a request to share content with the computer system (for example, a notification corresponding to receipt of: a message in a messaging application, an email in an email application, an incoming call in a phone application, and so on. In some embodiments, the second indication is not a request to share content with the electronic device.).

[0157] In some embodiments, in response to receiving the second indication, the computer system stops (810b) display of the second object in the three-dimensional environment. For example, the receipt of the notification event causes the second object (e.g., the user interface of the application displayed in the three-dimensional environment) to be stopped at the corresponding position in the three-dimensional environment. In some embodiments, the user interface element corresponding to the content (e.g., the notification corresponding to the shared content) is not stopped in the three-dimensional environment in response to receiving the second indication. In some embodiments, the notification corresponding to the second indication is displayed in the three-dimensional environment in response to receiving the second indication. For example, an object corresponding to a message notification, an email notification, an incoming phone call notification, etc. Maintaining the display of the notification of the shared content when a secondary notification event is received and stopping the display of other objects when a secondary notification event is received facilitates user input for interacting with the notification of the shared content, thereby improving user-device interaction.

[0158] In some embodiments, after completing an operation related to the content (812a) (e.g., after detecting an input directed to accepting or downloading a notification of shared content, as discussed below), based on determining that the content is first content having a first characteristic (e.g., the content is a hyperlink), such as Figure 7B In the content item 711 in the link, the first object corresponding to the content is of the first type (812b). For example, the content is associated with the first application and / or opened in the first application. In some embodiments, the first object is a window of a web browsing application that displays the content (e.g., text, image, or video) corresponding to the link. In some embodiments, in response to the completion of the download of the first content having the first characteristic, the first object is automatically displayed in the three-dimensional environment.

[0159] In some embodiments, based on determining that the content is second content having a second characteristic different from the first characteristic (e.g., the content is a contact), such as Figure 7BIn the content item 713 in the computer system, the first object corresponding to the content is a second type (812c) different from the first type. For example, content associated with a second application different from the first application and / or opened in the second application. In some embodiments, the first object is a contact card in the contact application that displays contact information (e.g., phone number, email address, mailing address, birthday, etc.) corresponding to the contact. In some embodiments, the first object is a photo or video clip, and the first object is a quick view window that displays the photo or video clip. In some embodiments, the first object is based on the type of content shared with the computer system (e.g., selected using the type of the content). Displaying shared content in an object configured to display shared content enables displaying shared content and therefore interacting with the shared content without displaying additional controls, thereby improving user-device interaction.

[0160] In some embodiments, displaying a user interface element includes (814a): based on determining that a first set of criteria is satisfied, including criteria that are satisfied when the content is shared by a user identified by the computer system (e.g., the first set of criteria are satisfied when the content is shared by a user trusted or known by the computer system. For example, the user is a contact of the user of the computer system. In some embodiments, the first set of criteria are not satisfied when the content is shared by an unidentified user), displaying a preview of the content in a user interface element corresponding to the content (814b), such as Figure 7B Preview 710 shown. For example, a user interface element (e.g., a notification) includes one or more portions of the content shared by the identified user. In some embodiments, the notification includes a still image corresponding to the content shared by the identified user. For example, when the content shared by the identified user is a photo, the notification includes one or more portions of the photo. When the content shared by the identified user is a video, the notification optionally includes one or more portions of a frame within the video. When the content shared by the identified user is a contact, the notification optionally includes a contact photo, the contact's name, and / or one or more portions of the contact's phone number, an email address, a home address, etc. When the content shared by the identified user is a song, the notification optionally includes one or more portions of the song cover art, the name of the song, the performer of the song, the name of the album of which the song is a part, etc.

[0161] In some embodiments, displaying the user interface element includes: based on determining that the first set of criteria is not met (e.g., because the content is shared by an unidentified user), abandoning displaying a preview of the content in a user interface element corresponding to the content (814c), such as Figure 7BThe display of the preview in the user interface element 706a is abandoned as shown. For example, the user interface element does not include one or more parts of the content shared by the unidentified user. In some embodiments, regardless of whether the sender of the shared content is identified by the computer system, the user interface element includes the identifier of the sender. For example, the user interface element includes the sender's name, the sender's phone number, the sender's email address and / or the sender's contact photo. Displaying a preview of the shared content in the notification of the shared content is conducive to deciding whether to accept or reject the request to share the content, thereby improving user-device interaction.

[0162] In some embodiments, the operation related to the content includes downloading the content to the computer system (816a) (e.g., saving and / or storing the content to a memory or storage device of the computer system. In some embodiments, as described below, the content can be used without downloading the content to the computer system.). In some embodiments, while displaying a user interface element corresponding to the content, the computer system receives (816b) via the one or more input devices a second input corresponding to a request to download the content to the computer system, such as, for example, Figure 7B A selection of the acceptance option 717 is shown (e.g., an air gesture directed to the user interface element, such as a tap or selection gesture (e.g., provided by an index finger of a user's hand) or a pinch gesture (e.g., provided by an index finger and thumb of a user's hand)). In some embodiments, the user interface element includes one or more selectable options for accepting and / or declining the request to share content with the computer system. The second input optionally corresponds to a selection of the one or more selectable options for accepting the request to share content with the computer system.

[0163] In some embodiments, in response to receiving the second input and while the content is being downloaded to the computer system, the computer system displays (816c) one or more visual indicators of the download progress of the content in a user interface element, such as Figure 7CDownload progress bar 728a-728d shown. For example, a user interface element (e.g., notification) includes a download progress bar or other status indicators (e.g., percentage, circular indicator, etc.) indicating the progress of downloading content to a computer system. In some embodiments, the one or more visual indicators of the download progress are displayed together with a preview of the content being downloaded. For example, the one or more visual indicators are displayed below, above, on the side, or partially within the preview of the content being downloaded. In some embodiments, once the download of the content is completed, the one or more visual indicators of the download progress are stopped from being displayed in the user interface element. Displaying an indicator of the download progress of the shared content in the notification of the shared content is conducive to finding that the content is being downloaded to the computer system and / or providing feedback on the status of the computer system, thereby improving user-device interaction.

[0164] In some embodiments, when the content is downloaded to the computer system (818a) (e.g., when the content is saved and / or stored in the memory of the computer system, as described above), the user of the computer system is determined to be paying attention (e.g., looking) (such as Figure 7C The user's gaze point 723 in the notification is directed to a user interface element corresponding to the content (e.g., the user's gaze is directed to a portion of the user interface element (e.g., notification) corresponding to the content. In some embodiments, the gaze is directed to one or more indicators of the download progress of the content displayed in the notification (e.g., a download progress bar).), the computer system displays (818b) via the display generation component a selectable option for pausing the download of the content, such as Figure 7C In some embodiments, the pause option is displayed in the user interface element. In some embodiments, the pause option is displayed adjacent to, above, below, or within a portion of the one or more indicators of the download progress of the content. In some embodiments, the selectable option ceases to be displayed when the user's gaze is no longer directed to the user interface element corresponding to the content.

[0165] In some embodiments, when the selectable options are displayed, the computer system detects (818c) a selection of the selectable option (such as a Figure 7C A third input (e.g., an air gesture or a pinch gesture pointing to a selectable option, as described above) corresponding to a selection of the pause option 727 in the display.

[0166] In some embodiments, in response to detecting a third input, the computer system pauses (818d) the download of the content according to the third input. For example, when the pause button is selected, the content (e.g., temporarily) stops downloading. In some embodiments, the appearance of the selectable option changes in response to the selection of the selectable option. For example, the pause button changes shape, size, color, opacity, thickness, etc. In some embodiments, the selectable option changes the appearance from the first icon type to the second icon type. For example, the selectable option changes from the pause icon to the play icon. In some embodiments, the appearance of the one or more indicators of the download progress of the content changes in response to the selection of the selectable option. For example, the progress indicator in the download progress bar stops moving in the download progress bar (e.g., filling the length of the download progress bar). In some embodiments, the subsequent selection of the selectable option resumes the download of the content. Providing a selectable option for pausing the download of the shared content when the gaze of the notification pointing to the shared content is detected reduces the number of inputs required to pause the download of the shared content, thereby improving user-device interaction.

[0167] In some embodiments, the content includes a first content item and a second content item (820a), such as Figure 7B In some embodiments, the first content item and the second content item are of the same type. For example, both the first content item and the second content item are photos. In some embodiments, the first content item and the second content item are of different types. For example, the first content item is a video and the second content item is a contact.) In some embodiments, the one or more visual indicators of the download progress of the content (e.g., a download progress bar) include (820b) a first visual indicator (820c) of the download progress of the first content item (e.g., a first download progress bar or other element indicating the download progress of the first content item) (e.g., a download progress bar or other element indicating the download progress of the first content item). Figure 7C and a second visual indicator (820d) of the download progress of the second content item (such as Figure 7CIn some embodiments, a first visual indicator and a second visual indicator are displayed in a user interface element corresponding to the content (e.g., a notification). In some embodiments, the first visual indicator is displayed above, below, on the side, or inside a portion of the first content item in the user interface element, and the second visual indicator is displayed above, below, on the side, or inside a portion of the second content item in the user interface element. Displaying indicators of the download progress of multiple shared content items in notifications of shared content items is conducive to discovering that content items are being downloaded to the computer system separately and / or providing feedback on the status of the computer system, thereby improving user-device interaction.

[0168] In some embodiments, the content includes a first content item and a second content item (822a) (e.g., the content shared with the computer system includes multiple files, such as photos, videos, hyperlinks, music files, etc. In some embodiments, as discussed above, the first content item and the second content item are of the same type. In some embodiments, the first content item and the second content item are of different types, as discussed above.), such as Figure 7B Content items 711 and 713 in .

[0169] In some embodiments, a first object corresponding to content includes a representation of a first content item and a representation of a second content item displayed in a predetermined arrangement within the first object (822b), such as Figure 7BThe first content item 711 and the second content item 713 are shown in a predetermined arrangement within the preview 710. For example, the first object includes all or part of each of the first content item and the second item. In some embodiments, the representation of the first content item and the representation of the second content item are displayed in the first object with the same size. For example, the representation of the first content item and the representation of the second content item are displayed side by side in the first object (for example, adjacently displayed in a longitudinal orientation), or the representation of the first content item and the representation of the second content item are displayed in the first object in a manner that one is stacked on top of the other (for example, in a horizontal orientation). In some embodiments, the representation of the first content item and the representation of the second content item are displayed in different sizes in the first object. In some embodiments, the content type of the first content and the second content determines the size of the representation of the first content item and the representation of the second content item in the first object. In some embodiments, the predetermined arrangement is based on the number of items shared simultaneously. For example, when two content items are shared with a computer system, the representation of the first content item and the representation of the second content item are displayed in the above-mentioned predetermined arrangement. When three content items are shared with a computer system, a representation of the first content item and a representation of the second content item are optionally displayed in a first size in a predetermined arrangement within a first object, and a representation of the third content item is optionally displayed in a second size different from the first size in a predetermined arrangement within the first object. In some embodiments, a representation of the first content item, a representation of the second content item, and a representation of the third content item are displayed in different sizes in a predetermined arrangement within the first object. Displaying representations of multiple content items in an object in a three-dimensional environment enables viewing of at least a portion of the multiple content items without displaying additional controls, thereby improving user-device interaction.

[0170] In some embodiments, when a first object including a representation of a first content item and a representation of a second content item is displayed, the computer system detects (824a) via the one or more input devices a second input corresponding to movement of the representation of the first content item away from the first object to a corresponding location in the three-dimensional environment (e.g., when a distance of a user's hand from the representation of the first content item is greater than a threshold distance (e.g., 0.2 cm, 0.5 cm, 1 cm, 2 cm, 3 cm, 5 cm, 10 cm, 12 cm, 24 cm, or 26 cm), while the user's gaze is directed toward the representation of the first content item). A pinch gesture of the index finger and thumb of the user's hand (e.g., the index finger and thumb are brought together and touched to form a pinch hand shape while maintaining contact with each other) and a subsequent movement of the hand in the pinch hand shape away from the first object, or a pinch gesture of the index finger and thumb of the user's hand pointing to the representation of the first content item and a subsequent movement of the hand in the pinch hand shape away from the first object when the distance of the user's hand from the representation of the first content item is less than a threshold distance, regardless of the user's gaze position. In some embodiments, the corresponding position in the three-dimensional environment is a position different from the position of the first object. ), such as Fig. 7E Content item 711 is shown moving away from object 734a.

[0171] In some embodiments, in response to detecting the second input (824b), the computer system displays (824c) a second object corresponding to the first content item at a corresponding position in the three-dimensional environment via a display generation component (e.g., a representation of the first content item is displayed as a new object at a corresponding position in the three-dimensional environment, where the new object displays a representation of the first content item. In some embodiments, the size of the second object corresponding to the representation of the first content item is larger than the size of the representation of the first content item in the first object in the three-dimensional environment.), such as Figure 7FAs shown, object 711a corresponding to content item 711 is displayed. In some embodiments, the computer system stops (824d) displaying the representation of the first content item in the first object (e.g., after generating a second object corresponding to the first content item and displaying the second object corresponding to the first content item at a corresponding position, stopping displaying the representation of the first content item in the first object. In some embodiments, due to stopping displaying the representation of the first content item in the first object, the representation of the second content item in the first object is displayed in a larger size in the first object. For example, the representation of the second content item expands in the first object to correspond to (e.g., or closely correspond to) the size of the first object in a three-dimensional environment. In some embodiments, due to stopping displaying the representation of the first content item in the first object in the first object, a predetermined arrangement of content items within the first object is changed to a different predetermined arrangement within the first object. For example, when the content includes a third content item, the predetermined arrangement within the first object is changed to display the representation of the second content item and the representation of the third content item in the first object, respectively, in a larger size), such as, Figure 7F The display of content item 711 in object 734a is shown stopped. Dragging the representation of the content item out of the first object in the three-dimensional environment causes the computer system to automatically perform object generation for the representation of the content item, thereby improving user-device interaction.

[0172] In some embodiments, when displaying a first object that includes a representation of a first content item and a representation of a second content item in a predetermined arrangement, the computer system detects (826a) via the one or more input devices a selection of the representation of the first content item in the first object (such as, for example, Fig.7D The second input (eg, an air gesture or a pinch gesture directed to a representation of the first content item as described above) corresponds to a selection of content item 711 as shown.

[0173] In some embodiments, in response to detecting the second input, the computer system visually emphasizes (826b) the representation of the first content item (e.g., displays the representation of the first content item in a larger size in the first object. In some embodiments, visually emphasizing the representation of the first content item includes expanding the representation of the first content item in the first object to a size corresponding to the size of the first object. In some embodiments, in response to visually emphasizing the representation of the first content item, additional detailed information corresponding to the first content item (e.g., additional portions of a photo, additional information for a contact, etc.) is displayed, such as Fig. 7E The display of the expanded content item 711 is shown, and the representation of the second content item in the first object is visually weakened, for example, as shown in FIG. Fig. 7EIn some embodiments, the display of the second content item is stopped. For example, the display of the representation of the second content item in the first object is stopped. In some embodiments, the representation of the second content item is displayed at a smaller size than the representation of the first content item. In some embodiments, in response to visually weakening the representation of the second content item in the second object, the corresponding detailed information (e.g., part of the photo, information of the contact, etc.) corresponding to the second content item is hidden and / or stopped. Additionally or alternatively, in some embodiments, the selection of the representation of the second content item causes the computer system to visually emphasize the representation of the second content item and visually weaken the representation of the first content item in the first object. The representation of the content item displayed in the object is emphasized in a three-dimensional environment to reduce the number of inputs required to display the additional detailed information corresponding to the content item in the three-dimensional environment, thereby improving user-device interaction.

[0174] In some embodiments, in response to detecting the second input (828a), the computer system displays (828b) via the display generation component a first option that can be selected to initiate a process for editing the representation of the first content item, such as Fig. 7E In the marking option 731. For example, when the representation of the first content item in the first object is visually emphasized in the three-dimensional environment, the marking option is displayed in the three-dimensional environment. In some examples, the marking option is displayed in a menu, a list, a toolbar, etc., including a plurality of options corresponding to the representation of the first content item. In some embodiments, the first option is displayed at, near, or inside a portion of the first object. In some embodiments, the selection of the first option causes the display of additional options and / or tools for editing the representation of the first content item (e.g., tools for performing editing operations such as clipping, cutting, erasing, copying, and / or filtering on the representation of the first content item). In some embodiments, the selection of the first option initiates the process of marking the first content item (e.g., the representation of the first content item) instead of the second content item (e.g., the representation of the second content item), regardless of whether both the representation of the first content item and the representation of the second content item are displayed. Similarly, in some embodiments, in response to detecting an input for selecting the representation of the second content item, the first option may be selected to initiate the process of marking the second content item (e.g., the representation of the second content item) instead of the first content item (e.g., the representation of the first content item). Displaying a markup option when a representation of a content item displayed in an object is selected reduces the amount of input required to edit the representation of the content item, thereby improving user-device interaction.

[0175] In some embodiments, in response to detecting the second input (830a), the computer system displays (830b) via the display generation component a first option that can be selected to initiate a process for sharing the first content item with the second computer system, such as Fig. 7E733 in the sharing options. For example, when the representation of the first content item in the first object is visually emphasized in the three-dimensional environment, the sharing options are displayed in the three-dimensional environment. In some examples, the sharing options are displayed in a menu, a list, a toolbar, etc., including a plurality of options corresponding to the representation of the first content item. In some embodiments, the first option is displayed at, near, or inside a portion of the first object. In some embodiments, the selection of the first option causes the display of additional options and / or tools for sharing the first content item (e.g., for performing editing operations such as sending, sending, and / or transmitting a copy of the first content item to a second computer system with a message). In some embodiments, the selection of the first option initiates a process of sharing the first content item (e.g., the representation of the first content item) with the second computer system instead of the second content item (e.g., the representation of the second content item), regardless of whether both the representation of the first content item and the representation of the second content item are displayed. Similarly, in some embodiments, in response to detecting an input for selecting the representation of the second content item, the first option may be selected to initiate a process of sharing the second content item (e.g., the representation of the second content item) with the second computer system instead of the first content item (e.g., the representation of the first content item). Displaying a share option when a representation of a content item displayed in an object is selected reduces the amount of input required to share the representation of the content item with a second computer system, thereby improving user-device interaction.

[0176] In some embodiments, when a user interface element corresponding to the content is displayed at a first position and before an operation related to the content is completed, the computer system receives (832a) corresponding input via the one or more input devices (e.g., an air gesture or a pinch gesture directed to a user interface element corresponding to the content (e.g., a notification) as described above. In some embodiments, the corresponding input corresponds to a request to perform a corresponding operation associated with the content.).

[0177] In some embodiments, in response to receiving the corresponding input (832b), based on determining that the corresponding input corresponds to a request to perform an operation associated with the content, the computer system initiates (832c) a process of performing an operation associated with the content without retaining the content in a permanent storage device of the computer system. For example, content shared with the computer system is displayed, such as viewing a full or expanded view of a photo, viewing all or a portion of a video clip, viewing all or a portion of a contact's details, or navigating to a web page associated with a hyperlink, without saving the content to the computer system (e.g., without storing the content in a memory / storage device of the computer system). In some embodiments, the shared content can be used while the content is downloaded to the computer system. Enabling shared content to be used by the computer system without downloading the shared content to the computer system reduces the number of inputs required to interact with the shared content, thereby improving user-device interaction.

[0178] In some embodiments, the content includes a first content item and a second content item (834a) (e.g., the content shared with the computer system includes multiple files, such as photos, videos, hyperlinks, music files, etc. In some embodiments, as discussed above, the first content item and the second content item are of the same type. In some embodiments, the first content item and the second content item are of different types, as discussed above.), such as Figure 7B In some embodiments, the first object includes a representation of the first content item having a first size in the three-dimensional environment (such as Figure 7B 711) and a representation of a second content item having a second size in a three-dimensional environment (such as, for example, Figure 7B The content item is shown displayed at a second size that is smaller than the first size 713) (834b) (e.g., the first object includes all or a portion of each of the first content item and the second item). In some embodiments, a representation of the first content item and a representation of the second content item are displayed at the same size in the first object. In some embodiments, a representation of the first content item and a representation of the second content item are displayed at different sizes in the first object.

[0179] In some embodiments, when the first object is displayed at a third location relative to the three-dimensional environment, the computer system receives (834c) via the one or more input devices a signal related to movement of a representation of the first content item away from the first object to a corresponding location in the three-dimensional environment (such as, for example, Fig. 7E In some embodiments, the corresponding position in the three-dimensional environment is a position different from the position of the first object.

[0180] In some embodiments, in response to receiving the second input, the computer system displays (834d) via the display generation component at a corresponding position in the three-dimensional environment a second object corresponding to the first content item having a third size, wherein the third size is different from the first size, such as Figure 7F The object 711a corresponding to the content item 711 is shown to be displayed in a larger size. For example, the representation of the first content item is displayed as a new object (e.g., a second object) at a corresponding position in the three-dimensional environment. In some embodiments, the size of the second object corresponding to the representation of the first content item is greater than the size of the representation of the first content item in the first object in the three-dimensional environment. In some embodiments, the size of the second object corresponding to the first content item is determined based on the content type of the first content item. In some embodiments, the size of the second object is determined based on the size of the first content item. Additionally or alternatively, in some embodiments, in response to receiving an input for removing the representation of the second content item from the first object, the representation of the second content item is displayed in a third object in the three-dimensional environment. In some embodiments, the third object is a size different from the second object. In some embodiments, the size of the third object is determined based on the size of the second content item. Changing the size of the content item when dragging the content item out of the object in the three-dimensional environment reduces the number of inputs required to expand the size of the content item in the three-dimensional environment, thereby improving user-device interaction.

[0181] In some embodiments, the computer system is coupled to a second computer system (such as Fig.7D In some embodiments, the second computer system is in communication with a computer system 730a in the display generation component (e.g., the second computer system is or includes a mobile device (e.g., a tablet, smart phone, media player, or wearable device) or a computer), and the second computer system is located at a corresponding location in the physical environment of the computer system (e.g., the display generation component and / or the physical environment surrounding the computer system) (836a). In some embodiments, the physical environment is visible through a transparent portion of the display generation component (e.g., real or realistic transmission). In some embodiments, a representation of the physical environment is displayed in a three-dimensional environment via the display generation component (e.g., virtual or video transmission).

[0182] In some embodiments, when the first object is displayed at a third location relative to the three-dimensional environment, the computer system receives (836b) via the one or more input devices an instruction to move the first object to a corresponding location in the three-dimensional environment that corresponds to a corresponding location of the second computer system in the physical environment (such as, for example, Fig.7DIn some embodiments, the corresponding position in the three-dimensional environment corresponds to the position of the second computer system in the physical environment. In some embodiments, the second computer system (e.g., or a representation of the second computer system) is visible in the three-dimensional environment at the corresponding position in the three-dimensional environment.

[0183] In some embodiments, in response to receiving the second input (838a), the computer system moves (838b) the first object to a corresponding position in the three-dimensional environment according to the second input, such as Fig. 7E The object 736a is shown displayed (e.g., the first object is displayed as moving toward and / or to a corresponding position in the three-dimensional environment according to the second input, the corresponding position corresponding to the position of the second computer system in the physical environment). In some embodiments, the computer system initiates (838c) a process for sharing the content corresponding to the first object with the second computer system at the corresponding position in the physical environment. For example, a copy of the content corresponding to the first object or metadata corresponding to the content is shared with / transferred to the second computer system. In some embodiments, the second computer system receives an indication of a request to share the content corresponding to the first object with the second computer system. In some embodiments, in response to accepting the request to share the content corresponding to the first object with the second computer system, the content corresponding to the first object is downloaded to the second computer system. In some embodiments, in response to sharing, the first object is displayed on a display in communication with the second computer system. For example, the first object is no longer displayed in the three-dimensional environment. In some embodiments, the first object is displayed both on a display in communication with the second computer system and in a corresponding position in the three-dimensional environment. In some embodiments, the appearance of the first object at the corresponding location in the three-dimensional environment changes to indicate that the content corresponding to the first object has been shared with the second computer system. Dragging the object corresponding to the content in the three-dimensional environment to the second computer system visible in the three-dimensional environment causes the computer system to perform content sharing with the second computer system, thereby improving user-device interaction.

[0184] In some embodiments, the computer system is coupled to a second computer system (such as Fig.7DThe three-dimensional environment includes a representation of a first content item stored at the second computer system, such as a first content item stored at the second computer system, and a second content item stored at the second computer system. Fig.7D In some embodiments, the representation of the first content item is displayed at the corresponding location in the three-dimensional environment such that the representation of the first content item is displayed relative to the second computer system. For example, the first content item is displayed around the perimeter of the second computer system.

[0185] In some embodiments, when displaying a representation of a first content item stored at the second computer system at a corresponding location in the three-dimensional environment, the computer system receives (840c) via the one or more input devices a command to move the representation of the first content item stored at the second computer system from the corresponding location to a fourth location in the three-dimensional environment (such as, for example, Fig.7D The second input (eg, a pinch gesture of the index finger and thumb of the user's hand and subsequent movement of the hand in the pinch hand shape, as described above) corresponding to a request to move the representation of the content item 732a shown.

[0186] In some embodiments, in response to receiving the second input (840d), the computer system moves (840e) the representation of the first content item stored at the second computer system to a fourth location in the three-dimensional environment according to the second input, such as Fig. 7EThe object 732a is shown (e.g., the representation of the first content item is shown as being moved toward and / or moved to a fourth position in the three-dimensional environment according to the second input). In some embodiments, the computer system initiates (840f) a process for receiving the first content item from the second computer system. For example, a copy of the first content item corresponding to the representation of the first content item or metadata corresponding to the first content item is shared with the computer system / transferred from the second computer system to the computer system. In some embodiments, the computer system receives an indication of a request to share the first content item with the computer system. In some embodiments, in response to accepting the request to share the first content item with the computer system, the first content item is downloaded to the computer system. In some embodiments, the representation of the first content item is displayed both on a display communicating with the second computer system and / or at a corresponding position in the three-dimensional environment, and at a fourth position in the three-dimensional environment. In some embodiments, the representation of the first content item stops being displayed at a corresponding position in the three-dimensional environment. In some embodiments, the appearance of the representation of the first content item at a corresponding position in the three-dimensional environment changes to indicate that the first content item has been shared with the computer system. Dragging an object corresponding to content stored at the second computer system to a corresponding position in the three-dimensional environment visible via the computer system causes the second computer system to perform content sharing with the computer system, thereby improving user-device interaction.

[0187] In some embodiments, the first position in the three-dimensional environment corresponds to a predetermined portion (e.g., the center) of the field of view of the user's viewpoint (842a) (e.g., the first position in the three-dimensional environment is defined as a predefined position of the user's field of view of the three-dimensional environment as seen from the user's current viewpoint of the three-dimensional environment (e.g., the center, upper left, or upper right of the user's field of view)). In some embodiments, in response to receiving the first indication and before completing the operation related to the content, a user interface element corresponding to the content is displayed at the first position with a first level of visual prominence (842b), such as, Fig. 7A User interface elements 704a and / or 706a are shown. For example, notifications are optionally displayed along the normal of the user's viewpoint, which is optionally defined by the posture of the user's head. In some embodiments, notifications are displayed based on the center of the user's head, regardless of the current location of the user's gaze. In some embodiments, user interface elements corresponding to the content are displayed at a first size, a first clarity / sharpness, a first orientation, etc. at a first position. Displaying notifications of shared content in the center of the field of view of the user's viewpoint facilitates discovery that content is being shared with the computer system and / or facilitates input for interacting with shared content, thereby improving user-device interaction.

[0188] In some embodiments, when a user interface element corresponding to the content is displayed at a first position in the three-dimensional environment with a first level of visual prominence, the computer system detects (844a) a user's attention directed to the user interface element via the one or more input devices, such as Fig. 7A 721 and / or 723 in the diagram (eg, the user's gaze is directed toward a portion of a user interface element (eg, a notification) corresponding to the content).

[0189] In some embodiments, in response to detecting a user's attention directed toward a user interface element, a computer system displays (844b) the user interface element at a first location in a three-dimensional environment with a second level of visual prominence that is greater than the first level of visual prominence, such as Figure 7B The user interface elements 704a and / or 706a are shown. For example, a notification is optionally displayed at a first position with a second size, a second clarity / brightness, a second orientation, etc. that is larger than before the user's attention is directed to the user interface element. In some embodiments, based on determining that the user's attention is no longer directed to the user interface element, the user interface element corresponding to the content is displayed again with the visual prominence of the first level. In some embodiments, detecting the user's attention corresponds to detecting the user's gaze. In some embodiments, detecting the user's attention corresponds to detecting the gaze pointing to the user interface element for a threshold time period (e.g., 0.5 seconds, 1 second, 1.5 seconds, 2 seconds, 3 seconds, 3.5 seconds, 4 seconds, or 5 seconds). In some embodiments, detecting the user's attention corresponds to detecting the user's gaze and the ready state of at least one hand of the user. Displaying the notification in the center of the field of view of the user's viewpoint with enhanced prominence in response to detecting the gaze of the notification pointing to the shared content reduces the number of inputs required to interact with the notification, thereby improving user-device interaction.

[0190] In some embodiments, in response to detecting the user's attention directed to the user interface element, the computer system displays (846) via the display generation component one or more options that can be selected to accept or deny the request to share content with the computer system, such as Figure 7BIn the embodiment of the present invention, the computer system further displays an acceptance option 717 and a rejection option 719 in the user interface element. For example, when the user interface element is displayed with a second level of visual prominence, the computer system also displays one or more options for accepting or rejecting the sharing request. In some embodiments, the one or more options are displayed above, below, on the side, or inside a portion of the user interface element. In some embodiments, the selection of the first option for accepting the sharing request in the one or more options causes the content to be downloaded to the computer system. In some embodiments, the selection of the second option for rejecting the sharing request in the one or more options causes the user interface element corresponding to the content to be stopped from being displayed in the three-dimensional environment. In some embodiments, the one or more options are displayed before attention is directed to the user interface element. For example, before the computer system detects attention, the user interface element is generated and displayed at a first position in the three-dimensional environment using the one or more options. Displaying one or more options for accepting or rejecting a request to share content with the computer system reduces the number of inputs required to accept or reject a request to share content with the computer system, thereby improving user-device interaction.

[0191] It should be understood that the specific order in which the operations in method 800 are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0192] 9A to 9E An example of a computer system that facilitates interaction with virtual controls associated with physical devices, the virtual controls displayed in a three-dimensional environment, is shown according to some embodiments.

[0193] Fig.9A The computer system (eg, electronic device) 101 is shown via a display generating component (eg, Figure 1 The display generation component 120 of the computer system 101 displays the three-dimensional environment 902 from the viewpoint of the user 926 shown in a top view (e.g., facing the back wall of the physical environment in which the computer system 101 is located). In some embodiments, the computer system 101 includes a display generation component (e.g., a touch screen) and multiple image sensors (e.g., Figure 3The image sensor 314). The image sensor optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the computer system 101 can use to capture one or more images of a user or a portion of a user (e.g., one or more hands of a user) when the user interacts with the computer system 101. In some embodiments, the user interface shown and described below can also be implemented on a head-mounted display, which includes a display generation component that displays a user interface or a three-dimensional environment to the user, and sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's attention (e.g., gaze) (e.g., internal sensors facing inward toward the user's face).

[0194] like Fig.9A As shown, the computer system 101 captures one or more images of the physical environment (e.g., operating environment 100) surrounding the computer system 101 (including one or more objects in the physical environment surrounding the computer system 101). In some embodiments, the computer system 101 displays a representation of the physical environment in a three-dimensional environment 902. For example, the three-dimensional environment 902 includes a representation 922a of a coffee table (corresponding to the table 922b in the top view), which is optionally a representation of a physical coffee table in the physical environment, and the three-dimensional environment 902 includes a representation 924a of a sofa (corresponding to the sofa 924b in the top view), which is optionally a representation of a physical sofa in the physical environment. In addition, the three-dimensional environment 902 includes a representation 928a of a round table (corresponding to the table 928b in the top view), which is optionally a representation of a physical round table in the physical environment, and the three-dimensional environment 902 includes a representation 942a of a side table (corresponding to the table 942b in the top view), which is optionally a representation of a physical side table in the physical environment. Representations of physical objects displayed and / or visible in the three-dimensional environment 902 described herein optionally correspond to renderings (e.g., realistic or otherwise) of those physical objects displayed by the computer system 101, or optionally correspond to those objects being visible via visual transmission of those objects through transparent or semi-transparent portions of the display generation component 120.

[0195] exist Fig.9A , the physical environment surrounding computer system 101 optionally includes one or more physical devices (e.g., a second computer system) configured to communicate with computer system 101. In some embodiments, representations of the one or more physical devices are displayed in three-dimensional environment 902 or otherwise visible via display generation component 120 of computer system 101. For example, in Fig.9AIn the embodiment of the present invention, the three-dimensional environment 902 includes a representation 904a of a first device (corresponding to the device 904b in the top view), which is optionally a representation of a physical computer device in the physical environment, and the three-dimensional environment 902 includes a representation 906a of a second device (corresponding to the device 906b in the top view), which is optionally a representation of a physical speaker device in the physical environment. In addition, the three-dimensional environment 902 optionally includes a representation 908a of a third device (corresponding to the device 908b in the top view), which is optionally a representation of a physical light-emitting device in the physical environment.

[0196] In addition, Fig.9A , the user 926's attention (e.g., gaze) is alternatively directed to representations of physical devices in the three-dimensional environment 902. For example, Fig.9A As shown, a first gaze point ("gaze 1") 911 points to a representation of a first device 904a, a second gaze point ("gaze 2") 913 points to a representation of a second device 906a, and a third gaze point ("gaze 3") 915 points to a representation of a third device 908a. 9A to 9E Multiple gaze points are shown in the figure, but such gaze points need not be detected simultaneously by the computer system 101; rather, in some embodiments, the computer system 101 independently responds to such gaze points in response to independently detecting the gaze points shown and described.

[0197] In some embodiments, in response to detecting the user's attention directed toward a representation of a physical device in the three-dimensional environment 902, the computer system 101 displays one or more virtual control elements associated with the physical device in the three-dimensional environment. In some embodiments, the one or more virtual control elements associated with the physical device can be selected to cause one or more corresponding operations involving the physical device to be performed. For example, in Fig.9A In response to detecting a gaze 911 directed toward a representation of a first device 904a in the three-dimensional environment 902, the computer system 101 displays one or more virtual control elements 914a associated with the first device in the three-dimensional environment 902 (which correspond to elements 914b in the overhead view). In response to detecting a gaze 913 directed toward a representation of a second device 906a in the three-dimensional environment 902, the computer system 101 optionally displays one or more virtual control elements 916a associated with the second device in the three-dimensional environment 902 (which correspond to elements 916b in the overhead view). In response to detecting a gaze 915 directed toward a representation of a third device 908a in the three-dimensional environment 902, the computer system 101 optionally displays one or more virtual control elements 918a associated with the third device in the three-dimensional environment 902 (which correspond to elements 918b in the overhead view).

[0198] In some embodiments, as mentioned above, the one or more virtual control elements associated with the physical device can be selected in the three-dimensional environment 902 to cause one or more corresponding operations involving the physical device to be performed. For example, the first device is a computer device, and the one or more virtual control elements 914a associated with the first device can be optionally selected to cause the computer system 101 to display a workspace (e.g., a virtual desktop) corresponding to the user interface of the first device in the three-dimensional environment 902, as discussed in more detail later. The second device is optionally a speaker device, and the one or more virtual control elements 916a associated with the second device can be optionally selected to control one or more playback operations (e.g., play audio) of the second device, as discussed in more detail later. The third device is optionally a light-emitting device, and the one or more virtual control elements 918a associated with the third device can be optionally selected to control one or more light-emitting operations (e.g., control brightness) of the third device, as discussed in more detail below.

[0199] In some embodiments, in response to detecting a user's attention directed toward a representation of a physical device in three-dimensional environment 902, computer system 101 displays state information corresponding to the physical device in three-dimensional environment 902. For example, in Fig.9A In response to detecting a gaze 911 directed toward a representation of a first device 904a in a three-dimensional environment 902, the computer system 101 displays status information 932a corresponding to the first device in the three-dimensional environment 902 (which corresponds to information 932b in the overhead view). In some embodiments, the status information 932a corresponding to the first device includes information indicating a battery level of the first device (e.g., an indication of a battery percentage or charge) and / or a name of the first device (e.g., "device 1"). In response to detecting a gaze 913 directed toward a representation of a second device 906a, the computer system 101 optionally displays status information 934a corresponding to the second device in the three-dimensional environment 902 (which corresponds to information 934b in the overhead view). In some embodiments, the status information 934a corresponding to the second device includes information indicating a current volume output of the second device (e.g., a volume level of audio emitted from the second device) and / or a name of the second device (e.g., "device 2"). In response to detecting a gaze 915 directed toward a representation of a third device 908a in the three-dimensional environment 902, the computer system 101 optionally displays state information 936a corresponding to the third device in the three-dimensional environment 902 (which corresponds to the state information 936b in the top view). In some embodiments, the state information 936a corresponding to the third device includes information indicating a current light output of the third device (e.g., the amount or intensity of light emitted from the third device) and / or a name of the third device (e.g., "Device 3").

[0200] In some embodiments, the one or more virtual control elements associated with the physical device and / or the status information corresponding to the physical device are world locked in the three-dimensional environment 902. For example, in response to detecting attention directed toward a representation of a physical device in the three-dimensional environment 902, the computer system 101 displays the one or more virtual control elements associated with the physical device and / or the status information corresponding to the physical device in the three-dimensional environment 902 relative to a portion of the representation of the physical device in the three-dimensional environment 902. In some embodiments, the portion of the representation of the physical device in the three-dimensional environment 902 includes a top portion and / or a bottom portion, or a side and / or an edge of the representation of the physical device. For example, in Fig.9A , one or more virtual control elements 914a and status information 932a are displayed around / along the perimeter (e.g., top and bottom edges) of the representation of the first device 904a in the three-dimensional environment 902. Similarly, in Fig.9A In the three-dimensional environment 902, one or more virtual control elements 916a and status information 934a are displayed around / along the perimeter (e.g., top and bottom edges) of the representation of the second device 906a, and one or more virtual control elements 918a and status information 936a are displayed around / along the perimeter (e.g., top edge and side portions) of the representation of the third device 908a.

[0201] In some embodiments, virtual control elements associated with a physical device and / or status information corresponding to the physical device are displayed in the three-dimensional environment 902 with a corresponding orientation based on the viewpoint of the user 926 (e.g., when the user's gaze is directed toward the representation of the physical device in the three-dimensional environment 902). For example, virtual control elements associated with a physical device and / or status information corresponding to the physical device are displayed with a corresponding orientation facing the viewpoint of the user 926. Fig.9AAs shown, virtual control element 914a and / or state information 932a displayed with the representation of first device 904a has a first orientation (e.g., a front-facing surface of virtual control element 914a and / or state information 932a facing the viewpoint of user 926 is tilted / slightly angled to the right relative to the viewpoint of user 926 to be oriented toward the viewpoint of user 926, as shown in 914b and / or 932b in the top view). Virtual control element 916a and / or state information 934a displayed with the representation of second device 906a has a second orientation different from the first orientation (e.g., a front-facing surface of virtual control element 916a and / or state information 934a facing the viewpoint of user 926 is substantially perpendicular relative to the viewpoint of user 926 (e.g., within 1 degree, 2 degrees, 5 degrees, 10 degrees, 12 degrees, or 15 degrees), as shown in 934b and / or 916b in the top view). The virtual control element 918a and / or state information 936a displayed with the representation of the third device 908a has a third orientation that is different from the first orientation and the second orientation (e.g., the front-facing surface of the virtual control element 918a and / or state information 936a facing the viewpoint of the user 926 is tilted / slightly angled to the left relative to the viewpoint of the user 926 to be oriented toward the viewpoint of the user 926, as shown by 918b and / or 936b in the top view). It should be understood that given the exemplary spatial positions of the representations of the physical devices in the three-dimensional environment 902, Fig.9A The orientations of control elements and / or status information in are exemplary only, and other orientations will optionally be provided for different spatial locations of representations of physical devices in three-dimensional environment 902.

[0202] In some embodiments, the corresponding orientation of the one or more virtual control elements associated with the physical device and / or state information corresponding to the physical device in the three-dimensional environment 902 changes in response to detecting a change in the user's viewpoint of the three-dimensional environment 902. For example, as discussed below, the change in the user's viewpoint of the three-dimensional environment 902 causes the computer system 101 to display the virtual control elements associated with the corresponding physical device and / or state information corresponding to the corresponding physical device in the three-dimensional environment 902 at a new orientation based on the new viewpoint of the user 926. In some embodiments, when the computer system 101 detects a change in the viewpoint of the user 926, the computer system 101 maintains display of the one or more virtual control elements and / or state information displayed with a representation of the corresponding physical device, as discussed below.

[0203] exist Fig.9A, hand 903a is moving computer system 101, which movement corresponds to a change in the user's viewpoint of three-dimensional environment 902 (e.g., and / or a change in the viewpoint of a portion of the physical environment visible via display generation component 120 of computer system 101). FIG. 9A to FIG. 9B , hand 903a is optionally moving computer system 101 to the right in the physical environment surrounding computer system 101. In some embodiments, the input for changing the viewpoint of user 926 corresponds to movement of user 926's head in the physical environment (e.g., movement of a head mounted display worn by user 926 in the physical environment). In some embodiments, computer system 101 detects that hand 903a is moving computer system 101 (e.g., and display generation component 120) to a new corresponding position in the physical environment. Additionally, Fig.9A , hand 905a (e.g., in hand state A) is moving a first device (e.g., device 1), and the movement corresponds to a change in the position of the first device in the physical environment. FIG. 9A to FIG. 9B , the hand 905a optionally moves the first device 904a closer to the viewpoint 1126 of the user 926. In some embodiments, this movement input includes the user's hand moving the first device toward the body of the user 926 while the hand is holding the first device. It should be understood that although in 9A to 9E Multiple hands and multiple corresponding inputs are shown, but such hands and inputs need not be detected simultaneously by the computer system 101; rather, in some embodiments, the computer system 101 independently responds to such hands and / or inputs in response to independently detecting the hands and / or inputs shown and described.

[0204] In response to the movement of the first device closer to the body of the user 926, the computer system 101 optionally displays a representation of the first device 904a (e.g., a portion thereof) at a location in the three-dimensional environment 902 closer to the viewpoint of the user 926, such as Fig. 9B As shown, the user's gaze 911 is directed toward the representation of the first device 904a, which optionally causes the computer system 101 to maintain the display of one or more virtual control elements 914a and / or state information 932a at the new position of the representation of the first device 904a in the three-dimensional environment 902. In some embodiments, the computer system 101 continuously updates the display of the representation of the first device 904a so that the representation of the first device 904a moves in the three-dimensional environment 902 as the first device moves in the physical environment. Additionally or alternatively, in some embodiments, the movement of the first device in the physical environment is visible via the display generation component 120. Additionally, in response to Fig.9AIn response to a change in the user's viewpoint of the three-dimensional environment 902 detected in the computer system 101, the computer system 101 optionally changes the corresponding orientation of the virtual control element associated with the physical device and / or the status information corresponding to the physical device in the three-dimensional environment 902, as discussed below.

[0205] like Fig. 9B As shown, virtual control elements associated with the physical devices and / or state information corresponding to the physical devices are displayed in the three-dimensional environment 902 with a new / updated orientation based on the new viewpoint of the user 926 according to the movement of the hand 903b. For example, virtual control elements associated with the corresponding physical devices and / or state information corresponding to the corresponding physical devices are displayed with a new corresponding orientation facing the new viewpoint of the user 926. Fig. 9B As shown, virtual control elements 914a and / or status information 932a displayed with the representation of first device 904a have a new first orientation (e.g., a front-facing surface of virtual control elements 914a and / or status information 932a facing the viewpoint of user 926 is tilted / slightly angled to the right relative to the viewpoint of user 926 to be oriented toward the viewpoint of user 926, as shown in 914b and / or 932b in the top view). Virtual control elements 916a and / or status information 934a displayed with the representation of second device 906a optionally have a new second orientation that is different from the previous second orientation (e.g., a front-facing surface of virtual control elements 916a and / or status information 934a facing the viewpoint of user 926 is tilted / slightly angled to the right relative to the viewpoint of user 926 to be oriented toward the viewpoint of user 926, as shown in 934b and / or 916b in the top view). The virtual control element 918a and / or status information 936a displayed along with the representation of the third device 908a optionally have a new third orientation that is different from the previous third orientation (e.g., the forward surface of the virtual control element 918a and / or status information 936a facing the viewpoint of the user 926 is substantially perpendicular (e.g., within 1 degree, 2 degrees, 5 degrees, 10 degrees, 12 degrees, or 15 degrees) relative to the viewpoint of the user 926, as shown by 918b and / or 936b in the top view).

[0206] As described above, in response to computer system 101 detecting attention (e.g., gaze) directed toward a representation of a physical device in three-dimensional environment 902, one or more virtual control elements associated with the physical device and / or state information corresponding to the physical device are optionally displayed in three-dimensional environment 902. For example, Fig. 9BAs shown, gaze 911 is directed toward the representation of the first device 904a, gaze 913 is directed toward the representation of the second device 906a, and gaze 915 is directed toward the representation of the third device 908a, which causes the computer system 101 to maintain the display of virtual control elements and / or status information corresponding to each of the first device, the second device, and the third device in the three-dimensional environment 902. In some embodiments, when the corresponding physical device moves in the physical environment, and therefore when the position of the representation of the corresponding physical device is updated in the three-dimensional environment 902 (for example, and / or when the position of the corresponding physical device in the physical environment is visible via the display generation component 120), the computer system 101 stops displaying the one or more virtual control elements and / or status information in the three-dimensional environment 902, regardless of whether the user 926's attention is directed toward the representation of the corresponding physical device in the three-dimensional environment 902. Fig. 9B , hand 905b (e.g., in hand state B) is moving the first device in the physical environment. For example, the user's hand 905b in hand state B moves to the right in the physical environment while the hand grasps (e.g., the fingers of the hand grasp) the first device. In some embodiments, when the computer system 101 detects the movement of the hand 905b, the portion of the first device that is visible in the three-dimensional environment 902 (i.e., the portion of the representation of the first device 904a) moves in the three-dimensional environment 902.

[0207] When the hand 905b moves the first device in the physical environment, such as Fig. 9C As shown, the computer system 101 optionally updates the display of the representation of the first device 904a to correspond to the current position of the first device in the physical environment. Additionally or alternatively, in some embodiments, when the hand 905b moves the first device in the physical environment, the movement of the first device is visible via the display generation component 120. Fig. 9C , gaze points 913 and 915 point to representations of second device 906a and third device 908a, respectively, which causes computer system 101 to display corresponding virtual control elements and / or status information in three-dimensional environment 902, as discussed above. Additionally, when hand 905c moves the first device in the physical environment, gaze 911 points to the representation of first device 904a in three-dimensional environment 902. However, because computer system 101 detects that the first device is being actively moved in the physical environment, the computer system optionally abandons displaying virtual control elements 914a and / or status information 932a together with the representation of first device 904a in three-dimensional environment 902. Fig. 9C , hand 905c (e.g., in hand state C) continues to move the first device in the physical environment. For example, hand 905c is moving the first device to a side table (e.g., its representation 942a) visible via display generation component 120.

[0208] In response to detecting movement of the first device to the side table in the physical environment, the computer system 101 optionally displays a representation of the first device 904a at the location of the representation of the side table 942a in the three-dimensional environment 902 (e.g., on top of the representation of the side table 942a), such as Fig.9D As shown. Fig.9D , the gaze 911 of the user 926 is directed to the representation of the first device 904a in the three-dimensional environment 902. Because the computer system 101 has detected that the movement of the first device in the physical environment (e.g., by the hand 905c) has stopped, the computer system 101 redisplays one or more virtual control elements 914a associated with the first device and / or state information 932a corresponding to the first device in the three-dimensional environment 902. For example, in Fig.9D In the example embodiment, a representation of a first electronic device 904a is displayed in a three-dimensional environment 902 along with one or more virtual control elements 914a and status information 932a.

[0209] As mentioned above Fig.9A As mentioned, in some embodiments, the one or more virtual control elements associated with the corresponding physical devices in the three-dimensional environment 902 can be selected to cause one or more corresponding operations involving the corresponding physical devices to be performed. Fig.9D, hand 905d (e.g., in hand state D) is providing selection input directed to one or more virtual control elements 914a associated with a first device, hand 907a (e.g., in hand state A) is providing selection input directed to one or more virtual control elements 916a associated with a second device, and hand 909a (e.g., in hand state A) is providing selection input directed to one or more virtual control elements 918a associated with a third device in three-dimensional environment 902. In hand state D (e.g., before the hands are in a pinch hand shape (e.g., when the fingertips of the thumb and index finger of the hands are not touching)), hand 905d optionally provides input for activating a first virtual control element of the one or more virtual control elements 914a in three-dimensional environment 902. In hand state A (e.g., before the hand is in a pinch hand shape (e.g., when the fingertips of the thumb and index finger of the hand are not touching)), hands 907a and 909a optionally provide input for activating a first virtual control element in one or more virtual control elements 916a and a first virtual control element in one or more virtual control elements 918a in the three-dimensional environment 902, respectively. For example, the computer system 101 detects that hands 905d, 907a, and 909a move away from the body of the user 926 and then provide a pinch gesture directed to each of their respective targets (e.g., as indicated by gaze points 911, 913, and 915, respectively). Additional or alternative details regarding such selection input are described with reference to method 1000. In some embodiments, the input provided by hands 905d, 907a, and / or 909a is an air gesture input.

[0210] In response to detecting a selection of a corresponding virtual control element of the one or more virtual control elements associated with a physical device in the three-dimensional environment 902, performing a corresponding action involving the physical device, such as Fig.9E For example, in response to detecting Fig.9D In response to the selection of a first virtual control element in one or more virtual control elements 914a associated with the first device provided by the hand 905d, the computer system 101 displays a virtual workspace 912a (which corresponds to the workspace 912b in the top view) in the three-dimensional environment 902, as shown in FIG. Fig.9EIn some embodiments, the virtual workspace 912a includes a user interface (e.g., user interface A) that corresponds to a user interface configured to be displayed via a first device in the physical environment (e.g., on its display). In some embodiments, the virtual workspace 912a is displayed in the three-dimensional environment 902 at a location different from where the display of the first device (e.g., its representation) is located in the three-dimensional environment 902. Interactions with the user interface of the workspace 912a in the three-dimensional environment 902 optionally correspond to interactions with a user interface configured to be displayed via the first device in the physical environment.

[0211] In response to detecting Fig.9D In response to a selection of a first virtual control element in one or more virtual control elements 916a provided by hand 907b, the second device optionally emits audio (e.g., audio) corresponding to content (e.g., a song, a podcast, an audiobook, etc.) in the physical environment, such as Fig.9E In some embodiments, audio emitted by a second device in the physical environment can be observed (e.g., heard) in the three-dimensional environment 902. Fig.9D In response to a selection of a first virtual control element of one or more virtual control elements 918a associated with a third device provided by hand 909b, the third device (e.g., a light) optionally emits light in the physical environment at a corresponding brightness (e.g., intensity) for a corresponding duration, such as Fig.9E In some embodiments, light emitted by a third device in the physical environment can be observed (eg, seen) in the three-dimensional environment 902 .

[0212] In some embodiments, based on determining that the user's 926 attention is no longer directed toward the representation of the corresponding physical device in the three-dimensional environment 902, the computer system 101 stops displaying one or more virtual control elements and / or status information associated with the corresponding physical device. Fig.9E As shown, the gaze point is no longer directed to the representation of the physical device in the three-dimensional environment 902. For example, gaze 911 of user 926 is no longer directed to the representation of the first device 904a, gaze 913 is no longer directed to the representation of the second device 906a, and gaze 915 is no longer directed to the representation of the third device 908a. In response to detecting that the gaze point is no longer directed to the representation of the physical device in the three-dimensional environment 902, the computer system 101 stops displaying the one or more virtual control elements and / or state information in the three-dimensional environment 902. For example, in Fig.9E, the computer system 101 no longer displays one or more virtual control elements 914a and / or state information 932a with the representation of the first device 904a, no longer displays one or more virtual control elements 916a and / or state information 934a with the representation of the second device 906a, and no longer displays one or more virtual control elements 918a and / or state information 936a with the representation of the third device 908a. It should be understood that as described above with reference to Fig.9A As discussed, if the user 926's attention (e.g., gaze) is redirected to the representation of the corresponding physical device in the three-dimensional environment 902, the computer system 101 will optionally redisplay the one or more virtual control elements associated with the corresponding physical device and / or status information corresponding to the corresponding physical device.

[0213] FIG. 10A to FIG. 10F 1 is a flow chart illustrating a method 1000 for facilitating interaction with virtual controls associated with a physical device, the virtual controls being displayed in a three-dimensional environment, according to some embodiments. In some embodiments, the method 1000 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, a smart phone, a wearable computer, or a head mounted device, the computer system including a display generating component (e.g., Figure 1 , Figure 3 and Figure 4 In some embodiments, method 1000 is performed by storing in a non-transitory computer-readable storage medium and by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., Figure 1 Some operations in method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0214] In some embodiments, method 1000 is performed at a computer system (e.g., computer system 101) that communicates with a display generation component (e.g., 120), one or more input devices (e.g., 314), and a second electronic device. In some embodiments, the computer system has one or more characteristics of the characteristics of the electronic device in method 800. In some embodiments, the display generation component has one or more characteristics of the characteristics of the display generation component in method 800. In some embodiments, the one or more input devices have one or more characteristics of the characteristics of the one or more input devices in method 800. In some embodiments, the second electronic device is an electronic device or component capable of exchanging information with an electronic device. For example, the second electronic device can be controlled by a computer system (e.g., via input received, detected, captured, etc. at an electronic device). Examples of the second electronic device include a smart speaker, a light-emitting device (e.g., a lamp), a mobile device, or a computer, etc. In some embodiments, the second electronic device is an accessory for providing input to a computer system, such as a touchpad for providing touch input to an electronic device.

[0215] In some embodiments, the computer system displays (1002a) a three-dimensional environment (e.g., Fig.9A The three-dimensional environment 902 includes a second electronic device (eg, Fig.9A One or more virtual control elements (e.g., Fig.9AIn some embodiments, the one or more virtual control elements 914a, 916a and / or 918a in the physical environment are displayed, wherein the one or more virtual control elements can be selected to perform one or more corresponding operations associated with the second electronic device (for example, one or more virtual buttons configured to control the second electronic device. In some embodiments, the one or more virtual control elements are displayed at or near the location of the representation of the second electronic device. For example, the one or more virtual control elements are displayed above the representation of the second electronic device in the three-dimensional environment or on top of the surface of the representation of the second electronic device. In some embodiments, the one or more corresponding operations are specific to the second electronic device in the physical environment. For example, if the second electronic device is a speaker, the one or more virtual buttons in the three-dimensional environment can be selected to adjust the volume of audio emitted by the speaker in the physical environment. If the second electronic device is a light-emitting device, the one or more virtual buttons can optionally be selected to adjust the amount of light emitted by the light-emitting device in the physical environment.), wherein the second electronic device is located at a first corresponding position (1002b) in a portion of the physical environment visible via a display generating component (for example, the environment surrounding the display generating component and / or the computer system includes the second electronic device (for example, the second electronic device is located on a table, chair, shelf, etc. in the portion of the physical environment)).

[0216] In some embodiments, the three-dimensional environment includes a representation of the portion of the physical environment, the representation of the portion including a representation of the second electronic device at a first location in the three-dimensional environment corresponding to a first corresponding location in the physical environment, such as Fig.9A In some embodiments, the representation of the portion of the physical environment and / or the physical objects included in the portion of the physical environment are displayed in the three-dimensional environment via the display generation component (e.g., virtual or video see-through). In some embodiments, the representation of the portion of the physical environment and / or the physical objects included in the portion of the physical environment are a view of the physical environment of the electronic device that is visible through the transparent portion of the display generation component (e.g., real or actual see-through). In some embodiments, the three-dimensional environment has one or more of the characteristics of the three-dimensional environment in method 800) (1002c) (e.g., the representation of the second electronic device is displayed in the three-dimensional environment at a position that is the same as or corresponding to the position of the second electronic device in the physical environment). In some embodiments, when the second electronic device is not located at the first corresponding position in the portion of the physical environment that is visible via the display generation component, the one or more virtual control elements are not displayed in the three-dimensional environment.

[0217] In some embodiments, when displaying a three-dimensional environment including a representation of a second electronic device at a first location in the three-dimensional environment and one or more virtual control elements associated with the second electronic device, the computer system detects (1002d) movement of the second electronic device from the first corresponding location to a second corresponding location in the physical environment via the one or more input devices, such as Fig. 9B Movement of the first device 904a shown (e.g., movement of the second electronic device to a new location in the physical environment (e.g., to a new location on a table, to a different chair, to a new location on a shelf, etc.), or reorientation of the second electronic device in the physical environment (e.g., rotation of the second electronic device from a first orientation to a second orientation different from the first orientation)). In some embodiments, the movement of the second electronic device corresponds to movement of the second electronic device in the physical environment caused by one or both hands of a user of the electronic device carrying / holding / pushing / etc.

[0218] In some embodiments, when movement of the second electronic device from the first corresponding location to the second corresponding location in the physical environment is detected, the computer system stops (1002e) display of the one or more virtual control elements in the three-dimensional environment when the representation of the second electronic device is moving from the first location to the second location in the three-dimensional environment, such as stopping Fig. 9C Display of a virtual control element 914a in the three-dimensional environment (for example, the representation of the second electronic device in the three-dimensional environment moves according to the movement of the second electronic device in the physical environment. In some embodiments, the movement of the second electronic device is visible via the display generation component. In some embodiments, the movement is displayed and updated in real time in the three-dimensional environment (for example, within 1ms, 2ms, 3ms, 5ms, 9ms, 10ms, 15ms, 20ms, or 30ms after the corresponding movement of the second electronic device in the real-world environment). In some embodiments, during the movement of the second electronic device, the one or more virtual control elements are stopped from being displayed in the three-dimensional environment.), wherein the second position in the three-dimensional environment corresponds to a second corresponding position in the physical environment (for example, the representation of the second electronic device is displayed at a new position in the three-dimensional environment that is the same as or corresponds to the position of the second electronic device in the physical environment).

[0219] In some embodiments, after detecting movement of the second electronic device from the first corresponding position to the second corresponding position in the physical environment (1002f) (e.g., in response to detecting the end of movement of the second electronic device), the computer system redisplays (1002g) one or more virtual control elements associated with the second electronic device in the three-dimensional environment when a representation of the second electronic device is visible at the second position in the three-dimensional environment via the display generation component, such as, Fig.9DThe virtual control element 914a is shown to be redisplayed (for example, the representation of the second electronic device has been moved to a second position in the three-dimensional environment based on the movement of the second electronic device by the user's hand in the physical environment). In some embodiments, after the movement of the second electronic device ends, the one or more virtual control elements are redisplayed in the three-dimensional environment. In some embodiments, the one or more virtual control elements are displayed at or near the new position of the representation of the second electronic device in the three-dimensional environment. Stopping the display of virtual control elements associated with a physical device in the physical environment in the three-dimensional environment during the movement of the physical device in the physical environment avoids the occlusion of portions of the three-dimensional environment by the virtual control elements during the movement of the physical device, thereby improving user-device interaction.

[0220] In some embodiments, the one or more virtual control elements (1004a) associated with the second electronic device are displayed relative to a corresponding portion of the representation of the second electronic device, such as, Fig.9A The virtual control element 914a is shown displayed relative to the representation of device 904a (for example, the one or more virtual control elements associated with the second electronic device are displayed at or along a portion of the perimeter of the representation of the second electronic device in a three-dimensional environment). In some embodiments, the one or more virtual control elements are displayed above, below, to the side, and / or in front of a portion of the representation of the second electronic device.

[0221] In some embodiments, when movement of the second electronic device from the first corresponding location to the second corresponding location in the physical environment is detected (e.g., and when display of the one or more virtual control elements in the three-dimensional environment is stopped, as discussed above), the corresponding portion of the representation of the second electronic device is visible via the display generation component (1004b), such as Fig. 9B The top portion of the representation of device 904a in the physical environment is displayed. For example, when the second electronic device moves in the physical environment, the portion of the representation of the second electronic device (e.g., before the movement is detected) displaying the one or more virtual control elements is visible in the three-dimensional environment. Stopping display of virtual control elements associated with a physical device in the physical environment in the three-dimensional environment during movement of the physical device in the physical environment avoids occlusion of portions of the three-dimensional environment by the virtual control elements during movement of the physical device, thereby improving user-device interaction.

[0222] In some embodiments, in response to detecting that a user of the computer system is directing attention (such as Fig.9AThe user's gaze is directed to at least a portion of the representation of the second electronic device (911, 913 and / or 915), and the one or more virtual control elements associated with the second electronic device are displayed in the three-dimensional environment (1006). For example, when the user's gaze is directed to at least a portion of the representation of the second electronic device, the one or more virtual control elements associated with the second electronic device are displayed in the three-dimensional environment. In some embodiments, when the user's gaze is no longer directed to at least a portion of the representation of the second electronic device, the one or more virtual control elements are stopped from being displayed in the three-dimensional environment. Displaying virtual control elements associated with physical devices in the physical environment in a three-dimensional environment when the user's gaze is directed to the representation of the physical device reduces the number of inputs required to initiate interactions with these virtual control elements, thereby improving user-device interaction.

[0223] In some embodiments, the one or more virtual control elements associated with the second electronic device are displayed relative to a corresponding portion of the representation of the second electronic device (1008), such as, Fig.9A The virtual control element 916a is shown relative to the representation of the device 906a. For example, the one or more virtual control elements associated with the second electronic device are world locked. In some embodiments, the one or more virtual control elements are displayed at a portion of the periphery of the representation of the second electronic device in a three-dimensional environment or along the portion. In some embodiments, the one or more virtual control elements are displayed above, below, on the side and / or in front of a portion of the representation of the second electronic device. Displaying virtual control elements associated with physical devices in a physical environment relative to the representation of physical devices in a three-dimensional environment is beneficial for user input for controlling physical devices via virtual control elements, thereby improving user-device interaction.

[0224] In some embodiments, when viewed from a first perspective of a user of the computer system (such as Fig.9A When displaying a three-dimensional environment from the viewpoint of user 926 as shown, the computer system detects (1010a) movement of the user's viewpoint from a first viewpoint to a second viewpoint different from the first viewpoint, such as Fig.9A Movement of the hand 903a shown (e.g., or movement of the user's head to a new position / orientation that causes the user's viewpoint of the three-dimensional environment to change), the three-dimensional environment including a representation of a second electronic device at a second position in the three-dimensional environment and one or more virtual control elements associated with the second electronic device. In some embodiments, detecting movement of the user's head does not necessarily correspond to detecting movement of the user's gaze in the three-dimensional environment. For example, the user's head moves to a new position / orientation, while the user's gaze remains at the same position in the three-dimensional environment as before the movement was detected.

[0225] In some embodiments, when movement of the user's viewpoint from the first viewpoint to the second viewpoint is detected, the computer system maintains (1010b) display of the one or more virtual control elements associated with the second electronic device when a representation of the second electronic device is visible at the second location in the three-dimensional environment via the display generation component, such as, Fig. 9B The display of virtual control element 916a is shown as maintained. For example, when the viewpoint of the three-dimensional environment changes, the user's gaze remains pointing to the representation of the second electronic device. In some embodiments, the one or more virtual controls associated with the second electronic device remain displayed in the three-dimensional environment (for example, displayed at the position of the representation of the second electronic device in the three-dimensional environment, near or adjacent to the position), because when the user's viewpoint changes, the gaze remains pointing to the representation of the second electronic device. In some embodiments, the display of the one or more virtual control elements has nothing to do with the user's gaze pointing to the representation of the second electronic device. For example, when the user's viewpoint for the three-dimensional environment changes, the one or more virtual control elements associated with the second electronic device remain displayed in the three-dimensional environment, regardless of the direction of the user's gaze when the user's viewpoint changes. When the viewpoint of the three-dimensional environment changes, the display of the virtual control elements associated with the physical device in the physical environment is maintained to enable interaction with the virtual control elements from the new viewpoint of the three-dimensional environment without displaying additional controls, thereby improving user-device interaction.

[0226] In some embodiments, the one or more virtual control elements associated with the second electronic device have a first orientation in the three-dimensional environment that points to the viewpoint (1012) of the user of the computer system, such as Fig.9A . For example, because the one or more virtual control elements associated with the second electronic device are world-locked, the one or more virtual control elements are oriented in the three-dimensional environment to be angled toward the viewpoint of the user of the computer system. In some embodiments, the orientation of the one or more virtual control elements associated with the second electronic device is angled horizontally in the three-dimensional environment (e.g., angled along a horizontal axis that intersects the center of the one or more virtual control elements (e.g., in an upward or downward direction)). In some embodiments, the orientation of the one or more virtual control elements is angled vertically in the three-dimensional environment (e.g., angled along a vertical axis of the one or more virtual control elements (e.g., in a left or right direction)). Angling virtual control elements associated with physical devices in the physical environment toward the user reduces the number of inputs required to easily interact with virtual control elements in the three-dimensional environment, thereby improving user-device interaction.

[0227] In some embodiments, when the three-dimensional environment is displayed from a first viewpoint of a user, the computer system detects (1014a) movement of the user's viewpoint from the first viewpoint to a second viewpoint different from the first viewpoint, such as Fig.9A Movement of the hand 903a shown (e.g., or movement of the user's head to a new position / orientation that causes the user's viewpoint of the three-dimensional environment to change), the three-dimensional environment including a representation of a second electronic device at a second position in the three-dimensional environment and the one or more virtual control elements associated with the second electronic device having a first orientation in the three-dimensional environment. In some embodiments, detecting movement of the user's head does not necessarily correspond to detecting movement of the user's gaze in the three-dimensional environment. For example, the user's head moves to a new position / orientation, while the user's gaze remains at the representation of the second electronic device at the second position in the three-dimensional environment before the movement was detected.

[0228] In some embodiments, in response to detecting movement of the user's viewpoint, the computer system changes (1014b) the orientation of the one or more virtual control elements associated with the second electronic device from a first orientation to a second orientation different from the first orientation (e.g., without changing the position of the one or more virtual control elements associated with the second electronic device in the three-dimensional environment), such as, Fig. 9B The new orientation of the virtual control elements 914a and / or 918a shown, wherein the second orientation is based on a second viewpoint, such as, for example Fig. 9B The viewpoint of user 926 shown. For example, the one or more virtual control elements associated with the second electronic device are reoriented in the three-dimensional environment to continue to be angled toward the viewpoint of the user of the computer system. In some embodiments, the orientation of the one or more virtual control elements associated with the second electronic device changes horizontally in the three-dimensional environment (e.g., re-angled along a horizontal axis that intersects the center of the one or more virtual control elements (e.g., in an upward or downward direction)). In some embodiments, the orientation of the one or more virtual control elements changes vertically in the three-dimensional environment (e.g., re-angled along the vertical axis of the one or more virtual control elements (e.g., in a left or right direction)). Changing the angle of the virtual control elements associated with the physical device in the physical environment to maintain an angle toward the user when the viewpoint of the three-dimensional environment changes reduces the number of inputs required to easily interact with the virtual control elements in the three-dimensional environment, thereby improving user-device interaction.

[0229] In some embodiments, when displaying a three-dimensional environment including a representation of a second electronic device at a second location in the three-dimensional environment and the one or more virtual control elements associated with the second electronic device, the computer system receives (1016a) via the one or more input devices a first input corresponding to a request to activate a first virtual control element of the one or more virtual control elements, such as, Fig.9D A selection of the virtual control element 914a shown (e.g., an air gesture pointing to a first virtual control element of the one or more virtual control elements, such as a tap or selection gesture (e.g., provided by an index finger of a user's hand) or a pinch gesture (e.g., provided by an index finger and thumb of a user's hand)). In some embodiments, the first virtual control element can be selected to cause the second electronic device to perform a corresponding operation corresponding to the first virtual control element in the physical environment.

[0230] In some embodiments, in response to receiving the first input (1016b), the computer system performs (1016c) a first operation involving the second electronic device corresponding to the first virtual control element according to the first input, such as, Fig.9E The display object 912a is shown. For example, the second electronic device performs a first operation (e.g., power on, wake up, emit light, emit sound / audio, display content, etc.) in a physical environment, and the first operation can be observed (e.g., can be viewed, can be heard, can be felt, etc.) in a three-dimensional environment (e.g., via a representation of the second electronic device). In some embodiments, the appearance of the first virtual control element of the one or more virtual control elements changes in response to receiving a first input. For example, the first virtual control element changes in shape, size, color, opacity, transparency, etc. For example, the first virtual control element can be selected to play a song using the second electronic device, and the first virtual control element is a play icon. In response to receiving the first input, the first virtual control element changes its appearance to a pause icon. In some embodiments, a subsequent selection of the first virtual control element causes the second electronic device to stop performing the first operation, or to perform a second operation different from the first operation. Providing a selection of a virtual control element associated with a physical device in a physical environment causes the physical device to automatically perform a corresponding operation corresponding to the virtual control element, thereby improving user-device interaction.

[0231] In some embodiments, the first input includes (1018a) a user's attention (such as Fig.9D 911, 913 and / or 915) is directed to the first virtual control element while a corresponding portion of the user of the computer system (e.g., Fig.9DIn some embodiments, the user's hand 905d, 907a, and / or 909a) provides an interactive input (1018b) corresponding to the selection of the first virtual control element. For example, while the user's gaze is directed to the first virtual control element, a pinch gesture of the index finger and thumb of the user's hand pointing to the first virtual control element of the one or more virtual control elements (e.g., the index finger and thumb are brought together and touched to form a pinch hand shape while maintaining contact with each other, or within a threshold distance of each other (e.g., within 0.1 cm, 0.5 cm, 1 cm, 1.5 cm, 2 cm, 3 cm, 3.5 cm, 4 cm, or 5 cm of each other)) and a subsequent release of the pinch gesture (e.g., the index finger and thumb move away from each other). In some embodiments, when the user's gaze is not directed to the first virtual control element, the pinch gesture pointing to the first virtual control element is not sufficient to activate the first control element (e.g., and cause the first operation to be performed). Requiring gaze to be directed to a virtual control element associated with a physical device in a physical environment in order for the physical device to perform a corresponding operation corresponding to the virtual control element prevents inadvertent selection of the virtual control element, thereby improving user-device interaction.

[0232] In some embodiments, the second electronic device is configured to display a virtual workspace (eg, a virtual desktop), such as Fig.9E In some embodiments, the virtual workspace is visible in a three-dimensional environment via a display generating component of a computer system. In some embodiments, the virtual workspace is not visible via the display generating component (e.g., the display communicating with the second electronic device is turned off (e.g., the second electronic device is in a turned-off state or a sleep state or a low-power state).), such as Fig.9E The display of device 904a in.

[0233] In some embodiments, the one or more corresponding operations associated with the second electronic device include (1020b): displaying a virtual workspace (1020c) of the second electronic device in a three-dimensional environment via a display generation component without using a display device that communicates with the second electronic device, such as, Fig.9EDisplay object 912a is shown. For example, a representation of a virtual workspace configured to be displayed on a display communicating with a second electronic device is displayed in a three-dimensional environment (e.g., a representation of a home screen or an application window). In some embodiments, when the representation of the virtual workspace is displayed in the three-dimensional environment, the virtual workspace is stopped from being displayed on the display communicating with the second electronic device. In some embodiments, one can interact with the representation of the virtual workspace in the three-dimensional environment to control one or more user interface elements (e.g., applications, content, windows, etc.) associated with operations on the second electronic device (e.g., so that the second computer system performs those operations). Displaying a functional representation of a workspace associated with a physical device in a physical environment in a three-dimensional environment reduces the number of inputs required to interact with the workspace associated with the physical device, thereby improving user-device interaction.

[0234] In some embodiments, a second electronic device (such as Fig.9A The device 906a in the second electronic device is configured to play the content (1022a) (for example, the second electronic device is a smart speaker, a display device, a computer device including a display, or other device configured to play the content). In some embodiments, the one or more corresponding operations associated with the second electronic device include controlling the playback of the content (1022b), such as, for example Fig.9E As shown, audio is emitted. For example, the playback of a content item on a display is initiated, or audio (e.g., songs, podcasts, audiobooks, etc.) is emitted from a smart speaker. In some embodiments, the one or more virtual control elements may be selected to perform corresponding functions related to the playback of content. For example, for a content item (e.g., a video clip, a TV series, or a movie), the one or more virtual controls may be selected to browse the content, jump to the next content item, or pause the content, and / or for a song, the one or more virtual controls may be selected to fast forward a song, rewind a song, jump to the next song, or pause a song. In some embodiments, as similar to the above, the second electronic device plays content in a physical environment, and the content can be observed (e.g., can be viewed, can be heard, can be felt, etc.) in a three-dimensional environment (e.g., via a representation of a second electronic device). Providing a selection of a virtual control element associated with a physical device configured to play content in a physical environment enables the physical device to automatically control the playback of the content according to the selection of the virtual control element, thereby improving user-device interaction.

[0235] In some embodiments, the second electronic device includes a light emitting device (1024a), such as Fig.9AIn some embodiments, the one or more corresponding operations associated with the second electronic device include controlling the light output of the light emitting device (1024b), such as, for example, Fig.9E As shown, light is emitted. For example, light is initiated and / or caused to be emitted from the light emitting device. In some embodiments, the one or more virtual control elements may be selected to perform corresponding functions involving the emission of light. For example, the one or more virtual controls may be selected to control the intensity (e.g., brightness) of the light emitted from the light emitting device, the duration of the emission of the light emitted from the light emitting device, or a trigger for causing light to be emitted from the light emitting device (e.g., at a certain time of the day at the second electronic device or in response to a motion detected by the second electronic device). In some embodiments, as similar to that described above, the light emitting device of the second electronic device emits light in a physical environment, and the light can be observed (e.g., can be viewed, can be felt, etc.) in a three-dimensional environment (e.g., via a representation of the second electronic device). Providing a selection of a virtual control element associated with a physical device configured to emit light in a physical environment enables the physical device to automatically control light output according to the selection of the virtual control element, thereby improving user-device interaction.

[0236] In some embodiments, the three-dimensional environment also includes status information corresponding to the second electronic device, such as Fig.9A The status information 932a, 934a, and / or 936a in the second electronic device (e.g., information corresponding to the battery level of the second electronic device, the name of the second electronic device, and / or the status (e.g., progress or completion) of the one or more corresponding operations associated with the second electronic device) is displayed (1026) with respect to the corresponding portion of the representation of the second electronic device, such as, Fig.9AThe state information 932a is shown relative to the representation of device 904a. For example, the state information corresponding to the second electronic device is world locked. In some embodiments, the state information is displayed at a portion of the periphery of the representation of the second electronic device or along the portion in a three-dimensional environment. In some embodiments, the state information is displayed above, below, on the side and / or in front of the representation of the second electronic device. In some embodiments, the state information corresponding to the second electronic device is displayed together with the one or more virtual control elements (for example, in a straight line with the one or more virtual control elements, adjacent to the one or more virtual control elements, above or below the one or more virtual control elements). In some embodiments, as similar to the above, when the user's attention (for example, gaze) points to at least a portion of the representation of the second electronic device, the state information corresponding to the second electronic device is displayed. In some embodiments, when the second electronic device is not located at the first corresponding position in the visible portion of the physical environment via the display generation component, the state information corresponding to the second electronic device is not displayed in the three-dimensional environment. Displaying the state information corresponding to the physical device in the physical environment provides feedback on the state of the physical device in the three-dimensional environment, thereby improving user-device interaction.

[0237] It should be understood that the specific order in which the operations in method 1000 are described is merely exemplary and is not intended to indicate that the described order is the only order in which these operations can be performed. One of ordinary skill in the art will appreciate that a variety of ways are reordered to the operations described herein. In some embodiments, aspects / operations of methods 800 and 1000 may be interchanged, replaced, and / or added between these methods. For example, the three-dimensional environment of method 800 and / or 1000, the objects moved in method 800 and / or 1000, and / or the selection of control elements and / or the options of method 800 and / or 1000 may be optionally interchanged, replaced, and / or added between these methods. For the sake of simplicity, these details are not repeated here.

[0238] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best illustrate the principles of the present invention and its practical application, so that other technical personnel in the field can best use the present invention and the various described embodiments with various modifications suitable for the specific purpose contemplated.

[0239] As described above, one aspect of the present technology is to collect and use data from various sources to improve the user's XR experience. The present disclosure contemplates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identification or personal information.

[0240] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to improve the user's XR experience. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0241] The disclosure anticipates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information data will comply with sound privacy policies and / or privacy measures. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easy for users to access and should be updated as changes in the collection and / or use of data occur. Personal information from users should be collected for legal and reasonable entity purposes and should not be shared or sold outside these legal purposes. In addition, such collection / sharing should be carried out after receiving the informed consent of the user. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data, and ensure that other entities with access to personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adapted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including considerations of special jurisdictions. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy measures should be advocated for different types of personal data in each country.

[0242] Regardless of the foregoing, the present disclosure also contemplates implementation scenarios in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates hardware elements and / or software elements to prevent or block access to such personal information data. For example, with respect to XR experiences, the technology of the present invention may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for a service. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then reminded again just before the personal information data is accessed by the application.

[0243] In addition, it is the intention of the present disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting the collection of data and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how the data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0244] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, an XR experience may be generated by inferring preferences based on non-personal information data or an absolute minimum amount of personal information (such as content requested by a device associated with a user, other non-personal information available to a service, or publicly available information).

Claims

1. A method comprising: At a computer system in communication with a display generating component, one or more input devices, and a second electronic device: Displaying a three-dimensional environment via the display generation component, the three-dimensional environment comprising: one or more virtual control elements associated with the second electronic device, wherein the one or more virtual control elements are selectable to cause performance of one or more corresponding operations associated with the second electronic device, wherein the second electronic device is located at a first corresponding location in a portion of a physical environment visible via the display generating component; and a representation of the portion of the physical environment, the representation of the portion comprising a representation of the second electronic device at a first location in the three-dimensional environment corresponding to the first corresponding location in the physical environment; detecting, via the one or more input devices, movement of the second electronic device from the first corresponding position to a second corresponding position in the physical environment while displaying the three-dimensional environment including the representation of the second electronic device at the first position in the three-dimensional environment and the one or more virtual control elements associated with the second electronic device; upon detecting the movement of the second electronic device from the first corresponding position to the second corresponding position in the physical environment, ceasing display of the one or more virtual control elements in the three-dimensional environment while the representation of the second electronic device is moving from the first position to a second position in the three-dimensional environment, wherein the second position in the three-dimensional environment corresponds to the second corresponding position in the physical environment; and After detecting the movement of the second electronic device from the first corresponding location to the second corresponding location in the physical environment: When the representation of the second electronic device is visible at the second position in the three-dimensional environment via the display generation component, the one or more virtual control elements associated with the second electronic device are redisplayed in the three-dimensional environment.

2. The method according to claim 1, wherein: displaying the one or more virtual control elements associated with the second electronic device relative to a corresponding portion of the representation of the second electronic device; and When the movement of the second electronic device from the first corresponding location to the second corresponding location in the physical environment is detected, the corresponding portion of the representation of the second electronic device is visible via the display generation component.

3. The method of claim 1, wherein in response to detecting attention of a user of the computer system directed toward the representation of the second electronic device, the one or more virtual control elements associated with the second electronic device are displayed in the three-dimensional environment. 4 . The method of claim 1 , wherein the one or more virtual control elements associated with the second electronic device are displayed relative to a corresponding portion of the representation of the second electronic device.

5. The method according to claim 1, further comprising: detecting movement of the user's viewpoint from a first viewpoint to a second viewpoint different from the first viewpoint while the three-dimensional environment is displayed from a first viewpoint of a user of the computer system, the three-dimensional environment including the representation of the second electronic device at the second location in the three-dimensional environment and the one or more virtual control elements associated with the second electronic device; as well as When the movement of the user's viewpoint from the first viewpoint to the second viewpoint is detected, the display of the one or more virtual control elements associated with the second electronic device is maintained when the representation of the second electronic device is visible at the second position in the three-dimensional environment via the display generation component.

6. The method of claim 1, wherein the one or more virtual control elements associated with the second electronic device have a first orientation in the three-dimensional environment, the first orientation pointing toward a viewpoint of a user of the computer system.

7. The method according to claim 6, further comprising: detecting movement of the user's viewpoint from a first viewpoint to a second viewpoint different from the first viewpoint while displaying the three-dimensional environment from a first viewpoint of a user, the three-dimensional environment including the representation of the second electronic device at the second position in the three-dimensional environment and the one or more virtual control elements associated with the second electronic device having the first orientation in the three-dimensional environment; as well as In response to detecting the movement of the viewpoint of the user, changing the orientation of the one or more virtual control elements associated with the second electronic device from the first orientation to a second orientation different from the first orientation, wherein the second orientation is based on the second viewpoint.

8. The method according to claim 1, further comprising: receiving, via the one or more input devices, a first input corresponding to a request to activate a first virtual control element of the one or more virtual control elements while displaying the three-dimensional environment including the representation of the second electronic device at the second location in the three-dimensional environment and the one or more virtual control elements associated with the second electronic device; as well as In response to receiving the first input: A first operation involving the second electronic device corresponding to the first virtual control element is performed according to the first input.

9. The method of claim 8, wherein the first input comprises: An interactive input corresponding to a selection of the first virtual control element is provided by a corresponding portion of a user of the computer system while the user's attention is directed toward the first virtual control element.

10. The method of claim 1, wherein: The second electronic device is configured to display a virtual workspace for interacting with the second electronic device via a display device in communication with the second electronic device; and The one or more corresponding operations associated with the second electronic device include: The virtual workspace of the second electronic device is displayed in the three-dimensional environment via the display generation component without using the display device in communication with the second electronic device.

11. The method according to claim 1, wherein: The second electronic device is configured to play content; and The one or more corresponding operations associated with the second electronic device include controlling playback of the content.

12. The method of claim 1, wherein: The second electronic device includes a light emitting device; and The one or more corresponding operations associated with the second electronic device include controlling a light output of the light emitting device.

13. The method of claim 1, wherein: The three-dimensional environment also includes state information corresponding to the second electronic device, the state information corresponding to the second electronic device being displayed relative to a corresponding portion of the representation of the second electronic device.

14. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 13.

15. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by one or more processors of a computer system in communication with a display generation component and one or more input devices, cause the computer system to perform the method according to any one of claims 1 to 13.

Images