Device, method, and graphical user interface for interacting with three-dimensional environment

Abstract

To provide a display generation component and a computer generation experience.SOLUTION: While displaying a virtual object at a first spatial location in a three-dimensional environment, a computer system detects a first movement of a hand performed by a user. According to a determination that the first movement of the hand satisfies a first gesture criterion, the computer system performs the first action in response to the first movement of the hand without moving the virtual object away from the first spatial location, and according to a determination that the first movement of the hand satisfies a second gesture criterion, the computer system displays a first visual indication that the virtual object has entered a reconstruction mode and further detects a second movement of the hand performed by the user. According to a determination that the second movement of the hand satisfies the first gesture criterion, the computer system moves the virtual object from the first spatial location to a second spatial location in response to the second movement of the hand.SELECTED DRAWING: Figure 7A

Description

Related Applications

[0001] This application claims priority to U.S. Provisional Application No. 62 / 907,614, filed September 28, 2019, and U.S. Application No. 17 / 030,219, filed September 23, 2020, and is a continuation of U.S. Application No. 17 / 030,219, filed September 23, 2020. [Technical field]

[0002] The present disclosure generally relates to computer systems having a display generation component, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display, and one or more input devices that provide a computer-generated experience. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays for computer systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include digital images, videos, text, icons, and virtual objects such as buttons and other graphics.

[0004] However, methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) that include at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback to perform actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulation of virtual objects is complex and error-prone cause significant cognitive burden to users and detract from the experience of the virtual / augmented reality environment. In addition, the methods are unnecessarily time-consuming, thereby wasting energy. The latter problem is particularly acute in battery-operated devices. Summary of the Invention

[0005] Thus, there is a need for a computer system having improved methods and interfaces for providing a user with a computer-generated experience that makes interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can complement or replace conventional methods of providing a user with a computer-generated reality experience. Such methods and interfaces reduce the number, extent and / or type of inputs from the user by helping the user understand the connection between the inputs provided and the device response to that input, thereby creating a more efficient human-machine interface.

[0006] The above deficiencies and other problems associated with user interfaces for computer systems having a display generating component and one or more input devices are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a watch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generating component, the output devices including one or more tactile output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a number of functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on a touch-sensitive surface, movements of the user's eyes and hands in space relative to the GUI or the user's body as captured by cameras and other movement sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, creating spreadsheets, playing games, making phone calls, video conferencing, emailing, instant messaging, training support, digital photography, digital videography, web browsing, playing digital music, taking notes, and / or playing digital videos. Executable instructions to perform those functions are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] There is a need for electronic devices with improved methods and interfaces for interacting with a three-dimensional environment. Such methods and interfaces can complement or replace conventional methods for interacting with a three-dimensional environment. Such methods and interfaces reduce the number, extent, and / or type of input from a user, creating a more efficient human-machine interface.

[0008] There is a need for electronic devices with improved methods and interfaces for generating computer-generated environments. Such methods and interfaces can complement or replace conventional methods for generating computer-generated environments. Such methods and interfaces can create more efficient human-machine interfaces, allowing users more control over the device, and allowing users to use the device in a safer, less cognitively intensive, and improved user experience.

[0009] In some embodiments, the method is performed in a computer system including a display generation component and one or more input devices, the method comprising: displaying a virtual object at a first spatial location within the three-dimensional environment; detecting a first hand movement performed by a user while displaying the virtual object at the first spatial location within the three-dimensional environment; in response to detecting the first hand movement performed by the user, in accordance with a determination that the first hand movement satisfies a first gesture criterion, performing a first action in accordance with the first hand movement without moving the virtual object away from the first spatial location; in accordance with a determination that the first hand movement satisfies a second gesture criterion, displaying a first visual indication that the virtual object has transitioned to a reconstruction mode; detecting a second hand movement performed by the user while displaying the virtual object with the first visual indication that the virtual object has transitioned to the reconstruction mode; and in response to detecting the second hand movement performed by the user, moving the virtual object from the first spatial location to the second spatial location in accordance with the second hand movement in accordance with a determination that the second hand movement satisfies the first gesture criterion.

[0010] According to some embodiments, a method is executed in a computer system including a display generation component and one or more input devices, comprising: displaying, via the display generation component, a three-dimensional scene including at least a first virtual object at a first location and a first physical surface at a second location distinct from the first location, wherein a first virtual object is displayed with a first value of a first display characteristic corresponding to a first portion of the first virtual object and a second value of the first display characteristic that is different from a second portion of the first virtual object, the second value of the first display characteristic being different from the first value of the first display characteristic; and, while displaying the three-dimensional scene including the first virtual object and the first physical surface, performing a display generation process on the display generation component. and generating, via the component, a first visual effect at a second location in the three-dimensional scene, where generating the first visual effect includes modifying a visual appearance of a first portion of a first physical surface in the three-dimensional scene in accordance with a first value of a first display characteristic corresponding to the first portion of the first virtual object, and modifying a visual appearance of a second portion of the first physical surface in the three-dimensional scene in accordance with a second value of the first display characteristic corresponding to the second portion of the first virtual object, where the visual appearance of the first portion of the first virtual object and the visual appearance of the second portion of the first physical surface are modified differently due to a difference between the first and second values ​​of the first display characteristic of the first and second portions of the first physical surface.

[0011] According to some embodiments, a method is executed in a computer system including a display generation component and one or more input devices, comprising: displaying, via the display generation component, a three-dimensional scene including a first set of physical elements and a first quantity of virtual elements, the first set of physical elements including at least physical elements corresponding to a first class of physical objects and physical elements corresponding to a second class of physical objects; detecting, while displaying, via the display generation component, a sequence of two or more user inputs, a first quantity of virtual elements; and in response to detecting a first user input of the sequence of two or more user inputs, successively increasing an amount of the virtual elements displayed in the three-dimensional scene in accordance with the first quantity of the virtual elements in response to detecting a first user input of the sequence of two or more user inputs, in accordance with a determination that the first user input satisfies a first criterion: displaying the three-dimensional scene with at least a first subset of the one or more physical elements of the first set and a second quantity of virtual elements, where the second quantity of virtual elements occupies a larger portion of the three-dimensional scene than the first quantity of virtual elements, including a first portion of the three-dimensional scene that was occupied by the first class of physical elements prior to the detection of the first user input; and in response to detecting a second user input of the sequence of two or more user inputs, in accordance with a determination that the second user input follows the first user input and satisfies the first criterion, displaying the three-dimensional scene with at least a second subset of the one or more physical elements of the first set and a third quantity of virtual elements, where a third quantity of virtual elements occupies a larger portion of the three-dimensional scene than the second quantity of virtual elements, where the third quantity of virtual elements occupies a larger portion of the three-dimensional scene than the second quantity of virtual elements, where the third quantity of virtual elements includes: a first portion of the three-dimensional scene that was occupied by the first class of physical elements prior to the detection of the first user input and a second portion of the three-dimensional scene that was occupied by the second class of physical elements prior to the detection of the second user input.

[0012] According to some embodiments, a method is performed in a computer system including a display generation component and one or more input devices, comprising: displaying, via the display generation component, a three dimensional scene including at least a first physical object or a representation thereof, the first physical object having at least a first physical surface, and a respective position of the first physical object or a representation thereof in the three dimensional scene corresponding to a respective position of the first physical object in a physical environment surrounding the display generation component; detecting, while displaying the three dimensional scene, that a first interaction criterion is met, the first criterion being met when a first level of user interaction between a user and the first physical object is detected; and, in response to detecting that the first interaction criterion is met, displaying a three dimensional scene including a first physical surface and a respective position of the first physical object or a representation thereof in the three dimensional scene; The method includes displaying, via the display generation component, a first user interface at a location corresponding to a location of a first physical surface of a first physical object in the three dimensional scene; detecting that a second interaction criterion is met while displaying the first user interface at a location corresponding to the location of the first physical surface of the first physical object or a representation thereof in the three dimensional scene, the second interaction criterion including a second criterion that is met when a second level of user interaction between the user and the first physical object is detected that is higher than a first level of user interaction between the user and the first physical object; and in response to detecting that the second interaction criterion is met, replacing the display of the first user interface with a display of the second user interface at a location corresponding to the location of the first physical surface of the first physical object or a representation thereof in the three dimensional scene.

[0013] According to some embodiments, the method is executed on a computer system including a display generation component and one or more input devices, and includes displaying a three-dimensional scene via the display generation component, the three-dimensional scene including at least a first physical object having a first physical surface and at least a first virtual object having a first virtual surface; detecting a request to activate a voice-based virtual assistant while displaying the three-dimensional scene including the first physical object and the first virtual object; activating a voice-based virtual assistant configured to receive voice commands in response to detecting the request to activate the voice-based virtual assistant; and displaying a visual representation of the voice-based virtual assistant with a first set value of a first display characteristic of the visual representation. Displaying a visual appearance of at least a portion of the first physical surface of the first physical object and at least a portion of the first virtual surface of the first virtual object according to the first set value of the first display characteristic of the visual representation of the voice-based virtual assistant.

[0014] According to some embodiments, a computer system includes a display generation component (e.g., a display, projector, head mounted display, etc.), one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), optionally one or more haptic output generators, one or more processors, and a memory storing one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a non-transitory computer readable storage medium has instructions stored therein that, when executed by a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, cause the device to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a graphical user interface of a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, a memory, and one or more processors executing one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, which elements are updated in response to input as described in any of the methods described herein. According to some embodiments, a computer system includes a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, and means for performing or causing to be performed the operations of any of the methods described herein.According to some embodiments, an information processing apparatus for use in a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, includes means for performing or causing to be performed any of the operations of the methods described herein.

[0015] Thus, computer systems having display generating components are provided with improved methods and interfaces for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments, thereby increasing the effectiveness, efficiency, and safety and satisfaction of such computer systems. Such methods and interfaces may complement or replace conventional methods for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments.

[0016] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, especially in view of the drawings, the specification, and the claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and description, and not to define or limit the subject matter of the present invention. [Brief description of the drawings]

[0017] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0018] [Figure 1]FIG. 1 is a block diagram illustrating a computer system operating environment for providing a CGR experience, according to some embodiments.

[0019] [Diagram 2] FIG. 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience, according to some embodiments.

[0020] [Diagram 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of a CGR experience to a user, according to some embodiments.

[0021] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0022] [Diagram 5] FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0023] [Figure 6] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0024] [Figure 7A] FIG. 1 is a block diagram illustrating user interaction with a computer-generated three-dimensional environment (including, for example, reconstructions and other interactions) in accordance with some embodiments. [Figure 7B] FIG. 1 is a block diagram illustrating user interaction with a computer-generated three-dimensional environment (including, for example, reconstructions and other interactions) in accordance with some embodiments.

[0025] [Figure 7C] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between physical and virtual objects) according to some embodiments. [Figure 7D] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between physical and virtual objects) according to some embodiments. [Figure 7E] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between physical and virtual objects) according to some embodiments. [Figure 7F] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between physical and virtual objects) according to some embodiments.

[0026] [Figure 7G] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the immersion level of the computer-generated experience based on user input) according to some embodiments. [Figure 7H] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the immersion level of the computer-generated experience based on user input) according to some embodiments. [Figure 7I] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the immersion level of the computer-generated experience based on user input) according to some embodiments. [Figure 7J]FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the immersion level of the computer-generated experience based on user input) according to some embodiments. [Figure 7K] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the immersion level of the computer-generated experience based on user input) according to some embodiments. [Figure 7L] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the immersion level of the computer-generated experience based on user input) according to some embodiments.

[0027] [Figure 7M] 1 is a block diagram illustrating a method for facilitating user interaction with a computer-generated environment (e.g., using interaction with a physical surface to control a device or interact with the computer-generated environment) according to some embodiments. [Figure 7N] 1 is a block diagram illustrating a method for facilitating user interaction with a computer-generated environment (e.g., using interaction with a physical surface to control a device or interact with the computer-generated environment) according to some embodiments. [Figure 7O] 1 is a block diagram illustrating a method for facilitating user interaction with a computer-generated environment (e.g., using interaction with a physical surface to control a device or interact with the computer-generated environment) according to some embodiments. [Figure 7P] 1 is a block diagram illustrating a method for facilitating user interaction with a computer-generated environment (e.g., using interaction with a physical surface to control a device or interact with the computer-generated environment) according to some embodiments. [Figure 7Q]1 is a block diagram illustrating a method for facilitating user interaction with a computer-generated environment (e.g., using interaction with a physical surface to control a device or interact with the computer-generated environment) according to some embodiments. [Figure 7R] 1 is a block diagram illustrating a method for facilitating user interaction with a computer-generated environment (e.g., using interaction with a physical surface to control a device or interact with the computer-generated environment) according to some embodiments.

[0028] [Figure 7S] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (including, for example, mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. [Figure 7T] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (including, for example, mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. [Figure 7U] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (including, for example, mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. [Figure 7V] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (including, for example, mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. [Figure 7W] FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (including, for example, mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. [Figure 7X]FIG. 1 is a block diagram illustrating a method for generating a computer-generated three-dimensional environment (including, for example, mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments.

[0029] [Figure 8] 1 is a flowchart of a method for interacting (eg, including reconstruction and other interactions) with a computer-generated three-dimensional environment, according to some embodiments.

[0030] [Figure 9] 1 is a flowchart of a method for generating a computer-generated three-dimensional environment (eg, including mimicking visual interactions between physical and virtual objects) according to some embodiments.

[0031] [Figure 10] 1 is a flowchart of a method for generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the level of the computer-generated experience based on user input) according to some embodiments.

[0032] [Figure 11] 1 is a flowchart of a method for facilitating user interaction with a computer-generated environment (e.g., using interaction with a physical surface to control a device or interact with the computer-generated environment), according to some embodiments.

[0033] [Figure 12] 1 is a flowchart of a method for generating a computer-generated three-dimensional environment (including, for example, mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user, according to some embodiments.

[0035] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways.

[0036] In some embodiments, the computer system allows a user to interact with virtual objects in the computer-generated three-dimensional environment by using various gesture inputs. A first predefined gesture (e.g., a swipe gesture, a tap gesture, a pinch gesture, a drag gesture, etc.) causes the computer system to execute a first action corresponding to the virtual object, while the same predefined gesture, when combined with a special modifying gesture (e.g., a reconfiguration gesture), causes the computer system to move the virtual object in the computer-generated three-dimensional environment from one position to another position. Specifically, in some embodiments, the predefined reconfiguration gesture causes the virtual object to enter a reconfiguration mode. While in the reconfiguration mode, the object is moved from one position to another position in the computer-generated environment in response to a first gesture configured to trigger a first type of interaction with the virtual object when the virtual object is not in the reconfiguration mode (e.g., activating the virtual object, navigating within the virtual object, or rotating the virtual object). In some embodiments, the reconfiguration gesture is not part of a gesture that moves a virtual object, and the virtual object optionally remains in the reconfiguration mode after entering the reconfiguration mode in response to detection of a previous reconfiguration mode. While the virtual object is in the reconfiguration mode, the computer system optionally responds to other gesture inputs directed to the computer-generated environment without causing the virtual object to leave the reconfiguration mode. The computer system moves the virtual object according to a first respective gesture that is also configured to trigger a first type of interaction with the virtual object when the virtual object is not in the reconfiguration mode. A visual indication of the virtual object entering and remaining in the reconfiguration mode is provided to help a user understand the internal status of the computer-generated environment and the virtual object and provide appropriate inputs to achieve a desired result.By using special reconfiguration gestures that cause virtual objects to enter a reconfiguration mode, utilizing gestures that would normally trigger other actions that reconfigure the environment and move virtual objects, and providing a visual indication of the virtual objects entering and remaining in the reconfiguration mode in response to the special reconfiguration gestures, the number, range, and / or nature of inputs from the user are reduced to produce a more efficient human-machine interface.

[0037] In some embodiments, the computer system generates a three-dimensional environment that includes both physical objects (e.g., appearing in the three-dimensional environment through transparent or semi-transparent portions of the display generation component or in a camera view of the physical environment) and virtual objects (e.g., user interface objects, computer-generated virtual objects that mimic physical objects, and / or objects that have no physical analogue in the real world). The computer system generates mimic visual interactions between the virtual objects and the physical objects according to mimic physical light propagation principles. Specifically, light (e.g., including brightness, color, hue, time variation, spatial pattern, etc.) emitted from the virtual objects appears to illuminate both the physical objects and the virtual objects in the environment. The computer system generates mimic lighting and shadows on various parts of the physical surface and various parts of the virtual surface caused by the virtual light emitted from the virtual objects. The lighting and shadows are generated taking into account the physical light propagation principles as well as the spatial position of the virtual objects relative to other physical and virtual surfaces in the environment, the mimic physical properties of the virtual surfaces (e.g., surface texture, optical properties, shape, and dimensions, etc.), and the actual physical properties of the physical surfaces (e.g., surface texture, optical properties, shape, and dimensions, etc.). Light emitted from different parts of a virtual object affects different parts of other virtual objects and different parts of other physical objects in the environment differently due to differences in their positions and physical properties. By generating realistic and detailed visual interactions between virtual and physical objects and making the virtual and physical objects react similarly to the lighting from the virtual objects, the computer system can make the three-dimensional environment more realistic, help the user become more accustomed to the computer-generated three-dimensional environment, and reduce the user's errors when interacting with the computer-generated three-dimensional environment.

[0038] In some embodiments, a user provides the computer system with a sequence of two or more predefined inputs to successively increase the immersiveness of the computer-generated experience provided by the computer system. When the user places the display generating component of the computer system in a predefined position relative to the user (e.g., placing a display in front of his / her eyes or a head-mounted device on his / her head), the user's view of the real world is blocked by the display generating component and the content presented by the display generating component dominates the user's view. Sometimes users benefit from a more gradual and controlled process to transition from the real world to the computer-generated experience. Thus, when displaying content to the user through the display generating component, the computer system displays a pass-through portion that includes a representation of at least a portion of the real world surrounding the user, and gradually increases the amount of virtual elements that replace the physical elements visible through the display generating component. Specifically, in response to each successive input of the sequence of two or more user inputs, different classes of physical elements are removed from view and replaced by newly displayed virtual elements (e.g., an extension of the existing virtual elements or the newly added virtual elements). The gradual transitions into and out of an immersive environment controlled by user input are intuitive and natural to the user, enhancing the user's experience and comfort when using a computer system for computer-generated immersive experiences. Dividing the physical elements that are replaced in their entirety in response to respective inputs into different classes of physical elements allows user control over multiple gradual transitions while still reducing the total number of user inputs required to transition into an immersive computer-generated environment.

[0039] In some embodiments, when the computer system displays a three-dimensional environment including a physical object (e.g., the physical object is visible through the display generating component) (e.g., in a camera view of the physical environment shown by the display generating component or as a virtual representation of the physical object in a mimicked reality environment rendered by the display generating component through a transparent pass-through portion of the display generating component). The physical object has a physical surface (e.g., a planar or smooth surface). When an interaction level between the physical object and the user is at a first predetermined level, the computer system displays a first user interface at a position corresponding to a position of the physical object in the three-dimensional environment (e.g., such that the first user interface appears to be overlaid on or standing on the physical surface). When an interaction level between the physical object and the user is at a second level, e.g., a level higher than the first level of interaction, the computer system displays a second user interface replacing the first user interface at a position corresponding to a position of the physical object in the three-dimensional environment (e.g., such that the second user interface appears to be overlaid on or standing on the physical surface). The second user interface provides more information and / or functionality associated with the physical object compared to the first user interface. The computer system allows the user to interact with the first and second user interfaces using various means for receiving information and controlling the first physical object. This technique allows the user to interact with the physical object with the aid of more information and control provided to the location in the computer-generated environment. The location of interaction in the computer-generated environment corresponds to the physical location of the physical object in the real world.By adjusting the information and level of control (e.g., provided in different user interfaces) according to the detected level of interaction between the user and the physical objects, the computer system does not provide information unnecessarily or clutter the computer-generated three-dimensional environment, reducing user confusion and user errors as the user interacts with the computer-generated environment. This technique also allows a user to utilize nearby physical surfaces to remotely control physical objects, according to some embodiments. In some embodiments, a user can remotely control physical objects or control information about physical objects to make the user's interaction with the physical objects and / or the three-dimensional environment more efficient.

[0040] In some embodiments, the computer system generates a three-dimensional environment that includes both physical objects (e.g., appearing in the three-dimensional environment through transparent or semi-transparent portions of the display generation component or in a camera view of the physical environment) and virtual objects (e.g., user interface objects, computer-generated virtual objects that mimic physical objects, and / or objects that have no physical analogue in the real world). The computer system also provides a voice-based virtual assistant. When the voice-based virtual assistant is activated, the computer system displays a visual representation of the activated virtual assistant. The computer system also modifies the appearance of the physical and virtual objects in the environment, and sometimes the background of the peripheral area of ​​the user's field of view or the screen, according to the values ​​of the display properties of the visual representation of the virtual assistant. Specifically, the light emitted from the visual representation of the virtual assistant (e.g., including brightness, color, hue, time variation, spatial pattern, etc.) appears to illuminate both the physical and virtual objects in the environment, and optionally the background of the field of view of the user or the peripheral area of ​​the screen. The computer system generates mimic lighting and shadows on various parts of the physical surface and various parts of the virtual surface caused by the virtual light emitted from the visual representation of the virtual assistant. Lighting and shadows are generated taking into account physical light propagation principles, the spatial location of the visual representation of the virtual assistant relative to other physical and virtual surfaces in the computer-generated environment, the mimicked physical properties of the virtual surfaces (e.g., surface texture, optical properties, shape, and dimensions, etc.), and the actual physical properties of the physical surfaces (e.g., surface texture, optical properties, shape, and dimensions, etc.). The lighting effects associated with the virtual assistant provide continuous and dynamic feedback to the user regarding the state of the voice-based virtual assistant (e.g., active or paused, listening, and / or responsive).The computer system can make the computer-generated three-dimensional environment more realistic and informative by generating realistic and detailed visual interactions between the visual representation of the virtual assistant and other virtual and physical objects in the computer-generated environment, thereby helping the user become more familiar with the computer-generated three-dimensional environment and reducing user errors when the user interacts with the computer-generated three-dimensional environment.

[0041] 1-6 illustrate an exemplary computer system for providing a CGR experience to a user. FIGS. 7A-7B are block diagrams illustrating user interaction with a computer-generated three-dimensional environment (e.g., including reconstructions and other interactions) according to some embodiments. FIGS. 7C-7F are block diagrams illustrating a method of generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between physical and virtual objects) according to some embodiments. FIGS. 7G-7L are block diagrams illustrating a method of generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the immersiveness of the computer-generated experience based on user input) according to some embodiments. FIGS. 7M-7R are block diagrams illustrating a method of facilitating user interaction with a computer-generated environment (e.g., utilizing interactions with physical surfaces to control a device or interact with the computer-generated environment) according to some embodiments. FIGS. 7S-7X are block diagrams illustrating a method of generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. FIG. 8 is a flowchart of a method of interacting with a computer-generated three-dimensional environment (e.g., including reconstruction and other interactions) according to some embodiments. FIG. 9 is a flowchart of a method of generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between physical and virtual objects) according to some embodiments. FIG. 10 is a flowchart of a method of generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the level of computer-generated experience based on user input) according to some embodiments. FIG. 11 is a flowchart of a method of facilitating user interaction with a computer-generated environment (e.g., utilizing interactions with physical surfaces to control a device or interact with the computer-generated environment) according to some embodiments.12 is a flowchart of a method for generating a computer-generated three-dimensional environment (including, for example, mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. The user interfaces of FIGS. 7A-7X are used to explain the processes of FIGS. 8-12.

[0042] 1, a CGR experience is provided to a user via an operating environment 100 that includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, tactile sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, speed sensors, etc.), and optionally one or more peripheral devices 195 (e.g., consumer electronics, wearable devices, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensors 190, and the peripheral devices 195 are integrated with the display generation component 120 (eg, within a head-mounted or handheld device).

[0043] When describing a CGR experience, various terms are used to individually refer to several related, but distinct, environments that a user senses and / or that the user can interact with (e.g., using inputs detected by computer system 101 that cause the computer system generating the CGR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the CGR experience). The following are a subset of these terms:

[0044] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can sense and / or interact with the physical environment directly, such as through sight, touch, hearing, taste, and smell.

[0045] Computer-Generated Reality: In contrast, a Computer-Generated Reality (CGR) environment refers to a wholly or partially mimicked environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's body movements or representations thereof are tracked, and in response, one or more properties of one or more virtual objects mimicked within the CGR environment are adjusted to behave with at least one law of physics. For example, a CGR system may detect a person's head rotation and in response adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to a property(s) of a virtual object(s) in a CGR environment may be made in response to a representation of a body movement (e.g., a voice command). A person may sense and / or interact with a CGR object using any one of these senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create an audio environment with 3D or spatial breadth that provides the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may sense and / or interact with only audio objects.

[0046] Examples of CGR include virtual reality and mixed reality.

[0047] Virtual Reality: A virtual reality (VR) environment refers to an emulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a number of virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through imitation of the person's presence in the computer-generated environment and / or through imitation of a subset of the person's physical movements in the computer-generated environment.

[0048] Mixed reality: A mixed reality (MR) environment refers to a mimetic environment designed to incorporate sensory input from a physical environment or a representation thereof in addition to including computer-generated sensory input (e.g., virtual objects) as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtual continuum, a mixed reality environment is anywhere between but not including a complete physical environment at one end and a virtual reality environment at the other end. In some MR environments, the computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track position and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items from the physical environment or their representations). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0049] Examples of mixed reality include augmented reality and augmented virtuality.

[0050] Augmented reality: An augmented reality (AR) environment refers to a mimicked environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or semi-transparent display, whereby a person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. A person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, the video of the physical environment shown on the opaque display is referred to as "pass-through video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a mimicked environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) other than the perspective captured by the image sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., enlarging) a portion thereof, such that the altered portion is a modified version that represents the original captured image but is non-photorealistic. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

[0051] Augmented Virtual: An augmented virtual (AV) environment refers to a mimicking environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0052] Hardware: There are many different types of electronic systems that allow a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination of these. In an embodiment, a transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects a graphical image onto a person's retina. A projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or as a physical surface. In some embodiments, the controller 110 is configured to manage and regulate the user's CGR experience.In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., physical setting / environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the display generation component 120 (e.g., an HMD, a display, a projector, a touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of one or more of the display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0053] In some embodiments, display generation component 120 is configured to provide a CGR experience (e.g., at least a visual component of a CGR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component 120.

[0054] According to some embodiments, the display generation component 120 provides a CGR experience to the user while the user is virtually and / or physically present within the scene 105.

[0055] In some embodiments, the display generation component is worn on a part of the user's body (e.g., head, hand, etc.). Thus, the display generation component 120 includes one or more CGR displays provided for displaying CGR content. For example, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (e.g., a smartphone or tablet) configured to present CGR content, where the user holds the device with a display pointed at the user's field of view and a camera pointed at the scene 105. In some embodiments, the handheld device is optionally located in a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is a CGR chamber, housing, or room configured to present CGR content without the user wearing or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing an interaction with CGR content that is triggered based on an interaction that occurs in the space in front of a handheld or tripod mounted device may be implemented similarly to an HMD where the interaction occurs in the space in front of the HMD and the CGR content response is displayed via the HMD. Similarly, a user interface showing an interaction with CGR content that is triggered based on movement of a handheld or tripod mounted device relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD where the interaction is triggered by movement of the HMD relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0056] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will appreciate from this disclosure that various other features have not been shown for purposes of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0057] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communications interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communications buses 204 for interconnecting these and various other components.

[0058] In some embodiments, the one or more communications buses 204 include circuitry for interconnecting and controlling communications between the system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0059] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or the non-transitory computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

[0060] Operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, an adjustment unit 246, and a data transmission unit 248.

[0061] 1 , and optionally one or more of the input devices 125, the output devices 155, the sensors 190, and / or the peripheral devices 195. To that end, in various embodiments, the data acquisition unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0062] In some embodiments, the tracking unit 244 is configured to map the scene 105 and track the position of at least the display generation component 120 relative to the scene 105 of FIG. 1, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To that end, in various embodiments, the tracking unit 244 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, the processing unit 244 includes a hand tracking unit 243 and / or an eye tracking unit 245. In some embodiments, the hand tracking unit 243 is configured to track the position of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the scene 105 of FIG. 1, relative to the display generation component 120, and / or relative to a coordinate system defined relative to the user's hand. The hand tracking unit 243 is described in more detail below with reference to FIG. 4. In some embodiments, eye tracking unit 245 is configured to track the position and movement of a user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)), or relative to CGR content displayed via display generation component 120. Eye tracking unit 245 is described in more detail below with reference to FIG.

[0063] In some embodiments, the coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by the display generation component 120, and optionally by one or more of the output devices 155 and / or peripheral devices 195. To that end, in various embodiments, the coordination unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0064] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0065] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0066] Additionally, FIG. 2 is intended more to illustrate the functionality of various features that may be present in certain embodiments, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in FIG. 2 may be implemented within a single module, and various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from embodiment to embodiment, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0067] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, and by way of non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communications interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communications buses 304 for interconnecting these and various other components.

[0068] In some embodiments, the one or more communication buses 304 include circuitry to interconnect and control communications between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), and the like.

[0069] In some embodiments, the one or more CGR displays 312 are configured to provide a CGR experience to the user. In some embodiments, the one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, the one or more CGR displays 312 correspond to a waveguide display, such as diffractive, reflective, polarized, holographic, etc. For example, the HMD 120 includes a single CGR display. In another example, the HMD 120 includes a CGR display for each eye of the user. In some embodiments, the one or more CGR displays 312 can present MR or VR content. In some embodiments, the one or more CGR displays 312 can present MR or VR content.

[0070] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene viewed by the user when the HMD 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0071] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or the non-transitory computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 330 and CGR presentation module 340:

[0072] The operating system 330 includes procedures for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. As such, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transmission unit 348.

[0073] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0074] In some embodiments, the CGR presentation unit 344 is configured to present CGR content via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0075] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate a computer-generated reality) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0076] In some embodiments, data transmission unit 348 is configured to transmit data (e.g., presentation data, position data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0077] Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as existing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0078] Moreover, Figure 3 is intended more to illustrate the functionality of various features that may be present in certain embodiments, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 3 may be implemented within a single module, and various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from embodiment to embodiment, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0079] 4 is a schematic diagram of an example embodiment of a hand tracking device 140. In some embodiments, the hand tracking device 140 (FIG. 1) is controlled by a hand tracking unit 243 (FIG. 2) to track the position of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to the scene 105 of FIG. 1 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hand). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0080] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for distinguishing fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or an image of the entire body, and can have either a dedicated sensor with zoom capabilities or high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0081] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to the controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives the display generation component 120 accordingly. For example, a user can interact with software running on the controller 110 by moving the hand 408 and changing hand postures.

[0082] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spots of the pattern. This approach is advantageous in that it does not require the user to hold or wear any kind of beacon, sensor, or other marker. It gives the depth coordinate of a point in the scene relative to a predefined reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x, y, z axes such that the depth coordinate of a point in the scene corresponds to the z component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

[0083] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps including the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors to patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D positions of the user's wrist joints and fingertips.

[0084] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with the motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. The program can, for example, move and modify an image presented on the display generation component 120, or perform other functions, in response to the pose and / or gesture information.

[0085] In some embodiments, the software may be downloaded in electronic form to the controller 110, for example over a network, or may alternatively be provided on a tangible non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in a memory associated with the controller 110. Alternatively or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in FIG. 4 as a separate unit from the image sensor 440, by way of example, some or all of the processing functionality of the controller may be associated with the image sensor 404 by a suitable microprocessor and software, or by a dedicated circuit within the housing of the hand tracking device 402, or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or with any other suitable computerized device, such as a game console or media player. The sensing functions of the image sensor 404 may similarly be integrated into a computer or other computerized device that is controlled by the sensor output.

[0086] FIG. 4 further includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels with respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The brightness of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing gray levels as depth increases. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the sequence of depth maps.

[0087] 4 also illustrates, in accordance with some embodiments, a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406. In FIG. 4, the skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, palm center, end of hand that connects to wrist, etc.) and optionally the wrist or arm connected to the hand are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0088] FIG. 5 illustrates an exemplary embodiment of eye tracking device 130 (FIG. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 245 (FIG. 2) to track the position and movement of the user's gaze relative to scene 105 or relative to CGR content displayed via display generation component 120. In some embodiments, eye tracking device 130 is integrated with display generation component 120. For example, in some embodiments, if display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed on a wearable frame, the head-mounted device includes both components for generating CGR content for viewing by the user and components for tracking the user's gaze relative to the CGR content. In some embodiments, eye tracking device 130 is separate from display generation component 120. For example, if display generation component is a handheld device or a CGR chamber, eye tracking device 130 is, optionally, a separate device from the handheld device or the CGR chamber. In some embodiments, eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a head-mounted or non-head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally used in combination with a head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally part of a non-head-mounted display generation component.

[0089] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or semi-transparent display that allows the user to view the physical environment directly and display virtual objects on the transparent or semi-transparent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, whereby an individual using the system can observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0090] As shown in FIG. 5, in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye tracking camera may be aimed at the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, may be aimed at a "hot" mirror disposed between the user's eye and a display panel that reflects the IR or NIR light from the eye to the eye tracking camera while allowing visual light to pass through. The eye tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a corresponding eye tracking camera and lighting source.

[0091] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for the particular operating environment 100, e.g., 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility prior to delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic calibration process or a manual calibration process. The user-specific calibration process may include estimation of the particular user's eye parameters, e.g., pupil position, central vision position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters have been determined for the eye tracking device 130, images captured by the eye tracking camera may be processed using a glint-assisted method to determine the user's current visual axis and viewpoint relative to the display.

[0092] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be located between the user's eye(s) 592 and the display 510 (e.g., a left or right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) and may be pointed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown at the top of FIG. 5), or may be pointed at the user's eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown at the bottom of FIG. 5).

[0093] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses eye-tracking input 542 from the eye-tracking camera 540 for various purposes, e.g., in processing the frames 562 for display. The controller 110 optionally estimates the user's gaze on the display 510 based on the eye-tracking input 542 obtained from the eye-tracking camera 540, using a glint-assisted method or other suitable method. The gaze estimated from the eye-tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0094] Below, some possible use cases of the user's current gaze direction are described, which are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 can generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller can position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller can display certain virtual content in a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The autofocus mechanism of the external camera can then focus on an object or surface in the environment that the user is currently viewing on the display 510. As another exemplary use case, the eyepiece 520 may be a focusable lens, and the eye tracking information is used by the controller to adjust the focus of the eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. The controller 110 can leverage the eye tracking information to orient and focus the eyepiece 520 so that a nearby object the user is looking at appears at the correct distance.

[0095] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0096] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR ranges, and therefore does not introduce noise into the eye tracking system. Note that the positions and angles of the eye tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wide field of view (FOV) and a camera 540 with a narrow FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0097] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality (e.g., including virtual reality and / or mixed reality) applications to provide a user with a computer-generated reality (e.g., including virtual reality, augmented reality, and / or augmented virtual) experience.

[0098] FIG. 6 illustrates a glint-assisted eye tracking pipeline according to some embodiments. In some embodiments, the eye tracking pipeline is realized by a glint-assisted eye tracking system (e.g., eye tracking device 130 as shown in FIG. 1 and FIG. 5). The glint-assisted eye tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the glint-assisted eye tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted eye tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0099] As shown in FIG. 6, an eye tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0100] At 610, if the tracking status is yes for the current captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eye.

[0101] At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and the pupil and glint information is passed to element 680 to estimate the user's gaze.

[0102] 6 is intended to serve as an example of eye-tracking technology that may be used in a particular implementation. As will be recognized by those skilled in the art, other eye-tracking technologies, either currently existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with a CGR experience according to various embodiments.

[0103] In the present disclosure, various input methods are described with respect to interaction with a computer system. If one example is provided using one input device or input method and another example is provided using another input device or input method, it should be understood that each example is compatible with and can be optionally used with the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. If one example is provided using one output device or output method and another example is provided using another output device or output method, it should be understood that each example is compatible with and can be optionally used with the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. If an example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and can be optionally used with the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples without exhaustively listing all features of the embodiments in the description of each embodiment. User Interface and Related Processes

[0104] Attention is now directed to embodiments of user interfaces ("UI") and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, with a display generating component, one or more input devices, and (optionally) one or more cameras.

[0105] 7A-7B are block diagrams illustrating user interaction with a computer-generated three-dimensional environment (including, for example, reconstructions and other interactions) according to some embodiments, and are used to illustrate the processes described below, including the process of FIG.

[0106] In some embodiments, the input gestures described with reference to FIGS. 7A-7B are detected by analyzing data and signals captured by a sensor system (e.g., sensor 190 of FIG. 1, image sensor 314 of FIG. 3). In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as a motion RGB camera, an infrared camera, a depth camera, etc.). For example, the one or more imaging sensors are components of or provide data to a computer system (e.g., computer system 101 of FIG. 1 (e.g., portable electronic device 7100 or HMD as shown in FIGS. 7A-7B)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4 (e.g., a touch screen display that functions as a display and a touch-sensitive surface, a stereoscopic display, a display with a pass-through portion, etc.)). In some embodiments, the one or more imaging sensors include one or more rear-facing cameras on a side of the device opposite the display of the device. In some embodiments, the input gestures are detected by a sensor system of a head-mounted system (e.g., a VR headset that includes a stereoscopic display that provides a left image for the user's left eye and a right image for the user's right eye). For example, one or more cameras that are components of the head mounted system are mounted on the front and / or underside of the head mounted system. In some embodiments, one or more imaging sensors are positioned in the space in which the head mounted system is used (e.g., arrayed around the head mounted system at various positions in a room) such that the imaging sensors capture images of the head mounted system and / or a user of the head mounted system. In some embodiments, the input gestures are detected by a sensor system of a head up device (e.g., a head up display, an automobile windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics). For example, one or more imaging sensors are mounted on an interior surface of an automobile.In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array). For example, the one or more depth sensors include one or more light-based (e.g., infrared) sensors and / or one or more acoustic-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal emitters, such as light emitters (e.g., infrared emitters) and / or sound emitters (e.g., ultrasonic emitters). For example, while light (e.g., light from an infrared light emitter array having a predetermined pattern) is projected onto a hand (e.g., hand 7200 as described with respect to FIGS. 7A-7B), images of the hand under illumination with the light are captured by one or more cameras, and the captured images are analyzed to determine the position and / or configuration of the hand. By determining input gestures using signals from an image sensor pointed at the hand, as opposed to using signals from a touch-sensitive surface or other direct contact or proximity-based mechanism, a user is free to choose whether to perform large movements or remain relatively still when providing input gestures with their hands, without experiencing constraints imposed by a particular input device or input area.

[0107] In some embodiments, multiple user interface objects 7208, 7210, and 7212 (e.g., in a menu or dock, or independent of each other) are displayed in a computer-generated three-dimensional environment (e.g., a virtual environment or a mixed reality environment). The multiple user interface objects are optionally displayed floating in space or above physical objects in the three-dimensional environment. Each user interface object optionally has one or more corresponding actions that can be performed in the three-dimensional environment or that cause an action in the physical environment in communication with the computer system (e.g., controlling another device (e.g., a speaker or a smart lamp) in communication with the device 7100). In some embodiments, the user interface objects 7208, 7210, and 7212 are displayed by a display of the computer system (e.g., device 7100 (FIGS. 7A-7B) or an HMD) along with (e.g., overlaying or replacing) at least a portion of a view of the physical environment captured by one or more rear-facing cameras of the computer system (e.g., device 7100). In some embodiments, user interface objects 7208, 7210, and 7212 are displayed on a transparent or semi-transparent display (e.g., a heads-up display, or HMD) of a computer system through which the physical environment is visible. In some embodiments, user interface objects 7208, 7210, and 7212 are displayed in a user interface that includes pass-through portions surrounded by virtual content (e.g., transparent or semi-transparent portions through which the physical surroundings are visible, or portions that display a camera view of the surrounding physical environment). In some embodiments, user interface objects 7208, 7210, and 7212 are displayed in a virtual reality environment (e.g., floating in virtual space or overlaid on a virtual surface).

[0108] In some embodiments, a representation of the hand 7200 is visible in the virtual reality environment (e.g., an image of the hand 7200 captured by one or more cameras is rendered in the virtual reality setting). In some embodiments, a representation 7200' of the hand 7200 (e.g., a cartoon version of the hand 7200) is rendered in the virtual reality setting. In some embodiments, the hand 7200 or a representation thereof is invisible (e.g., omitted) in the virtual reality environment. In some embodiments, the device 7100 (FIG. 7C) is invisible in the virtual reality environment (e.g., when the device 7100 is an HMD). In some embodiments, an image of the device 7100 or a representation of the device 7100 is visible in the virtual reality environment.

[0109] In some embodiments, one or more of the user interface objects 7208, 7210, and 7212 are application launch icons (e.g., for performing an action to launch a corresponding application, for displaying a quick action menu corresponding to the respective application, etc.). In some embodiments, one or more of the user interface objects 7208, 7210, and 7212 are controls for performing a respective action within the application (e.g., turn up the volume, turn down the volume, play, pause, fast forward, rewind, initiate communication with a remote device, terminate communication with a remote device, transmit communication with a remote device, start a game, etc.). In some embodiments, one or more of the user interface objects 7208, 7210, and 7212 are respective representations (e.g., avatars) of users of the remote devices (e.g., for performing an action to initiate communication with the respective users of the remote devices). In some embodiments, one or more of user interface objects 7208, 7210, and 7212 are representations (e.g., thumbnails, two-dimensional images, or album covers) of media items (e.g., images, virtual objects, audio files, and / or video files). For example, activating a user interface object that is a representation of an image causes the image to be displayed (e.g., at a location corresponding to a surface detected by one or more cameras and displayed in a computer-generated reality view (e.g., a location corresponding to a surface in a physical environment or a location corresponding to a surface displayed in a virtual space). Navigating within a user interface object that is an album (e.g., a music album, an image album, a flipbook album, etc.) switches the currently played or displayed item to another item in the album.

[0110] As shown in FIG. 7A, two separate actions are performed on user interface objects 7208, 7210, and 7212 in a three-dimensional environment in response to different types of gesture input provided by hand 7200, while the reconstruction mode is not activated for any of the user interface objects.

[0111] In FIG. 7A(a-1) and FIG. 7A(a-3), the thumb of the hand 7200 performs a tap gesture by moving along a vertical axis to contact the side of the index finger and moving upward away from the side of the index finger. The tap gesture is performed while a current selection indicator (e.g., a selector object or a moving visual effect such as highlighting the object by changing the outline or appearance of the object) is located on the user interface object 7208, indicating the currently selected status of the user interface object 7208. In some embodiments, in response to detecting a tap input by the hand 7200, the computer system (e.g., the device 7100) performs a first action (e.g., activates the user interface object 7208) that causes the virtual object 7202 to be displayed (e.g., as part of the user interface of the application represented by the user interface object 7208 or as content represented by the user interface object 7208). The visual appearance of the user interface object 7208 indicates that the first action is being performed (e.g., activated but not moved).

[0112] In FIG. 7A(a-1) and then in FIG. 7A(a-4)-FIG. 7A(a-5), the hand 7200 performs a drag gesture by moving laterally after the thumb of the hand touches the side of the index finger. The drag gesture is performed while a current selection indicator (e.g., a selector object or a moving visual effect such as highlighting the object by changing the object's outline or appearance) is located on the user interface object 7208, indicating the currently selected status of the user interface object 7208. In some embodiments, in response to detecting a drag input by the hand 7200, the computer system (e.g., device 7100) performs a second action on the user interface object 7208 (e.g., navigate away from the user interface object 7208 toward the user interface object 7210, or navigate within the user interface object 7208). The visual appearance of the user interface object indicates that the second action is being performed (e.g., navigation has occurred within the content of the user interface object or away from the user interface object, but the object has not moved within the three-dimensional environment).

[0113] FIG. 7B illustrates a scenario that is entirely different from that shown in FIG. 7A in that a reconfiguration gesture is performed (e.g., in combination with other gesture inputs (e.g., the gestures shown in FIG. 7A)) resulting in a three-dimensional environment being reconfigured (e.g., involving movement of user interface object 7208 in the three-dimensional environment).

[0114] As shown in the sequence of Figs. 7B(a-1)-7B(a-4), a wrist flick gesture is provided by the hand 7200 while the user interface object 7208 is currently in a selected state. In this example, the wrist flick gesture is a predefined reconfiguration gesture that causes the currently selected user interface object to enter a reconfiguration mode. In some embodiments, detecting the wrist flick gesture includes detecting a touchdown of the thumb on the side of the index finger, followed by an upward rotation of the hand around the wrist. Optionally, at the end of the wrist flick gesture, the thumb is lifted from the side of the index finger. In response to detecting the wrist flick gesture (e.g., by a previous input or by a gaze input focused on the user interface object 7208) while the user interface object 7208 is selected, the computer system (e.g., device 7100) activates a reconfiguration mode for the user interface object 7208. The computer system also displays a visual indication to inform the user that the user interface object 7208 is currently in a reconfiguration mode. In some embodiments, as shown in FIG. 7B(b-3), the user interface object is removed from its original position and, optionally, displayed with an altered appearance (e.g., semi-transparent, enlarged, and / or floating) to indicate that the user interface object 7208 is in the reconfiguration mode. In some embodiments, after the reconfiguration gesture is completed, the user interface object 7208 remains in the reconfiguration mode and the visual indication remains displayed in the three-dimensional environment. In some embodiments, the user interface object 7208 remains in the reconfiguration mode (e.g., floating in its original position with an altered appearance) while the computer system, optionally, responds to and provides interaction with the three-dimensional environment in accordance with other user input.In some embodiments, the computer system optionally allows the user to use a second wrist flick gesture to cause another currently selected user interface object (e.g., the user, optionally, selects another object with gaze or tap input) to enter the reconstruction mode while the user interface object 7208 remains in the reconstruction mode. In some embodiments, the computer system allows the user to look away or navigate to other parts of the three-dimensional environment without moving or interacting with the user interface object in the reconstruction mode while one or more user interface objects (e.g., the user interface object 7208) remain in the reconstruction mode. In some embodiments, in contrast to what is shown in FIGS. 7A(a-4)-7A(a-5), a subsequent drag gesture (e.g., performed by the hand 7200 moving laterally after the thumb of the hand touches the side of the index finger) may move the user interface object 7208 in the reconstruction mode from its current position to another position in the three-dimensional environment according to the hand movement (e.g., as shown in FIGS. 7B(a-5)-7B(a-6)). In some embodiments, moving the user interface object 7208 according to a drag gesture does not cause the user interface object 7208 to leave the reconfiguration mode. One or more additional drag gestures are optionally used to reposition the user interface object 7208 in the three-dimensional environment while the user interface object 7208 remains in the reconfiguration mode. In some embodiments, a predefined exit gesture (e.g., a downward wrist flick gesture (e.g., a downward wrist flick gesture performed at the end of a drag gesture, or a separate downward wrist flick gesture that is not part of another gesture) causes the user interface object 7208 to leave the reconfiguration mode.In some embodiments, once the user interface object 7208 exits the reconfiguration mode, its appearance is restored to its original state and settles into the destination position specified by the drag input(s) directed at the user interface object while in the reconfiguration mode.

[0115] As shown in the sequence of FIGS. 7B(a-5)-7B(a-6) following FIGS. 7B(a-1)-7B(a-2), a wrist flick gesture provided by hand 7200 is the beginning part of a composite gesture that ends with a drag gesture provided by hand 7200. The wrist flick gesture is detected while user interface object 7208 is currently in a selected state. In this example, the wrist flick gesture causes the currently selected user interface object to enter a reconfiguration mode and move to another position according to the movement of the drag gesture. In some embodiments, after a user interface object (e.g., user interface object 7208) enters the reconfiguration mode, the user interface object optionally remains in the reconfiguration mode after being moved from one position to another in the environment by the drag input.

[0116] In some embodiments, other types of gestures are optionally used as reconfiguration gestures to activate a reconfiguration mode of a currently selected user interface object. In some embodiments, a predefined gesture is configured to activate a reconfiguration mode of a respective class of user interface objects in the three-dimensional environment (e.g., cause multiple user interface objects of the same class (e.g., a class of application icons, a class of content items, a class of objects representing physical objects, etc.) to enter a reconfiguration mode to allow the respective classes of user interface objects to be moved individually or synchronously in the three-dimensional environment according to a subsequent movement input (e.g., a drag input). In some embodiments, the computer system activates a reconfiguration mode of a user interface object in response to detecting a tap input (e.g., on a finger or controller) while the user interface object is selected (e.g., by a previous input or gaze input). In some embodiments, the computer system activates a reconfiguration mode of a user interface object in response to detecting a swipe input (e.g., on a finger or controller) while the user interface object is selected (e.g., by a previous input or gaze input).

[0117] In some embodiments, while the user interface object is in the reconstruction mode, the computer system displays a visual indicator (e.g., a shadowed or semi-transparent image of the user interface object) that follows the user's gaze or finger movements to indicate the intended location of the user interface object within the three-dimensional environment. In response to detecting a subsequent commitment input (e.g., a downward wrist flick gesture or a tap input on a finger or controller), the computer system places the user interface object at the current location of the visual indicator.

[0118] In some embodiments, the drag input shown in Figures 7A and 7B is replaced by a swipe input on a finger or controller to cause the corresponding function to be performed.

[0119] In some embodiments, movement of user interface objects within the three-dimensional environment mimics movement of physical objects in the real world and is constrained by virtual and physical surfaces within the three-dimensional environment. For example, when a virtual object is moved in response to a drag input while the virtual object is in the reconfiguration mode, the virtual object slides across physical surfaces represented in the three-dimensional environment, and optionally also across virtual surfaces in the three-dimensional environment. In some embodiments, the user interface object jumps when switching between physical surfaces represented in the three-dimensional environment.

[0120] In some embodiments, the computer system optionally generates an audio output (eg, a continuous or one or more discrete audio outputs) while the user interface object is in the reconfiguration mode.

[0121] 7C-7F are block diagrams illustrating a method of generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between physical and virtual objects) according to some embodiments. Figures 7C-7F are used to illustrate the processes described below, including the process of Figure 9.

[0122] 7D-7F illustrate exemplary computer-generated environments that correspond to the physical environment shown in FIG. 7C. As described herein with reference to FIGS. 7D-7F, according to some embodiments, the computer-generated environment may optionally be an augmented reality environment including a camera view of the physical environment, or a computer-generated environment displayed on a display such that the computer-generated environment is superimposed on a view of the physical environment that is visible through a transparent portion of the display. As shown in FIG. 7C, a user 7302 is standing (e.g., holding a device 7100 or wearing an HMD) in a physical environment (e.g., scene 105) in which a computer system (e.g., computer system 101) is operated. In some embodiments, as in the examples shown in FIGS. 7C-7F, the device 7100 is a handheld device (e.g., a mobile phone, tablet, or other mobile electronic device) that includes a display, a touch-sensitive display, or the like. In some embodiments, the device 7100 may be a head-up display or a head-mounted display. 7302) surrounds the user 7302. In some embodiments, the physical environment represents, and optionally replaces, a wearable headset including a headset display, etc. In some embodiments, the physical environment includes one or more physical surfaces and physical objects surrounding the user 7302 (e.g., the walls of a room, (e.g., front wall 7304 and side walls 7306), floor 7308, and furniture 7310). In some embodiments, one or more physical surfaces of physical objects in the environment (e.g., front surface 8312 of furniture 7310) are visible through a display generation component of a computer system (e.g., on a display of device 7100 or via an HMD).

[0123] 7D-7F, a computer-generated three-dimensional environment that corresponds to a physical environment (e.g., a portion of the physical environment that is within the field of view of one or more cameras of device 7100 or visible through a transparent portion of a display of device 7100) is displayed on device 7100. The physical environment includes physical objects having representations that correspond to the computer-generated three-dimensional environment shown by a display generation component of the computer system. For example, in the computer-generated environment shown on the display, the front wall 7304 is represented by a front wall representation 7304', the side wall 7306 is represented by a side wall representation 7306', the floor 7308 is represented by a floor representation 7308', the furniture 7310 is represented by a furniture representation 7310', and the front surface 7312 of the furniture 7310 is represented by a front surface representation 7312' (e.g., the computer-generated environment is an augmented reality environment that includes representations 7304', 7306', 7308', 7310', and 7312' of physical objects that are part of a live view of one or more cameras of the device 7100 or are visible through a transparent portion of a display of the device 7100). In some embodiments, the computer-generated environment shown on the display also includes virtual objects. In some embodiments, as the device 7100's field of view of the physical environment changes (e.g., as the field of view angle of the device 7100 or one or more cameras of the device 7100 with respect to the physical environment changes in response to movement and / or rotation of the device 7100 within the physical environment), the field of view of the computer-generated environment displayed on the device 7100 changes accordingly (e.g., including a change in the field of view of physical surfaces and physical objects (e.g., walls, floors, furniture, etc.)).

[0124] 7E, a first virtual object (e.g., a virtual window 7332) is displayed at a first location (e.g., a location in the three dimensional environment corresponding to a location on a side wall 7306 in the physical environment) in response to, e.g., a user input that adds virtual content to the three dimensional environment. The first virtual object (e.g., virtual window 7332) has respective spatial relationships to representations of physical objects in the three dimensional environment (e.g., front wall representation 7304', furniture representation 7310', physical surface representation 7312', and floor representation 7308') that are determined by respective spatial relationships between the side wall 7306 to other physical objects (e.g., front wall 7304, furniture 7310, physical surface 7312, and floor 7308). 7E, a first virtual object (e.g., virtual window 7332) is displayed with a first appearance (e.g., having first brightness and / or color values ​​in first portions 7332-b and 7332-c and second brightness and / or color values ​​in second portions 7332-a and 7332-d of the first virtual object). In some embodiments, these internal variations in display characteristics within various portions of the first virtual object reflect content shown in the first virtual object, which may change due to external factors, predetermined conditions, or time.

[0125] As shown in FIG. 7E, the computer system generates an imitation lighting pattern on the representation of the physical object in the three-dimensional environment based on the virtual light emitted from various parts of the virtual object 7332. According to some embodiments, the imitation lighting pattern is generated according to the relative spatial positions of the virtual object and the representation of the physical object in the three-dimensional environment, as well as the physical properties of the virtual and physical objects (e.g., surface shape, texture, optical properties, etc.). As shown in FIG. 7E, the lighting pattern generated on the representation of the physical object observes the imitation physical light propagation principle. For example, the shape, brightness, color, hue, etc. of the lighting patterns (e.g., lighting patterns 7334, 7336, and 7340) on the representation of the physical object (e.g., representations 7304′, 7310′, 7312′, and 7308′) mimic the lighting pattern on the physical object (e.g., physical object / surfaces 7304, 7310, 7312, and 7308) that would be made by a real window with similar properties as the virtual window 7332 on the side wall 7306.

[0126] 7E, in some embodiments, the computer system generates the mimicked lighting pattern 7334 of the front wall 7304 by modifying the visual appearance (e.g., luminance and color values) of the first portions 7334-b and 7334-c of the front wall representation 7304' in the three-dimensional scene according to the luminance and color values ​​of the first portions 7332-b and 7332-c of the first virtual object 7332. Similarly, the computer system generates the mimicked lighting pattern 7336 of the physical surface 7312 by modifying the visual appearance (e.g., luminance and color values) of the first portions 7336-b and 7336-c of the physical surface representation 7312' in the three-dimensional scene according to the luminance and color values ​​of the first portions 7332-b and 7332-c of the first virtual object 7332. Similarly, the computer system generates an mimicked lighting pattern 7340 of the floor 7308 by modifying the visual appearance (e.g., brightness and color values) of first portions 7340-b and 7340-c of the floor representation 7308' of the three-dimensional scene according to the brightness and color values ​​of the first portions 7332-b and 7332-c of the first virtual object 7332.

[0127] As shown in FIG. 7E , the visual appearance of the first portion of the physical surface and the visual appearance of the second portion of the physical surface are modified differently according to, for example, the mimicked spatial relationships between the first virtual object and the various physical surfaces, the real and mimicked physical properties of the virtual object and the various physical surfaces, and differences in brightness and color values ​​in various portions of the first virtual object.

[0128] As shown in FIG. 7E, in addition to adding mimicked lighting patterns 7334, 7336, and 7340 at locations in the three-dimensional environment corresponding to the locations of physical surfaces in the physical environment (e.g., front wall 7304, physical surface 7312 of furniture 7310, and floor 7308), the computer system also generates an mimicked shadow 7338 at a location in the three-dimensional environment (e.g., on floor representation 7338') that corresponds to the location of an actual shadow (e.g., on floor 7308) that would have been cast by furniture 7310 if illuminated by an actual light source in the same position and characteristics of virtual object 7332 (e.g., an actual window on side wall 7306).

[0129] FIG. 7F shows that dynamic changes of various parts of a virtual object affect the representation of various parts of a physical environment differently, as compared to FIG. 7E. For example, the size and internal content of a first virtual object have changed in FIG. 7E from those shown in FIG. 7F. Here, the first virtual object is represented as virtual object 7332'. The first parts of the first virtual objects 7332-b and 7332-c in FIG. 7E become first parts 7332-b' and 7332-c' in FIG. 7F, respectively. The second parts 7332-a and 7332-d in FIG. 7E become second parts 7332-a' and 7332-d' in FIG. 7F, respectively. The center positions of the first parts 7332-b' and 7332-c' in FIG. 7F and the second parts 7332-a' and 7332-d' in FIG. 7F, respectively, have also shifted relative to the center positions shown in FIG. 7E. As a result, for many positions on the sidewall representation 7306', the brightness and color values ​​of the corresponding positions on the first virtual object 7332 have changed (e.g., from the values ​​shown in FIG. 7E to the values ​​shown in FIG. 7F). Similarly, for many positions on the lighting patterns 7334, 7336, and 7340 cast on the representations 7304', 7312', and 7308', the brightness and color values ​​of the lighting patterns have also changed (e.g., from the values ​​shown in FIG. 7E to the values ​​shown in FIG. 7F). For example, for a first position on the sidewall representation 7306', the brightness and color values ​​of the corresponding positions on the first virtual object (e.g., a virtual window, or a virtual movie screen) may switch from 1 to 0.5 and from yellow to blue, respectively, and for a second position on the sidewall representation 7306', the brightness and color values ​​of the corresponding positions on the first virtual object may switch from 0.5 to 1 and from blue to yellow, respectively. In some embodiments, a change in size of the first virtual object or a movement of the first virtual object causes the brightness and color corresponding to some locations on the side wall representation 7306' to change because the first virtual object has expanded or moved to those locations, while the brightness and color corresponding to some other locations on the side wall representation 7306' to change because the first virtual object has moved or shrunk from those locations.Furthermore, in some embodiments, the direction of light coming from different parts of the first virtual object also optionally changes (e.g., the light direction changes according to the time of day or according to the view shown in the virtual window). As a result, changes in brightness and color at different locations on the first virtual object cause different changes in lighting at different locations on the representation of the nearby physical surface. Various relationships are used to modify the appearance of the representation of the nearby physical surface based on the appearance of the first virtual object.

[0130] As shown in FIG. 7F, the first portion 7332-b' provides illumination 7334-b' on the front wall representation 7304' but does not provide any illumination on the front surface representation 7312', and the illumination 7336-a' provided by the second portion 7332-a' covers the area previously covered by the illumination 7334-b provided by the first portion 7332-b (FIG. 7E). Similarly, the second portion 7332-d' provides illumination 7334-d' on the front wall representation 7304' but does not provide any illumination on the front surface representation 7312', and the illumination 7336-c' provided by the first portion 7332-c' covers the area previously covered by the illumination 7334-d provided by the second portion 7332-d (FIG. 7E). Similarly, in the front wall representation 7304', some parts that were previously covered by the lighting 7334-c provided by the first portion 7332-c and the lighting 7334-a provided by the second portion 7332-a are no longer covered by any lighting. Similarly, in the floor representation 7308', some parts that were previously covered by the lighting 7334-c provided by the first portion 7332-c and the lighting 7334-a provided by the second portion 7332-a are no longer covered by any lighting, since the first virtual object has contracted. Now, some positions on the floor representation 7308' that were previously covered by higher lighting are now covered by lower lighting, and now other positions on the floor representation 7308' that were previously covered by lower lighting are now covered by higher lighting. In FIG. 7F, the shadow 7338 cast on the floor representation 7308′ also appears darker than the shadow 7308 in FIG. 7E due to the reduced amount of illumination caused by the reduced size of the first virtual object 7332.

[0131] In some embodiments, the first virtual object is a virtual window showing a virtual landscape. Light emitted from the virtual window is based on the virtual landscape shown in the virtual window. In some embodiments, the virtual window casts a lighting pattern onto the representation of the nearby physical surface of the three-dimensional environment to mimic how light from a real window illuminates the nearby physical surface (e.g., based on the spatial relationship between the window and the physical surface, the physical properties of the physical surface, and physical light propagation principles). In some embodiments, the virtual landscape displayed in the virtual window changes based on parameters such as the time of day, the location of the landscape, and the size of the virtual window.

[0132] In some embodiments, the first virtual object is a virtual screen or hologram showing a moving image. As the moving image playback progresses, the virtual light emitted from the virtual screen or hologram changes as the scene in the moving image changes. In some embodiments, the virtual screen or hologram casts a lighting pattern onto a representation of a nearby physical surface of the three-dimensional environment to mimic how light from a real moving image screen or hologram illuminates the nearby physical surface (e.g., based on the spatial relationship between the screen or hologram and the physical surface, the physical properties of the physical surface, and physical light propagation principles).

[0133] In some embodiments, the first virtual object is a virtual assistant, and the light emitted from the virtual assistant changes during different interaction modes between the user and the virtual assistant. For example, the visual representation of the virtual assistant has a first color and intensity when first activated by the user, changes to a different color when asking or responding to a question, and changes to a different color when performing a task or waiting for the completion of a task or an answer from the user. In some embodiments, the virtual assistant casts a lighting pattern on the representation of nearby physical surfaces in the three-dimensional environment to mimic how light from an actual light source illuminates the nearby physical surfaces (e.g., based on the spatial relationship between the light source and the physical surface, the physical properties of the physical surface, and physical light propagation principles). Additional aspects of how the visual representation of the virtual assistant affects the appearance of nearby physical and virtual objects in the three-dimensional environment are described with reference to Figures 7S-7X and 12.

[0134] In some embodiments, the computer system also generates virtual reflections and virtual shadows on the representation of the physical surface based on light emitted from virtual objects near the physical surface.

[0135] 7G-7L are block diagrams illustrating methods of generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting the immersiveness of the computer-generated experience based on user input) according to some embodiments. Figures 7G-7L are used to illustrate the processes described below, including the process of Figure 10.

[0136] FIG. 7G illustrates an exemplary computer-generated environment that corresponds to a physical environment. As described herein with reference to FIG. 7G, the computer-generated environment may be an augmented reality environment or a computer-generated environment displayed on a display, where the computer-generated environment is superimposed on a view of the physical environment that is visible through a transparent portion of the display. As shown in FIG. 7G, a user 7302 is present (e.g., holding a device 7100 or wearing an HMD) in a physical environment (e.g., scene 105) in which a computer system (e.g., computer system 101) is operated. In some embodiments, such as the example shown in FIG. 7G, the device 7100 is a handheld device (e.g., a mobile phone, tablet, or other mobile electronic device) that includes a display, a touch-sensitive display, or the like. In some embodiments, the device 7100 represents, and is optionally replaced by, a wearable headset that includes a head-up display, a head-mounted display, or the like. In some embodiments, the physical environment includes one or more physical surfaces and physical objects surrounding the user (e.g., the walls of a room (represented by a front wall representation 7304', a side wall representation 7306'), a floor (e.g., represented by a floor representation 7308'), furniture (e.g., represented by a furniture representation 7310), and physical surfaces of the furniture 7312 (e.g., represented by a physical surface representation 7312')).

[0137] 7G-7L, a computer-generated three-dimensional environment corresponding to a physical environment (e.g., a portion of the physical environment that is within the field of view of one or more cameras of device 7100 or visible through a transparent portion of a display of device 7100) is displayed on device 7100. The computer-generated environment shown on device 7100 is a three-dimensional environment, and according to some embodiments, as the perspective of device 7100 with respect to the physical environment changes (e.g., as the viewing angle of device 7100 or one or more cameras of device 7100 with respect to the physical environment changes in response to movement and / or rotation of device 7100 within the physical environment), the perspective of the computer-generated environment displayed on device 7100 changes accordingly (e.g., including a change in the perspective of physical surfaces and physical objects (e.g., walls, floors, furniture, etc.)).

[0138] As shown in FIG. 7G, initially, the three-dimensional environment is shown with a first set of physical elements, including representations of a front wall 7304, a side wall 7306, a floor 7308, and furniture 7310. Optionally, the three-dimensional environment may include a first amount of virtual elements. For example, when the three-dimensional environment is first displayed, or when the display generating component of the computer system is first turned on or worn on the user's head or in front of the user's eyes, no virtual elements, or only a minimal amount of virtual elements, are displayed in the three-dimensional environment. This allows the user to start with a view of the three-dimensional environment that is very similar to a direct view of the real world, without the display generating component blocking the user's eyes.

[0139] 7G and 7H, the computer system detects a first predefined gesture input (e.g., a thumb flick gesture or swipe gesture performed by hand 7200, an upward wave gesture in the air, a swipe gesture on a controller, etc., as represented by representation 7200′ on the display generating component) that increases the immersion of the three-dimensional environment. In response to detecting the first predefined gesture, the computer system displays a virtual element 7402 (e.g., a virtual landscape or a virtual window) that blocks a view of a front wall 7304 in the three-dimensional environment (e.g., the virtual element 7402 replaces the display of representation 7304′ of the front wall 7304 on the display, or the virtual element 7402 is displayed in a position that blocks a view of the front wall 7304 through a previously transparent portion of the display (the portion that now displays the virtual element 7402)). In some embodiments, as shown in FIG. 7H, a view of the furniture 7310 in front of the front wall 7304 is not affected even though the view of the front wall 7304 is occluded by the display of the virtual element 7402. In other words, the first predefined gesture causes a first class of physical object or surface (e.g., the front wall) to be replaced or occluded by a newly displayed virtual element or a newly displayed portion of an existing virtual element. In some embodiments, an animated transition is displayed to show the virtual element 7402 gradually expanding (e.g., as shown in FIG. 7H) or becoming more opaque and saturated to cover or occlude the view of the front wall 7304 (e.g., replacing the representation 7304′ in the three-dimensional environment).

[0140] In some embodiments, in response to the first predefined gesture, the computer system also adds another virtual element (e.g., virtual object 7404) to the three-dimensional environment, optionally without replacing an entire class of physical elements. The virtual object 7404 is optionally a user interface object such as a menu (e.g., a menu of an application, a document, etc.), a control (e.g., a display brightness control, a display focus control, etc.), or other object that can be manipulated by user input or that provides information or feedback to the three-dimensional environment (e.g., a virtual assistant, a document, a media item, etc.). In some embodiments, as shown in FIG. 7I, the virtual object 7404 is added to the three-dimensional environment (e.g., blocking a portion of the floor 7308 or replacing a portion of the floor representation 7308′) without gaining input focus and / or without being specifically inserted into the three-dimensional environment (e.g., dragged from a menu or drawn with a drawing tool). In some embodiments, the computer system allows the user to introduce each virtual element into the three-dimensional environment individually (e.g., adding new furniture, throwing virtual confetti into the room, etc.) using the user interface currently provided in the three-dimensional environment, but this type of input does not change the immersiveness of the three-dimensional environment or replace the view of an entire class of physical elements with a single action.

[0141] Figure 7I, which follows Figure 7H, shows that the front wall 7304 has been completely occluded or replaced by the virtual element 7402. A view of the furniture 7310 in front of the front wall 7304 is still shown in the three-dimensional environment. The virtual element 7404 occludes a portion of the floor representation 7308'. A representation 7306' of the side wall 7306 and a representation 7308' of the floor 7308 are visible in the three-dimensional environment after the virtual elements 7402 and 7404 have been added to the three-dimensional environment in response to the first predefined gesture input.

[0142] As shown in Figures 7I and 7J, the computer system detects a first predefined gesture input (e.g., as shown in Figure 7G) and then detects a second predefined gesture input (e.g., a thumb flick gesture or a swipe gesture performed by hand 7200 represented by representation 7200' on the display generating component, a swipe gesture on a controller, etc.) to enhance the immersion of the three-dimensional environment. In response to detecting the second predefined gesture, the computer system maintains the display of a virtual element 7402 (e.g., a virtual landscape, or a virtual window) that blocks a view of a front wall 7304 in the three-dimensional environment and displays a virtual element 7406. The virtual element 7406 occludes the view of the side wall 7306 in the three-dimensional environment (e.g., the virtual element 7406 replaces the display of a representation 7306' of the side wall 7306 on the display, or the virtual element 7406 is displayed in a position that blocks the view of the side wall 7306 through a previously transparent portion of the display (e.g., the portion now displaying the virtual element 7406). In FIGS. 7I-7J, a second predefined gesture causes an additional class of physical object or surface (e.g., the side wall) to be replaced or blocked by a newly displayed virtual element or a newly displayed portion of an existing virtual element. In some embodiments, an animated transition is displayed such that the virtual element 7406 gradually expands or becomes more opaque, covering or blocking the view of the side wall 7306 (e.g., replacing the representation 7306' in the three-dimensional environment).

[0143] Figure 7K, which follows Figure 7J, shows that the front wall 7304 and side wall 7306 have been completely occluded or replaced by virtual elements 7402 and 7406. A view of the furniture 7310 in front of the front wall 7304 is still shown in the three-dimensional environment. The virtual element 7404 occludes a portion of the floor representation 7308'. After the virtual elements 7402, 7404, and 7406 have been added to the three-dimensional environment in response to the first and second predetermined gesture inputs, the representation 7308' of the floor 7308 is still visible in the three-dimensional environment.

[0144] As shown in Figures 7K and 7L, after detecting the first and second predefined gesture inputs (e.g., as shown in Figures 7G and 7I), the computer system detects a third predefined gesture input (e.g., a thumb flick gesture or a swipe gesture performed by hand 7200 represented by representation 7200' on the display generating component, a swipe gesture on a controller, etc.) to enhance the immersion of the three-dimensional environment. In response to detecting the third predefined gesture input, the computer system maintains the display of virtual elements 7402 and 7406 (e.g., a virtual landscape or a virtual window) that block views of front wall 7304 and side wall 7306 in the three-dimensional environment, and displays virtual elements 7408 and 7410. The virtual element 7408 occludes the view of the floor 7308 in the three dimensional environment (e.g., the virtual element 7408 replaces the display of a representation 7308' of the floor 7308 on the display, or the virtual element 7408 is displayed in a position that blocks the view of the floor 7306 through a previously transparent portion of the display (e.g., the portion now displaying the virtual element 7408). In Figures 7K-7L, a third predefined gesture causes an additional class of physical object or surface (e.g., a floor) to be replaced or occluded by a newly displayed virtual element or a newly displayed portion of an existing virtual element. In some embodiments, an animated transition is displayed such that the virtual element 7408 gradually expands or becomes more opaque, covering or occluding the view of the floor 7308 (e.g., replacing the representation 7308' in the three dimensional environment).

[0145] In some embodiments, in response to the third predetermined gesture, the computer system also adds another virtual element (e.g., virtual element 7410) to the three-dimensional environment, optionally without replacing an entire class of physical elements. The virtual element 7410 is optionally a user interface object such as a menu (e.g., a menu of an application, a document, etc.), a control (e.g., a display brightness control, a display focus control, etc.), or other object that can be manipulated by user input or that provides information or feedback to the three-dimensional environment (e.g., a virtual assistant, a document, a media item, etc.), or a texture that changes the appearance of a physical object (e.g., a decorative feature, a photo, etc.). In some embodiments, as shown in FIG. 7L, the virtual object 7410 is added to the three-dimensional environment (e.g., overlaps a portion of the front surface 7312 of the furniture 7310 or replaces a portion of the physical surface representation 7312').

[0146] In some embodiments, after successive input gestures of a predefined gesture type to increase the immersion of the three-dimensional environment, additional amounts of virtual elements are optionally introduced into the three-dimensional environment to replace or block views of additional classes of physical elements that were previously visible in the three-dimensional environment, in some embodiments, the entirety of the three-dimensional environment is replaced with the virtual elements, and the view into the physical world is completely replaced by the view of the virtual elements within the three-dimensional environment.

[0147] In some embodiments, virtual elements 7402 and 7406 are virtual windows that are displayed in place of corresponding portions of the front wall representation 7304' and the side wall representation 7306', respectively. In some embodiments, light emitted from the virtual windows casts an emulated lighting pattern on physical surfaces (e.g., floors or furniture) that are still visible or represented in the three-dimensional environment. Additional details of the influence of light from virtual elements on surrounding physical surfaces, according to some embodiments, are described with reference to Figures 7C-7F and 9.

[0148] In some embodiments, the content or appearance of virtual elements 7402 and 7406 (e.g., a virtual window or virtual screen) changes in response to additional gesture input (e.g., a horizontal swipe of a hand in the air or a directional swipe around a finger). In some embodiments, the size of the virtual elements, the position of the virtual scenery displayed within the virtual elements, the media items displayed within the virtual elements, etc. change in response to the additional gesture input.

[0149] In some embodiments, the gesture input for increasing or decreasing the immersion of the three-dimensional environment is a vertical swipe gesture in opposite directions (e.g., up to increase the immersion / amount of virtual elements and down to decrease the immersion / amount of virtual elements). In some embodiments, the gesture for changing the content of the virtual elements is a horizontal swipe gesture (e.g., a horizontal swipe gesture switches the content displayed in the virtual elements backwards and / or forwards through multiple positions or times).

[0150] In some embodiments, the sequence of the first predefined gesture input, the second predefined gesture input, and the third predefined gesture input for increasing the immersion of the three-dimensional environment is optionally replaced by one successive input to change the immersion to multiple levels, each successive portion of the successive input corresponding to a respective one of the first, second, and third predefined gesture inputs shown in Figures 7G-7L according to some embodiments.

[0151] In some embodiments, the floor 7308 or floor representation 7308' remains visible at all times in the three-dimensional environment, even when other physical surfaces, such as walls, are replaced or overlaid by virtual elements. This helps ensure that the user feels safe and does not fall when navigating within the three-dimensional environment by walking around the physical world.

[0152] In some embodiments, some furniture or parts of furniture surfaces remain visible at all times even when other physical surfaces such as walls and floors are replaced or overlaid by virtual elements, helping to ensure that the user maintains a natural relationship with the environment when immersed in the three-dimensional environment.

[0153] According to this example, in Figures 7G, ​​7I, and 7K, a representation 7200' of a hand 7200 is displayed in the computer-generated environment. The computer-generated environment does not include a representation of the user's right hand (e.g., because the right hand is not within the field of view of one or more cameras of device 7100). Furthermore, in some embodiments, for example in the example shown in Figure 7I where device 7100 is a handheld device, the user can see parts of the surrounding physical environment separately from any representation of the physical environment displayed on device 7100. For example, parts of the user's hand are visible to the user outside of the display of device 7100. In some embodiments, device 7100 in these examples represents or can be replaced by a headset having a display (e.g., a head-mounted display) that completely blocks the user's view of the surrounding physical environment. In some such embodiments, no part of the physical environment is directly visible to the user, but instead the physical environment is visible to the user via a representation of parts of the physical environment displayed by the device. In some embodiments, the user's hand(s) are not visible to the user, either directly or via the display of device 7100, while the current state of the user's hands is continuously or periodically monitored by the device to determine whether the user's hand(s) have entered a state ready for providing a gesture input. In some embodiments, the device displays an indicator of whether the user's hands are in a state ready for providing an input gesture, provides feedback to the user, and alerts the user to adjust the position of their hands if they wish to provide an input gesture.

[0154] 7M-7R are block diagrams illustrating methods for facilitating user interaction with a computer-generated environment (e.g., using interactions with physical surfaces to control a device or interact with the computer-generated environment) according to some embodiments. Figures 7M-7R are used to illustrate the processes described below, including the process of FIG.

[0155] FIG. 7N illustrates an exemplary computer-generated environment that corresponds to the physical environment illustrated in FIG. 7M. As described herein with reference to FIGS. 7M-7R, according to some embodiments, the computer-generated environment may optionally be an augmented reality environment including a camera view of the physical environment, or a computer-generated environment displayed on a display such that the computer-generated environment is superimposed on a view of the physical environment that is visible through a transparent portion of the display. As illustrated in FIG. 7M, a user 7302 is standing (e.g., holding a device 7100 or wearing an HMD) in a physical environment (e.g., scene 105) in which a computer system (e.g., computer system 101) is operated. In some embodiments, as in the examples illustrated in FIGS. 7M-7R, the device 7100 is a handheld device (e.g., a mobile phone, tablet, or other mobile electronic device) that includes a display, a touch-sensitive display, or the like. In some embodiments, the device 7100 may be a head-up display, a head-mounted display, or the like. 7100) through a display generating component of the computer system (e.g., on a display of device 7100 or through an HMD).

[0156] 7M-7R, a computer-generated three-dimensional environment that corresponds to a physical environment (e.g., a portion of the physical environment that is within the field of view of one or more cameras of device 7100 or visible through a transparent portion of a display of device 7100) is displayed on device 7100. The physical environment includes physical objects having representations that correspond to the computer-generated three-dimensional environment shown by a display generation component of the computer system. For example, front wall 7304 is represented by a front wall representation 7304', side wall 7306 is represented by a side wall representation 7306', floor 7308 is represented by a floor representation 7308', and boxes 7502 and 7504 are represented by box representations 7502' and 7504' in a computer-generated environment shown on a display (e.g., the computer-generated environment is an augmented reality environment that includes representations 7304', 7306', 7308', 7502', and 7504' of physical objects as part of a live view of one or more cameras of device 7100 or physical objects visible through transparent portions of a display of device 7100). In some embodiments, the computer-generated environment shown on a display also includes virtual objects. In some embodiments, as the device 7100's field of view of the physical environment changes (e.g., as the field of view angle of the device 7100 or one or more cameras of the device 7100 with respect to the physical environment changes in response to movement and / or rotation of the device 7100 within the physical environment), the field of view of the computer-generated environment displayed on the device 7100 changes accordingly (e.g., including a change in the field of view of physical surfaces and physical objects (e.g., walls, floors, furniture, etc.)).

[0157] In some embodiments, when the level of interaction between the user 7302 and the three-dimensional environment falls below a first predetermined level (e.g., the user simply views the three-dimensional environment without focusing on a specific location within the three-dimensional environment), the computer system displays an initial state of the three-dimensional environment, as shown in FIG. 7N, in which representations 7502' and 7504 of boxes 7502 and 7504 are not displayed along with any corresponding user interface or virtual objects.

[0158] In Figures 7O and 7P, the computer system detects that the level of interaction between the user and the three-dimensional environment has risen above a first predetermined level. In particular, in Figure 7O, gaze input is detected on the representation 7502' of the box 7502 (e.g., a speaker or tabletop) without a simultaneous gesture input or an indication that gesture input is about to be provided (e.g., the user's hands are not in a ready state to provide gesture input). In response to detecting the gaze input on the representation 7502' of the box 7502 in the three-dimensional environment, the computer system determines that the level of interaction between the user and the box 7502 or representation 7502' has reached a first predetermined level (but has not reached a second predetermined level that exceeds the first predetermined level). In response to determining that the level of interaction with the box 7502 or representation 7502' has reached the first predetermined level, the computer system displays a first user interface 7510 corresponding to the box 7502 at a location in the three-dimensional environment that corresponds to the location of the box 7502 in the physical environment. For example, as shown in FIG. 7O, multiple user interface objects (e.g., user interface objects 7506 and 7508) are displayed to appear to overlay or replace portions of representation 7502' on top of box 7502. In some embodiments, box 7502 is a table and user interface objects 7506 and 7508 include one or more of a virtual newspaper, a virtual screen, notifications from an application or communication channel, a keyboard and display, a sketch pad, etc. In some embodiments, box 7502 is a speaker and user interface objects 7506 and 7508 include a volume indicator, a play / pause control, the name of the currently playing song / album, today's weather, etc. In some embodiments, box 7502 is a smart lamp or appliance and user interface objects 7506 and 7508 include one or more of a brightness or temperature control, a start / stop or on / off button, a timer, etc.

[0159] 7P, the gaze input shifts from a representation 7502' of box 7502 (e.g., a tabletop, speaker, smart lamp, or appliance) to a representation 7504' of box 7504 (e.g., a smart medicine cabinet) without a simultaneous gesture input or an indication that gesture input is about to be provided (e.g., the user's hands are not ready to provide gesture input). In response to detecting that the gaze input shifts from the representation 7502' of box 7502 to the representation 7504' of box 7504 within the three-dimensional environment, the computer system determines that a level of interaction between the user and box 7504 or representation 7504' has reached a first predetermined level (but has not reached a second predetermined level that is above the first predetermined level) or that a level of interaction between the user and box 7502 or representation 7502' has dropped below the first predetermined level. In accordance with determining that the level of interaction between the user and box 7502 or representation 7502' has fallen below the first predetermined level, the computer system stops displaying first user interface 7510 corresponding to box 7502. In response to determining that the level of interaction with box 7504 or representation 7504' has reached the first predetermined level, the computer system displays first user interface 7512 corresponding to box 7504 at a location in the three-dimensional environment that corresponds to the location of box 7504 in the physical environment. For example, as shown in FIG. 7P, multiple user interface objects (e.g., user interface objects 7514 and 7516) are displayed to overlay in front of box 7504 or replace a portion of representation 7504'. In some embodiments, box 7504 is a smart medicine shelf and multiple user interface objects (e.g., user interface objects 7514 and 7516) include one or more of the medicine shelf's statuses (e.g., an indicator that a particular medicine or supply is low and needs to be replenished, or a reminder whether today's date's medicine has been taken).

[0160] In Figures 7Q and 7R, the computer system detects that the level of interaction between the user and the three-dimensional environment has risen above a second predetermined level that exceeds the first predetermined level. In particular, in Figure 7Q, in addition to detecting gaze input on the representation 7502' of the box 7502 (e.g., a speaker, or a tabletop), the computer system also detects an indication that a gesture input is about to be provided (e.g., the user's hand is found in a state ready to provide a gesture input). In response to determining that the level of interaction between the user and the box 7502 or the representation 7502' has reached the second predetermined level, the computer system displays a second user interface 7510' that is optionally an enhanced version of the first user interface 7510 corresponding to the box 7502. The second user interface 7510' corresponding to the box 7502 is displayed at a location in the three-dimensional environment that corresponds to the location of the box 7502 in the physical environment. 7Q, a number of user interface objects (e.g., user interface objects 7506, 7518, 7520, 7522, and 7524) are displayed such that they appear to overlay or replace portions of representation 7502' on top of box 7502. In some embodiments, box 7502 is a table, and user interface objects 7506, 7518, 7520, 7522, and 7524 include one or more of the user interface objects shown in first user interface 7510, as well as one or more other user interface objects not included in first user interface 7510 (e.g., an extended display, a keyboard with additional keys not available in first user interface 7510, a virtual desktop with application icons and a document list, etc.).In some embodiments, box 7502 is a speaker and user interface objects 7506, 7518, 7520, 7522, and 7524 include one or more of the user interface objects shown in the first user interface 7510 and one or more of other user interface objects not included in the first user interface 7510 (e.g., output routing controls, a browseable media database, a search input field with a corresponding virtual keyboard, etc.) In some embodiments, box 7502 is a smart lamp or appliance and user interface objects 7506, 7518, 7520, 7522, and 7524 include one or more of the user interface objects shown in the first user interface 7510 and one or more of other user interface objects not included in the first user interface 7510 (e.g., various settings of the smart lamp or appliance, color controls, scheduling controls, etc.).

[0161] In some embodiments, Figure 7Q follows Figure 7O, where the second user interface 7510' is displayed in response to the user's hand becoming ready while the user's gaze is focused on box 7502. In some embodiments, Figure 7Q follows Figure 7P, where the user's hand becomes ready and the user interface is displayed in response to the user's gaze moving from box 7504 to box 7502 (e.g., the display of the first user interface 7512 is stopped after the gaze input moves away from box 7504).

[0162] In FIG. 7R, the gaze input is shifted from a representation 7502′ of a box 7502 (e.g., a tabletop, a speaker, a smart lamp or an appliance) to a representation 7504′ of a box 7504 (e.g., a smart medicine cabinet) while the user's hand is in a ready state to provide a gesture input. In response to detecting that the gaze input has shifted from the representation 7502′ of the box 7502 to the representation 7504′ of the box 7504 in the three-dimensional environment, the computer system determines that a level of interaction between the user and the box 7504 or the representation 7504′ has reached a second predetermined level and determines that a level of interaction between the user and the box 7502 or the representation 7502′ has dropped below the second predetermined level and below the first predetermined level. In accordance with determining that a level of interaction between the user and the box 7502 or the representation 7502′ has dropped below the first predetermined level, the computer system stops displaying the second user interface 7510′ corresponding to the box 7502. In response to determining that the level of interaction with the box 7504 or the representation 7504' has reached a second predetermined level, the computer system displays a second user interface 7512' corresponding to the box 7504 at a location in the three-dimensional environment corresponding to the location of the box 7504 in the physical environment. For example, as shown in FIG. 7R, a plurality of user interface objects (e.g., user interface objects 7514, 7516, 7526, 7528, and 7530) are displayed to appear to overlay or replace a portion of the representation 7504' in front of the box 7504. In some embodiments, the box 7504 is a smart medicine cabinet, and the plurality of user interface objects (e.g., user interface objects 7514 and 7516) include one or more of the user interface objects shown in the first user interface 7510 and one or more other user interface objects not included in the first user interface 7510, such as a list of medications or supplies in the medicine cabinet, scheduling settings for today's medications, medicine cabinet temperature and authentication settings, etc.

[0163] In some embodiments, FIG. 7R follows FIG. 7Q, where the user's hand remains in the ready state and the user interface 7512 is displayed in response to the user's gaze moving from box 7502 to box 7504 (e.g., the display of the second user interface 7512′ is stopped after the gaze input is moved away from box 7502). In some embodiments, FIG. 7R follows FIG. 7P, where the second user interface 7512′ is displayed in response to the user's hand becoming in the ready state while the user's gaze is focused on box 7504. In some embodiments, FIG. 7R follows FIG. 7O, where the user's hand becomes in the ready state and the user interface 7512′ is displayed in response to the user's gaze moving from box 7502 to box 7504 (e.g., the display of the first user interface 7510 is stopped after the gaze input is moved away from box 7502).

[0164] In some embodiments, when the computer system detects that the user's hand is hovering above the physical object (e.g., box 7502 or 7504) (e.g., the distance between the user's finger and the physical object is within a threshold distance), the computer system determines that a third level of interaction has been reached and displays a third user interface corresponding to the physical object (e.g., box 7502 or 7504) with more information and / or user interface objects than the second user interface corresponding to the physical object. In some embodiments, in response to the user's hand moving away from the physical object (e.g., the distance between the user's finger and the physical object increases beyond a threshold distance), the third user interface shrinks and returns to the second user interface corresponding to the physical object.

[0165] In some embodiments, the computer system performs an action in response to touch input provided on a physical surface that is on a physical object (e.g., box 7502 or 7504). For example, the touch input is optionally detected by a sensor, such as one or more cameras of the computer system, as opposed to a touch sensor on a physical surface on the physical object. In some embodiments, the location of the input on the physical surface is mapped to the location of a user interface object in a first / second / third user interface that corresponds to the physical object, such that the computer system can determine which action to perform according to the location of the touch input on the physical surface.

[0166] In some embodiments, the user uses a gaze in the first / second / third user interface to select a user interface object in the first / second / third user interface that corresponds to a physical object (e.g., box 7502 or 7504). The computer performs an action corresponding to the currently selected user interface object in response to a gesture input for activating the user interface object detected while the gaze input is over the currently selected user interface object.

[0167] In some embodiments, a user optionally utilizes nearby physical surfaces to control physical objects that are distant from the user. For example, the user can swipe on a nearby physical surface (e.g., the back or palm of the user's hand, the arm of an armchair, a controller, etc.) and the user's gesture input is detected by one or more sensors (e.g., one or more cameras of a computer system) and used to interact with the currently displayed first / second / third user interface.

[0168] According to this example, in Figures 7Q and 7R, a representation 7200' of hand 7200 is displayed in the computer-generated environment. The computer-generated environment does not include a representation of the user's right hand (e.g., because the right hand is not within the field of view of one or more cameras of device 7100). Furthermore, in some embodiments, for example in the examples shown in Figures 7Q and 7R where device 7100 is a handheld device, the user can see parts of the surrounding physical environment separately from any representation of the physical environment displayed on device 7100. For example, parts of the user's hand are visible to the user outside of the display of device 7100. In some embodiments, device 7100 in these examples represents or can be replaced by a headset having a display (e.g., a head-mounted display) that completely blocks the user's view of the surrounding physical environment. In some such embodiments, no part of the physical environment is directly visible to the user, but instead the physical environment is visible to the user via a representation of parts of the physical environment displayed by the device. In some embodiments, the user's hand(s) are not visible to the user, either directly or via the display of device 7100, while the current state of the user's hands is continuously or periodically monitored by the device to determine whether the user's hand(s) have entered a state ready for providing a gesture input. In some embodiments, the device displays an indicator of whether the user's hands are in a state ready for providing an input gesture, provides feedback to the user, and alerts the user to adjust the position of their hands if they wish to provide an input gesture.

[0169] 7S-7X are block diagrams illustrating a method of generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. 7S-7X are used to illustrate the processes described below, including the process of FIG.

[0170] Figures 7T-7X show example computer-generated environments that correspond to the physical environment shown in Figure 7S. As described herein with reference to Figures 7T-7X, according to some embodiments, the computer-generated environment may optionally be an augmented reality environment that includes a camera view of the physical environment, or a computer-generated environment that is displayed on a display such that the computer-generated environment is superimposed on a view of the physical environment that is visible through a transparent portion of the display. As shown in FIG. 7T, a user 7302 is standing (e.g., holding a device 7100 or wearing an HMD) in a physical environment (e.g., scene 105) in which a computer system (e.g., computer system 101) is operating. In some embodiments, such as the examples shown in FIGS. 7T-7X, the device 7100 is a handheld device (e.g., a mobile phone, tablet, or other mobile electronic device) including a display, a touch-sensitive display, or the like. In some embodiments, the device 7100 represents, and is optionally replaced by, a wearable headset including a head-up display, a head-mounted display, or the like. In some embodiments, the physical environment includes one or more physical surfaces and physical objects surrounding the user 7302 (e.g., walls of a room, (e.g., front wall 7304, side walls 7306), floor 7308, and furniture 7310). In some embodiments, one or more physical surfaces of physical objects in the environment are visible through a display generation component of the computer system (e.g., on the display of the device 7100 or via the HMD).

[0171] 7T-7X, a computer-generated three-dimensional environment that corresponds to a physical environment (e.g., a portion of the physical environment that is within the field of view of one or more cameras of device 7100 or visible through a transparent portion of a display of device 7100) is displayed on device 7100. The physical environment includes physical objects having representations that correspond to the computer-generated three-dimensional environment shown by a display generation component of the computer system. For example, in the computer-generated environment shown on the display, the front wall 7304 is represented by a front wall representation 7304', the side wall 7306 is represented by a side wall representation 7306', the floor 7308 is represented by a floor representation 7308', the furniture 7310 is represented by a furniture representation 7310', and the front surface 7312 of the furniture 7310 is represented by a front surface representation 7312' (e.g., the computer-generated environment is an augmented reality environment that includes representations 7304', 7306', 7308', 7310', and 7312' of physical objects that are part of a live view of one or more cameras of the device 7100 or are visible through a transparent portion of the display of the device 7100). In some embodiments, the computer-generated environment shown on the display also includes virtual objects (e.g., a virtual object 7404 that is stationary in a portion of the display that corresponds to a portion of the floor representation 7308' of the floor 7308). In some embodiments, as the device 7100's field of view of the physical environment changes (e.g., as the field of view angle of the device 7100 or one or more cameras of the device 7100 with respect to the physical environment changes in response to movement and / or rotation of the device 7100 within the physical environment), the field of view of the computer-generated environment displayed on the device 7100 changes accordingly (e.g., including a change in the field of view of physical surfaces and physical objects (e.g., walls, floors, furniture, etc.)).

[0172] In FIG. 7T, the computer system detects an input corresponding to a request to activate the voice-based virtual assistant. For example, the user provides the voice-based startup command "Assistant!" to the computer system. In some embodiments, the user optionally turns and looks at a predetermined location in the three-dimensional environment that corresponds to the home position of the voice-based virtual assistant, and / or provides an activation input (e.g., a tap input of the user's finger or controller, a gaze input, etc.).

[0173] In FIG. 7U and FIG. 7W, in response to detecting an input corresponding to a request to activate a voice-based virtual assistant in a three-dimensional environment, the computer system displays a visual representation of the virtual assistant in the three-dimensional environment. In some embodiments, the visual representation of the virtual assistant is a virtual object 7602. For example, the virtual object 7602 is an avatar of the virtual assistant (e.g., a light-emitting oval or animated character, etc.). In some embodiments, the visual indication is not necessarily an object with a virtual surface, but a visual effect such as lighting around the peripheral area of ​​the display, the peripheral area of ​​the user's field of view, or the peripheral area of ​​the target area of ​​the gaze input. In some embodiments, other visual effects (e.g., darkening or blurring the background of the virtual assistant or the entire display) are displayed in conjunction with the display of the visual indication of the virtual assistant.

[0174] As shown in FIG. 7U and FIG. 7W, the visual representation of the virtual assistant has a first set of values ​​of a first display characteristic (e.g., brightness, color) of the visual representation when the virtual assistant is activated. For example, the visual representation is a light-emitting ellipse having a first brightness value distribution and a first color value distribution over various parts of the visual representation. The computer system modifies the visual appearance of the first physical surface 7312 of the physical object 7310 in the three-dimensional environment or its representation 7312', and the visual appearance of the first virtual surface of the virtual object 7404 in the three-dimensional environment according to the first set of values ​​of the first display characteristic. For example, as shown in FIG. 7U and 7W, the computer system generates mimic lighting at a position in the three-dimensional environment that is mapped to the surface of the physical object 7310 in the physical world, and the value of the first display characteristic of the lighting takes into account the spatial relationship between the virtual object 7602 and the representation 7310' in the three-dimensional world, the surface characteristics of the physical object 7310, and the mimic physical light propagation principle. In Figures 7U and 7W, the front representation 7312' of the furniture representation 7310' appears to be illuminated by the virtual object 7602, and the left side of the front representation 7312' appears to be more strongly illuminated (e.g., with higher brightness and higher color saturation from the virtual object 7602) than the right side of the front representation 7312' because the virtual object 7602 is closer to the left side of the front representation 7312' in the three-dimensional environment. Similarly, as shown in Figures 7U and 7W, the computer system generates simulated lighting at a location in the three-dimensional environment that is mapped to a surface of the virtual object 7404 in the three-dimensional environment, and the value of the first display characteristic of the lighting takes into account the spatial relationship between the virtual object 7602 and the virtual object 7404 in the three-dimensional world, the surface characteristics of the virtual object 7404, and the simulated physical light propagation principles.In Figures 7U and 7W, the top surface of virtual object 7404 appears to be illuminated by the visual representation of a virtual assistant (e.g., virtual object 7602), which is closer to the top of the virtual object 7404 than the middle of the surface of the virtual object 7404 in the three-dimensional environment, so that the middle area of ​​the surface of virtual object 7404 appears to be less brightly lit (e.g., with lower brightness and lower color saturation from virtual object 7602) than the top of the virtual object 7404.

[0175] In some embodiments, as shown in Figures 7U and 7W, the computer system also generates simulated shadows for the physical and virtual objects under the illumination of the visual representation of the virtual assistant. For example, the computer generates a shadow 7606 behind the furniture representation 7310' in the three-dimensional environment based on the spatial relationship between the virtual object 7602 and the furniture representation 7310' in the three-dimensional environment, the surface characteristics of the furniture 7310, and the simulated physical light propagation principle. The computer also generates a shadow 7604 under the virtual object 7404 in the three-dimensional world based on the spatial relationship between the virtual object 7602 and the virtual object 7404 in the three-dimensional environment, the simulated surface characteristics of the virtual object 7404, and the simulated physical light propagation principle.

[0176] As shown in FIG. 7V, which follows FIG. 7U, in some embodiments, the position of the visual indication of the virtual assistant (e.g., virtual object 7602) is fixed relative to the display (e.g., touch-sensitive display) or the user's head (e.g., represented by an HMD) and moves relative to the three-dimensional environment according to the movement of the display relative to the physical world, or according to the movement of the user's head (or HMD) relative to the physical world. In FIG. 7V, as the user's head (e.g., the three-dimensional environment is shown via an HMD) or the display (e.g., a touch-sensitive display) moves within the physical environment, the mimicked lighting on the physical and virtual objects in the three-dimensional environment is adjusted because the spatial relationship between the visual representation of the virtual assistant (e.g., virtual object 7602) in the three-dimensional environment and the representations of the physical objects (e.g., furniture representation 7310') and virtual objects (e.g., virtual object 7404) has changed in response to the movement. For example, the front representation 7312' of the furniture representation 7310' is strongly lit (with higher brightness and color saturation from the visual representation of the virtual assistant (e.g., virtual object 7602)) because the visual representation (e.g., virtual object 7602) is now closer to the front representation 7312' than before the movement. Correspondingly, the top surface of the virtual object 7404 is weakly lit (e.g., with lower brightness and color saturation from the visual representation of the virtual assistant (e.g., virtual object 7602)) because the visual representation (e.g., virtual object 7602) is now farther away from the virtual object 7404 than before the movement.

[0177] In contrast to the example shown in FIG. 7V, in some embodiments, the position of the visual indication of the virtual assistant (e.g., virtual object 7602) is fixed relative to the three-dimensional environment, rather than to the display (e.g., touch-sensitive display) or the user's head (e.g., represented by an HMD). Thus, the spatial relationship between the visual representation of the virtual assistant (e.g., visual representation 7602) and the physical objects (e.g., furniture representation 7310') and virtual objects (e.g., virtual object 7404) represented in the three-dimensional environment does not change when the display moves relative to the physical world or when the user moves his or her head (or HMD) relative to the physical world. In FIG. 7V, when the user's head (e.g., the three-dimensional environment is shown via an HMD) or the display (e.g., a touch-sensitive display) moves in the physical environment, the mimetic lighting on the physical objects and virtual objects in the three-dimensional environment does not change, because the spatial relationship between the visual representation of the virtual assistant in the three-dimensional environment and the representations of the physical objects and virtual objects does not change in response to the movement. However, the view of the three-dimensional world shown on the display is altered due to the movement.

[0178] In some embodiments, various examples and embodiments described herein optionally include discrete, small movement gestures that are performed by moving a user's finger(s) relative to other finger(s) or portion(s) of the user's hand, without requiring the user's entire hand or arm to move significantly away from their natural position(s) and posture(s) to perform an action immediately before or during the gesture, according to some embodiments, to interact with a virtual or mixed reality environment.

[0179] In some embodiments, the input gesture is detected by analyzing data and signals captured by a sensor system (e.g., sensor 190 of FIG. 1, image sensor 314 of FIG. 3). In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as a motion RGB camera, an infrared camera, a depth camera, etc.). For example, the one or more imaging sensors are components of or provide data to a computer system (e.g., computer system 101 of FIG. 1 (e.g., portable electronic device 7100 or HMD)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4 (e.g., a touch screen display that functions as a display and a touch-sensitive surface, a stereoscopic display, a display with a pass-through portion, etc.)). In some embodiments, the one or more imaging sensors include one or more rear-facing cameras on a side of the device opposite the display of the device. In some embodiments, the input gesture is detected by a sensor system of a head-mounted system (e.g., a VR headset that includes a stereoscopic display that provides a left image for the user's left eye and a right image for the user's right eye). For example, one or more cameras that are components of the head-mounted system are mounted on the front and / or underside of the head-mounted system. In some embodiments, one or more imaging sensors are positioned in the space in which the head-mounted system is used (e.g., arrayed around the head-mounted system at various positions in a room) such that the imaging sensors capture images of the head-mounted system and / or a user of the head-mounted system. In some embodiments, the input gesture is detected by a sensor system of a head-up device (e.g., a head-up display, an automobile windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics). For example, the one or more imaging sensors are mounted on an interior surface of an automobile. In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array).For example, the one or more depth sensors include one or more light-based (e.g., infrared) sensors and / or one or more acoustic-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal emitters, such as light emitters (e.g., infrared emitters) and / or sound emitters (e.g., ultrasonic emitters). For example, while light (e.g., light from an infrared light emitter array having a predetermined pattern) is projected onto a hand (e.g., hand 7200), an image of the hand under the illumination of the light is captured by one or more cameras, and the captured image is analyzed to determine the position and / or configuration of the hand. By determining input gestures using signals from an image sensor directed at the hand, as opposed to using signals of a touch-sensitive surface or other direct contact or proximity-based mechanism, a user can freely choose to perform large movements or remain relatively still when providing input gestures with their hand without experiencing constraints imposed by a particular input device or input area.

[0180] In some embodiments, a micro-tap input indicates a thumb tap input on the index finger of a user's hand (e.g., on the side of the index finger adjacent to the thumb). In some embodiments, the tap input is detected without having to lift the thumb from the side of the index finger. In some embodiments, the tap input is detected according to a determination that a downward movement of the thumb is followed by an upward movement of the thumb and the thumb is in contact with the side of the index finger for less than a threshold time. In some embodiments, a tap-hold input is detected according to a determination that the thumb moves from an up position to a touch-down position and remains in the touch-down position for at least a first threshold time (e.g., a tap-time threshold or another time threshold longer than the tap-time threshold). In some embodiments, the computer system requires that the entire hand remain substantially stationary in a position for at least a first threshold time to detect a thumb-on-index tap-hold input. In some embodiments, the touch-hold input is detected without requiring the hand to remain substantially stationary (e.g., the entire hand can move while the thumb is resting on the side of the index finger). In some embodiments, a taphole drag input is detected when the thumb touches the side of the index finger and the entire hand moves while the thumb remains stationary on the side of the index finger.

[0181] In some embodiments, a micro-flick gesture indicates a push or flick input with the thumb moving across the index finger (e.g., from the palm side to the back side of the index finger). In some embodiments, the extension movement of the thumb is accompanied by an upward movement away from the side of the index finger, e.g., an upward flick input with the thumb. In some embodiments, the index finger moves in the opposite direction to the thumb while the thumb moves forward and upward. In some embodiments, a reverse flick input is performed by the thumb moving from an extended position to a retracted position. In some embodiments, the index finger moves in the opposite direction to the thumb while the thumb moves backward and downward.

[0182] In some embodiments, the microswipe gesture is a swipe input by movement of the thumb along the index finger (e.g., along the side of the index finger adjacent to the thumb or along the side of the palm). In some embodiments, the index finger is optionally in an extended state (e.g., substantially straight) or a bent state. In some embodiments, the index finger moves between an extended state and a bent state during the movement of the thumb in the swipe input gesture.

[0183] In some embodiments, different phalanges of various fingers correspond to different inputs. A microtap input of the thumb across various phalanges of various fingers (e.g., index, middle, ring, and optionally pinky) is optionally mapped to different actions. Similarly, in some embodiments, different push or click inputs can be performed by the thumb across different fingers and / or different parts of the fingers to trigger different actions at each user interface contact. Similarly, in some embodiments, different swipe inputs performed by the thumb along different fingers and / or in different directions (e.g., towards the distal or proximal ends of the fingers) trigger different actions in each user interface context.

[0184] In some embodiments, the computer system processes tap, flick, and swipe inputs as different types of inputs based on the type of thumb movement. In some embodiments, the computer system processes inputs with different finger positions tapped, touched, or swiped by the thumb as different sub-input types (e.g., proximal, intermediate, distal sub-types, or index, middle, ring, or pinky sub-types) of a given input type (e.g., tap input type, flick input type, swipe input type, etc.). In some embodiments, the amount of movement performed by the moving finger (e.g., thumb) and / or other movement measures associated with the finger movement (e.g., velocity, initial velocity, ending velocity, duration, direction, movement pattern, etc.) are used to quantitatively affect the action triggered by the finger input.

[0185] In some embodiments, the computer system recognizes combination input types that combine a series of movements by the thumb, such as a tap-swipe input (e.g., the thumb touches down on another finger, then swipes along the side of the finger), a tap-flick input (e.g., the thumb touches down on another finger, then flicks across the finger from the side of the palm to the back of the finger), and a double-tap input (e.g., two consecutive taps on the side of a finger in approximately the same location).

[0186] In some embodiments, the gesture input is performed by the index finger instead of the thumb (e.g., the index finger performs a tap or swipe on the thumb, or the thumb and index finger move towards each other to perform a pinch gesture). In some embodiments, a wrist movement (e.g., a wrist flick in a horizontal or vertical direction) is performed immediately before, immediately after (e.g., within a threshold time), or simultaneously with the finger movement input to trigger an additional, different, or modified action in the current user interface context compared to a finger movement input without a modified wrist movement input. In some embodiments, a finger input gesture performed with the palm of the user facing the user's face is treated as a different type of gesture than a finger input gesture performed with the palm of the user facing away from the user's face. For example, a tap gesture performed with the palm of the user facing the user performs an action with added (or reduced) privacy protection compared to an action (e.g., the same action) performed in response to a tap gesture performed with the palm of the user facing away from the user's face.

[0187] While one type of finger input may be used to trigger an action type in the examples provided in this disclosure, in other embodiments, other types of finger input are optionally used to trigger the same type of action.

[0188] Additional explanation regarding Figures 7A-7X is provided below with reference to methods 8000, 9000, 10000, 11000, and 12000 described with respect to Figures 8-12 below.

[0189] 8 is a flowchart of a method 8000 of interacting (e.g., including reconstruction and other interactions) with a computer-generated three-dimensional environment, according to some embodiments. In some embodiments, the method 8000 is performed on a computer system (e.g., computer system 101 of FIG. 1 ) that includes a display generating component (e.g., display generating component 120 of FIGS. 1 , 3, and 4 ) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more input devices (e.g., one or more cameras (e.g., cameras facing downward at the user's hand or facing forward from the user's head (e.g., color sensors, infrared sensors, and other depth-sensing cameras)), controllers, touch-sensitive surfaces, joysticks, buttons, etc.). In some embodiments, the method 8000 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of method 8000 are optionally combined and / or the order of some operations is optionally changed.

[0190] In method 8000, a computer system displays (8002) a virtual object (e.g., virtual objects 7208(a-1) and 7B(a-1) of FIG. 7A ) at a first spatial location within a three-dimensional environment (e.g., a physical environment viewed through a display generation component, a mimicked reality environment, a virtual reality environment, an augmented reality environment, a mixed reality environment, etc.). While displaying the virtual object (e.g., virtual object 7208) at the first spatial location within the three-dimensional environment, the computer system detects (8004) a first hand movement performed by a user (e.g., detects a movement of the user's fingers and / or wrist that meets one or more gesture recognition criteria). In response to detecting a first hand movement performed by the user (8006), according to a determination that the first hand movement satisfies a first gesture criterion (e.g., the first hand movement is a pinch and drag gesture (e.g., a pinch movement of a finger resulting from moving the entire hand laterally) or a swipe gesture (e.g., a micro-swipe gesture with a finger across another finger surface or controller)), the computer system performs a first action in accordance with the first hand movement (e.g., rotate the virtual object, adjust a control associated with the virtual object, navigate the virtual object (turn a page of a virtual book), etc.) without moving the virtual object from the first spatial location (e.g., a pinch and drag gesture prior to entering the reconfiguration mode does not move the object from one position to another). This is shown, for example, in FIGS. 7A(a-1)-7A(a-3) and 7A(a-1) followed by FIGS. 7A(a-4) and 7A(a-5).In response to detecting a first hand movement performed by the user (8006), in accordance with a determination that the first hand movement satisfies a second gesture criterion (e.g., a pinch gesture followed by a wrist flick gesture (e.g., a pinch movement of the fingers resulting from rotating the hand around the wrist (e.g., flicking upwards or to the side)), the computer system displays a first visual indication that the virtual object has transitioned to a reconfiguration mode (e.g., the device activates a reconfiguration mode for the virtual object, the virtual object is removed from its original position, and / or becomes semi-transparent and floats above its original position). This is shown, for example, in FIGS. 7B(a-1) to 7B(a-3). While displaying the virtual object with the first visual indication that the virtual object has transitioned to a reconfiguration mode, the computer The computer system detects a second hand movement performed by the user (8008). In response to detecting the second hand movement performed by the user, according to a determination that the second hand movement satisfies a first gesture criterion (e.g., the first hand movement is a pinch and drag gesture (e.g., a pinch finger movement results from the entire hand moving laterally)), the computer system moves the virtual object from the first spatial position to the second spatial position (e.g., without performing the first action) according to the second hand movement (8010) (e.g., once in the reconfiguration mode, wrist flicks no longer need to be continued and a simple pinch and drag gesture moves the object from one position to another). This is shown, for example, in Figures 7B(a-3) to 7B(a-6) following Figure 7B(a-2), or in Figures 7B(a-5) and 7B(a-6).

[0191] In some embodiments, in method 8000, in response to detecting a first hand movement performed by the user, and in accordance with a determination that the first hand movement satisfies a third gesture criterion (e.g., the first hand movement is a micro-tap gesture without lateral and rotational movement of the entire hand), the computer system performs a second operation corresponding to the virtual object (e.g., activates a function corresponding to the virtual object (e.g., launches an application, a communication session, displays content, etc.)). In some embodiments, in response to detecting a second hand movement performed by the user, and in accordance with a determination that the second hand movement satisfies a third gesture criterion (e.g., the second hand movement is a micro-tap gesture without lateral and rotational movement of the entire hand), the device stops displaying the first visual indication that the virtual object has transitioned to a reconfiguration mode to indicate that the virtual object has left the reconfiguration mode (e.g., the device deactivates the reconfiguration mode for the virtual object and returns the virtual object to its original position if the virtual object has not been moved, or places the virtual object in a new position to restore the original appearance of the virtual object if the virtual object has been moved by user input). In some embodiments, in response to detecting a second hand movement performed by the user, following a determination that the second hand movement does not satisfy the first gesture criterion (e.g., the second hand movement is a free hand movement without pinching the fingers together or satisfying another predetermined gesture criterion), the device maintains the virtual object in the reconstruction mode without moving the virtual object. In other words, while the virtual object is in the reconstruction mode, the user can move the user's hand in a manner that does not correspond to a gesture that moves the virtual object and does not cause the virtual object to leave the reconstruction mode. For example, the user can use this opportunity to explore the three-dimensional environment and prepare a suitable position to subsequently move the virtual object.

[0192] In some embodiments, the second hand movement does not meet the second gesture criteria (e.g., the second hand movement is not a pinch gesture followed by a wrist flick gesture (e.g., a finger pinch movement results from rotating the hand around the wrist (e.g., flicking upwards or to the side)).

[0193] In some embodiments, the second gesture criterion includes a requirement that it be satisfied by a pinch gesture followed by a wrist flick gesture (e.g., the second gesture criterion is satisfied with respect to the virtual object when the thumb and index finger of a hand are moved to a position in three-dimensional space that corresponds to the position of a virtual object and touch each other, and then the entire hand is rotated about the wrist while the thumb and index finger continue to touch each other).

[0194] In some embodiments, the second gesture criterion includes a requirement that the second gesture criterion be met by a wrist flick gesture detected while the object selection criterion is met (e.g., the second gesture criterion is met for a virtual object when the entire hand is quickly rotated about the wrist while the virtual object is currently selected (e.g., by a previous selection input (e.g., gaze input directed at the virtual object, a pinch gesture directed at the virtual object, a two-finger tap gesture directed at the virtual object, etc.)). In some embodiments, the previous selection input may be in progress (e.g., in the case of a pinch gesture or gaze input) or may have ended (e.g., in the case of a two-finger tap gesture for selecting a virtual object) when the wrist flick gesture is detected.

[0195] In some embodiments, the first gesture criterion includes a requirement that the first gesture criterion be satisfied by a movement input (e.g., a lateral movement of a finger across the air or a surface (e.g., the surface of a controller or the surface of a hand or another finger), or a tapping movement of a finger in the air or on a surface (e.g., the surface of a controller or the surface of a hand or another finger)) provided by one or more fingers of a hand (e.g., a single finger or multiple fingers moving laterally in unison).

[0196] In some embodiments, while displaying the virtual object with a first visual indication that the virtual object has transitioned to the reconstruction mode, the computer system detects a predetermined input specifying a destination location of the virtual object in the three-dimensional environment (e.g., detecting the predetermined input includes detecting a movement of the user's gaze from the first spatial location to the second spatial location, or detecting a tap input by a finger of a hand (tapping on the controller or in the air or on a surface of the same hand) while the user's gaze is focused on the second spatial location in the three-dimensional space). In response to detecting the predetermined input specifying the destination location of the virtual object in the three-dimensional environment, the computer system displays a second visual indication (e.g., a glowing or shaded overlay (e.g., in the shape of the virtual object)) at the destination location before moving the virtual object from the first spatial location to the destination location (e.g., the second spatial location or a location different from the second spatial location). In some embodiments, the second visual indication is displayed at the destination location in response to detecting the predetermined input before a second hand movement that actually moves the virtual object is detected. In some embodiments, the second hand movement that satisfies the first gesture criterion is a tap input, finger flick input, hand swipe input, or pinch and drag input, etc., that is detected after a destination location of the virtual object is specified by a predefined input (e.g., gaze input, tap input) provided while the virtual object is in the reconstruction mode. In some embodiments, the predefined input is detected before the second hand movement is detected (e.g., if the predefined input is a gaze input or tap input that selects the destination location of the virtual object (e.g., the user may look away from the destination location after providing the predefined input), the second hand movement is a small finger flick or finger tap without whole hand movement that initiates movement of the virtual object toward the destination location).In some embodiments, the predetermined input is detected simultaneously with a second hand movement (e.g., the predetermined input is a gaze input focused on the destination location of the virtual object (e.g., the user maintains gaze on the destination location while a second movement (e.g., a small finger flick or finger tap without full hand movement) initiates movement of the virtual object towards the destination location)). In some embodiments, the predetermined input is a second hand movement (e.g., the predetermined input is a pinch gesture that grabs the virtual object and drags the virtual object to the destination location).

[0197] In some embodiments, detecting a predetermined input specifying a destination location of a virtual object within the three-dimensional environment includes detecting movement in the predetermined input (e.g., movement of a gaze input or movement of a finger prior to the finger tap), and displaying a second visual indication (e.g., an illuminated or shaded overlay (e.g., in the shape of a virtual object)) at the destination location includes updating a position of the second visual indication based on the movement of the predetermined input (e.g., a position of the illuminated or shaded overlay (e.g., in the shape of a virtual object) is continuously and dynamically changed according to movement of the gaze input and / or the position of the finger prior to the input tap).

[0198] In some embodiments, after completion of the second hand movement that satisfies the first gesture criterion and while the virtual object remains in the reconfiguration mode (e.g., after the object is moved according to the second hand movement and while the virtual object is displayed with a first visual indication that the virtual object has transitioned to the reconfiguration mode), the computer system detects a third hand movement that satisfies the first gesture criterion (e.g., a microswipe gesture in which the thumb swipes across the side of the index finger of the same hand, or a swipe gesture by a finger on the touch-sensitive surface of the controller). In response to detecting the third hand movement, the computer system moves the virtual object from the virtual object's current location to a third spatial location according to the third hand movement.

[0199] In some embodiments, the three-dimensional environment includes one or more planes (e.g., a surface of a physical object, a mimicked surface of a virtual object, a surface of a virtual object representing a physical object, etc.), and moving the virtual object from the first spatial location to the second spatial location in accordance with the second hand movement includes constraining a movement path of the virtual object to a first plane of the one or more planes during movement of the virtual object in accordance with the second hand movement (e.g., if the first spatial location and the second spatial location are on the same plane, the virtual object slides along the plane even when the movement path of the second hand movement does not strictly follow the plane).

[0200] The method of any one of claims 1 to 10. In some embodiments, the three-dimensional environment includes at least a first plane and a second plane (e.g., a surface of a physical object, a mimicked surface of a virtual object, a surface of a virtual object representing a physical object, etc.), and moving the virtual object from the first spatial location to the second spatial location according to the second hand movement includes constraining a path of movement of the virtual object to the first plane during a first portion of the movement of the virtual object according to the second hand movement, constraining a path of movement of the virtual object to the second plane during a second portion of the movement of the virtual object according to the second hand movement, and increasing an elevation of the virtual object (e.g., the object jumps up when switching between real-world planes) during a third portion of the movement of the virtual object between the first and second portions of the movement of the virtual object.

[0201] In some embodiments, in response to detecting a first hand movement performed by the user, following a determination that the first hand movement satisfies a second gesture criterion (e.g., a pinch gesture followed by a wrist flick gesture (e.g., a pinch movement of the fingers results from the hand rotating around the wrist (e.g., flicking upwards or to the side)), the computer system generates an audio output in conjunction with displaying a first visual indication indicating that the virtual object has transitioned to a reconfiguration mode (e.g., the device generates a discrete audio output (e.g., a beep or chirp) providing an indication that the virtual object has been removed from its original position and / or generates a continuous audio output (e.g., continuous music or sound waves) while the virtual object remains in the reconfiguration mode).

[0202] In some embodiments, while the virtual object is in the reconstruction mode, the computer system detects a second hand movement, moves the virtual object according to the second movement, and then detects a fourth hand movement. In response to detecting the fourth hand movement, in accordance with a determination that the fourth hand movement satisfies the first gesture criterion, the computer system moves the virtual object from the second spatial location to a third spatial location according to the fourth hand movement, and in accordance with a determination that the fourth hand movement satisfies the fourth gesture criterion (e.g., a pinch gesture followed by a wrist flick gesture (e.g., a pinch movement of fingers resulting from a hand rotating about a wrist (e.g., flicking downward)), the computer system stops displaying the first visual indication to indicate that the virtual object has exited the reconstruction mode. In some embodiments, in conjunction with stopping displaying the first visual indication (e.g., restoring the virtual object's normal appearance), the device displays an animation of the virtual object being placed at the third spatial location in the three-dimensional environment.

[0203] It should be understood that the particular order described for the operations in FIG. 8 is merely an example, and the order described is not intended to indicate that the operations are the only order in which they can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 9000, 10000, 11000, and 12000) can also be applied in a similar manner to method 8000 described above in connection with FIG. 8. For example, the gestures, gaze inputs, physical objects, user interface objects, and / or animations described above with reference to method 8000 optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, and / or animations described herein with reference to other methods described herein (e.g., methods 9000, 10000, 11000, and 12000). For the sake of brevity, those details will not be repeated here.

[0204] FIG. 9 is a flowchart of a method 9000 for generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between physical and virtual objects) according to some embodiments. In some embodiments, the method 9000 is performed on a computer system (e.g., computer system 101 of FIG. 1 ) that includes a display generation component (e.g., display generation component 120 of FIGS. 1 , 3, and 4 ) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more input devices (e.g., a camera (e.g., a camera facing downward at the user's hand or facing forward from the user's head (e.g., color sensors, infrared sensors, and other depth-sensing cameras)), a controller, a touch-sensitive surface, a joystick, buttons, etc.). In some embodiments, the method 9000 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of method 9000 are optionally combined and / or the order of some operations is optionally changed.

[0205] In method 9000, the computer system, via a display generation component, displays (9002) a three-dimensional scene including at least a first virtual object (e.g., virtual object 7332 of FIGS. 7E and 7F ) at a first location (e.g., a virtual window on a wall, a virtual screen on a wall displaying a video) and a first physical surface (e.g., a front wall 7304, a side wall 7306, a floor 7308, furniture 7310, or representations thereof) at a second location separate from the first location (e.g., a bookshelf in a room away from the wall, a wall, or a floor of the room), where the first virtual object and the first physical surface are separated by a real or simulated free space, and the virtual object is The virtual object is displayed with a first value for a first display characteristic corresponding to a first portion of the object (e.g., luminance and color values ​​in first portions 7332-b and 7332-c, and 7332-b' and 7332-c') and a second value for the first display characteristic corresponding to a second portion of the virtual object (e.g., luminance and color values ​​in second portions 7332-a and 7332-d, and 7332-a' and 7332-d') (e.g., the virtual object has different luminance values ​​or colors in various portions of the virtual object, and the first display characteristic is unrelated to the shape or dimensions of the virtual object), and the second value of the first display characteristic is different from the first value of the first display characteristic. While displaying the three-dimensional scene including the first virtual object and the first physical surface, the computer system generates (9004) a first visual effect at a second location of the three-dimensional scene (e.g., a location of the physical surface in the scene) via the display generation component.Generating the first visual effect includes altering a visual appearance of a first portion of a first physical surface in the three-dimensional scene according to a first value of a first display characteristic corresponding to the first portion of the first virtual object, and modifying a visual appearance of a second portion of the first physical surface in the three-dimensional scene according to a second value of the first display characteristic corresponding to the second portion of the first virtual object, where the visual appearance of the first portion of the first physical surface and the visual appearance of the second portion of the first physical surface are modified differently (e.g., according to mimicked spatial relationships between the virtual object and the physical surface, real and mimicked physical properties of the virtual object and the physical surface, and mimicked physical principles) due to differences in the first and second values ​​of the first display characteristic at the first and second portions of the first virtual object (e.g., different color and brightness values ​​of various portions of the virtual object cause the color and brightness of various portions of the physical surface to change differently due to spatial relationships between the various portions of the virtual object and the various portions of the physical surface). This is shown, for example, in FIGS. 7E-7F.

[0206] In some embodiments, the computer system detects a change in appearance of the first virtual object, including a change in a value of a first display characteristic at the first and second portions of the first virtual object. In response to detecting the change in appearance of the first virtual object, the computer system modifies a visual appearance of the first physical surface at various portions of the first physical surface according to the change in appearance of the first virtual object. The modifying includes modifying a visual appearance of the first portion of the first physical surface according to a first relationship between the first display characteristic and the visual appearance of the first portion of the first physical surface, and modifying a visual appearance of the second portion of the first physical surface according to a second relationship between the first display characteristic and the visual appearance of the second portion of the first virtual object, the first relationship and the second relationship corresponding to different physical characteristics of the first and second portions of the first physical surface. For example, the first relationship and the second relationship are based on mimicking physical laws of light emitted from a virtual object interacting with a first physical surface, but vary due to distances, shapes, surface textures, and optical properties corresponding to various portions of the first physical surface, and / or different spatial relationships between various portions of the first physical surface and corresponding portions of the first virtual object.

[0207] In some embodiments, the first virtual object includes a virtual overlay (e.g., a virtual window showing a virtual landscape (e.g., as seen through a window) on a second physical surface (e.g., a wall) at a position corresponding to the first position in the three-dimensional scene) (the first virtual object is a virtual window displayed at a position corresponding to a portion of a physical window or physical wall in the real world), and the computer system changes the appearance of the virtual overlay (e.g., changes the appearance of the landscape shown in the virtual overlay) according to changes in the respective values ​​of one or more parameters including at least one of time of day, location, and size of the virtual overlay. For example, as the time of day changes in the real world or in a setting set by the user, the device changes the virtual landscape (e.g., a view of a city, nature, landscape, factory, etc.) shown in the virtual overlay (e.g., the virtual window) according to the change in time. In another example, the user or the device specifies a scene location of the virtual landscape shown in the virtual overlay, and the virtual landscape is selected from a database of landscapes based on the scene location. In another embodiment, the user requests the computer system to increase or decrease the size of the virtual overlay (e.g., go from a small virtual window to a large virtual window, replace an entire wall with a virtual window, etc.), and the computer system changes the amount of virtual scenery presented through the virtual overlay.

[0208] In some embodiments, generating the first visual effect includes modifying the visual appearance of a first portion of a first physical surface (e.g., an opposing wall or floor in the real world) according to changes in content shown in the first portion of the virtual overlay, and modifying the visual appearance of a second portion of the first physical surface according to changes in content shown in the second portion of the virtual overlay. For example, on a real-world floor surface, the amount of light, color of light, and direction of light coming from various portions of a virtual window superimposed on the physical wall (e.g., depending on the time of day) creates different simulated lighting on the floor surface in front of the virtual window. The computer system generates a second virtual overlay for the floor surface that mimics the different amounts, colors, and directions of lighting in various portions of the second virtual overlay corresponding to the various portions of the floor surface. For example, as the time of day changes, the amount of light and direction of light corresponding to the virtual window changes, and accordingly, the amount of simulated lighting shown in the second virtual overlay on the floor changes (e.g., different light directions and different colors and tones of light in the morning, midday, and evening).

[0209] In some embodiments, the first virtual object includes a virtual screen displaying media content (e.g., a flat virtual screen displaying a movie or video, a three-dimensional space or dome surface displaying a three-dimensional movie or an immersive holographic experience from the user's perspective) at a position corresponding to the first position in the three-dimensional scene (e.g., the virtual screen is freestanding and not attached to any physical surface, or is overlaid on a physical surface such as a wall or a television screen), and the computer system modifies the content shown on the virtual screen according to the progress of the playback of the media items. For example, as the movie or video playback progresses, content is shown on the virtual screen (e.g., 2D or 3D, or immersive) according to the current playback position of the movie or video.

[0210] The method of claim 18. In some embodiments, generating the first visual effect includes modifying the visual appearance of a first portion of a first physical surface (e.g., an opposing wall or floor in the real world) according to changes in content shown on the first portion of the virtual screen, and modifying the visual appearance of a second portion of the first physical surface according to changes in content shown on the second portion of the virtual screen. For example, on a surface of a physical object of the surrounding environment (e.g., a floor, a wall, a couch, the user's body, etc.), the amount of light, color of light, and direction of light coming from various portions of the virtual screen cause different mimicking lighting on the surface of the physical object in the surrounding environment. The device generates a virtual overlay for the surrounding physical surfaces that mimics the different amount, color, and direction of lighting in various portions of the virtual overlay that correspond to the various portions of the physical surface. As the video scene changes, the amount of light, color of light, and direction of light also change, modifying the mimicking lighting superimposed on the surrounding physical surfaces.

[0211] In some embodiments, the first virtual object is a virtual assistant that interacts with the user through speech (e.g., the virtual assistant is activated in various contexts to provide assistance to the user with various tasks and interactions with the electronic device), and the computer system changes the appearance of the virtual assistant according to the operation mode of the virtual assistant. For example, when the virtual assistant is performing various tasks or various operation modes (e.g., waiting, listening to user instructions, being moved from one position to another, performing a task according to a user's command, completing a task, performing various types of tasks, etc.), the color, size, hue, brightness, etc. of the virtual assistant changes. As a result of the change in the appearance of the virtual assistant, the device generates mimicking lighting on the physical surface at a position corresponding to the position surrounding the virtual assistant.

[0212] In some embodiments, generating the first visual effect includes modifying the visual appearance of the first portion of the first physical surface according to an emulated reflection of the first virtual object on the first portion of the first physical surface (e.g., the emulated reflection is generated according to surface properties of the first portion of the first physical surface, a relative position of the first virtual object in the three-dimensional scene and the first portion of the first physical surface, emulated physical properties of light emitted from the first virtual object, and physical light propagation principles that define how light is reflected and transmitted and how objects are lit by real-world light). In some embodiments, generating the first visual effect further includes modifying the visual appearance of the second portion of the first physical surface according to an emulated reflection of the first virtual object on the second portion of the first physical surface (e.g., the emulated reflection is generated according to surface properties of the second portion of the first physical surface, a relative position of the first virtual object in the three-dimensional scene and the second portion of the first physical surface, emulated physical properties of light emanating from the first virtual object, and physical light propagation principles that define how light is reflected and transmitted and how objects are illuminated by real-world light).

[0213] In some embodiments, generating the first visual effect includes modifying a visual appearance of the first portion of the first physical surface (e.g., a non-reflective physical surface) according to an emulated shadow cast by the first virtual object on the first portion of the first physical surface (e.g., the emulated shadow is generated by the device according to surface characteristics of the first portion of the first physical surface, a relative position of the first virtual object in the three-dimensional scene and the first portion of the first physical surface, emulated physical characteristics (e.g., shape, size, etc.) of the first virtual object, real light sources, emulated light sources present in the three-dimensional scene, and physical light propagation and refraction principles). In some embodiments, generating the first visual effect further includes modifying a visual appearance of the second portion of the first physical surface (e.g., a non-reflective physical surface) according to an emulated shadow of the first virtual object on the second portion of the first physical surface (e.g., the emulated shadow is generated by the device according to surface characteristics of the second portion of the first physical surface, a relative position of the first virtual object in the three-dimensional scene and the second portion of the first physical surface, emulated physical characteristics (e.g., shape, size, etc.) of the first virtual object, real light sources, emulated light sources present in the three-dimensional scene, and physical light propagation principles that define how light is reflected and transmitted and how objects are illuminated by this light in the real world).

[0214] It should be understood that the particular order described for the operations in FIG. 9 is merely an example, and the order described is not intended to indicate that the operations are the only order in which they can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 10000, 11000, and 12000) can also be applied in a similar manner to method 9000 described above in connection with FIG. 9. For example, the gestures, gaze inputs, physical objects, user interface objects, and / or animations described above with reference to method 9000 optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, and / or animations described herein with reference to other methods described herein (e.g., methods 8000, 10000, 11000, and 12000). For the sake of brevity, those details will not be repeated here.

[0215] FIG. 10 is a flowchart of a method 10000 for generating a computer-generated three-dimensional environment and facilitating user interaction with the three-dimensional environment (e.g., including gradually adjusting a level of computer-generated experience based on user input) according to some embodiments. In some embodiments, the method 10000 is performed on a computer system (e.g., computer system 101 of FIG. 1 ) that includes a display generation component (e.g., display generation component 120 of FIGS. 1 , 3, and 4 ) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more input devices (e.g., a camera (e.g., a camera that faces downward at the user's hand or forward from the user's head (e.g., color sensors, infrared sensors, and other depth-sensing cameras)), a controller, a touch-sensitive surface, a joystick, a button, etc.). In some embodiments, the method 10000 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of method 10000 are optionally combined and / or the order of some operations is optionally changed.

[0216] In method 10000, a computer system displays (10002) a three-dimensional scene via a display generation component, the three-dimensional scene including a first set of physical elements (e.g., the physical objects shown in FIG. 7G or representations thereof) (e.g., physical objects visible through a transparent portion of the display generation component or represented by their images in a camera view of the physical environment of the physical objects, where the positions of each of the physical elements in the three-dimensional scene correspond to the positions of each of the physical objects in the physical environment surrounding the display generation component) and a first amount of virtual elements (e.g., no virtual objects or very simple virtual objects representing user interface elements and controls). The first set of physical elements includes at least physical elements corresponding to a first class of physical objects (e.g., walls or walls directly facing the display generating component, windows, etc.) and physical elements corresponding to a second class of physical objects (side walls distinct from walls directly facing the display generating component, ceilings and floors distinct from walls, walls distinct from windows, physical objects in the room, vertical physical surfaces inside the room, horizontal surfaces inside the room, surfaces larger than a pre-set threshold, surfaces of real furniture in the room, etc.). While displaying the three-dimensional scene with the first amount of virtual elements via the display generating component, the computer system detects (10004) a sequence of two or more user inputs (e.g., a sequence of two or more swipe inputs, a sequence of two or more snaps, an input corresponding to a user placing the HMD on the head, followed by the user releasing the HMD from the hands, followed by the user sitting down with the HMD on the head) (e.g., the two or more user inputs are distinct from a user input dragging and / or dropping a particular virtual object (by input focus on the particular virtual object) into the three-dimensional scene).In response to detecting successive user inputs in the sequence of two or more user inputs, the computer system successively increases (10006) an amount of virtual elements displayed in the three-dimensional scene in accordance with the successive inputs in the sequence of two or more user inputs (e.g., successively increasing the immersiveness of the three-dimensional scene by substituting additional classes of physical elements in the three-dimensional scene in response to successive user inputs of the same type or in a sequence of related inputs). Specifically, in response to detecting a first user input (e.g., an input by hand 7200 in FIG. 7G ) of a sequence of two or more user inputs, the computer system, in accordance with a determination that the first user input satisfies a first criterion (e.g., a criterion for detecting a gesture that increases the immersiveness of a computer-generated experience), displays the three-dimensional scene with at least a first subset of the one or more physical elements of the first set (e.g., some, but not all, of the one or more physical elements of the first set are obscured or blocked by the newly added virtual elements) and a second amount of virtual elements (e.g., virtual object 7402 in FIGS. 7H and 7I ). The second amount of virtual elements occupies a larger portion of the three-dimensional scene than the first amount of virtual elements (e.g., displaying virtual elements such as a virtual landscape or a virtual window that blocks a view of the first set of physical surfaces (e.g., walls, etc.) in the three-dimensional scene), including a first portion of the three-dimensional scene that was occupied by the first class of physical elements (e.g., walls, etc.) prior to detection of the first user input. Additionally, in response to detecting a second user input (e.g., an input by hand 7200 in FIG. 7I ) in a sequence of two or more user inputs, in accordance with a determination that the second user input follows the first user input and satisfies a first criterion, the computer system displays a three-dimensional scene having at least a second subset of one or more physical elements of the first set (e.g., more or all of the one or more physical elements of the first set are obscured or blocked by the newly added virtual elements) and a third amount of virtual elements (e.g., virtual objects 7402 and 7406 in FIGS. 7J and 7K ).The third amount of virtual elements occupies a larger portion of the three-dimensional scene than the second amount of virtual elements, including a first portion of the three-dimensional scene that was occupied by the first class of physical elements prior to detection of the first user input and a second portion of the three-dimensional scene that was occupied by the second class of physical elements prior to detection of the second user input (e.g., continuing to display virtual elements such as a virtual landscape or virtual window that blocks a view of a first set of physical surfaces (e.g., walls, etc.) in the three-dimensional scene, and displaying additional virtual elements such as virtual decorations or virtual surfaces that block a view of a second set of physical surfaces (e.g., table tops, shelves, equipment surfaces, etc.)). This is shown, for example, in FIGS. 7G-7L.

[0217] In some embodiments, displaying the second amount of virtual elements in response to detecting a first user input of the sequence of two or more user inputs includes displaying a first animated transition that gradually replaces an increasing amount of a first class of physical elements in the three-dimensional scene with virtual elements (e.g., new virtual elements and / or augmentations of existing virtual elements) (e.g., replacing a display of an object visible via bus-through video or obscuring an object directly visible through a transparent or partially transparent display). Displaying a third amount of virtual elements in response to detecting a second user input of the sequence of two or more user inputs includes displaying a second animated transition that gradually replaces an increasing amount of a second class of physical elements in the three-dimensional scene with virtual elements (e.g., new virtual elements and / or augmentations of existing virtual elements) while displaying the first class of physical elements in place of the existing virtual elements (e.g., the second amount of virtual elements) in the three-dimensional scene. For example, in response to a first input (e.g., a first swipe input on the controller or the user's hand), the device replaces a view of a first physical wall (e.g., a wall directly facing the display generating component) that is visible in the three-dimensional scene with a virtual landscape of a forest, leaving the other physical walls, the physical ceiling, and the physical floor visible in the three-dimensional scene. When replacing the view of the first physical wall, the device displays an animated transition that gradually fades in with the virtual landscape of the forest. In response to a second input (e.g., a second swipe input on the controller or the user's hand), the device replaces a view of the remaining physical walls (e.g., walls not directly facing the display generating component) that are visible in the three-dimensional scene with the virtual landscape of the forest extending from the portion already visible in the three-dimensional scene, leaving only the physical ceiling and the physical floor visible in the three-dimensional scene. When replacing the view of the remaining physical walls, the device displays an animated transition that gradually extends the existing view of the virtual forest from the location of the first physical wall to the remaining portions of the walls.In some embodiments, in response to a third input (e.g., a third swipe input on the controller or the user's hand), the device gradually replaces the view of the ceiling (and optionally the floor) that is still visible in the three-dimensional scene with a virtual landscape of the forest that gradually extends from the location of the surrounding physical walls toward the center of the ceiling (and optionally toward the center of the floor) from the existing view of the virtual forest (e.g., showing a portion of the virtual sky visible through a cut in the virtual forest) (e.g., showing a portion of the ground visible through a cut in the virtual forest). In response to a fourth input (e.g., a fourth swipe input on the controller or the user's hand), the device replaces the view of other physical objects that are still visible in the three-dimensional scene with virtual overlays that gradually fade in and become increasingly opaque and saturated on the surfaces of the physical objects.

[0218] In some embodiments, when the amount of virtual elements is successively increased according to successive inputs of the sequence of two or more user inputs, the computer system, in response to detecting a third user input of the sequence of two or more user inputs, displays the three-dimensional scene with a fourth amount of virtual elements according to a determination that the third user input follows the second user input and satisfies the first criterion. The fourth amount of virtual elements occupies a larger portion of the three-dimensional scene (e.g., the fourth amount occupies all of the three-dimensional scene) than the third amount of virtual elements, including a first portion of the three-dimensional scene occupied by a first class of physical elements (e.g., a physical window or a wall facing the display generating component) before the detection of the first user input, a second portion of the three-dimensional scene occupied by a second class of physical elements (e.g., a wall or a wall not facing the display generating component) before the detection of the second user input, and a third portion of the three-dimensional scene occupied by a third class of physical elements (e.g., a physical object in a room) before the detection of the third user input.

[0219] In some embodiments, in response to detecting a second user input of the sequence of two or more user inputs, in accordance with a determination that the second user input follows the first user input and satisfies the first criterion, the computer system displays a third animated transition between displaying the second amount of virtual elements and displaying a third amount of virtual elements. In some embodiments, the rendering of the second amount of virtual elements is more artificial and less realistic, and the rendering of the third amount of virtual elements (including the previously displayed second amount of virtual elements and the additional virtual elements) is more realistic and presents a more immersive computer-generated reality experience.

[0220] In some embodiments, the second amount of virtual elements includes at least a view into the first virtual environment (e.g., a virtual window showing a scene (e.g., a real-time video feed or a mimicked scene) in a different geographic location) displayed on a first subset of the one or more physical elements of the first set. The view into the first virtual environment has a first set of values ​​for a first display characteristic (e.g., luminance distribution, color, hue, etc.) of a portion of the first virtual environment represented in the view (e.g., the virtual window showing pink morning light reflected from a snowy mountain top). The computer system modifies the visual appearance of at least a portion of a first subset of one or more physical elements of the first set in accordance with a first set of values ​​of a first display characteristic of a portion of the first virtual environment represented in the view into the first virtual environment (e.g., the correspondence between the first set of values ​​for the first display characteristic of the view of the first virtual environment shown in the virtual window and the change in visual appearance of the physical elements of the first subset is based on emulated physical principles such as physical light propagation principles that define how light is reflected and transmitted and how objects are illuminated by this light in the real world, actual or emulated surface properties of the physical elements of the first subset, and the relative position of the virtual window with respect to the physical elements of the first subset within the three-dimensional scene, etc.).

[0221] In some embodiments, while displaying a second amount of virtual elements (e.g., a virtual window showing a scene (e.g., a real-time video feed or a mimicked scene) of a different geographic location) including a view into the first virtual environment displayed at least at a first subset of the one or more physical elements of the first set, the computer system detects an input that meets a second criterion (e.g., a criterion for displaying a navigation menu for changing the view into the virtual environment without changing the immersion) (e.g., a criterion for detecting a long press gesture by a user's finger or hand). In response to detecting an input that meets a second criterion distinct from the first criterion (e.g., a long press input that is maintained for at least a time threshold), the computer system displays a plurality of selectable options for changing the view into the first virtual environment (e.g., including menu options for changing the virtual environment represented in the virtual window (e.g., by changing a location, a time of day, lighting, weather conditions, zoom level, viewpoint, season, date, etc.)). In some embodiments, the computer system detects an input selecting one of the displayed selectable options, and in response, the computer system replaces the view into the first virtual environment with a view into a second virtual environment that is different from the first virtual environment (e.g., an ocean or a cave, etc.), or updates the view to show the first virtual environment with at least one changed parameter (e.g., time of day, season, date, location, zoom level, viewing angle, etc.) that changes the appearance of the first virtual environment.

[0222] In some embodiments, while displaying a second amount of virtual elements (e.g., a virtual window showing a scene (e.g., a real-time video feed or a mimicked scene) in a different geographic location) including a view into the first virtual environment displayed at least with a first subset of the one or more physical elements of the first set, the computer system detects an input that satisfies a third criterion (e.g., a criterion for changing the view into the virtual environment without changing the immersion) (e.g., a criterion for detecting a swipe gesture by a user's finger or hand). In response to detecting an input that satisfies the third criterion, the computer system replaces the view into the first virtual environment with a view into a second virtual environment (e.g., an ocean or a cave) that is separate from the first virtual environment. In some embodiments, as the content of the view changes (e.g., with changes in time of day, location, zoom level, viewing angle, season, etc.), the computer system also modifies the visual appearance of at least a portion of the first subset of one or more physical elements of the first set in accordance with changed values ​​of a first display characteristic of the portion of the virtual environment represented in the content of the view (e.g., the correspondence between the first set of values ​​for the first display characteristic of the view of the virtual environment shown in the virtual window and the changes in visual appearance of the physical elements of the first subset is based on emulated physical principles such as physical light propagation principles that define how light is reflected and transmitted and how objects are illuminated by this light in the real world, real or emulated surface properties of the physical elements of the first subset, and the relative position of the virtual window with respect to the physical elements of the first subset in the three-dimensional scene, etc.).

[0223] In some embodiments, while displaying a second amount of virtual elements (e.g., a virtual window showing a scene (e.g., a real-time video feed or a mimicked scene) in a different geographic location) including a view into the first virtual environment displayed at least at a first subset of the one or more physical elements of the first set, the computer system detects an input that satisfies a third criterion (e.g., a criterion for changing the view into the virtual environment without changing the immersion) (e.g., a criterion for detecting a swipe gesture by a user's finger or hand). In response to detecting an input that satisfies the third criterion, the computer system updates the view to show the first virtual environment with at least one changed parameter (e.g., time of day, season, date, location, zoom level, viewing angle, etc.) that changes the appearance of the first virtual environment. In some embodiments, as the content of the view changes (e.g., with changes in time of day, location, zoom level, viewing angle, season, etc.), the computer system also modifies the visual appearance of at least a portion of the first subset of one or more physical elements of the first set in accordance with changed values ​​of a first display characteristic of the portion of the virtual environment represented in the content of the view (e.g., the correspondence between the first set of values ​​for the first display characteristic of the view of the virtual environment shown in the virtual window and the changes in visual appearance of the physical elements of the first subset is based on emulated physical principles such as physical light propagation principles that define how light is reflected and transmitted and how objects are illuminated by this light in the real world, real or emulated surface properties of the physical elements of the first subset, and the relative position of the virtual window with respect to the physical elements of the first subset in the three-dimensional scene, etc.).

[0224] In some embodiments, the first criterion includes a first directional criterion (e.g., the input is a horizontal swipe input) and the second criterion includes a second directional criterion distinct from the first directional criterion (e.g., the input is a vertical swipe input). For example, in some embodiments, a horizontal swipe gesture changes the view represented in the virtual window without changing the size of the window or changes the immersion (e.g., without changing the amount of virtual elements in the three-dimensional scene), while a vertical swipe gesture increases or decreases the immersion (e.g., increases or decreases the amount of virtual elements in the three-dimensional scene).

[0225] In some embodiments, displaying the first amount of virtual elements includes displaying a first virtual window in the three-dimensional scene, displaying the second amount of virtual elements includes expanding the first virtual window in the three-dimensional scene, and displaying the third amount of virtual elements includes replacing a view of one or more physical walls with the virtual elements. In some embodiments, an additional user input of the sequence of two or more user inputs causes an additional amount of virtual elements to be introduced into the three-dimensional scene occupying a portion of the scene previously occupied by the physical elements. For example, a third input that meets the first criterion replaces a plurality of remaining walls and a ceiling with virtual elements. A fourth input that meets the first criterion replaces a floor with a virtual element.

[0226] In some embodiments, the sequence of two or more user inputs includes repeated inputs of a first input type (eg, the same input type, such as a vertical / upward swipe input).

[0227] In some embodiments, the sequence of two or more user inputs includes successive portions of a continuous input (e.g., a vertical / upward swipe input that starts from a first position and includes successive movement in a predetermined direction through multiple threshold positions / distances, or a press input of successively increasing intensity that exceeds multiple intensity thresholds, etc.), each portion of the continuous input corresponding to a respective user input of the sequence of two or more user inputs (e.g., by satisfying a corresponding input threshold of the multiple input thresholds).

[0228] In some embodiments, a first subset of the one or more physical elements of the first set includes at least the walls and floors of the physical environment, and a second subset of the one or more physical elements of the first set includes the floors of the physical environment without including the walls of the physical environment. For example, in some embodiments, the virtual element replaces one or more walls of the physical environment represented in the three-dimensional scene, but does not replace the floors of the physical environment.

[0229] In some embodiments, a first subset of the one or more physical elements of the first set includes at least walls and one or more pieces of furniture in the physical environment, and a second subset of the one or more physical elements of the first set includes one or more pieces of furniture in the physical environment without including walls of the physical environment. For example, in some embodiments, the virtual element replaces one or more walls of the physical environment represented in the three-dimensional scene, but does not replace at least some of the furniture in the physical environment.

[0230] It should be understood that the particular order described for the operations in FIG. 10 is merely an example, and the order described is not intended to indicate that the operations are the only order in which they can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 9000, 11000, and 12000) can also be applied in a similar manner to method 10000 described above in connection with FIG. 10. For example, the gestures, gaze inputs, physical objects, user interface objects, and / or animations described above with reference to method 10000 optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, and / or animations described herein with reference to other methods described herein (e.g., methods 8000, 9000, 11000, and 12000). For the sake of brevity, those details will not be repeated here.

[0231] FIG. 11 is a flowchart of a method 11000 for facilitating user interaction with a computer-generated environment (e.g., using interaction with a physical surface to control a device or interact with the computer-generated environment) according to some embodiments. In some embodiments, the method 11000 is performed on a computer system (e.g., computer system 101 of FIG. 1 ) that includes a display generating component (e.g., display generating component 120 of FIGS. 1 , 3, and 4 ) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more input devices (e.g., a camera (e.g., a camera facing downward at the user's hand or facing forward from the user's head (e.g., color sensors, infrared sensors, and other depth-sensing cameras), a controller, a touch-sensitive surface, a joystick, buttons, etc.). In some embodiments, the method 11000 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of the method 11000 are optionally combined and / or the order of some operations is optionally changed.

[0232] In method 11000, a computer system display displays (11002) a three-dimensional scene via a display generation component, the three-dimensional scene including at least a first physical object (e.g., box 7502 or box 7504 of FIG. 7M) or a representation thereof (e.g., representation 7502′ or representation 7504′ of FIG. 7N). The first physical object has at least a first physical (substantially flat and / or smooth) surface (e.g., the first physical object is visible in the three-dimensional scene through a camera or a transparent display). A respective position of the first physical object or its representation in the three-dimensional scene corresponds to a respective position of the first physical object in a physical environment surrounding the display generation component (e.g., the first physical object is visible through a transparent pass-through portion of a head-up display or HMD, or the representation of the first physical object includes an image of the first physical object in a camera view of the physical environment displayed on the display or HMD). While displaying the three-dimensional scene, the computer system detects (11004) that first interaction criteria are met, the first interaction criteria including a first criterion that is met when a first level of user interaction between the user and the first physical object is detected (e.g., when the user's gaze is directed toward the first physical object without other gestures or actions (e.g., hand movements, verbal commands, etc.) indicating that the user intends to perform an action on the first physical object). In response to detecting that the first interaction criterion is satisfied, the computer system, via the display generation component, displays (11006) a first user interface (e.g., first user interface 7510 of FIG. 7O or first user interface 7516 of FIG. 7P) (e.g., a simplified user interface or information interface) at a location corresponding to the location of a first physical surface of a first physical object (e.g., box 7502 or box 7504) in the three-dimensional scene (e.g., the first user interface is displayed on or overlaying at least a portion of the first physical surface of the first physical object or a representation thereof).While displaying the first user interface at a position corresponding to the position of a first physical surface of the first physical object or a representation thereof within the three-dimensional scene, the computer system detects (11008) that second interaction criteria are met, the second interaction criteria including a second criterion that is met when a second level of user interaction between the user and the first physical object is detected that is higher than the first level of user interaction (e.g., when the user or the user's hand approaches the first physical object while the user's gaze is still directed toward the first physical object) (e.g., a level of user interaction that meets the second criterion also meets the first criterion, but a level of user interaction that meets the first criterion does not meet the second criterion). In response to detecting that the second interaction criteria is met, the computer system replaces (11010) the first user interface with a second user interface (e.g., second user interface 7510′ of FIG. 7Q or second user interface 7512′ of FIG. 7R) (e.g., an enhanced user interface or a user interface with control elements) at a location corresponding to a location of a first physical surface of the first physical object or a representation thereof in the three-dimensional scene (e.g., the second user interface corresponds to an enhanced user interface corresponding to the first physical object compared to the first physical object). In some embodiments, when the user's hand is detected near the first physical object (e.g., a hover input is detected), the computer system displays more information (past / future songs, enhanced controls) in a third user interface that replaces the second user interface. In some embodiments, the first user interface includes an indication of a keyboard, the second user interface includes a keyboard with keys for text input, and the first physical surface is a tabletop of a physical table. An indication of the keyboard is displayed when the user looks at the table top, and the keyboard is displayed when the user looks at the table top and has their hands hovering above the table top in a typing position.In some embodiments, the keys of the keyboard pop up from a position in the three-dimensional scene that corresponds to the tabletop when the user hovers their hands above the tabletop. In some embodiments, when the user's fingers press or touch the tabletop, keys appear to be depressed and, optionally, larger, at positions on the tabletop that correspond to the touched positions. When the user's fingers are lifted off the tabletop, the keys are restored to their original size.

[0233] In some embodiments, while displaying the second user interface at a location corresponding to a location of the first physical surface of the first physical object or a representation thereof in the three-dimensional scene, the computer system detects that the first interaction criterion is met (e.g., the level of user interaction returns to the first level of user interaction). In response to detecting that the first interaction criterion is met after displaying the second user interface, the computer system replaces the display of the second user interface with a display of the first user interface at a location corresponding to the location of the first physical surface of the first physical object or a representation thereof in the three-dimensional scene. For example, once the level of user interaction falls below a threshold level required to display the enhanced user interface, the display of the enhanced user interface is stopped. In some embodiments, if the level of user interaction further decreases and the first interaction criterion is also not met, the computer system stops displaying the first user interface as well.

[0234] In some embodiments, while displaying the first user interface (e.g., a media playback user interface) at a location corresponding to a location of a first physical surface of a first physical object (e.g., a speaker) or a representation thereof in the three-dimensional scene, the computer detects that a third interaction criterion is met, the third interaction criterion including a third criterion that is met when a first level of user interaction between the user and a second physical object (e.g., a smart lamp) separate from the first physical object (e.g., when the user or the user's hand is not moving but the user's gaze is moving from the first physical object to the second physical object) is detected. In response to detecting that the third interaction criterion is met, the computer system stops displaying the first user interface (e.g., a media playback user interface) at a location corresponding to the location of the first physical surface of the first physical object (e.g., a speaker) or a representation thereof in the three-dimensional scene, and the computer system displays a third user interface (e.g., a lighting control user interface) at a location corresponding to the location of the second physical surface of the second physical object (e.g., a smart lamp) or a representation thereof in the three-dimensional scene. For example, when a user's gaze shifts from a first physical object to a second physical object, while the user's hand is hovering in the air without moving in the vicinity of both the first physical object and the second physical object, the computer system stops displaying a user interface corresponding to the first physical object overlaying a surface of the first physical object, and instead displays a user interface corresponding to the second physical object overlaying a surface of the second physical object in the three-dimensional scene.

[0235] In some embodiments, while displaying the first user interface at a location corresponding to a location of a first physical surface of a first physical object or a representation thereof within the three-dimensional scene, the computer system detects a first input that satisfies a first action criterion, the first action criterion corresponding to an activation of a first option included in the first user interface (e.g., the first activation criterion is a criterion for detecting a tap input). In response to detecting the first input that satisfies the first action criterion while the first user interface is displayed, the computer system performs a first operation that corresponds to the first option included in the first user interface (e.g., activates a play / pause function of a media player associated with the first physical object (e.g., a speaker or stereo)). In some embodiments, while displaying the first user interface at a position corresponding to a position of a first physical surface of the first physical object or a representation thereof within the three-dimensional scene, the computer system detects a second input that satisfies a second action criterion, the second action criterion corresponding to activation of a second option included in the first user interface (e.g., the second action criterion is a criterion for detecting a swipe input or a criterion for detecting a twist input), and in response to detecting the second input that satisfies the second action criterion while the first user interface is displayed, the computer system performs a second action corresponding to the second option included in the first user interface (e.g., activating a fast forward or rewind function of a media player associated with the first physical object (e.g., speakers or stereo) or adjusting the volume or output level of the first physical object).

[0236] In some embodiments, while displaying the second user interface at a location corresponding to a location of a first physical surface of the first physical object or a representation thereof within the three-dimensional scene, the computer system detects a second input satisfying a third action criterion, the third action criterion corresponding to activation of a third option included in the second user interface (e.g., the third action criterion is a criterion for detecting a tap input together with a gaze input directed at a first user interface object included in the second user interface). In response to detecting the first input satisfying the third action criterion while the second user interface is displayed, the computer system performs a third operation corresponding to a third option included in the second user interface (e.g., switching to a different album in a media player associated with the first physical object (e.g., a speaker or a stereo)). In some embodiments, while displaying the second user interface at a position corresponding to a position of a first physical surface of the first physical object or a representation thereof within the three-dimensional scene, the computer system detects a fourth input that satisfies a fourth action criterion, the fourth action criterion corresponding to activation of a fourth option included in the second user interface (e.g., the fourth action criterion is a criterion for detecting a swipe input accompanied by a gaze input directed toward a second user interface object included in the second user interface), and in response to detecting a fourth input that satisfies the fourth action criterion while the second user interface is displayed, the computer system performs a fourth operation that corresponds to the fourth option included in the second user interface (e.g., activating one or more other associated physical objects of the first physical object (e.g., activating one or more other associated speakers) or sending an output of the first physical object to another physical object).

[0237] In some embodiments, the first physical object is a speaker, and the first user interface provides a first set of one or more playback control functions associated with the speaker (e.g., play / pause control functions, fast forward functions, rewind functions, stop functions, etc.). In some embodiments, the first user interface includes user interface objects corresponding to these control functions. In some embodiments, the first user interface does not include user interface objects corresponding to at least some of the control functions provided on the first user interface at a given time, and the user interface objects displayed on the first user interface are selected in response to user input detected while the first user interface is displayed. For example, when a user provides a swipe input while the first user interface is displayed, the first user interface displays a fast forward or rewind symbol depending on the direction of the swipe input. When a user provides a tap input while the first user interface is displayed, the first user interface displays a play / pause indicator depending on the current state of playback. When the user provides a pinch and twist input with their fingers, the first user interface displays a volume control that adjusts the speaker volume level according to the direction of the twist input. In some embodiments, the first user interface also provides information such as a list of recently played or next songs / albums from which the user can select.

[0238] In some embodiments, the first user interface includes one or more notifications corresponding to the first physical object. For example, when a user has a first level of interaction with the first physical object (e.g., the user looks at a speaker or a smart lamp), the computer system displays one or more notifications overlaid on a first physical surface of the first physical object (e.g., notifications related to a status or warning corresponding to the speaker or smart lamp (e.g., "battery low," "set timer for 20 minutes")).

[0239] In some embodiments, the second user interface includes a keyboard including a plurality of character keys for text input. For example, when the user has a second level of interaction with the first physical object (e.g., the user looks at the speaker and raises both hands), the computer displays a search interface together with the keyboard in which the user inputs search keywords for searching a music database associated with the speaker.

[0240] In some embodiments, the first user interface displays an indication of an internal state of the first physical object. For example, when a user has a first level of interaction with the first physical object (e.g., the user looks at a speaker or a smart lamp), the computer system displays the internal state of the first physical object (e.g., name of the currently playing album / song, "battery low," "timer set to 20 minutes," etc.) overlaying a first physical surface of the first physical object.

[0241] In some embodiments, the second user interface provides at least a subset of the functionality or information provided in the first user interface and includes at least one functionality or item of information not available in the first user interface. For example, when a user has a first level of interaction with the first physical object (e.g., the user looks at a speaker or smart lamp), the computer system displays an internal state of the first physical object overlaying the first physical surface of the first physical object (e.g., name of the currently playing album / song, "battery low", "timer set to 20 minutes", etc.), and when a user has a second level of interaction with the first physical object (e.g., the user looks at a speaker or smart lamp and raises his / her hand in a ready gesture to provide input or approaches the first physical object), the computer system displays a user interface displaying the internal state of the first physical object, as well as one or more controls for changing the internal state of the first physical object (e.g., a control for changing the currently playing song / album, a control for routing output to an associated speaker, etc.).

[0242] In some embodiments, while the first user interface is displayed at a location corresponding to a location of a first physical surface of a first physical object in the three-dimensional scene (e.g., the first user interface is displayed on or overlying at least a portion of the first physical surface of the first physical object or a representation thereof), the computer system detects a user input that satisfies a fifth criterion corresponding to a request to dismiss the first user interface (e.g., the criterion for detecting a swipe input while gaze input is focused on the first user interface). In response to detecting the user input that satisfies the fifth criterion, the computer system stops displaying the first user interface (e.g., without replacing the first user interface with the second user interface). Similarly, in some embodiments, while displaying the second user interface at a position corresponding to a position of a first physical surface of a first physical object in the three-dimensional scene (e.g., the second user interface is displayed on or overlying at least a portion of the first physical surface of the first physical object or a representation thereof), the computer system detects user input that satisfies a sixth criterion corresponding to a request to dismiss the second user interface (e.g., the criterion for detecting a swipe input while gaze input is focused on the second user interface), and in response to detecting the user input that satisfies the sixth criterion, the computer system stops displaying the second user interface (e.g., without replacing the second user interface with the first user interface).

[0243] In some embodiments, while displaying the first user interface or the second user interface at a position corresponding to a position of a first physical surface of a first physical object in the three-dimensional scene (the first / second user interface is displayed on or overlying at least a portion of the first physical surface of the first physical object or a representation thereof), the computer system detects a user input on the first physical surface of the first physical object (e.g., using one or more sensors on the physical surface, such as touch sensors or proximity sensors, and / or one or more sensors on the device, such as a camera or depth sensor). In response to detecting the user input on the first physical surface of the first physical object, the computer system performs a first operation corresponding to the first physical object in accordance with a determination that the user input on the first physical surface of the first physical object satisfies a sixth criterion (e.g., a first set of criteria in a respective set of criteria for detecting a swipe input, a tap input, a long press input, a double tap input, or the like). In accordance with a determination that the user input on the first physical surface of the first physical object satisfies a sixth criterion (e.g., a second set of criteria in a respective set of criteria for detecting a swipe input, a tap input, a long press input, a double tap input, etc.), the computer system performs a second action corresponding to the first physical object, separate from the first action.

[0244] In some embodiments, while displaying the first user interface or the second user interface at a location corresponding to a location of a first physical surface of a first physical object in the three-dimensional scene (e.g., the first / second user interface is displayed on or overlying at least a portion of the first physical surface of the first physical object or a representation thereof), the computer system detects a gesture input (e.g., a hand gesture in the air, on the controller, or on the user's hand) while the gaze input is directed toward the first physical surface of the first physical object. In response to detecting the gesture input while the gaze input is directed toward the first physical surface of the first physical object, in accordance with a determination that the gesture input and the gaze input satisfy a seventh criterion (e.g., the gesture is a tap input while the gaze input is on a button in the user interface), the computer system performs a third operation corresponding to the first physical object (e.g., a function associated with the button). Following a determination that the gesture input and gaze input meet an eighth criterion (e.g., the gesture is a swipe input while the gaze input is over a slider in the user interface), the computer system performs a fourth action corresponding to the first physical object that is separate from the third action (e.g., adjusting a value associated with the slider).

[0245] In some embodiments, while displaying the first user interface or the second user interface at a position corresponding to a position of a first physical surface of a first physical object in the three-dimensional scene (e.g., the first / second user interface is displayed on or overlaying the first physical surface of the first physical object or at least a portion of a representation thereof), the computer system detects (using one or more sensors on the physical surface, such as touch sensors or proximity sensors, and / or one or more sensors on the device, such as a camera or depth sensor) a gesture input on a second physical surface of a second physical object separate from the first physical object (e.g., the second physical object is a tabletop or a controller in the vicinity of the user's hand) while the gaze input is directed at the first physical surface of the first physical object (e.g., the first physical object is far away from the user's hand). In response to detecting a gesture input on a second physical surface of a second physical object while the gaze input is directed toward the first physical surface of the first physical object, in accordance with a determination that the gesture input and the gaze input satisfy a ninth criterion (e.g., the gesture is a tap input while the gaze input is on a button in the user interface), the computer system performs a fifth action corresponding to the first physical object (e.g., a function associated with the button), and in accordance with a determination that the gesture input and the gaze input satisfy a tenth criterion (e.g., the gesture is a swipe input while the gaze input is on a slider in the user interface), the computer system performs a sixth action corresponding to the first physical object that is separate from the fifth action (e.g., adjusting a value associated with the slider).

[0246] It should be understood that the particular order described for the operations in FIG. 11 is merely an example, and the order described is not intended to indicate that the operations are the only order in which they can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 9000, 10000, and 12000) can also be applied in a similar manner to method 11000 described above in connection with FIG. 11. For example, the gestures, gaze inputs, physical objects, user interface objects, and / or animations described above with reference to method 11000 optionally have one or more of the characteristics of the gestures, gaze inputs, physical objects, user interface objects, and / or animations described herein with reference to other methods described herein (e.g., methods 8000, 9000, 10000, and 12000). For the sake of brevity, those details will not be repeated here.

[0247] FIG. 12 is a flowchart of a method 12000 for generating a computer-generated three-dimensional environment (e.g., including mimicking visual interactions between a voice-based virtual assistant and physical and virtual objects in the environment) according to some embodiments. In some embodiments, method 12000 is performed on a computer system (e.g., computer system 101 of FIG. 1 ) that includes a display generation component (e.g., display generation component 120 of FIGS. 1 , 3, and 4 ) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more input devices (e.g., a camera (e.g., a camera (e.g., a camera that faces downward at the user's hand or faces forward from the user's head (e.g., color sensors, infrared sensors, and other depth-sensing cameras)), a controller, a touch-sensitive surface, a joystick, a button, etc.). In some embodiments, method 12000 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of method 12000 are optionally combined and / or the order of some operations is optionally changed.

[0248] In method 12000, the computer system, via a display generation component, displays (12002) a three-dimensional scene, the three-dimensional scene including a first physical object (e.g., furniture 7310 of FIG. 7T) having at least a first physical surface (e.g., front surface 7312 of furniture 7310 of FIG. 7T) (e.g., the first physical object is visible in the three-dimensional scene via a camera or a transparent display (e.g., initially visualized and with inherent optical properties such as color, texture, reflectance, and transparency) and a first virtual object (e.g., virtual object 7404 of FIG. 7T) having at least a first virtual surface (e.g., a computer-rendered three-dimensional object having a computer-generated surface with mimicked surface optical properties (e.g., mimicked reflectance, mimicked surface texture, etc.), such as a computer-rendered three-dimensional vase or table top). The first physical object and the first virtual object are While displaying the three-dimensional scene including the voice-based virtual assistant, the computer system detects a request to activate the voice-based virtual assistant (12004), for example, as shown in FIG. 7T. In response to detecting the request to activate the voice-based virtual assistant (12006), the computer system activates the voice-based virtual assistant configured to receive voice commands (e.g., to interact with the three-dimensional scene). The computer system also displays a visual representation of the voice-based virtual assistant (e.g., the luminous ellipse 7602 of FIGS. 7U and 7W) in the three-dimensional scene, including displaying the visual representation of the voice-based virtual assistant with a first set of values ​​(e.g., a single value, continuous values, or separate and discrete values ​​for various portions of the visual representation) of a first display characteristic (e.g., color, or brightness) of the visual representation (e.g., the luminous ellipse 7602 has a first range of brightness and a first color).The computer system modifies the visual appearance of at least a portion of the first physical surface of the first physical object (e.g., the front surface 7312 of the furniture 7310 in FIGS. 7U and 7W or a representation thereof) and at least a portion of the first virtual surface of the first virtual object (e.g., the top surface of the virtual object 7404 in FIGS. 7U and 7W) according to the first set of values ​​of the first display characteristics of the visual representation of the voice-based virtual assistant (e.g., the correspondence between the first set of values ​​of the first display characteristics of the visual representation of the voice-based virtual assistant and the change in the visual appearance of the first physical surface and the first virtual surface is based on emulated physical principles such as light propagation principles that define how light is reflected and transmitted and how an object is illuminated by this light in the real world, the actual or emulated surface properties of the first physical surface and the first virtual surface, and the relative position of the virtual assistant with respect to the first physical surface and the first virtual surface). For example, as shown in FIG. 7U, when the voice-based assistant's representation begins to emit light at a first level of brightness, the appearance of the front of the furniture 7310 is modified to appear illuminated with mimic lighting emitted from the voice-based assistant's light-emitting representation. The mimic lighting on the front of the rectangular box is stronger / brighter the closer it is to the voice-based assistant's light-emitting representation, and weaker / dimmer the further it is from the voice-based assistant's light-emitting representation. In some embodiments, the mimic lighting is generated according to the physical characteristics (e.g., surface texture, reflectivity, etc.) of the front of the rectangular box in the real world and the mimic distance between the voice-based assistant's light-emitting representation and the rectangular box in the three-dimensional scene. In some embodiments, the device also modifies the appearance of the three-dimensional scene by adding a mimic shadow cast by the rectangular box (e.g., on the physical wall behind the rectangular box) under the mimic lighting by the voice-based virtual assistant's light-emitting representation next to the rectangular box. In some embodiments, in addition to modifying the appearance of the front surface of the rectangular box (e.g., using a semi-transparent overlay in a location that corresponds to the physical surface or by directly modifying the displayed pixel values ​​of the representation of the physical surface), the device also modifies the appearance of the top surface of the virtual elliptical object to appear to be lit by mimic lighting emitted from an emissive representation of the voice-based assistant.The mimicked lighting on the top surface of the virtual elliptical object is stronger / brighter in positions closer to the light-emitting representation of the voice-based assistant and weaker / dimmer in positions further from the light-emitting representation of the voice-based assistant. In some embodiments, the mimicked lighting is generated according to the mimicked physical properties (e.g., surface texture, reflectivity, etc.) of the top surface of the virtual elliptical object and the mimicked distance between the light-emitting representation of the voice-based assistant and the virtual elliptical object in the three-dimensional scene. In some embodiments, the device also modifies the appearance of the three-dimensional scene by adding a mimicked shadow next to the virtual elliptical object or modifying an existing mimicked shadow cast by the virtual elliptical object according to the mimicked lighting by the light-emitting representation of the voice-based assistant.

[0249] In some embodiments, modifying the visual appearance of at least a portion of the first virtual surface of the first virtual object (e.g., the top surface of virtual object 7404 in FIG. 7U and FIG. 7W) according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant includes increasing the brightness of each of at least a portion of the first virtual surface of the first virtual object according to an increased brightness value of the visual representation of the voice-based virtual assistant (e.g., a portion of the first virtual surface of the first virtual object (e.g., a portion that may be invisible to the user on the display generation component)).

[0250] In some embodiments, modifying the visual appearance of at least a portion of the first virtual surface of the first virtual object (e.g., the top surface of virtual object 7404 in FIG. 7U and FIG. 7W) according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant includes changing the color of each of at least a portion of the first virtual surface of the first virtual object according to the changed color value of the visual representation of the voice-based virtual assistant (e.g., a portion of the first virtual surface of the first virtual object (e.g., a portion invisible to the user on the display generation component)).

[0251] In some embodiments, modifying the visual appearance of at least a portion of the first physical surface of the first physical object (e.g., the front 7312 of the furniture 7310 in FIG. 7U and FIG. 7W or a representation thereof) according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant includes increasing the brightness of each of the portions of the three-dimensional scene corresponding to at least a portion of the first physical surface of the first physical object according to the increased brightness value of the visual representation of the voice-based virtual assistant (e.g., a portion of the first physical surface of the first physical object (e.g., a portion that may be invisible to the user on the display generation component)).

[0252] In some embodiments, modifying the visual appearance of at least a portion of the first physical surface of the first physical object (e.g., the front 7312 of the furniture 7310 in FIG. 7U and FIG. 7W or a representation thereof) according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant includes changing the color of each of the portions of the three-dimensional scene corresponding to at least a portion of the first physical surface of the first physical object according to the changed color value of the visual representation of the voice-based virtual assistant (e.g., a portion of the first physical surface of the first physical object (e.g., a portion that may be invisible to the user on the display generation component)).

[0253] In some embodiments, in response to detecting a request to activate the voice-based virtual assistant, the computer system modifies the visual appearance of the peripheral area of ​​the portion of the three-dimensional scene currently being displayed via the display generation component according to a first set of values ​​of a first display characteristic of the visual representation of the voice-based virtual assistant (e.g., increasing the brightness of the peripheral area or changing the color or hue). For example, if the virtual assistant is represented in the three-dimensional scene by a luminous purple oval, the peripheral area of ​​the user's field of view is displayed with a blurred luminous border to indicate that a voice command to the voice-based virtual assistant will be performed on an object or objects in the portion of the three-dimensional scene currently in the user's field of view. For example, as the user looks around the room, the central area of ​​the user's field of view is transparent and surrounded by a purple vignette, and the objects in the central area of ​​the user's field of view are targets of the voice command or provide context for the voice command detected by the voice-based virtual assistant (e.g., "Turn this on" or "Change this photo").

[0254] In some embodiments, detecting a request to activate the voice-based virtual assistant includes detecting a gaze input that meets a first criterion, the first criterion including a criterion that is met when the gaze input is directed at a position corresponding to a visual representation of the voice-based virtual assistant in a three-dimensional scene (e.g., the virtual assistant is activated when the user gazes at the visual representation of the virtual assistant). In some embodiments, the first criterion also includes a criterion that is met when the gaze input meets a predetermined gaze fixation and duration threshold. In some embodiments, the request to activate the voice-based virtual assistant includes a predetermined trigger command "Hey, Assistant".

[0255] In some embodiments, in response to detecting a request to activate the voice-based virtual assistant (e.g., in response to detecting a gaze input that meets a first criterion), displaying a visual representation of the voice-based virtual assistant (e.g., the luminous ellipse 7602 in FIG. 7U and FIG. 7W ) in the three-dimensional scene includes moving the visual representation of the voice-based virtual assistant from a first position to a second position in the three-dimensional scene (e.g., when a user gazes at a dormant virtual assistant, the virtual assistant pops up from its original position (e.g., in the center of the user's field of view or a short distance away from its original position to indicate that it has been activated).

[0256] In some embodiments, in response to detecting a request to activate the visual representation of the voice-based virtual assistant (e.g., in response to detecting a gaze input that meets a first criterion), displaying the voice-based virtual assistant in the three-dimensional scene (e.g., the luminous ellipse 7602 in FIG. 7U and FIG. 7W) includes resizing the visual representation of the voice-based virtual assistant in the three-dimensional scene (e.g., when a user gazes at a dormant virtual assistant, the virtual assistant expands in size and then returns to its original size or maintains the expanded size until it is deactivated again).

[0257] In some embodiments, in response to detecting a request to activate the voice-based virtual assistant (e.g., in response to detecting a gaze input that meets a first criterion), displaying a visual representation of the voice-based virtual assistant in a three-dimensional scene (e.g., the light-emitting ellipse 7602 in FIG. 7U and FIG. 7W ) includes changing a first set of values ​​of a first display characteristic of the visual representation of the voice-based virtual assistant in the three-dimensional scene (e.g., when a user gazes at a dormant virtual assistant, the virtual assistant glows and / or has a different color or hue).

[0258] In some embodiments, in response to detecting a request to activate the voice-based virtual assistant (e.g., in response to detecting a gaze input that meets a first criterion), the computer system changes a second set of values ​​of a first display characteristic of a portion of the three-dimensional scene at a location surrounding the visual representation of the voice-based virtual assistant (e.g., when the virtual assistant is invoked). Obscure (blur, darken, etc.) the background (e.g., the area around the virtual assistant or the area around the entire screen).

[0259] In some embodiments, detecting a request to activate the voice-based virtual assistant includes detecting a gaze input that meets a first criterion and a voice input that meets a second criterion, the first criterion including a criterion that is met when the gaze input is directed at a position corresponding to a visual representation of the voice-based virtual assistant in a three-dimensional scene, and the second criterion including a criterion that is met when the voice input is detected while the gaze input meets the first criterion (e.g., the virtual assistant is activated when a user gazes at a visual representation of the virtual assistant and speaks a voice command). In some embodiments, after the voice-based virtual assistant is activated, the device processes the voice input to determine a user command for the voice assistant and provides the user command to the virtual assistant as an input to trigger the performance of a corresponding operation by the virtual assistant. In some embodiments, if the gaze input does not meet the first criterion or the voice input does not meet the second criterion, the virtual assistant does not perform an operation corresponding to the voice command of the voice input.

[0260] In some embodiments, while displaying a visual representation of the voice-based virtual assistant in the three-dimensional scene (e.g., the luminous ellipse 7602 in FIG. 7K and FIG. 7L), the computer system detects a first input corresponding to a request for the voice-based assistant to perform a first operation (e.g., change a photo in a virtual photo frame in the scene, start a communication session, start an application, etc.), the first input being for a first period of time (e.g., the first input is a speech input, a gaze input, a gesture input, or a combination of two or more of the above). In response to detecting the first input, the computer system changes a first display characteristic of the visual representation of the voice-based virtual assistant from a first set of values ​​(e.g., a single value, a continuous range of values ​​for various portions of the visual representation, or discrete and separate values ​​for various portions of the visual representation) during the first input to a second set of values ​​separate from the first set of values. In some embodiments, the device also modifies the visual appearance of at least a portion of the first physical surface of the first physical object (e.g., the front of or representation of furniture 7310 in FIG. 7U and FIG. 7W) and at least a portion of the first virtual surface of the first virtual object (e.g., the top surface of virtual object 7602 in FIG. 7U and FIG. 7W) according to the second set of values ​​of the first display characteristics of the visual representation of the voice-based virtual assistant when the value of the first display characteristic of the visual representation of the voice-based virtual assistant changes during the first input. For example, while the user is speaking to the virtual assistant, the visual representation of the virtual assistant glows with pulsating light, various colors, or dynamic colors / light patterns.

[0261] In some embodiments, while displaying a visual representation of the voice-based virtual assistant in the three-dimensional scene (e.g., the luminous ellipse 7602 in FIG. 7U and FIG. 7W), the computer system detects a second input corresponding to a request for the voice-based assistant to perform a second operation (e.g., change a photo in a virtual photo frame in the scene, start a communication session, start an application, etc.) (e.g., the second input is a speech input, a gaze input, a gesture input, or a combination of two or more of the above). In response to detecting the second input, the computer system begins performing the second operation (e.g., launch an application, play a media file, generate an audio output such as a request for additional information or an answer to a question). The computer system also changes a first display characteristic of the visual representation of the voice-based virtual assistant from a first set of values ​​(e.g., a single value, a continuous range of values ​​for various portions of the visual representation, or separate and discrete values ​​for various portions of the visual representation) to a third set of values ​​separate from the first set of values ​​during the execution of the second operation. In some embodiments, the device also modifies the visual appe...

Claims

1. 1. A method comprising:

1. A computer system including a display generation component and one or more input devices, comprising: Displaying a virtual object at a first spatial location within the three-dimensional environment; detecting a first hand movement performed by a user while displaying the virtual object at the first spatial location within the three-dimensional environment; in response to detecting the first hand movement performed by the user; performing a first action in accordance with the first hand movement without moving the virtual object away from the first spatial location in accordance with a determination that the first hand movement satisfies a first gesture criterion; displaying a first visual indication that the virtual object has transitioned to a reconstruction mode in accordance with a determination that the first hand movement satisfies a second gesture criterion; and detecting a second hand movement performed by the user while displaying the virtual object with the first visual indication that the virtual object has transitioned to the reconstruction mode; in response to detecting the second hand movement performed by the user; moving the virtual object from the first spatial location to a second spatial location in accordance with the second hand movement in accordance with a determination that the second hand movement satisfies the first gesture criterion; A method comprising:

2. in response to detecting the first hand movement performed by the user; The method of claim 1 , further comprising: performing a second action corresponding to the virtual object in accordance with a determination that the first hand movement satisfies a third gesture criterion.

3. The method of claim 1 or 2, wherein the second hand movement does not satisfy the second gesture criterion.

4. The method of claim 1 , wherein the second gesture criterion comprises a requirement to be satisfied by a pinch gesture followed by a wrist flick gesture.

5. The method of claim 1 , wherein the second gesture criterion includes a requirement that a wrist flick gesture be satisfied while an object selection criterion is satisfied.

6. The method of claim 1 , wherein the first gesture criterion comprises a requirement that the movement input be satisfied by one or more fingers.

7. detecting a predetermined input specifying a destination location of the virtual object within the three-dimensional environment while displaying the virtual object with the first visual indication that the virtual object has transitioned to the reconstruction mode; in response to detecting the predetermined input specifying the destination location of the virtual object within the three-dimensional environment, displaying a second visual indication at the destination location prior to moving the virtual object from the first spatial location to the destination location; The method of any one of claims 1 to 6, comprising:

8. 8. The method of claim 7, wherein detecting the predetermined input specifying the destination location of the virtual object within the three-dimensional environment includes detecting a movement of the predetermined input, and wherein displaying the second visual indication at the destination location includes updating a position of the second visual indication based on the movement of the predetermined input.

9. detecting a third hand movement that satisfies the first gesture criterion while the virtual object remains in the reconstruction mode after completion of the second hand movement that satisfies the first gesture criterion; in response to detecting the third hand movement, moving the virtual object from a current position of the virtual object to a third spatial position in accordance with the third hand movement; The method of any one of claims 1 to 8, comprising:

10. 10. The method of claim 1, wherein the three-dimensional environment includes one or more planes, and wherein moving the virtual object from the first spatial location to the second spatial location in accordance with the second hand movement includes constraining a movement path of the virtual object to a first of the one or more planes during the movement of the virtual object in accordance with the second hand movement.

11. the three-dimensional environment includes at least a first plane and a second plane, and moving the virtual object from the first spatial location to the second spatial location in accordance with the second hand movement comprises constraining a movement path of the virtual object to the first plane during a first portion of the movement of the virtual object in accordance with the second hand movement; constraining the path of movement of the virtual object to the second plane during a second portion of the movement of the virtual object according to the second hand movement; and increasing an altitude of the virtual object during a third portion of the movement of the virtual object between the first portion of the movement and the second portion of the movement of the virtual object.

12. in response to detecting the first hand movement performed by the user; 12. The method of claim 1, further comprising: generating an audio output in conjunction with displaying the first visual indication that the virtual object has transitioned to the reconstruction mode in accordance with a determination that the first hand movement satisfies the second gesture criterion.

13. detecting a second hand movement while the virtual object is in the reconstruction mode, and detecting a fourth hand movement after moving the virtual object according to the second movement; in response to detecting the fourth hand movement; moving the virtual object from the second spatial location to a third spatial location in accordance with the fourth hand movement in accordance with a determination that the fourth hand movement satisfies the first gesture criterion; ceasing display of the first visual indication to indicate that the virtual object has exited the reconstruction mode in accordance with a determination that the fourth hand movement satisfies a fourth gesture criterion; 13. The method of any one of claims 1 to 12, comprising:

14. A computer-readable storage medium storing executable instructions that, when executed by a computer system having one or more processors and a display generating component, cause the computer system to perform the method of any one of claims 1 to 13.

15. 1. A computer system comprising: one or more processors; A display generating component; a memory storing one or more programs configured to be executed by the one or more processors; wherein the one or more programs contain instructions for performing the method according to any one of claims 1 to 13. A computer system comprising:

16. 1. A computer system having one or more processors and a display generation component, comprising: A computer system comprising means for carrying out the method according to any one of claims 1 to 13.

17. 1. An information processing device for use in a computer system having one or more processors and a display generation component, comprising: An information processing device comprising means for carrying out the method according to any one of claims 1 to 13.

18. 1. A method comprising:

1. A computer system having a display generation component and one or more input devices, comprising: displaying, via the display generation component, a three-dimensional scene including at least the first virtual object at a first location and a first physical surface at a second location distinct from the first location, wherein a first virtual object is displayed with a first value of a first display characteristic corresponding to a first portion of the first virtual object and a second value of the first display characteristic corresponding to a second portion of the first virtual object, the second value of the first display characteristic being different from the first value of the first display characteristic; and generating, via the display generation component, a first visual effect at the second location of the three-dimensional scene while displaying the three-dimensional scene including the first virtual object and the first physical surface, wherein generating the first visual effect includes: modifying a visual appearance of a first portion of the first physical surface in the three-dimensional scene according to the first value of the first display characteristic corresponding to the first portion of the first virtual object; and modifying a visual appearance of a second portion of the first physical surface in the three-dimensional scene according to the second value of the first display characteristic corresponding to the second portion of the first virtual object, wherein the visual appearance of the first portion of the first physical surface and the visual appearance of the second portion of the first physical surface are modified differently due to a difference between the first and second values ​​of the first display characteristic for the first and second portions of the first virtual object.

19. detecting a change in an appearance of the first virtual object, the change including a value change of the first display property in the first portion and the second portion of the first virtual object; in response to detecting the change in appearance of the first virtual object, modifying the visual appearance of the first physical surface at various portions of the first physical surface according to the change in the appearance of the first virtual object; modifying the visual appearance of the first portion of the first physical surface according to a first relationship between the first display characteristic and the visual appearance of the first portion of the first physical surface; modifying the visual appearance of the second portion of the first physical surface according to a second relationship between the first display characteristic and the visual appearance of the second portion of the first virtual object; 20. The method of claim 18, comprising: wherein the first relationship and the second relationship correspond to different physical characteristics of the first portion and the second portion of the first physical surface.

20. 20. The method of claim 18 or 19, wherein the first virtual object includes a virtual overlay on a second physical surface at a position corresponding to the first position in the three-dimensional scene, and the computer system modifies an appearance of the virtual overlay according to changes in respective values ​​of one or more parameters including at least one of time, position, and size of the virtual overlay.

21. Generating the first visual effect includes: modifying the visual appearance of the first portion of the first physical surface in accordance with changes in content shown in the first portion of the virtual overlay; modifying the visual appearance of the second portion of the first physical surface in accordance with changes in content shown in the second portion of the virtual overlay; 21. The method of claim 20, comprising:

22. 20. The method of claim 18 or 19, wherein the first virtual object comprises a virtual screen displaying media content at the position corresponding to the first position in the three-dimensional scene, and the computer system is adapted to modify the content shown on the virtual screen according to the playback progress of a media item.

23. Generating the first visual effect includes: modifying the visual appearance of the first portion of the first physical surface in accordance with changes in content shown on the first portion of the virtual screen; modifying the visual appearance of the second portion of the first physical surface in accordance with changes in content shown on the second portion of the virtual screen; 23. The method of claim 22, comprising:

24. 20. The method according to claim 18 or 19, wherein the first virtual object is a virtual assistant that interacts with a user through speech, and the computer system changes the appearance of the virtual assistant according to the operation mode of the virtual assistant.

25. Generating the first visual effect includes: modifying the visual appearance of the first portion of the first physical surface according to an simulated reflection of the first virtual object on the first portion of the first physical surface; modifying the visual appearance of the second portion of the first physical surface according to an simulated reflection of the first virtual object on the second portion of the first physical surface; 25. The method of any one of claims 18 to 24, comprising:

26. Generating the first visual effect includes: modifying the visual appearance of the first portion of the first physical surface according to an imitated shadow cast by the first virtual object onto the first portion of the first physical surface; modifying the visual appearance of the second portion of the first physical surface according to an imitated shadow of the first virtual object on the second portion of the first physical surface; 26. The method of any one of claims 18 to 25, comprising:

27. 27. A computer readable storage medium storing executable instructions that, when executed by a computer system having one or more processors and a display generating component, cause the computer system to perform the method of any one of claims 18 to 26.

28. 1. A computer system comprising: one or more processors; A display generating component; a memory storing one or more programs configured to be executed by the one or more processors; wherein the one or more programs contain instructions for performing the method according to any one of claims 18 to 26. A computer system comprising:

29. 1. A computer system having one or more processors and a display generation component, comprising: A computer system comprising means for carrying out the method according to any one of claims 18 to 26.

30. 1. An information processing device for use in a computer system having one or more processors and a display generation component, comprising:

27. An information processing device comprising means for carrying out the method according to any one of claims 18 to 26.

31. 1. A method comprising:

1. A computer system having a display generation component and one or more input devices, comprising: displaying, via the display generation component, a three-dimensional scene comprising a first set of physical elements and a first quantity of virtual elements, the first set of physical elements comprising at least physical elements corresponding to a first class of physical objects and physical elements corresponding to a second class of physical objects; detecting a sequence of two or more user inputs while displaying the three-dimensional scene including the first amount of virtual elements via the display generation component; in response to detecting successive user inputs of the sequence of two or more user inputs, successively increasing an amount of virtual elements displayed in the three-dimensional scene in accordance with the successive inputs of the sequence of two or more user inputs; in response to detecting a first user input of the sequence of the two or more user inputs, in accordance with a determination that the first user input satisfies a first criterion, displaying the three-dimensional scene with at least a first subset of the one or more physical elements of the first set and the second amount of virtual elements, where the second amount of virtual elements occupies a greater portion of the three-dimensional scene than the first amount of virtual elements, including a first portion of the three-dimensional scene occupied by physical elements of a first class prior to detection of the first user input; in response to detecting a second user input of the sequence of two or more user inputs, in accordance with a determination that the second user input follows the first user input and satisfies the first criterion, displaying the three-dimensional scene with at least a second subset of the one or more physical elements of the first set and a third amount of virtual elements, the third amount of virtual elements comprising: the first portion of the three-dimensional scene that was occupied by physical elements of the first class prior to the detection of the first user input; and a second portion of the three-dimensional scene that was occupied by physical elements of a second class prior to the detection of the second user input, the third amount of virtual elements occupying a larger portion of the three-dimensional scene than the second amount of virtual elements; Including, and a method including.

32. displaying the second amount of virtual elements in response to detecting the first user input of the sequence of two or more user inputs includes displaying a first animated transition gradually replacing increasing amounts of the first class of physical elements in the three-dimensional scene with virtual elements; 32. The method of claim 31 , wherein displaying the third amount of virtual elements in response to detecting the second user input of the sequence of two or more user inputs comprises displaying a second animated transition gradually replacing increasing amounts of the second class of physical elements in the three dimensional scene with virtual elements while the first class of physical elements is displayed in place of existing virtual elements in the three dimensional scene.

33. Successively increasing the amount of virtual elements according to the successive inputs of a sequence of two or more user inputs; 33. The method of claim 31 or 32, comprising, in response to detecting a third user input of the sequence of two or more user inputs, and in accordance with a determination that the third user input follows the second user input and satisfies the first criterion, displaying the three-dimensional scene with a fourth amount of virtual elements that occupy a larger portion of the three-dimensional scene than a third amount of virtual elements, the fourth amount of virtual elements comprising: a first portion of the three-dimensional scene that was occupied by physical elements of the first class before the detection of the first user input; a second portion of the three-dimensional scene that was occupied by physical elements of the second class before the detection of the second user input; and a third portion of the three-dimensional scene that was occupied by physical elements of a third class before the detection of the third user input.

34. 34. The method of claim 31, comprising, in response to detecting the second user input of the sequence of two or more user inputs, displaying a third animation transition between displaying the second amount of a virtual element and displaying the third amount of a virtual element in accordance with the determination that the second user input follows the first user input and satisfies the first criterion.

35. The second amount of virtual elements includes at least a view into a first virtual environment displayed with a first subset of one or more physical elements of the first set, the view into the first virtual environment having a first set of values ​​for a first display characteristic of a portion of the first virtual environment represented in the view, and the method further comprises:

35. A method according to any one of claims 32 to 34, comprising modifying the visual appearance of at least a portion of the first subset of one or more physical elements of the first set in accordance with the first set of values ​​of the first display characteristics of the portion of the first virtual environment represented in the view into the first virtual environment.

36. detecting an input that satisfies a second criterion while displaying the second amount of virtual elements including the view into the first virtual environment displayed at least at the first subset of one or more physical elements of the first set; in response to detecting the input satisfying the second criterion distinct from the first criterion, displaying a plurality of selectable options for modifying the view into the first virtual environment; 36. The method of claim 35, comprising:

37. detecting an input that satisfies a third criterion while displaying the second amount of virtual elements including the view into the first virtual environment displayed at least at the first subset of the one or more physical elements of the first set; in response to detecting the input satisfying the third criterion, replacing the view into the first virtual environment with a view into a second virtual environment distinct from the first virtual environment; 37. The method of claim 35 or 36, comprising:

38. detecting an input that satisfies a third criterion while displaying the second amount of virtual elements including at least a view into the first virtual environment displayed at the first subset of the one or more physical elements of the first set; in response to detecting the input that satisfies the third criterion, updating the view to show the first virtual environment with at least one changed parameter that modifies the appearance of the first virtual environment; 37. The method of claim 35 or 36, comprising:

39. 39. The method of claim 37 or 38, wherein the first criterion comprises a first directional criterion and the second criterion comprises a second directional criterion distinct from the first directional criterion.

40. 40. The method of claim 37, wherein displaying the first amount of virtual elements comprises displaying a first virtual window in the three-dimensional scene, displaying the second amount of virtual elements comprises expanding the first virtual window in the three-dimensional scene, and displaying the third amount of virtual elements comprises replacing a view of one or more physical walls with virtual elements.

41. 41. The method of any one of claims 31 to 40, wherein the sequence of two or more user inputs includes repetitive inputs of a first input type.

42. 41. The method of any one of claims 31 to 40, wherein the sequence of two or more user inputs comprises consecutive portions of consecutive inputs, each portion of the consecutive input corresponding to a respective user input of the sequence of two or more user inputs.

43. 43. The method of any one of claims 31 to 42, wherein the first subset of one or more physical elements of the first set includes at least a wall and a floor of a physical environment, and the second subset of one or more physical elements of the first set includes the floor of the physical environment without including the walls of the physical environment.

44. 44. The method of any one of claims 31 to 43, wherein the first subset of the one or more physical elements of the first set includes at least walls and one or more pieces of furniture in a physical environment, and the second subset of the one or more physical elements of the first set includes the one or more pieces of furniture in the physical environment without including the walls of the physical environment.

45. 45. A computer readable storage medium storing executable instructions that, when executed by a computer system having one or more processors and a display generating component, cause the computer system to perform the method of any one of claims 31 to 44.

46. 1. A computer system comprising: one or more processors; A display generating component; a memory storing one or more programs configured to be executed by the one or more processors; wherein the one or more programs contain instructions for performing the method of any one of claims 31 to 44. A computer system comprising:

47. 1. A computer system having one or more processors and a display generation component, comprising:

45. A computer system comprising means for carrying out the method of any one of claims 31 to 44.

48. 1. An information processing device for use in a computer system having one or more processors and a display generation component, comprising:

45. An information processing device comprising means for carrying out a method according to any one of claims 31 to 44.

49. 1. A method comprising:

1. A computer system having a display generation component and one or more input devices, comprising: displaying, via the display generation component, a three-dimensional scene including at least a first physical object or a representation of the first physical object, the first physical object having at least a first physical surface, and a respective position of the first physical object or the representation of the first physical object in the three-dimensional scene corresponding to a respective position of the first physical object in a physical environment surrounding the display generation component; detecting that a first interaction criterion has been satisfied while displaying the three-dimensional scene, the first interaction criterion including a first criterion that is satisfied when a first level of user interaction between a user and the first physical object is detected; in response to detecting that the first interaction criterion is satisfied, displaying, via the display generation component, a first user interface at a location corresponding to a location of the first physical surface of the first physical object within the three dimensional scene; detecting that a second interaction criterion is satisfied while displaying the first user interface at the position corresponding to the position of the first physical surface or a representation of the first physical surface of the first physical object within the three dimensional scene, the second interaction criterion including a second criterion that is satisfied when a second level of user interaction between the user and the first physical object is detected that is higher than the first level of user interaction between the user and the first physical object; responsive to detecting that the second interaction criterion is satisfied, replacing a display of the first user interface with a display of a second user interface at the location corresponding to the location of the first physical surface or the representation of the first physical surface of the first physical object in the three dimensional scene; A method comprising:

50. detecting that a first interaction criterion is satisfied while displaying the second user interface at the position corresponding to the position of the first physical surface or the representation of the first physical surface of the first physical object in the three-dimensional scene; responsive to detecting that the first interaction criterion has been satisfied after display of the second user interface, replacing the display of the second user interface with a display of the first user interface at the location corresponding to the location of the first physical surface or the representation of the first physical surface of the first physical object in the three dimensional scene; 50. The method of claim 49, comprising:

51. detecting that a third interaction criterion is satisfied while displaying the first user interface at the position corresponding to the position of the first physical surface or the representation of the first physical surface of the first physical object within the three dimensional scene, the third interaction criterion including a third criterion that is satisfied when the first level of user interaction between the user and a second physical object distinct from the first physical object is detected; in response to detecting that the third interaction criterion is satisfied, ceasing to display the first user interface at the location corresponding to the location of the first physical surface or the representation of the first physical surface of the first physical object in the three dimensional scene; displaying a third user interface at a location corresponding to a location of a second physical surface or a representation of the second physical surface of the second physical object in the three dimensional scene; 51. The method of claim 49 or 50, comprising:

52. detecting a first input that satisfies a first action criterion while displaying the first user interface at a position that corresponds to the position of the first physical surface or the representation of the first physical surface of the first physical object in the three dimensional scene, where a first action criterion corresponds to activation of a first option included in the first user interface; responsive to detecting the first input satisfying the first action criterion while the first user interface is displayed, performing a first operation corresponding to the first option included in the first user interface; 52. The method of any one of claims 49 to 51, comprising:

53. detecting a second input that satisfies a third action criterion while displaying the second user interface at a position that corresponds to the position of the first physical surface or the representation of the first physical surface of the first physical object in the three dimensional scene, where a third action criterion corresponds to activation of a third option included in the second user interface; in response to detecting the first input satisfying the third action criterion while the second user interface is displayed, performing a third operation corresponding to the third option included in the second user interface; 53. The method of claim 52, comprising:

54. 54. The method of any one of claims 49 to 53, wherein the first physical object is a speaker, and the first user interface provides a first set of one or more playback control functions associated with the speaker.

55. 55. The method of any one of claims 49 to 54, wherein the first user interface includes one or more notifications corresponding to the first physical object.

56. 56. The method of any one of claims 49 to 55, wherein the second user interface includes a keyboard including a plurality of character keys for text input.

57. 57. The method of any one of claims 49 to 56, wherein the first user interface displays an indication of an internal state of the first physical object.

58. 58. A method according to any one of claims 49 to 57, wherein the second user interface provides at least a subset of the functionality or information provided in the first user interface and includes at least one functionality or item of information not available in the first user interface.

59. detecting a user input satisfying a fifth criterion corresponding to a request to dismiss the first user interface while displaying the first user interface at the location corresponding to the location of the first physical surface of the first physical object within the three dimensional scene; ceasing display of the first user interface in response to detecting the user input satisfying the fifth criterion; and 59. The method of any one of claims 49 to 58, comprising:

60. detecting a user input on a first physical surface of the first physical object while displaying the first user interface or the second user interface at the location corresponding to the location of the first physical surface of the first physical object in the three dimensional scene; in response to detecting the user input on the first physical surface of the first physical object. performing a first action corresponding to the first physical object in accordance with a determination that the user input on the first physical surface of the first physical object satisfies a sixth criterion; and performing a second action corresponding to the first physical object, distinct from the first action, in response to a determination that the user input on the first physical surface of the first physical object satisfies a sixth criterion; and 60. The method of any one of claims 49 to 59, comprising:

61. while displaying the first user interface or the second user interface at the position corresponding to the position of the first physical surface of the first physical object in the three dimensional scene, while a gaze input is directed toward the first physical surface of the first physical object. in response to detecting the gesture input while the gaze input is directed toward the first physical surface of the first physical object. performing a third action corresponding to the first physical object in accordance with a determination that the gesture input and the gaze input satisfy a seventh criterion; and performing a fourth action corresponding to the first physical object, distinct from the third action, in accordance with a determination that the gesture input and the gaze input satisfy an eighth criterion; and 60. The method of any one of claims 49 to 59, comprising:

62. detecting a gesture input on a second physical surface of a second physical object, separate from the first physical object, while a gaze input is directed toward the first physical surface of the first physical object while displaying the first user interface or the second user interface at the position corresponding to the position of the first physical surface of the first physical object within the three-dimensional scene; in response to detecting the gesture input on the second physical surface of the second physical object while the gaze input is directed at the first physical surface of the first physical object, performing a fifth action corresponding to the first physical object in accordance with a determination that the gesture input and the gaze input satisfy a ninth criterion; and performing a sixth action corresponding to the first physical object, distinct from the fifth action, in accordance with a determination that the gesture input and the gaze input satisfy a tenth criterion; and 60. The method of any one of claims 49 to 59, comprising:

63. 63. A computer readable storage medium storing executable instructions that, when executed by a computer system having one or more processors and a display generating component, cause the computer system to perform the method of any one of claims 49 to 62.

64. 1. A computer system comprising: one or more processors; A display generating component; a memory storing one or more programs configured to be executed by the one or more processors; wherein the one or more programs contain instructions for carrying out the method of any one of claims 49 to 62. A computer system comprising:

65. 1. A computer system having one or more processors and a display generation component, comprising:

63. A computer system comprising means for carrying out the method of any one of claims 49 to 62.

66. 1. An information processing device for use in a computer system having one or more processors and a display generation component, comprising:

63. An information processing device comprising means for carrying out a method according to any one of claims 49 to 62.

67. 1. A method comprising:

1. A computer system having a display generation component and one or more input devices, comprising: displaying, via a display generation component, a three-dimensional scene, the three-dimensional scene including a first physical object having at least a first physical surface and a first virtual object having at least a first virtual surface; Detecting a request to activate a voice-based virtual assistant while displaying the three-dimensional scene including the first physical object and the first virtual object; In response to detecting the request to activate the voice-based virtual assistant, activating the voice-based virtual assistant configured to receive voice commands; Displaying a visual representation of the voice-based virtual assistant in the three-dimensional scene, the visual representation including displaying the visual representation of the voice-based virtual assistant with a first set of values ​​of a first display characteristic of the visual representation; Modifying the visual appearance of at least a portion of the first physical surface of the first physical object and at least a portion of the first virtual surface of the first virtual object according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant; A method comprising:

68. Modifying the visual appearance of at least a portion of the first virtual surface of the first virtual object according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant; 68. The method of claim 67, comprising increasing a brightness of at least a portion of the first virtual surface of the first virtual object in accordance with the increased brightness value of the visual representation of the voice-based virtual assistant.

69. Modifying the visual appearance of at least a portion of the first virtual surface of the first virtual object according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant; 69. The method of claim 67 or 68, comprising changing a color of at least a portion of the first virtual surface of the first virtual object according to the changed color value of the visual representation of the voice-based virtual assistant.

70. Modifying the visual appearance of at least a portion of the first physical surface of the first physical object according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant; and increasing the brightness of a portion of the three-dimensional scene corresponding to at least a portion of the first physical surface of the first physical object in accordance with the increased brightness value of the visual representation of the voice-based virtual assistant. The method of any one of claims 67 to 69.

71. Modifying the visual appearance of at least a portion of the first physical surface of the first physical object according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant; 71. The method of any one of claims 67 to 70, comprising changing a color of a portion of the three-dimensional scene corresponding to at least a portion of the first physical surface of the first physical object according to the changed color value of the visual representation of the voice-based virtual assistant.

72. In response to detecting the request to activate the voice-based virtual assistant, 72. The method according to any one of claims 67 to 71, further comprising modifying the visual appearance of the surrounding area of ​​the portion of the three-dimensional scene currently displayed via the display generation component according to the first set of values ​​of the first display characteristic of the visual representation of the voice-based virtual assistant.

73. Detecting the request to activate the voice-based virtual assistant 73. The method of any one of claims 67 to 72, comprising detecting the gaze input that satisfies a first criterion, the first criterion including a criterion that is met when the gaze input is directed at a position corresponding to the visual representation of the voice-based virtual assistant in the three-dimensional scene.

74. In response to detecting the request to activate the voice-based virtual assistant, displaying the visual representation of the voice-based virtual assistant within the three-dimensional scene, 74. The method of any one of claims 67 to 73, comprising moving the visual representation of the voice-based virtual assistant from a first position to a second position within the three-dimensional scene.

75. In response to detecting the request to activate the voice-based virtual assistant, displaying the visual representation of the voice-based virtual assistant within the three-dimensional scene, 75. The method of any one of claims 67 to 74, comprising resizing the visual representation of the voice-based virtual assistant within the three-dimensional scene.

76. In response to detecting the request to activate the voice-based virtual assistant, displaying the visual representation of the voice-based virtual assistant within the three-dimensional scene, 76. The method of any one of claims 67 to 75, comprising changing the first set of values ​​for the first display characteristic of the visual representation of the voice-based virtual assistant in the three-dimensional scene.

77. In response to detecting the request to activate the voice-based virtual assistant, the method includes changing values ​​of a second set of the first display characteristics of a portion of the three-dimensional scene at a position surrounding the visual representation of the voice-based virtual assistant.

78. Detecting the request to activate the voice-based virtual assistant The first criterion includes a criterion that is met when a gaze input is directed at a position corresponding to the visual representation of the voice-based virtual assistant in the three-dimensional scene, and the second criterion includes a criterion that is met when a voice input is detected while the gaze input satisfies the first criterion. Detecting the gaze input that satisfies the first criterion and the voice input that satisfies the second criterion. The method of any one of claims 67 to 77.

79. Detecting a first input corresponding to a request for the voice-based assistant to perform a first operation while displaying the visual representation of the voice-based virtual assistant in the three-dimensional scene, the first input spanning a first time period; In response to detecting the first input, changing the first display characteristic of the visual representation of the voice-based virtual assistant during the first input from the first set of values ​​to a second set of values ​​distinct from the first set of values; 79. The method of any one of claims 67 to 78, comprising:

80. Detecting a second input corresponding to a request for the voice-based assistant to perform a second action while displaying the visual representation of the voice-based virtual assistant in the three-dimensional scene; In response to detecting the second input, commencing execution of the second operation; and During execution of the second operation, changing the first display characteristic of the visual representation of the voice-based virtual assistant from the first set of values ​​to a third set of values ​​distinct from the first set of values; 80. The method of any one of claims 67 to 79, comprising:

81. 81. The method of any one of claims 67 to 80, wherein the visual representation of the voice-based virtual assistant has a predetermined location in a three-dimensional environment including the first virtual object and the first physical object, and the request to activate the voice-based virtual assistant includes an input directed to the predetermined location.

82. Detecting the movement of the display generation component while displaying the visual representation of the voice-based virtual assistant in the three-dimensional scene including the first physical object and the first virtual object via the display generation component, wherein the visual representation of the voice-based virtual assistant is displayed at a first position in the three-dimensional scene corresponding to a first portion of the display generation component at the start of the movement of the display generation component; In response to detecting the movement of the display generation component, moving the visual representation of the voice-based virtual assistant to a second position in the three-dimensional scene different from the first position, the second position being updated in accordance with the movement of the display generation component to correspond to the first portion of the display generation component; 82. The method of any one of claims 67 to 81, comprising:

83. Detecting the movement of the display generation component while displaying the visual representation of the voice-based virtual assistant in the three-dimensional scene including the first physical object and the first virtual object via the display generation component, wherein the visual representation of the voice-based virtual assistant is displayed at a first position in the three-dimensional scene relative to the first virtual object and the first physical object at the start of the movement of the display generation component; In response to detecting the movement of the display generation component, While maintaining the visual representation of the voice-based virtual assistant at the first position within the three-dimensional scene, modifying the viewpoint of the display generation component relative to the three-dimensional scene in accordance with the movement of the display generation component; 82. The method of any one of claims 67 to 81, comprising:

84. 84. A computer readable storage medium storing executable instructions that, when executed by a computer system having one or more processors and a display generating component, cause the computer system to perform the method of any one of claims 67 to 83.

85. 1. A computer system comprising: one or more processors; A display generating component; a memory storing one or more programs configured to be executed by the one or more processors; wherein the one or more programs contain instructions for performing the method of any one of claims 67 to 83. A computer system comprising:

86. 1. A computer system having one or more processors and a display generation component, comprising:

84. A computer system comprising means for carrying out the method of any one of claims 67 to 83.

87. 1. An information processing device for use in a computer system having one or more processors and a display generation component, comprising:

84. An information processing device comprising means for carrying out a method according to any one of claims 67 to 83.

Images