System for, method of, and graphical user interface for mutual interaction with augmented reality environment and virtual reality environments

Abstract

To provide mutual interaction with augmented reality environment and virtual reality environment.SOLUTION: A computer system simultaneously displays, in an augmented reality environment, at least a part of representation of visual fields of one or more cameras having corresponding physical objects and a corresponding virtual user interface object at corresponding positions in the virtual user interface determined based on the positions of the corresponding physical objects in the visual field. While detecting any input at the position corresponding to the displayed corresponding virtual user interface objects, in response to detection of movement of an input to the corresponding physical objects in the one or more visual fields of the cameras, the system adjusts appearance of the corresponding virtual user interface objects in accordance with magnitude of the movement of the input to the corresponding physical objects.SELECTED DRAWING: Figure 5A1

Description

[Technical Field]

[0001] This application relates generally to computer systems for virtual / augmented reality, including but not limited to electronic devices for interacting with augmented reality and virtual reality environments. [Background technology]

[0002] The development of computer systems for virtual / augmented reality has progressed significantly in recent years. Exemplary virtual / augmented reality environments include at least some virtual elements that replace or augment the physical world. Interaction with virtual / augmented reality environments involves the use of input devices, such as touch-sensitive surfaces of computer systems and other electronic computing devices. Exemplary touch-sensitive surfaces include touchpads, touch-sensitive remote controls, and touchscreen displays. These surfaces are used to manipulate user interfaces and objects therein on the display. Exemplary user interface objects include digital images, video, text, icons, and control elements such as buttons and other graphics.

[0003] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., augmented reality, mixed reality, and virtual reality environments) are cumbersome, inefficient, and limited. For example, using a series of inputs to select one or more user interface objects (e.g., one or more virtual elements in a virtual / augmented reality environment) and perform one or more actions related to the selected user interface objects is tedious, creates significant cognitive burden for the user, and detracts from the experience of the virtual / augmented reality environment. Additionally, these methods take longer than necessary, thereby wasting energy. The latter problem is particularly acute in battery-operated devices. Summary of the Invention

[0004] Thus, there is a need for computer systems with improved methods and interfaces for interacting with augmented and virtual reality environments. Such methods and interfaces optionally complement or replace conventional methods for interacting with augmented and virtual reality environments. Such methods and interfaces reduce the number, range, and / or type of inputs from a user, creating a more efficient human-machine interface. For battery-operated devices, such methods and interfaces conserve power and extend the time between battery charges.

[0005] The above drawbacks and other problems associated with user interfaces for virtual / augmented reality are reduced or eliminated by the disclosed computer system. In some embodiments, the computer system comprises a desktop computer. In some embodiments, the computer system is portable (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system comprises a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the computer system has (and / or is in communication with) a touchpad. In some embodiments, the computer system has (and / or is in communication with) a touch-sensitive display (also known as a "touch screen" or "touchscreen display"). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing multiple functions. In some embodiments, a user interacts with the GUI, in part, through contacts and gestures with a stylus and / or fingers on a touch-sensitive surface. In some embodiments, the functionality optionally includes game playing, image editing, drawing, presenting, word processing, spreadsheet creation, telephony, video conferencing, email, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note taking, and / or digital video playback, and executable instructions to perform those functionality are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0006] According to some embodiments, a method is implemented on a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying representations of at least a portion of the field of view of one or more cameras including a corresponding physical object, the representation updating as content of the field of view of the one or more cameras changes, and a corresponding virtual user interface object at a corresponding position within the representation of the field of view of the one or more cameras, the corresponding virtual user interface object having a position determined based on the corresponding physical object within the field of view of the one or more cameras. The method also includes detecting an input at a position corresponding to the corresponding virtual user interface object while displaying the augmented reality environment. The method further includes detecting, while continuing to detect the input, movement of the input relative to the corresponding physical object within the field of view of the one or more cameras, and, in response to detecting the movement of the input relative to the corresponding physical object within the field of view of the one or more cameras, adjusting an appearance of the corresponding virtual user interface object according to a magnitude of the movement of the input relative to the corresponding physical object.

[0007] According to some embodiments, a method is implemented on a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying representations of at least a portion of the field of view of the one or more cameras including corresponding physical objects, the representations updating as content of the field of view of the one or more cameras changes, and corresponding virtual user interface objects at corresponding positions within the representations of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras. The method also includes detecting, while displaying the augmented reality environment, an input that modifies a virtual environment setting of the augmented reality environment. In response to detecting the input that modifies the virtual environment setting, the method further includes adjusting an appearance of the corresponding virtual user interface object according to changes made to the virtual environment setting of the augmented reality environment and applying a filter to at least a portion of the representations of the field of view of the one or more cameras, the filter being selected based on the changes made to the virtual environment setting.

[0008] According to some embodiments, a method is implemented on a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying representations of at least a portion of the field of view of the one or more cameras including corresponding physical objects, the representations updating as content of the field of view of the one or more cameras changes, and a first virtual user interface object in a virtual model displayed at a corresponding position within the representation of the field of view of the one or more cameras, the first virtual user interface object having a position determined based on the corresponding physical objects in the field of view of the one or more cameras. The method also includes detecting, while displaying the augmented reality environment, a first input corresponding to a selection of the first virtual user interface object, and displaying a simulated view of the virtual model from the perspective of the first virtual user interface object in the virtual model in response to detecting the first input corresponding to the selection of the first virtual user interface object.

[0009] According to some embodiments, a method is executed on a computer system having a display generation component and an input device. The method includes displaying a first virtual user interface object within a virtual three-dimensional space via the display generation component. The method also includes, while displaying the first virtual user interface object within the virtual three-dimensional space, detecting a first input via the input device, the first input including a selection of a corresponding portion of the first virtual user interface object and a movement of the first input in two dimensions. In response to detecting the first input including the movement of the first input in two dimensions, and in accordance with a determination that the corresponding portion of the first virtual user interface object is a first portion of the first virtual user interface object, adjusting an appearance of the first virtual user interface object in a first direction determined based on the movement of the first input in two dimensions and the selected first portion of the first virtual user interface object, wherein the adjustment of the first virtual user interface object in the first direction is constrained to movement within a first set of two dimensions of the virtual three-dimensional space; and and adjusting an appearance of the first virtual user interface object in a second direction different from the first direction in accordance with determining that the corresponding portion of the first input is a second portion of the first virtual user interface object separate from the first portion of the first virtual user interface object, wherein the second direction is determined based on movement of the first input in two dimensions and the selected second portion of the first virtual user interface object, and the adjusting of the first virtual user interface object in the second direction is constrained to movement within a second set of two dimensions of the virtual three-dimensional space different from the first set of two dimensions of the virtual three-dimensional space.

[0010] According to some embodiments, a method is performed on a computer system having a display generation component, one or more orientation sensors, and an input device. The method includes displaying, via the display generation component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, where displaying the simulated environment in the first viewing mode includes displaying a first virtual user interface object in a virtual model displayed at a first corresponding position in the simulated environment associated with the physical environment of the computer system. The method also includes detecting, while displaying the simulated environment, a first change in orientation of at least a portion of the computer system relative to the physical environment via the one or more orientation sensors, and, in response to detecting the first change in orientation of the portion of the computer system, modifying an appearance of the first virtual user interface object in the virtual model to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment. The method further includes, after modifying the appearance of the first virtual user interface object based on a first change in a posture of a portion of the computer system, detecting, via the input device, a first gesture corresponding to an interaction with the simulated environment, and, in response to detecting the first gesture corresponding to the interaction with the simulated environment, performing an action within the simulated environment corresponding to the first gesture.Additionally, the method includes detecting, after performing an action corresponding to the first gesture, a second change in attitude of the portion of the computer system relative to the physical environment via one or more attitude sensors; and, in response to detecting the second change in attitude of the portion of the computer system, transitioning from displaying the simulated environment including the virtual model in a first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode in accordance with determining that the first gesture satisfied mode change criteria, the mode change criteria including a requirement that the first gesture correspond to an input that changes a spatial parameter of the simulated environment relative to the physical environment, wherein the virtual model is displayed in the simulated environment in the second viewing mode. displaying the virtual model includes refraining from changing an appearance of the first virtual user interface object to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; and continuing to display the first virtual model within the simulated environment in the first viewing mode in accordance with a determination that the first gesture did not satisfy the mode change criterion, wherein displaying the virtual model in the first viewing mode includes changing an appearance of the first virtual user interface object within the virtual model in response to a second change in an attitude of the portion of the computer system relative to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment.

[0011] According to some embodiments, a method is performed on a first computer system having a first display generation component, one or more first orientation sensors, and a first input device, the method including displaying, via the first display generation component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, where displaying the simulated environment includes simultaneously displaying a first virtual user interface object in a virtual model displayed at a corresponding position in the simulated environment associated with the first physical environment of the first computer system and a visual indication of a second computer system's viewing perspective of the simulated environment, the second computer system having a second display generation component, one or more second orientation sensors, and a second input device, and where the second computer system is a computer system displaying, via the second display generation component of the second computer system, a view of the simulated environment oriented relative to the second physical environment of the second computer system. The method also includes, while displaying the simulated environment via the first display generation component of the first computer system, detecting a change in a viewing viewpoint of the second computer system of the simulated environment based on a change in an attitude of the portion of the second computer system relative to a second physical environment of the second computer system. The method further includes, in response to detecting a change in a viewing viewpoint of the second computer system of the simulated environment based on a change in an attitude of the portion of the second computer system relative to the physical environment of the second computer system, updating a visual indication of a viewing viewpoint of the second computer system of the simulated environment displayed via the first display generation component of the first computer system in accordance with the change in the viewing viewpoint of the second computer system of the simulated environment.

[0012] According to some embodiments, a method is executed on a computer system having a display generation component, one or more orientation sensors, and an input device. The method includes displaying a simulated environment via the display generation component. The method also includes, while displaying the simulated environment, detecting a first input via the input device directed at a corresponding location within the simulated environment. The method also includes, in response to detecting the first input directed at the corresponding location within the simulated environment, displaying an insertion cursor at the first location in accordance with a determination that the first input is of a first input type and that the first input is detected at a first location within the simulated environment other than a current location of the insertion cursor within the simulated environment; and inserting a first object at the second location and moving the insertion cursor to a third location over the first object in accordance with a determination that the first input is of the first input type and that the first input is detected at a second location within the simulated environment corresponding to the current location of the insertion cursor.

[0013] According to some embodiments, a method is performed on a computer system having a display generation component, one or more cameras, and one or more orientation sensors. The method includes displaying, via the display generation component, an augmented reality environment, where displaying the augmented reality environment includes simultaneously displaying representations of at least a portion of the field of view of the one or more cameras, the representations including physical objects, and virtual user interface objects at corresponding positions within the representations of the field of view of the one or more cameras, the virtual user interface objects being determined based on fixed spatial relationships between the virtual user interface objects and the physical objects included in the representations of the field of view of the one or more cameras. The method also includes detecting, while displaying the augmented reality environment, a first change in orientation of at least a portion of the computer system relative to a physical environment of the computer system via the one or more orientation sensors.The method also includes, in response to detecting a first change in an attitude of the portion of the computer system relative to the computer system's physical environment, updating the augmented reality environment according to the first change in attitude of the portion of the computer system; wherein, in accordance with determining that the augmented reality environment is displayed in a non-stable operating mode, updating the augmented reality environment according to the first change in attitude of the portion of the computer system includes updating representations of the portion of the field of view of the one or more cameras by a first adjustment amount based on the first change in attitude of the portion of the computer system relative to the computer system's physical environment and updating corresponding positions of virtual user interface objects to positions selected to maintain a fixed spatial relationship between the virtual user interface objects and physical objects included in the representations of the field of view of the one or more cameras; and, in accordance with determining that the augmented reality environment is displayed in a stable operating mode, updating the augmented reality environment according to the first change in attitude of the portion of the computer system includes updating representations of the portion of the field of view of the one or more cameras by a second adjustment amount based on the first change in attitude of the portion of the computer system relative to the computer system's physical environment and that is less than the first adjustment amount and updating corresponding positions of the virtual user interface objects to positions selected to maintain a fixed spatial relationship between the virtual user interface objects and physical objects included in the representations of the field of view of the one or more cameras.

[0014] According to some embodiments, a computer system includes (and / or is in communication with) a display generation component (e.g., a display, projector, heads-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of content within the camera's field of view, and optionally generates a video output including one or more streams of image frames capturing the content within the camera's field of view), one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touchscreen display that also functions as a display generation component, a mouse, joystick, wand controller, and / or a camera that tracks the position of one or more features of a user, such as the user's hands), optionally one or more posture sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, one or more processors, and a memory that stores one or more programs, the one or more programs configured to be executed by the one or more processors, and the one or more programs including instructions for performing or causing to be performed any of the operations of the methods described herein. According to some embodiments, a computer-readable storage medium has stored therein instructions that, when executed by a computer system including (and / or in communication with) a display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators, cause the computer system to perform or cause the performance of any of the operations of the methods described herein.According to some embodiments, a graphical user interface on a computer system including (and / or in communication with) a display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, memory, and one or more processors running one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, and the elements are updated in response to input as described in any of the methods described herein. According to some embodiments, the computer system includes (and / or is in communication with) the display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, and means for performing or causing to be performed the operations of any of the methods described herein. According to some embodiments, an information processing device for use in a computer system including (and / or in communication with) a display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators, includes means for performing or causing to be performed the operations of any of the methods described herein.

[0015] Thus, a computer system having (and / or in communication with) a display generation component, one or more cameras, one or more input devices, optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators, provides improved methods and interfaces for interacting with augmented and virtual reality environments, thereby increasing the effectiveness, efficiency, and user satisfaction of such computer systems. Such methods and interfaces can complement or replace conventional methods for interacting with augmented and virtual reality environments. [Brief explanation of the drawings]

[0016] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description of the Invention in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0017] [Figure 1A] 1 is a block diagram illustrating a portable multifunction device having a touch-sensitive display in accordance with some embodiments.

[0018] [Figure 1B] FIG. 2 is a block diagram illustrating exemplary components for event processing according to some embodiments.

[0019] [Figure 2] 1 illustrates a portable multifunction device with a touch screen according to some embodiments.

[0020] [Figure 3A] FIG. 1 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.

[0021] [Figure 3B]FIG. 1 is a block diagram of an exemplary computer system according to some embodiments. [Figure 3C] FIG. 1 is a block diagram of an exemplary computer system according to some embodiments.

[0022] [Figure 4A] 1 illustrates an exemplary user interface for a menu of applications on a portable multifunction device in accordance with some embodiments.

[0023] [Figure 4B] 1 illustrates an exemplary user interface for a multifunction device having a touch-sensitive surface separate from a display in accordance with some embodiments.

[0024] [Figure 4C] 1 illustrates an example of a dynamic intensity threshold, according to some embodiments. [Figure 4D] 1 illustrates an example of a dynamic intensity threshold, according to some embodiments. [Figure 4E] 1 illustrates an example of a dynamic intensity threshold, according to some embodiments.

[0025] [Figure 5A1] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A2] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A3] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A4] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A5] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A6] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A7] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A8]1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A9] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A10] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A11] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A12] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A13]1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A14] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A15] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A16] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A17] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A18]1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A19] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A20] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A21] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A22] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A23]1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A24] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A25] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A26] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A27] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A28]1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A29] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A30] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A31] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A32] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A33]1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A34] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A35] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A36] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A37] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A38]1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A39] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. [Figure 5A40] 1 illustrates an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments.

[0026] [Figure 5B1] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B2] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B3] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B4] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B5]1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B6] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B7] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B8] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B9] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B10] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B11] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B12] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B13] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B14] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B15] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B16]1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B17] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B18] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B19] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B20] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B21] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B22] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B23] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B24] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B25] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B26] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B27]1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B28] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B29] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B30] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B31] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B32] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B33] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B34] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B35] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B36] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B37] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B38]1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B39] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B40] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 5B41] 1 illustrates an example system and user interface for three-dimensional manipulation of virtual user interface objects, according to some embodiments.

[0027] [Figure 5C1] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C2] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C3] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C4] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C5] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C6] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C7]1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C8] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C9] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C10] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C11] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C12] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C13] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C14] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C15] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C16] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C17] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C18] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C19] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C20] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C21] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C22] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C23] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C24] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C25] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C26] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C27] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C28] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C29] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 5C30] 1 illustrates an example system and user interface for transitioning between viewing modes of a displayed simulated environment, according to some embodiments.

[0028] [Figure 5D1] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D2] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D3a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D3b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D3c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D4a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D4b]1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D4c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D5a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D5b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D5c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D6a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D6b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D6c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D7a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D7b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D7c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D8a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D8b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D8c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D9a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D9b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D9c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D10a]1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D10b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D10c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D11a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D11b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D11c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D12a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D12b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D13a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D13b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D13c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D13d] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D14a] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D14b] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. [Figure 5D14c] 1 illustrates an example system and user interface for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments.

[0029] [Figure 5E1] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E2] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E3] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E4]1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E5] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E6] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E7] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E8] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E9] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E10] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E11] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E12] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E13] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E14] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E15] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E16] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E17]1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E18] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E19] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E20] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E21] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E22] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E23] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E24] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E25] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E26] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E27] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E28] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E29] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E30]1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E31] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments. [Figure 5E32] 1 illustrates an example system and user interface for insertion cursor placement according to some embodiments.

[0030] [Figure 5F1] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F2] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F3a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F3b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F4a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F4b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F5a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F5b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F6a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F6b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F7a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F7b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F8a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F8b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F9a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F9b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F10a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F10b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F11a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F11b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F12a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F12b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F13a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F13b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F14a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F14b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F15a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F15b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F16a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F16b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F17a] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 5F17b] 1 illustrates an example system and user interface for displaying an augmented reality environment in a stable operating mode, according to some embodiments.

[0031] [Figure 6A]FIG. 1 is a flow diagram of a process for adjusting the appearance of a virtual user interface object in an augmented reality environment, according to some embodiments. [Figure 6B] FIG. 1 is a flow diagram of a process for adjusting the appearance of a virtual user interface object in an augmented reality environment, according to some embodiments. [Figure 6C] FIG. 1 is a flow diagram of a process for adjusting the appearance of a virtual user interface object in an augmented reality environment, according to some embodiments. [Figure 6D] FIG. 1 is a flow diagram of a process for adjusting the appearance of a virtual user interface object in an augmented reality environment, according to some embodiments.

[0032] [Figure 7A] FIG. 1 is a flow diagram of a process for applying filters to live images captured by one or more cameras of a computer system in an augmented reality environment, according to some embodiments. [Figure 7B] FIG. 1 is a flow diagram of a process for applying filters to live images captured by one or more cameras of a computer system in an augmented reality environment, according to some embodiments. [Figure 7C] FIG. 1 is a flow diagram of a process for applying filters to live images captured by one or more cameras of a computer system in an augmented reality environment, according to some embodiments.

[0033] [Figure 8A] FIG. 1 is a flow diagram of a process for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments. [Figure 8B] FIG. 1 is a flow diagram of a process for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments. [Figure 8C] FIG. 1 is a flow diagram of a process for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments.

[0034] [Figure 9A] FIG. 1 is a flow diagram of a process for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 9B] FIG. 1 is a flow diagram of a process for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 9C] FIG. 1 is a flow diagram of a process for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 9D] FIG. 1 is a flow diagram of a process for three-dimensional manipulation of virtual user interface objects, according to some embodiments. [Figure 9E] FIG. 1 is a flow diagram of a process for three-dimensional manipulation of virtual user interface objects, according to some embodiments.

[0035] [Figure 10A] FIG. 1 is a flow diagram of a process for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 10B] FIG. 1 is a flow diagram of a process for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 10C] FIG. 1 is a flow diagram of a process for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 10D] FIG. 1 is a flow diagram of a process for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. [Figure 10E]FIG. 1 is a flow diagram of a process for transitioning between viewing modes of a displayed simulated environment, according to some embodiments.

[0036] [Figure 11A] FIG. 1 is a flow diagram of a process for updating an indication of a viewing viewpoint of a second computer system within a simulated environment displayed by a first computer system, according to some embodiments. [Figure 11B] FIG. 1 is a flow diagram of a process for updating an indication of a viewing viewpoint of a second computer system within a simulated environment displayed by a first computer system, according to some embodiments. [Figure 11C] FIG. 1 is a flow diagram of a process for updating an indication of a viewing viewpoint of a second computer system within a simulated environment displayed by a first computer system, according to some embodiments.

[0037] [Figure 12A] FIG. 10 is a flow diagram of a process for placement of an insertion cursor according to some embodiments. [Figure 12B] FIG. 10 is a flow diagram of a process for placement of an insertion cursor according to some embodiments. [Figure 12C] FIG. 10 is a flow diagram of a process for placement of an insertion cursor according to some embodiments. [Figure 12D] FIG. 10 is a flow diagram of a process for placement of an insertion cursor according to some embodiments.

[0038] [Figure 13A] FIG. 1 is a flow diagram of a process for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 13B] FIG. 1 is a flow diagram of a process for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 13C]FIG. 1 is a flow diagram of a process for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 13D] FIG. 1 is a flow diagram of a process for displaying an augmented reality environment in a stable operating mode, according to some embodiments. [Figure 13E] FIG. 1 is a flow diagram of a process for displaying an augmented reality environment in a stable operating mode, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0039] An augmented reality environment is an environment in which reality is augmented with supplemental information that provides a user with additional information not available in the physical world. Conventional methods of interacting with an augmented reality environment (e.g., to access the supplemental information) often require multiple separate inputs (e.g., a series of gestures and button presses) to achieve an intended result. Furthermore, conventional input methods are often limited in scope (e.g., by the size of a computer system's touch-sensitive display). Embodiments herein provide an intuitive way for a user to interact with an augmented reality environment (e.g., by adjusting the appearance of virtual user interface objects based on a combination of movement of the computer system and movement of contacts on an input device (e.g., a touchscreen display) of the computer system, and by applying filters selected based on the virtual environment settings of the augmented reality environment in real time to live images captured by one or more cameras of the computer system).

[0040] Additionally, traditional interactions with virtual / augmented reality environments are generally limited to a single perspective (e.g., from the perspective of a user wearing / holding a device). Embodiments herein provide a more immersive and intuitive way to experience a virtual / augmented reality environment by presenting a simulated view of a virtual model (e.g., of a physical object) within a virtual reality environment from the perspective of a virtual user interface object (e.g., from the perspective of a car or person within the augmented reality environment).

[0041] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways, such as making it easier to display an augmented reality environment and adjust the appearance of the augmented reality environment and / or objects therein in response to different inputs, to transition between viewing a virtual model in the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, and to manipulate virtual user interface objects in three dimensions.

[0042] 1A-1B, 2, and 3A-3C provide a description of an exemplary device. Figures 4A-4B, 5A1-5A40, 5B1-5B41, 5C1-5C30, 5D1-5D14, 5E1-5E32, and 5F1-5F17 illustrate example systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to some embodiments. Figures 4A-4B, 5A1-5A40, 5B1-5B41, 5C1-5C30, 5D1-5D14, 5E1-5E32, and 5F1-5F17 illustrate example user interfaces for interacting with augmented reality and virtual reality environments, according to some embodiments, including displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects in the augmented reality environment in response to different inputs, transitioning between viewing a virtual model in the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, and three-dimensional manipulation of virtual user interface objects. Figures 6A-6D illustrate a flow diagram of a method for adjusting the appearance of a virtual user interface object in an augmented reality environment, according to some embodiments. 7A-7C show a flow diagram of a method for applying filters to live images captured by one or more cameras of a computer system in an augmented reality environment, according to some embodiments. 8A-8C show a flow diagram of a method for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments. 9A-9E show a flow diagram of a method for three-dimensional manipulation of virtual user interface objects, according to some embodiments. 10A-10E show a flow diagram of a method for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. 11A-11C show a flow diagram of a method for updating an indication of a viewing viewpoint of a second computer system in a simulated environment displayed by a first computer system, according to some embodiments. 12A-12D show a flow diagram of a method for insertion cursor placement, according to some embodiments.13A-13E illustrate a flow diagram of a method for displaying an augmented reality environment in a stable operating mode, according to some embodiments.

[0043] The user interfaces of Figures 5A1-5A40, 5B1-5B41, 5C1-5C30, 5D1-5D14, 5E1-5E32, and 5F1-5F17 are used to illustrate the processes of Figures 6A-6D, 7A-7C, 8A-8C, 9A-9E, 10A-10E, 11A-11C, 12A-12D, and 13A-13E. Exemplary Devices

[0044] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments being described. However, it will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0045] In this specification, terms such as "first," "second," etc. are used to describe various elements in some examples, but it will be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first contact can be referred to as a second contact, and similarly, a second contact can be referred to as a first contact, without departing from the scope of the various embodiments being described. Although a first contact and a second contact are both contacts, they are not the same contact unless the context clearly dictates otherwise.

[0046] The terminology used in the description of the various embodiments set forth herein is for the purpose of describing particular embodiments only and is not intended to be limiting. In the description of the various embodiments set forth and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. Also, as used herein, the term "and / or" should be understood to refer to and include any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and / or "comprising," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0047] As used herein, the term "if" is optionally interpreted to mean "when," "upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]," depending on the context.

[0048] A computer system for virtual / augmented reality includes an electronic device that generates a virtual / augmented reality environment. Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer that has a touch-sensitive surface (e.g., touchscreen displays and / or touchpads) and also includes or is in communication with one or more cameras.

[0049] In the following description, a computer system is described that includes an electronic device having (and / or in communication with) a display and a touch-sensitive surface. However, it should be understood that the computer system optionally includes one or more other physical user interface devices, such as a physical keyboard, a mouse, a joystick, a wand controller, and / or a camera that tracks the position of one or more features of a user, such as the user's hands.

[0050] The device typically supports a variety of applications, such as one or more of gaming applications, note-taking applications, drawing applications, presentation applications, word processing applications, spreadsheet applications, phone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0051] Various applications running on the device optionally use at least one common physical user-interface device, such as a touch-sensitive surface. One or more features of the touch-sensitive surface and corresponding information displayed by the device are optionally adjusted and / or changed for each application and / or within each application. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports various applications with user interfaces that are intuitive and transparent to the user.

[0052] Attention now turns to embodiments of portable devices with touch-sensitive displays. FIG. 1A is a block diagram illustrating portable multifunction device 100 having touch-sensitive display system 112, according to some embodiments. Touch-sensitive display system 112 may conveniently be referred to as a "touch screen" or simply a touch-sensitive display. Device 100 includes memory 102 (optionally including one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input or control devices 116, and external port 124. Device 100 optionally includes one or more light sensors 164 (e.g., as part of one or more cameras). Device 100 optionally includes one or more intensity sensors 165 that detect the intensity of a contact on device 100 (e.g., a touch-sensitive surface, such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 163 that generate tactile output on device 100 (e.g., generate tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0053] As used herein and in the claims, the term “tactile output” refers to a physical displacement of a device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, that will be detected by a user with the user's sense of touch. For example, in a situation where a device or a component of a device is in contact with a touch-sensitive surface of a user (e.g., the fingers, palm, or other part of the user's hand), the tactile output produced by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical property of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a “downclick” or “upclick” of a physical actuator button. In some cases, a user feels a tactile sensation such as a “downclick” or “upclick” even when no movement of a physical actuator button associated with the touch-sensitive surface is physically pressed (e.g., displaced) by the user's action. As another example, movement of a touch-sensitive surface is optionally interpreted or perceived by a user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. While such user interpretation of touch depends on the user's personal sensory perception, there are many sensory perceptions of touch that are common to a majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "upclick," "downclick," "roughness"), unless otherwise specified, the generated tactile output corresponds to a physical displacement of the device, or a component of the device, that produces the described sensory perception for a typical (or average) user.Using tactile output to provide haptic feedback to the user improves usability of the device, makes the user-device interface more efficient (e.g., by helping the user provide appropriate inputs and reducing user errors when operating / interacting with the device), and additionally reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

[0054] It should be understood that device 100 is only one example of a portable multifunction device, and that device 100 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of its components. The various components shown in FIG. 1A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0055] Memory 102 optionally includes high-speed random access memory, and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as CPU(s) 120 and peripherals interface 118, is optionally controlled by memory controller 122.

[0056] A peripheral interface 118 may be used to couple input and output peripherals of the device with the CPU(s) 120 and memory 102. The one or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions and process data for the device 100.

[0057] In some embodiments, peripheral interface 118, CPU(s) 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0058] RF (radio frequency) circuitry 108 transmits and receives RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to or from electromagnetic signals and communicates with communication networks and other communication devices via electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, memory, etc. RF circuitry 108 optionally communicates wirelessly with networks, such as the Internet, also known as the World Wide Web (WWW), an intranet, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs), and with other devices. Radio options include Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), and code division multiple access (CDMA).Wireless technologies include, but are not limited to, standard IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n, voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP)), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), and Instant Messaging and Presence Services (IMP). The present invention may use any of a number of communication standards, protocols, and technologies, including, but not limited to, Intermediate Message Service (IMPS), and / or Short Message Service (SMS), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0059] Audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100. Audio circuit 110 receives audio data from peripherals interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves audible to humans. Audio circuit 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuit 110 converts the electrical signals into audio data and transmits the audio data to peripherals interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to memory 102 and / or RF circuit 108 by peripherals interface 118. In some embodiments, audio circuit 110 also includes a headset jack (e.g., 212, FIG. 2 ). The headset jack provides an interface between audio circuitry 110 and a detachable audio input / output peripheral, such as an output-only headphone or a headset with both an output (e.g., mono or binaural headphones) and an input (e.g., a microphone).

[0060] I / O subsystem 106 couples input / output peripherals on device 100, such as touch-sensitive display system 112 and other input or control devices 116, with peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, light sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from and send electrical signals to other input or control devices 116. Other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, input controller(s) 160 are optionally coupled to any (or none) of a keyboard, infrared port, USB port, stylus, and / or pointer device such as a mouse. The one or more buttons (e.g., 208, FIG. 2) optionally include up / down buttons for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., 206, FIG. 2).

[0061] Touch-sensitive display system 112 provides an input and output interface between the device and a user. Display controller 156 receives electrical signals from and / or sends electrical signals to touch-sensitive display system 112. Touch-sensitive display system 112 displays visual output to the user. This visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term "affordance" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to input directed towards the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, a button, a slider, an icon, a selectable menu item, a switch, a hyperlink, or other user interface control.

[0062] Touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and / or tactile contact. Touch-sensitive display system 112 and display controller 156 (along with any associated modules and / or instruction sets in memory 102) detect contacts (and any movement or breaking of contact) on touch-sensitive display system 112 and translate the detected contacts into interactions with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touch-sensitive display system 112. In some embodiments, the point of contact between touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0063] Touch-sensitive display system 112 optionally uses liquid crystal display (LCD), light emitting polymer display (LPD), or light emitting diode (LED) technology, although other display technologies are used in other embodiments. Touch-sensitive display system 112 and display controller 156 optionally detect contact and any movement or disruption thereof using any of a number of now-known or later-developed touch sensing technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with touch-sensitive display system 112. In some embodiments, projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, California.

[0064] Touch-sensitive display system 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or higher). A user optionally contacts touch-sensitive display system 112 using any suitable object or accessory, such as a stylus, finger, or the like. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on a touchscreen than that of a stylus. In some embodiments, the device translates coarse finger-based input into precise pointer / cursor positions or commands to perform actions desired by the user.

[0065] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touchscreen.

[0066] Device 100 also includes a power system 162 that provides power to the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, power failure detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of electrical power within a portable device.

[0067] Device 100 also optionally includes one or more light sensors 164 (e.g., as part of one or more cameras). FIG. 1A shows a light sensor coupled to light sensor controller 158 in I / O subsystem 106. Light sensor(s) 164 optionally include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Light sensor(s) 164 receive light from the environment, projected through one or more lenses, and convert the light into data representing an image. In conjunction with imaging module 143 (also called a camera module), light sensor(s) 164 optionally capture still images and / or video. In some embodiments, the light sensor(s) is located on the back of device 100, opposite touch-sensitive display system 112 on the front of the device, so that the touchscreen can be used as a viewfinder for still and / or video image acquisition. In some embodiments, another light sensor is placed on the front of the device so that an image of the user is captured (e.g., for a selfie, for a video conference while the user is viewing other video conference participants on the touchscreen, etc.).

[0068] Device 100 also optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor coupled to intensity sensor controller 159 in I / O subsystem 106. Contact intensity sensor(s) 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor(s) 165 receive contact intensity information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with or proximate to the touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of device 100, opposite touchscreen display system 112, which is located on the front of device 100.

[0069] Device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows proximity sensor 166 coupled to peripherals interface 118. Alternatively, proximity sensor 166 is coupled to input controller 160 in I / O subsystem 106. In some embodiments, the proximity sensor turns off and disables touch-sensitive display system 112 when the multifunction device is placed near a user's ear (e.g., when the user is making a phone call).

[0070] Device 100 also optionally includes one or more tactile output generators 163. FIG. 1A shows tactile output generators coupled to haptic feedback controller 161 in I / O subsystem 106. In some embodiments, tactile output generator(s) 163 include one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear movement, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on the device). Tactile output generator(s) 163 receive tactile feedback generation instructions from haptic feedback module 133 and generate tactile outputs on device 100 that can be sensed by a user of device 100. In some embodiments, at least one tactile output generator is located on or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112) and generates a tactile output, optionally by moving the touch-sensitive surface vertically (e.g., in / out of the surface of device 100) or horizontally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100, opposite touch-sensitive display system 112, which is located on the front of device 100.

[0071] Device 100 also optionally includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (e.g., as part of an inertial measurement unit (IMU)) to obtain information regarding the device's position (e.g., attitude). FIG. 1A shows sensors 167, 168, and 169 coupled to peripherals interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on a touchscreen display in portrait or landscape view based on analysis of data received from the one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) to obtain information regarding device 100's location.

[0072] In some embodiments, software components stored in memory 102 include an operating system 126, a communications module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and applications (or instruction sets) 136. Additionally, in some embodiments, as shown in Figures 1A and 3, memory 102 stores device / global internal state 157. Device / global internal state 157 includes one or more of: active application state, which indicates which applications, if any, are currently active; display state, which indicates which applications, views, or other information occupy various areas of touch-sensitive display system 112; sensor state, which includes information obtained from the device's various sensors and other input or control devices 116; and position and / or location information regarding the device's position and / or orientation.

[0073] Operating system 126 (e.g., an embedded operating system such as iOS, Android, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) includes various software components and / or drivers for controlling and managing overall system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0074] Communications module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by RF circuitry 108 and / or external port 124. External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) is adapted to couple to other devices directly or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector identical to, similar to, and / or compatible with the 30-pin connector used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external port is a Lightning connector identical to, similar to, and / or compatible with the Lightning connector used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external port is a USB Type-C connector that is the same as, similar to, and / or compatible with the USB Type-C connector used in some electronic devices from Apple Inc. of Cupertino, California.

[0075] Contact / motion module 130 optionally detects contact with touch-sensitive display system 112 (in cooperation with display controller 156) and with other touch-sensitive devices (e.g., a touchpad or physical click wheel). Contact / motion module 130 includes software components for performing various operations related to detecting contact (e.g., by a finger or stylus), such as determining if contact has occurred (e.g., detecting a finger-down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining if there is contact movement and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining if the contact has stopped (e.g., detecting a finger-up event or an interruption of contact). Contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact, as represented by the series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact. These actions are optionally applied to a single contact (e.g., a single finger contact or a stylus contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, contact / motion module 130 and display controller 156 detect contacts on the touchpad.

[0076] Contact / motion module 130 optionally detects gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different movements, timing, and / or strength of the detected contacts). Thus, gestures are optionally detected by detecting particular contact patterns. For example, detecting a finger tap gesture includes detecting a finger down event, followed by detecting a finger up (lift-off) event at the same location (or substantially the same location) as the finger down event (e.g., at the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger down event, followed by detecting one or more finger drag events, followed by detecting a finger up (lift-off) event. Similarly, taps, swipes, drags, and other gestures are optionally detected with respect to a stylus by detecting particular contact patterns with respect to the stylus.

[0077] In some embodiments, detecting a finger tap gesture depends on the length of time between detecting a finger-down event and detecting a finger-up event, but is not related to the intensity of the finger contact between detecting the finger-down event and detecting the finger-up event. In some embodiments, a tap gesture is detected according to determining that the length of time between the finger-down event and the finger-up event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the intensity of the finger contact during the tap meets a given intensity threshold (greater than a nominal contact-detection intensity threshold), such as a light or deep pressure intensity threshold. Thus, a finger tap gesture can satisfy certain input criteria that do not require the characteristic intensity of the contact to meet a given intensity threshold for the specific input criteria to be met. For clarity, finger contacts in a tap gesture generally need to meet a nominal contact-detection intensity threshold below which the contact is not detected in order to detect a finger-down event. A similar analysis applies to detecting a stylus tap gesture or other contact. In cases where the device is capable of detecting contact of a finger or stylus hovering over the touch-sensitive surface, the nominal contact-detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0078] In a similar manner, the same concepts apply to other types of gestures. For example, swipe gestures, pinch gestures, de-pinch gestures, and / or long press gestures are optionally detected based on meeting criteria that are either unrelated to the intensity of the contacts included in the gesture or that do not require the contacts performing the gesture to reach an intensity threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts, a pinch gesture is detected based on the movement of two or more contacts toward each other, a de-pinch gesture is detected based on the movement of two or more contacts away from each other, and a long press gesture is detected based on the duration of contact on the touch-sensitive surface that is less than a threshold amount of movement. Thus, a statement that a particular gesture recognition criterion does not require the intensity of a contact(s) to meet a corresponding intensity threshold in order for the particular gesture recognition criterion to be met means that the particular gesture recognition criterion can be met when the contact(s) in the gesture do not reach the corresponding intensity threshold, and can also be met in situations where one or more of the contacts in the gesture reach or exceed the corresponding intensity threshold. In some embodiments, a tap gesture is detected based on a determination that a finger-down event and a finger-up event are detected within a predetermined period of time, regardless of whether the contacts are above or below the respective intensity thresholds during the predetermined period, and a swipe gesture is detected based on a determination that a movement of the contact is greater than a predetermined magnitude, even if the contacts exceed the respective intensity thresholds at the end of the movement of the contacts. Even in implementations in which gesture detection is affected by the intensity of the contact performing the gesture (e.g., the device detects long presses more quickly when the intensity of the contact exceeds an intensity threshold, or the device is slow to detect tap inputs when the intensity of the contact is higher), detection of those gestures does not require the contact to reach a particular intensity threshold, as long as the criteria for recognizing the gesture can be met in situations in which the contact does not reach the particular intensity threshold (e.g., even if the amount of time required to recognize the gesture varies).

[0079] The contact intensity threshold, duration threshold, and movement threshold may, in some circumstances, be combined in various different combinations to create heuristics for distinguishing between two or more different gestures directed at the same input element or region, thereby enabling multiple different interactions with the same input element to provide a richer set of user interactions and responses. A statement that a particular set of gesture recognition criteria does not require that the intensity of a contact(s) meet a corresponding intensity threshold for that particular gesture recognition criterion to be satisfied does not preclude the simultaneous evaluation of other intensity-dependent gesture recognition criteria to identify other gestures with criteria that are satisfied when the gesture includes a contact having an intensity above the corresponding intensity threshold. For example, in some circumstances, a first gesture recognition criterion for a first gesture that does not require that the intensity of a contact(s) meet a corresponding intensity threshold for that first gesture recognition criterion to be satisfied competes with a second gesture recognition criterion for a second gesture that depends on the contact(s) reaching a corresponding intensity threshold. In such a competition, a gesture is optionally not recognized as satisfying the first gesture recognition criteria for the first gesture if the second gesture recognition criteria for the second gesture are satisfied first. For example, if the contact reaches the corresponding intensity threshold before moving a predetermined amount of movement, a deep press gesture is detected rather than a swipe gesture. Conversely, if the contact moves a predetermined amount of movement before reaching the corresponding intensity threshold, a swipe gesture is detected rather than a deep press gesture. Even in such a situation, the first gesture recognition criteria for the first gesture still do not require that the intensity of the contact(s) meet the corresponding intensity threshold for the first gesture recognition criteria to be satisfied, because if the contact remained below the corresponding intensity threshold until the end of the gesture (e.g., a swipe gesture with a contact that does not increase in intensity above the corresponding intensity threshold), the gesture would have been recognized by the first gesture recognition criteria as a swipe gesture.In this way, certain gesture recognition criteria that do not require the intensity of the contact(s) to meet a corresponding intensity threshold for the particular gesture recognition criterion to be satisfied are still dependent on the intensity of the contact with respect to an intensity threshold, in the sense that (A) in some circumstances, they ignore the intensity of the contact with respect to the intensity threshold (e.g., for a tap gesture), and / or (B) in some circumstances, the particular gesture recognition criterion (e.g., for a long press gesture) will not function if a competing set of intensity-dependent gesture recognition criteria (e.g., for a deep press gesture) recognizes an input as corresponding to an intensity-dependent gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input (e.g., for a long press gesture that competes with a deep press gesture for recognition).

[0080] In conjunction with accelerometer 167, gyroscope 168, and / or magnetometer 169, attitude module 131 optionally detects attitude information about the device, such as the device's attitude (e.g., roll, pitch, and / or yaw) within a particular coordinate system. Attitude module 131 includes software components for performing various operations related to detecting the device's position and detecting changes in the device's attitude.

[0081] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other display, including components for modifying the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, video, and animation.

[0082] In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 132 receives one or more codes specifying the graphics to be displayed, including coordinate data and other graphic characteristic data, as needed, from an application or the like, and then generates screen image data to output to display controller 156.

[0083] The haptic feedback module 133 includes various software components that generate instructions (e.g., instructions used by the haptic feedback controller 161) that use the tactile output generator(s) 163 to create tactile outputs at one or more locations on the device 100 in response to user interaction with the device 100.

[0084] Text input module 134 is optionally a component of graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0085] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for use in location-based calling, to the camera 143 as photo / video metadata, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and maps / navigation widgets).

[0086] Virtual / Augmented Reality Module 145 provides virtual and / or augmented reality logic to Application 136, which implements augmented reality functionality, and in some embodiments, virtual reality functionality. Virtual / Augmented Reality Module 145 facilitates the overlay of virtual content, such as virtual user interface objects, onto a representation of at least a portion of the field of view of one or more cameras. For example, with assistance from Virtual / Augmented Reality Module 145, the representation of at least a portion of the field of view of one or more cameras may include corresponding physical objects, and the virtual user interface objects may be displayed in a displayed augmented reality environment determined based on the corresponding physical objects in the field of view of the one or more cameras, or at a position within the virtual reality environment determined based on an orientation of at least a portion of a computer system (e.g., the orientation of a display device used to display a user interface to a user of the computer system).

[0087] Application 136 optionally includes the following modules (or sets of instructions), or a subset or superset thereof: • a contacts module 137 (sometimes called an address book or contact list); ●Telephone module 138, ●Video conferencing module 139, ● an email client module 140; ● Instant messaging (IM) module 141; ●Training support module 142, a camera module 143 for still and / or video images, ● Image management module 144; ● Browser module 147, ● Calendar module 148, a widget module 149, optionally including one or more of a weather widget 149-1, a stock price widget 149-2, a calculator widget 149-3, an alarm clock widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6; a widget creator module 150 for creating user-created widgets 149-6; ● Search module 151, • a video and music player module 152, optionally consisting of a video player module and a music player module; ● Memo module 153, Map module 154, and / or ●Online video module 155.

[0088] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice duplication.

[0089] Contacts module 137, along with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, includes executable instructions (e.g., stored in memory 102 or in application internal state 192 of contacts module 137 in memory 370) for managing an address book or contact list, including adding name(s) to the address book, removing name(s) from the address book, associating phone number(s), email address(es), street address(es), or other information with names, associating images with names, categorizing and sorting names, providing phone numbers and / or email addresses to initiate and / or facilitate communication by telephone 138, video conference 139, email 140, or IM 141, etc.

[0090] In cooperation with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 includes executable instructions for entering a series of characters corresponding to a telephone number, accessing one or more telephone numbers in address book 137, modifying an entered telephone number, dialing each telephone number, conducting a conversation, and disconnecting or hanging up when the conversation is completed. As noted above, wireless communication optionally uses any of a number of communication standards, protocols, and technologies.

[0091] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions to initiate, conduct, and terminate video conferences between a user and one or more other participants according to the user's commands.

[0092] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 contains executable instructions for composing, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 greatly facilitates the creation and sending of emails with still or video images captured by camera module 143.

[0093] In cooperation with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant message module 141 includes executable instructions for entering a series of characters corresponding to an instant message, modifying previously entered characters, sending each instant message (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for telephone-based instant messaging, or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, sent and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0094] In cooperation with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the video and music player module 152, the training support module 142 includes executable instructions to create workouts (e.g., with time, distance, and / or calorie burn goals), communicate with training sensors (in the sports device and smartwatch), receive training sensor data, calibrate sensors used to monitor workouts, select and play music for workouts, and display, store, and transmit workout data.

[0095] Camera module 143, along with touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, and image management module 144, includes executable instructions to capture still images or video (including video streams) and store them in memory 102, change characteristics of the still images or video, and / or delete the still images or video from memory 102.

[0096] Image management module 144, along with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or album), and storing still and / or video images.

[0097] Browser module 147, along with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, contains executable instructions for browsing the Internet according to user commands, including retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0098] Calendar module 148, along with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, includes executable instructions to create, display, modify, and store calendars and data associated with calendars (e.g., calendar entries, to-do lists, etc.) according to user commands.

[0099] Widget modules 149, along with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, are optionally mini-applications downloaded and used by a user (e.g., weather widget 149-1, stock quotes widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5), or mini-applications created by a user (e.g., user-created widget 149-6). In some embodiments, widgets include Hypertext Markup Language (HTML) files, Cascading Style Sheets (CSS) files, and JavaScript files. In some embodiments, widgets include Extensible Markup Language (XML) files and JavaScript files (e.g., Yahoo! Widgets).

[0100] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget creator module 150 contains executable instructions for creating widgets (e.g., turning user-specified portions of a web page into widgets).

[0101] In cooperation with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions to search memory 102 for text, music, sound, images, video, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with a user's instructions.

[0102] In cooperation with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 contains executable instructions that enable a user to download and play recorded music or other sound files stored in one or more file formats, such as MP3 or AAC files, as well as executable instructions to display, present, or otherwise play videos (e.g., on touch-sensitive display system 112 or on an external display connected wirelessly or via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).

[0103] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, notes module 153 contains executable instructions for creating and managing notes, to-do lists, and the like according to user commands.

[0104] In cooperation with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions to receive, display, modify, and store maps and data associated with maps (e.g., driving directions, data about businesses and other points of interest at or near a particular location, and other location-based data) in accordance with user commands.

[0105] In cooperation with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 contains executable instructions that enable a user to access, view, receive (e.g., by streaming and / or downloading), and play (e.g., on touchscreen 112 or on an external display connected wirelessly or via external port 124) online videos in one or more file formats, such as H.264, and send and otherwise manage emails with links to particular online videos. In some embodiments, instant messaging module 141 is used to send links to particular online videos, rather than email client module 140.

[0106] Each of the above-identified modules and applications corresponds to executable instruction sets that perform one or more of the functions described above and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). The modules (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules; thus, various subsets of the modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 102 optionally stores additional modules and data structures not described above.

[0107] In some embodiments, device 100 is a device in which operation of a predetermined set of functions on the device is performed exclusively through a touchscreen and / or touchpad. Using the touchscreen and / or touchpad as the primary input control device for operation of device 100 optionally reduces the number of physical input control devices (push buttons, dials, etc.) on device 100.

[0108] The set of predetermined functions performed only through the touchscreen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 to a main menu, home menu, or root menu from any user interface displayed on device 100. In such embodiments, a "menu button" is implemented using the touch-sensitive surface. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touch-sensitive surface.

[0109] 1B is a block diagram illustrating exemplary components for event processing, according to some embodiments. In some embodiments, memory 102 (in FIG. 1A) or 370 (in FIG. 3A) includes an event sorter 170 (e.g., within operating system 126) and a respective application 136-1 (e.g., any of applications 136, 137-155, 380-390 described above).

[0110] Event sorter 170 receives the event information and determines application 136-1 and application view 191 of application 136-1 to which the event information should be delivered. Event sorter 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192 that indicates the current application view(s) that are displayed on touch-sensitive display system 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event sorter 170 to determine which application(s) are currently active, and application internal state 192 is used by event sorter 170 to determine application view 191 to which the event information should be delivered.

[0111] In some embodiments, application internal state 192 includes additional information such as one or more of resume information used when application 136-1 resumes execution, user interface state information indicating information being displayed or ready to be displayed by application 136-1, a state queue that allows the user to return to a previous state or view of application 136-1, and a redo / undo queue of previous actions taken by the user.

[0112] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user's touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer(s) 167, and / or microphone 113 (via audio circuitry 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0113] In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or for longer than a predetermined period of time).

[0114] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognizer determination module 173 .

[0115] Hit view determination module 172 provides software procedures for determining where in one or more views a sub-event occurred when touch-sensitive display system 112 displays more than one view. A view consists of the controls and other elements that a user can see on the display.

[0116] Another aspect of a user interface associated with an application is the set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the respective application) in which the touch is detected optionally corresponds to a programmatic level within the application's programmatic or view hierarchy. For example, the lowest-level view in which the touch is detected is optionally referred to as the hit view, and the set of events that are recognized as appropriate inputs is optionally determined based at least in part on the hit view of the initial touch that initiates the touch gesture.

[0117] Hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should process the sub-events. In most situations, the hit view is the lowest-level view in which the initiating sub-event occurs (i.e., the first sub-event in a series of sub-events that form an event or potential event). Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0118] Active event recognizer determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that contain the physical location of the sub-events are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy remain actively participating views.

[0119] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores event information obtained by each event receiver module 182 in an event queue.

[0120] In some embodiments, operating system 126 includes event sorter 170. Alternatively, application 136-1 includes event sorter 170. In still other embodiments, event sorter 170 is a stand-alone module or part of another module stored in memory 102, such as contact / motion module 130.

[0121] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each containing instructions for processing touch events that occur within a respective view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, each application view 191 includes multiple event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handler 190 includes one or more of a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or invokes data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Instead, one or more of the application views 191 include one or more respective event handlers 190. Also, in some embodiments, one or more of the data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0122] Each event recognizer 180 receives event information (e.g., event data 179) from event sorter 170 and identifies an event from the event information. Event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event recognizer 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (optionally including sub-event delivery instructions).

[0123] The event receiver 182 receives event information from the event sorter 170. The event information includes information about a sub-event, for example, information about a touch or a touch movement. Depending on the sub-event, the event information also includes additional information, such as the position of the sub-event. When the sub-event involves a touch movement, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape or vice versa), and the event information includes corresponding information about the device's current orientation (also called the device's attitude).

[0124] The event comparator 184 compares the event information with predefined event or sub-event definitions and determines the event or sub-event, or determines or updates the state of the event or sub-event, based on the comparison. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes a definition of an event (e.g., a predetermined sequence of sub-events), such as Event 1 (187-1) and Event 2 (187-2). In some embodiments, the sub-events in Event 187 include, for example, a touch start, a touch end, a touch movement, a touch cessation, and multiple touches. In one example, the definition for Event 1 (187-1) is a double tap on a displayed object. The double tap includes, for example, a first touch (touch start) for a predetermined stage on the displayed object, a first lift-off (touch end) for a predetermined stage on the displayed object, a second touch (touch start) for a predetermined stage on the displayed object, and a second lift-off (touch end) for a predetermined stage. In another example, a definition of event 2 (187-2) is a drag on a displayed object. Drag includes, for example, a touch (or contact) of a predetermined degree on a displayed object, movement of the touch across the touch-sensitive display system 112, and lift-off of the touch (end of the touch). In some embodiments, the event also includes information about one or more associated event handlers 190.

[0125] In some embodiments, event definition 187 includes a definition of the event for each user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view in which three user interface objects are displayed on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, event comparator 184 uses the results of the hit test to determine which event handler 190 to activate. For example, event comparator 184 selects the event handler associated with the sub-event and object that triggers the hit test.

[0126] In some embodiments, each event 187 definition also includes a delay action that delays delivery of the event information until it is determined whether a set of sub-events corresponds to the event recognizer's event type.

[0127] If the respective event recognizer 180 determines that the sequence of sub-events does not match any event in the event definition 186, the respective event recognizer 180 enters an event disabled, event failed, or event finished state and thereafter ignores subsequent sub-events of the touch gesture. In this situation, any other event recognizers that remain active for the hit view continue to track and process sub-events of the ongoing touch gesture.

[0128] In some embodiments, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate to actively participating event recognizers how the event delivery system should perform sub-event delivery. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact with each other or how event recognizers are enabled to interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how sub-events are delivered to various levels in a view or programmatic hierarchy.

[0129] In some embodiments, each event recognizer 180 activates an event handler 190 associated with an event when one or more specific sub-events of the event are recognized. In some embodiments, each event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is separate from sending (and deferring sending) sub-events to the respective hit view. In some embodiments, the event recognizer 180 pops a flag associated with the recognized event, and the event handler 190 associated with that flag captures the flag and performs a predetermined process.

[0130] In some embodiments, the event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with a set of sub-events or to an actively participating view. The event handler associated with the set of sub-events or the actively participating view receives the event information and performs predetermined processing.

[0131] In some embodiments, data updater 176 creates and updates data used by application 136-1. For example, data updater 176 updates phone numbers used by contacts module 137 or stores video files used by video and music player module 152. In some embodiments, object updater 177 creates and updates objects used by application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on the touch-sensitive display.

[0132] In some embodiments, event handler(s) 190 include or have access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0133] It should be understood that the foregoing description of event processing of a user's touch on a touch-sensitive display also applies to other forms of user input for operating multifunction device 100 using input devices, not all of which are initiated on the touchscreen. For example, mouse movements and mouse button presses, contact movements such as tapping, dragging, scrolling on a touchpad, optionally coordinated with single or multiple keyboard presses or holds, pen stylus input, input based on real-time analysis of video images captured by one or more cameras, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof, optionally utilize as inputs corresponding to sub-events that define the recognized event.

[0134] FIG. 2 illustrates portable multifunction device 100 having a touchscreen (e.g., touch-sensitive display system 112, FIG. 1A ) according to some embodiments. The touchscreen optionally displays one or more graphics within user interface (UI) 200. In these embodiments, as well as embodiments described below, a user is enabled to select one or more of the graphics by making a gesture on the graphics, for example, with one or more fingers 202 (not drawn to scale) or one or more styluses 203 (not drawn to scale). In some embodiments, selection of one or more graphics is performed when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (left to right, right to left, upward and / or downward), and / or rolling of a finger in contact with device 100 (right to left, left to right, upward and / or downward). In some implementations or situations, accidental contact with a graphic does not select the graphic, for example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon optionally does not select the corresponding application.

[0135] Device 100 also optionally includes one or more physical buttons, such as a "home" or menu button 204. As mentioned above, menu button 204 is optionally used to navigate to any application 136 in a set of applications optionally running on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key within a GUI displayed on a touchscreen display.

[0136] In some embodiments, device 100 includes a touchscreen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for powering the device on / off and locking the device, volume control button(s) 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and an external docking / charging port 124. Push button 206 is optionally used to power the device on / off by pressing and holding the button down for a predetermined period of time, to lock the device by pressing and releasing the button before the predetermined period of time has elapsed, and / or to unlock the device or begin the unlocking process. In some embodiments, device 100 also accepts verbal input through microphone 113 to activate or deactivate some features. Device 100 also optionally includes one or more contact intensity sensors 165 that detect the intensity of a contact on touch-sensitive display system 112 and / or one or more tactile output generators 163 that generate a tactile output for a user of device 100.

[0137] FIG. 3A is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a gaming system, a laptop computer, a desktop computer, a tablet computer, a multimedia playback device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes referred to as a chipset) that interconnects and controls communication between system components. Device 300 includes input / output (I / O) interfaces 330, including a display 340, which is optionally a touchscreen display. I / O interface 330 optionally also includes a keyboard and / or mouse (or other pointing device) 350, as well as a touchpad 355, a tactile output generator 357 for generating tactile outputs on device 300 (e.g., similar to tactile output generator(s) 163 described above with reference to FIG. 1A ), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors similar to contact intensity sensor(s) 165 described above with reference to FIG. 1A ). Memory 370 includes high-speed random-access memory such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310.In some embodiments, the memory 370 stores programs, modules, and data structures similar to those stored in the memory 102 of the portable multifunctional device 100 (FIG. 1A), or subsets thereof. Further, the memory 370 optionally stores additional programs, modules, and data structures that do not exist in the memory 102 of the portable multifunctional device 100. For example, the memory 370 of the device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while the memory 102 of the portable multifunctional device 100 (FIG. 1A) optionally does not store those modules.

[0138] Each of the elements in FIG. 3A identified above is optionally stored in one or more of the previously mentioned memory devices. Each of the modules identified above corresponds to a set of instructions for performing the functions described above. The modules or programs (e.g., sets of instructions) identified above need not be implemented as separate software programs, procedures, or modules, and thus, various subsets of these modules are optionally combined or otherwise reconfigured in various embodiments. In some embodiments, the memory 370 optionally stores a subset of the modules and data structures identified above. Further, the memory 370 optionally stores additional modules and data structures not described above.

[0139] FIGS. 3B - 3D are block diagrams of an exemplary computer system 301 according to some embodiments.

[0140] In some embodiments, the computer system 301 includes and / or communicates with the following. input device(s) (302 and / or 307, e.g., a touch-sensitive surface such as a touch-sensitive remote control or a touchscreen display that also functions as a display generation component, a mouse, joystick, wand controller, and / or a camera that tracks the position of one or more features of the user, such as the user's hand); Virtual / Augmented Reality Logic 303 (e.g., Virtual / Augmented Reality Module 145); display generation component(s) (304 and / or 308, e.g., display, projector, heads-up display, etc.) for displaying virtual user interface elements to the user; ● camera(s) (e.g., 305 and / or 311) for capturing images of the device's field of view, e.g., images used to determine placement of virtual user interface elements, determine the device's pose, and / or display a portion of the physical environment in which the camera(s) are located; • Orientation sensor(s) (e.g., 306 and / or 311) for determining the device's orientation and / or changes in the device's orientation relative to the physical environment.

[0141] In some computer systems (e.g., 301-a in FIG. 3B), input device(s) 302, virtual / augmented reality logic 303, display generation component(s) 304, camera(s) 305, and orientation sensor(s) 306 are all integrated into the computer system (e.g., portable multifunction device 100 of FIGS. 1A-1B or device 300 of FIG. 3, such as a smartphone or tablet).

[0142] In some computer systems (e.g., 301-b), in addition to integrated input device(s) 302, virtual / augmented reality logic 303, display generation component(s) 304, camera(s) 305, and orientation sensor(s) 306, the computer system also communicates with additional devices separate from the computer system, such as separate input device(s) 307, such as a touch-sensitive surface, wand, or remote control, and / or separate display generation component(s) 308, such as a virtual reality headset or augmented reality glasses that overlay virtual objects on the physical environment.

[0143] In some computer systems (e.g., 301-c in FIG. 3C), the input device(s) 307, the display generation component(s) 309, the camera(s) 311, and / or the orientation sensor(s) 312 are separate from and in communication with the computer system. In some embodiments, other combinations of components within the computer system 301 and in communication with the computer system are used. For example, in some embodiments, the display generation component(s) 309, the camera(s) 311, and the orientation sensor(s) 312 are incorporated into a headset that is either integrated with or in communication with the computer system.

[0144] In some embodiments, all of the operations described below with reference to Figures 5A1-5A40 and 5B1-5B41 are performed on a single computing device (e.g., computer system 301-a described below with reference to Figure 3B) that has virtual / augmented reality logic 303. It should be understood, however, that often different computing devices are linked together to perform the operations described below with reference to Figures 5A1-5A40 and 5B1-5B41 (e.g., a computing device having virtual / augmented reality logic 303 communicates with a separate computing device having display 450 and / or a separate computing device having touch-sensitive surface 451). In any of these embodiments, the computing device described below with reference to Figures 5A1-5A40 and 5B1-5B41 is the computing device(s) that includes virtual / augmented reality logic 303. In addition, it should be understood that in various embodiments, virtual / augmented reality logic 303 may be divided among separate modules or computing devices. However, for purposes of description herein, the virtual / augmented reality logic 303 will be primarily referred to as residing within a single computing device so as not to unnecessarily obscure other aspects of the embodiment.

[0145] In some embodiments, the virtual / augmented reality logic 303 includes one or more modules (e.g., one or more event handlers 190 including one or more object updaters 177 and one or more GUI updaters 178, as described in more detail above with reference to FIG. 1B ) that receive the interpreted inputs and, in response to these interpreted inputs, generate instructions for updating the graphical user interface according to the interpreted inputs, which are then used to update the graphical user interface on the display. In some embodiments, the interpreted inputs relative to inputs detected (e.g., by the contact movement module 130 of FIGS. 1A and 3 ), recognized (e.g., by the event recognizer 180 of FIG. 1B ), and / or delivered (e.g., by the event sorter 170 of FIG. 1B ) are used to update the graphical user interface on the display. In some embodiments, the interpreted inputs are generated by a module in the computing device (e.g., the computing device receives raw contact input data so as to identify gestures from the raw contact input data). In some embodiments, some or all of the interpreted input is received by a computing device as the interpreted input (e.g., a computing device including touch-sensitive surface 451 processes the raw contact input data to identify a gesture from the raw contact input data and sends information indicative of the gesture to a computing device including virtual / augmented reality logic 303).

[0146] In some embodiments, both the display and the touch-sensitive surface are integrated with a computer system (e.g., 301-a in FIG. 3B) that includes virtual / augmented reality logic 303. For example, the computer system may be a desktop or laptop computer with an integrated display (e.g., 340 in FIG. 3) and touchpad (e.g., 355 in FIG. 3). As another example, the computing device may be portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) with a touchscreen (e.g., 112 in FIG. 2).

[0147] In some embodiments, the touch-sensitive surface is integrated with a computer system, and the display is not integrated with the computer system that includes virtual / augmented reality logic 303. For example, the computer system may be device 300 (e.g., a desktop or laptop computer) with an integrated touchpad (e.g., 355 in FIG. 3 ) connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, a television, etc.). As another example, the computer system may be portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) with a touchscreen (e.g., 112 in FIG. 2 ) connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, a television, etc.).

[0148] In some embodiments, the display is integrated with the computer system, and the touch-sensitive surface is not integrated with the computer system that includes virtual / augmented reality logic 303. For example, the computer system may be device 300 (e.g., a desktop computer, a laptop computer, a television with an integrated set-top box) with an integrated touch display (e.g., 340 in FIG. 3 ) connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device, etc.). As another example, the computer system may be portable multifunction device 100 (e.g., a smartphone, a PDA, a tablet computer, etc.) with a touchscreen (e.g., 112 in FIG. 2 ) connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, another portable multifunction device with a touchscreen that functions as a remote touchpad, etc.).

[0149] In some embodiments, neither the display nor the touch-sensitive surface is integrated with the computer system (e.g., 301-c in FIG. 3C ) that includes virtual / augmented reality logic 303. For example, the computer system may be a stand-alone computing device 300 (e.g., a set-top box, a game console, etc.) that is connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device, etc.) and a separate display (e.g., a computer monitor, a television, etc.).

[0150] In some embodiments, the computer system has an integrated audio system (e.g., audio circuitry 110 and speakers 111 in portable multifunction device 100). In some embodiments, the computing device is in communication with an audio system that is separate from the computing device. In some embodiments, the audio system (e.g., an audio system integrated in a television unit) is integrated with a separate display. In some embodiments, the audio system (e.g., a stereo system) is a stand-alone system that is separate from the computer system and the display.

[0151] Attention is now directed to embodiments of a user interface (“UI”) that is optionally implemented on portable multifunction device 100.

[0152] 4A shows an exemplary user interface for a menu of applications on portable multifunction device 100, according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof: signal strength indicator(s) for wireless communication(s), such as cellular and Wi-Fi signals; ●Time, ●Bluetooth (registered trademark) indicator, ● Battery status indicator, A tray 408 with icons of frequently used applications, such as: An icon 416 for the phone module 138, labeled "Phone," optionally including an indicator 414 of the number of missed calls or voicemail messages; icon 418 of the email client module 140, labeled "Mail," optionally including an indicator 410 of the number of unread emails; ○ An icon 420 for the browser module 147, labeled "Browser"; ○ An icon 422 for the video and music player module 152 labeled "Music", and ● Icons of other applications, such as: ○ Icon 424 of IM module 141, labeled "Messages"; icon 426 of the calendar module 148, labeled "Calendar"; ○ Icon 428 of the image management module 144, labeled "Photos" ○ An icon 430 of the camera module 143, labeled "camera"; ○ Icon 432 of the online video module 155, labeled "Online Video"; Icon 434 of Stock Price Widget 149-2, labeled "Stock Price" ○ Icon 436 of the map module 154, labeled "Map" Icon 438 of weather widget 149-1, labeled "Weather" ○ Icon 440 of alarm clock widget 149-4, labeled "Clock" ○ Icon 442 of Training Support Module 142, labeled "Training Support"; icon 444 of the Notes module 153, labeled "Notes"; and ○ An icon 446 for a settings application or module labeled “Settings” that provides access to settings for the device 100 and its various applications 136.

[0153] 4A are merely examples. For example, other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label for a particular application icon is different from the name of the application corresponding to that particular application icon.

[0154] FIG. 4B shows an example user interface on a device (e.g., device 300, FIG. 3A) that has touch-sensitive surface 451 (e.g., tablet or touchpad 355, FIG. 3A) that is separate from display 450. While many of the following examples are given with reference to input on touchscreen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as shown in FIG. 4B. In some embodiments, the touch-sensitive surface (e.g., 451 in FIG. 4B) has a major axis (e.g., 452 in FIG. 4B) that corresponds to a major axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). According to these embodiments, the device detects contact with touch-sensitive surface 451 (e.g., 460 and 462 in FIG. 4B) at locations that correspond to respective locations on the display (e.g., in FIG. 4B, 460 corresponds to 468, and 462 corresponds to 470). In this manner, when the touch-sensitive surface is separate from the display, user input (e.g., contacts 460 and 462, and their movement) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4B ) is used by the device to operate a user interface on the display (e.g., 450 in FIG. 4B ) of the multifunction device. It should be understood that similar methods are optionally used for the other user interfaces described herein.

[0155] Additionally, while the following examples are given primarily with reference to finger input (e.g., finger touches, finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of those finger inputs are replaced with input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a touch) followed by cursor movement along the path of the swipe (e.g., instead of a touch movement). As another example, a tap gesture is optionally replaced with a mouse click while the cursor is positioned over the tap gesture location (e.g., instead of detecting a touch and then ceasing contact detection). Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice are optionally used simultaneously, or a mouse and finger touches are optionally used simultaneously.

[0156] As used herein, the term “focus selector” refers to an input element that indicates the current portion of a user interface with which a user is interacting. In some implementations involving a cursor or other location marker, the cursor functions as a “focus selector” such that when input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in FIG. 3A or touch-sensitive surface 451 in FIG. 4B ) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations involving a touchscreen display (e.g., touch-sensitive display system 112 in FIG. 1A or touchscreen in FIG. 4A ) that enables direct interaction with user interface elements on the touchscreen display, a contact detected on the touchscreen functions as a “focus selector” such that when input (e.g., a press input by contact) is detected on the touchscreen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one region of the user interface to another region of the user interface (e.g., by using the tab key or arrow keys to move focus from one button to another) without a corresponding cursor movement or contact movement on the touchscreen display. In these implementations, the focus selector moves to follow the movement of focus between various regions of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on a touchscreen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device the element of the user interface that the user intends to interact with).For example, while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button indicates that the user intends to activate that corresponding button (rather than other user interface elements shown on the device's display). In some embodiments, a focus indicator (e.g., a cursor or selection indicator) is displayed via a display device to indicate the current portion of the user interface that will be affected by input received from one or more input devices.

[0157] In some embodiments, the device's response to an input detected by the device depends on criteria based on the intensity of the contact during the input. For example, for some "light press" inputs, the intensity of the contact during the input that exceeds a first intensity threshold triggers a first response. In some embodiments, the device's response to an input detected by the device depends on criteria including both the intensity of the contact during the input and time-based criteria. For example, for some "deep press" inputs, the intensity of the contact during the input that exceeds a second intensity threshold that is greater than the first intensity threshold for a light press triggers a second response only if a delay time has elapsed between meeting the first intensity threshold and meeting the second intensity threshold. This delay time is typically less than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, with the delay time increasing as the second intensity threshold increases). This delay time helps avoid accidental recognition of deep press inputs. As another example, for some "deep press" inputs, there is a period of reduced sensitivity that occurs after the first intensity threshold is met. During the period of decreased sensitivity, the second intensity threshold is increased. This temporary increase in the second intensity threshold also helps to avoid accidental deep pressure inputs. For other deep pressure inputs, the response to deep pressure input detection does not depend on a time-based criterion.

[0158] In some embodiments, one or more of the input intensity threshold and / or corresponding output vary based on one or more factors, such as user settings, contact movement, input timing, running application, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), position of focus selector, etc. Exemplary factors are described in U.S. Patent Application Publication Nos. 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entireties.

[0159] For example, FIG. 4C illustrates a dynamic intensity threshold 480 that varies over time based in part on the intensity of touch input 476 over time. Dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predetermined delay time p1 from when touch input 476 is first detected, and a second component 478 that tracks the intensity of touch input 476 over time. The initial, high intensity threshold of first component 474 reduces accidental triggering of a "deep press" response while still allowing an immediate "deep press" response if touch input 476 provides sufficient intensity. The second component 478 reduces unintentional triggering of a "deep press" response due to gradual intensity variations of the touch input. In some embodiments, a "deep press" response is triggered when touch input 476 meets dynamic intensity threshold 480 (e.g., at point 481 in FIG. 4C ).

[0160] FIG. 4D illustrates another dynamic intensity threshold 486 (e.g., intensity threshold ITD). FIG. 4D also illustrates two other intensity thresholds, a first intensity threshold ITH and a second intensity threshold ITL. In FIG. 4D, a touch input 484 satisfies the first intensity threshold ITH and the second intensity threshold ITL before time p2, but no response is provided until delay time p2 has elapsed at time 482. Also in FIG. 4D, the dynamic intensity threshold 486 decays over time, with the decay beginning at time 488 after a predefined delay time p1 has elapsed from time 482 (when a response associated with the second intensity threshold ITL is triggered). This type of dynamic intensity threshold reduces the inadvertent triggering of a response associated with the dynamic intensity threshold ITD immediately after or simultaneously with the triggering of a response associated with a lower intensity threshold, such as the first intensity threshold ITH or the second intensity threshold ITL.

[0161] 4E illustrates yet another dynamic intensity threshold 492 (e.g., intensity threshold ITD). In FIG. 4E, a response associated with intensity threshold ITL is triggered after a delay time p2 from when touch input 490 is first detected. At the same time, dynamic intensity threshold 492 decays after a predetermined delay time p1 from when touch input 490 is first detected. Thus, a decrease in the intensity of touch input 490 after triggering a response associated with intensity threshold ITL, followed by an increase in the intensity of touch input 490 without releasing touch input 490, can trigger a response associated with intensity threshold ITD (e.g., at time 494) even when the intensity of touch input 490 falls below another intensity threshold, e.g., intensity threshold ITL. User Interface and Related Processes

[0162] Attention is now directed to embodiments of user interfaces (“UI”) and associated processes that may be implemented on a computer system (e.g., portable multifunction device 100 or device 300) that includes (and / or is in communication with) a display generation component (e.g., a display, projector, heads-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of the content within the camera's field of view, and optionally generates a video output including one or more streams of image frames capturing the content within the camera's field of view), one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touchscreen display that also functions as a display generation component, a mouse, joystick, wand controller, and / or a camera that tracks the position of one or more features of a user, such as the user's hand), optionally one or more orientation sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators.

[0163] 5A1-5A40 show exemplary user interfaces for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model, according to some embodiments. The user interfaces in these figures are used to illustrate processes described below, including those in FIGS. 6A-6D, 7A-7C, and 8A-8C. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. Similarly, similar operations are performed on a computer system having a headset 5008 and a separate input device 5010 with a touch-sensitive surface (e.g., as shown in FIG. 5A2) in response to detecting contact on the touch-sensitive surface of input device 5010 while displaying the illustrated user interface on the display of headset 5008, optionally with a focus indicator.

[0164] 5A1-5A27 illustrate exemplary user interfaces for displaying an augmented reality environment and adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, according to some embodiments.

[0165] 5A1 and 5A2 illustrate a situation in which the user interface described with respect to FIGS. 5A3 to 5A40 is used.

[0166] 5A1 illustrates a physical space in which a user 5002, a table 5004, and a physical architectural model 5006 are located. User 5002 holds device 100 and views physical architectural model 5006 via a display of device 100 (e.g., on a touch-sensitive display system 112 of device 100, which may be referred to as a “touchscreen display 112,” “touchscreen 112,” “display 112,” or “touch-sensitive display 112” as shown in FIGS. 1A , 4A , and 5A4 ). One or more cameras of device 100 (which may also be referred to as the “cameras” of device 100) continuously provide a live preview of content within the camera's field of view, including one or more physical objects in the physical space (e.g., wallpaper 5007 in a room of the physical space, table 5004, etc.). The device 100 displays an augmented reality environment including a representation of at least a portion of the camera's field of view, including a physical object (e.g., a physical architectural model 5006) and one or more virtual objects (e.g., a virtual model of a building covering the physical architectural model 5006, a virtual tree, etc.), and the user 5002 interacts with the augmented reality environment using the touchscreen display of the device 100.

[0167] 5A2 shows an alternative method in which a user 5002 views a physical building model 5006 using a computer system including a headset 5008 and a separate input device 5010 having a touch-sensitive surface. In this example, the headset 5008 displays an augmented reality environment, and the user 5002 interacts with the augmented reality environment using the separate input device 5010. In some embodiments, the device 100 is used as the separate input device 5010. In some embodiments, the separate input device 5010 is a touch-sensitive remote control, a mouse, a joystick, a wand controller, or the like. In some embodiments, the separate input device 5010 includes one or more cameras that track the position and movement of one or more features of the user 5002, such as the user's hands.

[0168] 5A3 and 5A4 show views of the augmented reality environment displayed on the touchscreen 112 of device 100. FIG. 5A3 shows the position of device 100 relative to table 5004 and physical building model 5006 from the perspective of user 5002. FIG. 5A4 shows a closer view of device 100 from FIG. 5A3. Device 100 displays the augmented reality environment, including a live view of the physical space as captured by the camera of device 100 and a virtual user interface object (virtual building model 5012). Here, virtual building model 5012 is a three-dimensional virtual model of physical building model 5006 that appears to be attached to or cover (e.g., replaces) physical building model 5006 in the field of view of the camera. The displayed augmented reality environment also includes virtual objects that do not correspond to physical objects in the camera's field of view (e.g., virtual trees, virtual bushes, virtual people, and virtual cars), as well as physical objects that are in the camera's field of view (e.g., wallpaper 5007). In some embodiments, device 100 displays one or more buttons (e.g., button 5014, button 5016, and button 5018, sometimes referred to as virtual buttons or displayed buttons) for interacting with the augmented reality environment (e.g., as described below with respect to FIGS. 5A25-5A27).

[0169] 5A5 and 5A6 show different views of the augmented reality environment displayed on touchscreen 112 of device 100 after user 5002 moves from in front of table 5004 (e.g., as shown in FIG. 5A3) to the side of table 5004 (e.g., as shown in FIG. 5A5). FIG. 5A5 shows the position of device 100 relative to table 5004 and physical building model 5006 from the perspective of user 5002. FIG. 5A6 shows a closer view of device 100 from FIG. 5A5. As shown in FIGS. 5A5 and 5A6, virtual building model 5012 remains fixed to physical building model 5006, and the view of virtual building model 5012 changes as the position, shape, and / or orientation of physical building model 5006 changes within the camera's field of view.

[0170] 5A7-5A14 illustrate adjusting the appearance of the virtual building model 5012 in the augmented reality environment based on a combination of movement of a contact on the touchscreen 112 and movement of the device 100. Reference box 5019 indicates the position of the device 100 relative to the table 5004 and the physical building model 5006 from the perspective of the user 5002.

[0171] 5A7, device 100 displays the augmented reality environment when device 100 is in a first position (e.g., as shown in reference box 5019) relative to table 5004 and physical building model 5006. In FIG. 5A8, device 100 detects an input on virtual building model 5012 (e.g., by detecting a touch input by contact 5020-a on the roof of virtual building model 5012). In FIGS. 5A9-5A11, while continuing to detect the input (e.g., while contact 5020 is maintained on touch screen 112), device 100 detects movement of the input relative to physical building model 5006 (e.g., a drag gesture by contact 5020) and adjusts the appearance of virtual building model 5012 (e.g., lifting virtual roof 5012-a from the virtual building model) according to the magnitude of the movement of the input relative to physical building model 5006. In FIG. 5A9, when contact 5020-b moves a relatively small amount, virtual roof 5012-a lifts a corresponding small amount. In FIG. 5A10, when contact 5020-c moves a larger amount, virtual roof 5012-a lifts a corresponding larger amount. In some embodiments, as shown in FIG. 5A11, as virtual roof 5012-a continues to lift, the floors of virtual building model 5012 lift and extend (e.g., showing virtual first floor 5012-d, virtual second floor 5012-c, and virtual third floor 5012-b). As shown in FIGS. 5A9-5A11, as contact 5020 moves upward, device 100 updates the display of virtual building model 5012 to maintain the display of the initial contact point on virtual roof 5012-a at the location of contact 5020.

[0172] 5A12 and 5A13, while contact 5020-d is maintained on touchscreen 112 and remains stationary, device 100 detects movement of device 100 in physical space (e.g., movement 5022 from a first position lower relative to physical building model 5006, as shown in reference box 5019 in FIG. 5A12, to a second position higher relative to physical building model 5006, as shown in reference box 5019 in FIG. 5A13). In response to the input movement (from device 100 moving in physical space), device 100 adjusts the appearance of virtual building model 5012 by further raising virtual roof 5012-a according to the magnitude of the movement. In some embodiments, as shown in FIG. 5A13, when the appearance of virtual model 5012 is adjusted according to the magnitude of the movement, virtual roof 5012-a is displayed at a position beyond the maximum limit of virtual roof 5012-a's stationary state.

[0173] In Figure 5A14, device 100 stops detecting input (e.g., contact 5020 lifts off) and displays virtual roof 5012-a at a position corresponding to the maximum limit of rest. In some embodiments, device 100 displays an animated transition (e.g., from Figure 5A13 to Figure 5A14) from virtual roof 5012-a at a position beyond the maximum limit of rest (e.g., in Figure 5A13) to a position corresponding to the maximum limit of rest (e.g., in Figure 5A14).

[0174] 5A15 and 5A16 illustrate movement of device 100 in physical space (e.g., movement 5024) when no input is detected on touchscreen 112 (e.g., no tactile touch input is detected on touchscreen 112). Because no input is detected, the movement of device 100 changes the field of view of the camera of device 100 from a first position that is lower relative to physical building model 5006 (e.g., as shown in reference box 5019 of FIG. 5A15) to a second position that is higher relative to physical building model 5006 (e.g., as shown in reference box 5019 of FIG. 5A16) without adjusting the appearance of virtual building model 5012.

[0175] 5A15 and 5A16, FIGS. 5A17 and 5A18 illustrate movement of device 100 in physical space (e.g., movement 5028) when an input is detected on touchscreen 112 (e.g., touch input via contact 5026-a is detected on touchscreen 112). While continuing to detect the input (e.g., while contact 5026-a is maintained on touchscreen 112 and remains stationary), device 100 detects movement of device 100 in physical space (from a first position lower relative to physical building model 5006, as shown in reference box 5019 in FIG. 5A17, to a second position higher relative to physical building model 5006, as shown in reference box 5019 in FIG. 5A18). In response to the movement of the input (from device 100's movement in physical space), device 100 adjusts the appearance of virtual building model 5012 by raising virtual roof 5012-a according to the magnitude of the movement.

[0176] 5A19 and 5A20, while continuing to detect the input (e.g., while contact 5026 is maintained on touchscreen 112), device 100 detects movement of the input (e.g., a drag gesture with contact 5026) relative to physical building model 5006 and adjusts the appearance of virtual building model 5012 (e.g., lifting virtual roof 5012-a further from virtual building model 5012) according to the magnitude of the movement of the input relative to physical building model 5006. In some embodiments, as shown in FIG. 5A20, the floors of virtual building model 5012 lift and extend (e.g., showing first floor 5012-d, second floor 5012-c, and third floor 5012-b) as virtual roof 5012-a continues to lift.

[0177] As shown in Figures 5A17-5A20, as the input moves up (whether the input movement is due to movement of device 100 while the contact (e.g., contact 5026-a) is maintained and remains stationary on touchscreen 112, or whether the input movement is due to movement of the contact across touchscreen 112 while device 100 is held substantially stationary in physical space), device 100 updates the display of virtual building model 5012 to maintain the display of the initial contact point on virtual roof 5012-a at the location of contact 5026.

[0178] 5A21-5A24 illustrate changing a virtual environment setting (e.g., the time of day) of an augmented reality environment in response to input that navigates through time within the augmented reality environment. In FIGS. 5A21-5A24, device 100 detects input that changes the virtual environment setting (e.g., a left-to-right swipe gesture with contact 5030), and in response, device 100 changes the time of day within the augmented reality environment (e.g., by adjusting the appearance of virtual building model 5012 and applying a filter to portions of the representation of the camera's field of view that are not obscured by virtual building model 5012). In FIG. 5A21, the time of day in the augmented reality environment is 12:00 AM, due to the shadow of virtual building model 5012 and the shadows of virtual objects (e.g., a virtual tree, a virtual bush, a virtual person, and a virtual car) to the right of the objects. As contact 5030 moves from left to right, the time of day within the augmented reality environment changes (e.g., according to the speed and / or distance of the input movement) from morning to night (e.g., from morning in FIG. 5A21 to noon in FIG. 5A22 to afternoon in FIG. 5A23 to night in FIG. 5A24). In some embodiments, device 100, in addition to adjusting the appearance of the virtual scene, applies a filter (e.g., to wallpaper 5007) to portions of the live view that are not obscured by the virtual scene. For example, in FIG. 5A24 (e.g., when the virtual environment setting is changed to night mode), a different filter is applied to wallpaper 5007 (e.g., indicated by the first shading pattern) in addition to adjusting the appearance of the virtual scene in night mode (e.g., indicated by the second shading pattern).

[0179] 5A25-5A27 illustrate changing a virtual environment setting of an augmented reality environment in response to an input (e.g., a tap input on a displayed button) to switch between different virtual environments (e.g., a default virtual environment such as a landscape view, an interior view, or a day / night view) for a virtual user interface object (e.g., virtual building model 5012), where different virtual environments are associated with different interactions for exploring the virtual user interface object. In FIG. 5A25, the landscape button 5014 is selected, and a landscape view of virtual building model 5012 is displayed (e.g., with virtual trees, virtual bushes, virtual people, and a virtual car). 5A26 and 5A27, device 100 detects input on interior button 5016, such as a tap gesture with contact 5032, and in response displays an interior view of virtual building model 5012 (e.g., showing an expanded view of virtual building model 5012 without the virtual trees, virtual bushes, virtual people, and virtual cars, but instead with a virtual first floor 5012-d, a virtual second floor 5012-c, a virtual third floor 5012-b, and a virtual roof 5012-a). In some embodiments, when the virtual environment setting is changed (e.g., to an interior view), the surrounding physical environment is blurred (e.g., using a filter). For example, although not shown in FIG. 5A27, in some embodiments, wallpaper 5007 is blurred when the virtual environment setting is changed to an interior view.

[0180] 5A28-5A40 show exemplary user interfaces for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments.

[0181] 5A28, similar to FIG. 5A4, shows a view of an augmented reality environment displayed on touchscreen 112 of device 100, including a live view of the physical space as captured by the camera of device 100, a virtual building model 5012, a virtual vehicle 5050, and a virtual person 5060. In addition, reference box 5019 in FIG. 5A28 indicates the position of device 100 relative to table 5004 and physical building model 5006 from the perspective of user 5002 (e.g., as shown in FIGS. 5A1 and 5A2).

[0182] Figures 5A29-5A31 show the transition from Figure 5A28. Specifically, Figures 5A29-5A31 show the transition from a view of the augmented reality environment (e.g., as shown in Figure 5A28) to a simulated view of the virtual model from the perspective of a virtual vehicle 5050 in the virtual model.

[0183] FIG. 5A29 shows input 5052 (eg, a tap gesture on the touchscreen 112 of device 100 or a selection using a separate input device with a focus indicator) detected at a location corresponding to vehicle 5050.

[0184] 5A30 and 5A31 illustrate a transition from a view of the augmented reality environment to a simulated view of the virtual model from the perspective of vehicle 5050, displayed in response to detecting input 5052. Specifically, FIG. 5A30 illustrates a view shown on device 100 during an animated transition from the view shown in FIG. 5A29 to a simulated perspective view from vehicle 5050 (e.g., from the perspective of a person, such as a driver or passenger within vehicle 5050), and FIG. 5A31 illustrates the simulated perspective view from vehicle 5050.

[0185] In some embodiments, the transition from the view of the augmented reality environment to the simulated perspective view includes an animated transition. Optionally, the transition includes an animation of flying from a position viewing the augmented reality environment to a position of the vehicle 5050 (e.g., a position of a person in the vehicle 5050). For example, FIG. 5A30 shows a view of the virtual model from a position between the user's position and the position of the vehicle 5050 in FIG. 5A29 (e.g., halfway through the animated transition), even though the user did not move the device 100 (e.g., the position of the device 100 relative to the physical building model 5006 as shown in reference box 5019 in FIG. 5A30 is the same as in FIG. 5A29).

[0186] In some embodiments, a portion of the field of view of device 100 (e.g., a camera of device 100) continues to be displayed during the animated transition to the perspective view from vehicle 5050. For example, as shown in FIG. 5A30, wallpaper 5007 and an edge of table 5004 are displayed during the animated transition to the simulated perspective view (e.g., as if viewed from a position corresponding to the view shown in FIG. 5A30, between the user's position in FIG. 5A29 and the position of vehicle 5050). In some embodiments, the field of view of the camera ceases to be displayed during the animated transition to the perspective view of vehicle 5050 (e.g., wallpaper 5007 and an edge of table 5004 are not displayed during the animated transition, and, optionally, corresponding portions of the virtual model are displayed instead).

[0187] In FIG. 5A31 , the simulated perspective view from vehicle 5050 also shows controls 5054 including directional arrows (up, down, left, and right) for controlling the movement (e.g., direction of movement) of vehicle 5050 (e.g., the virtual object for which the simulated perspective view is displayed). In the example shown in FIG. 5A31 , up arrow 5056 controls the forward movement of vehicle 5050. Thus, in some embodiments, a user can control the movement of a corresponding virtual object (e.g., vehicle 5050) while a simulated view from the perspective of that virtual object is displayed. In some embodiments, a user cannot control the movement of a corresponding virtual object (e.g., virtual vehicle 5050 and / or virtual person 5060) in the virtual model while the view of the augmented reality environment is displayed. For example, in some embodiments, a user cannot control the movement of vehicle 5050 in the view of the augmented reality environment of FIG. 5A28. In some embodiments, vehicle 5050 moves autonomously within the virtual model while the view of the augmented reality environment (e.g., FIG. 5A28) is displayed.

[0188] Figures 5A32 and 5A33 illustrate the transition from Figure 5A31. Specifically, Figures 5A32 and 5A33 illustrate user-controlled movement of vehicle 5050 within the virtual model. Figure 5A32 illustrates input 5058 detected at a location corresponding to up arrow 5056 (shown in Figure 5A31) of control 5054. In response to input 5058 on up arrow 5056, vehicle 5050 moves forward within the virtual model. Accordingly, Figure 5A33 illustrates an updated simulated perspective view of the virtual model being displayed corresponding to the forward movement of vehicle 5050 within the virtual model. For example, in the updated simulated perspective view of Figure 5A33, virtual building model 5012 appears less visible and person 5060 appears closer than in Figure 5A32.

[0189] Figures 5A34 and 5A35 show the transition from Figure 5A33. Specifically, Figures 5A34 and 5A35 show the transition from a simulated view of the virtual model from the perspective of a vehicle 5050 to a simulated view of the virtual model from the perspective of a virtual person 5060. Figure 5A34 shows input 5062 detected at a location corresponding to person 5060. Figure 5A35 shows the simulated view of the virtual model from the perspective of person 5060 displayed in response to detecting input 5062. In some embodiments, device 100 displays an animated transition between the simulated perspective view from vehicle 5050 and the simulated perspective view from person 5060 (e.g., as if a user were moving from a location of (e.g., within) vehicle 5050 to a location of person 5060).

[0190] Figures 5A36 and 5A37 illustrate the transition from Figure 5A35. Specifically, Figures 5A36 and 5A37 illustrate changing views of a virtual model from the perspective of person 5060 (e.g., a selected virtual object) in response to movement of device 100 (e.g., in physical space).

[0191] FIG. 5A36 shows arrow 5064 indicating movement of device 100 toward the left and rotation of device 100 about the z-axis (e.g., causing the right edge of device 100 to move closer to the user and the left edge of device 100 to move farther away from the user). FIG. 5A37 shows an updated simulated perspective view of the virtual model from the perspective of person 5060 displayed in response to detecting movement of device 100. The updated simulated perspective view of FIG. 5A37 corresponds to a view of the virtual model if person 5060 had moved toward the left and slightly rotated their head toward the right relative to the person's position in FIG. 5A36. Reference box 5019 in FIG. 5A37 indicates the new position of device 100 relative to physical building model 5006 after device 100 has been moved as indicated by arrow 5064.

[0192] In some embodiments, control 5054 (e.g., shown in FIG. 5A31 but not shown in FIGS. 5A35, 5A36) is displayed while displaying a simulated view from the perspective of person 5060, thereby allowing a user to use arrows on control 5054 to control the movement of person 5060 within the virtual model while a simulated view from the perspective of person 5060 is displayed (e.g., FIG. 5A35).

[0193] Figures 5A38-5A40 show the transition from Figure 5A37. Specifically, Figures 5A38-5A40 show the transition from the simulated perspective view shown in Figure 5A37 back to the view of the augmented reality environment.

[0194] 5A38 illustrates input 5066. In the example illustrated in 5A38, input 5066 is a pinch gesture (e.g., from the minimum zoom level of the simulated perspective view of the virtual model). In some embodiments, input 5066 is a gesture (e.g., a tap) on an "empty" location in the virtual model (e.g., a location where a simulated perspective view is unavailable, such as a patch of grass). In some embodiments, input 5066 is a gesture (e.g., a tap) on an affordance for displaying or undisplaying the augmented reality environment (e.g., an icon such as an "X" to exit the simulated perspective view).

[0195] 5A39 and 5A40 show a transition from a simulated view of a virtual model from the perspective of person 5060 to a view of an augmented reality environment displayed in response to detecting input 5066. Specifically, FIG. 5A39 shows a view shown on device 100 during an animated transition from the view shown in FIG. 5A38 to a view of the augmented reality environment, and FIG. 5A40 shows a view of the augmented reality environment. In some embodiments, the transition from the simulated perspective view to the view of the augmented reality environment includes an animated transition that optionally includes an animation of flying from the location of the virtual object (from which the simulated perspective view is shown) to a location viewing the augmented reality environment.

[0196] Because device 100 is in a different position relative to physical building model 5006 in Figures 5A38-5A40 than in Figures 5A28-5A30, the augmented reality view, such as that shown in Figure 5A40, corresponds to the new position of device 100 and is different from that shown in Figure 5A28. Similarly, Figure 5A39 shows a view of the virtual model from a position between (e.g., midway through the animated transition) the position of person 5060 in Figure 5A38 and the position of the user in Figure 5A40, even though the user did not move device 100 (e.g., the position of device 100 relative to physical building model 5006, as shown in reference box 5019, is the same in each of Figures 5A38-5A40).

[0197] Similar to the animated transition to the simulated perspective view described above with reference to Figures 5A29-5A31, in some embodiments, a portion of the field of view of device 100 (e.g., the camera of device 100) is visible during the animated transition from the simulated perspective view to the view of the augmented reality environment. For example, as shown in Figure 5A39, wallpaper 5007 is displayed during the animated transition from the simulated perspective view (e.g., as if viewed from a position corresponding to the view shown in Figure 5A39, between the position of person 5060 and the user's position in Figure 5A40). In some embodiments, the camera's field of view ceases to be displayed during the animated transition to the view of the augmented reality environment (e.g., wallpaper 5007 is not displayed during the animated transition, and, optionally, a corresponding portion of the virtual model is displayed instead).

[0198] 5B1-5B41 show example systems and user interfaces for three-dimensional manipulation of virtual user interface objects, according to some embodiments. The user interfaces in these figures are used to illustrate processes described below, including those in FIGS. 6A-6D, 7A-7C, and 8A-8C. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. Similarly, similar operations are performed on a computer system having headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 5B2) in response to detecting contact on the touch-sensitive surface of input device 5010 while displaying the illustrated user interface on the display of headset 5008, optionally with a focus indicator.

[0199] 5B1-5B4 illustrate situations in which the user interfaces described with respect to FIGS. 5B5-5B41 are used.

[0200] 5B1 shows a physical space 5200 in which a user 5202 and a table 5204 are located. The device 100 is held by the user 5202 in the user's hand 5206. A reference mat 5208 is located on the table 5204.

[0201] FIG. 5B2 shows a view of a virtual three-dimensional space displayed on display 112 of device 100. A reference mat 5208 is within the field of view of one or more cameras (e.g., light sensors 164) of device 100 (hereinafter referred to as "cameras" to refer to one or more cameras of device 100). Display 112 shows a live view of physical space 5200 as captured by the cameras, including a displayed version 5208b of a physical reference mat 5208a. A virtual user interface object (virtual box 5210) is displayed within the virtual three-dimensional space displayed on display 112. In some embodiments, virtual box 5210 is anchored to reference mat 5208b, such that the view of virtual box 5210 changes as the displayed view 5208b of the reference mat changes in response to movement of reference mat 5208a within physical space 5200 (e.g., as shown in FIGS. 5B2-5B3). Similarly, the view of the virtual box 5210 changes as the view of the displayed version 5208b changes in response to the movement of the device 100 relative to the reference mat 5208a.

[0202] 5B3, the reference mat 5208 has been rotated so that the longer side of the reference mat 5208a is adjacent to the device 100 (in FIG. 5B2, the shorter side of the reference mat 5208a was adjacent to the device 100). The rotation of the displayed version 5208b of the reference mat from FIG. 5B2 to FIG. 5B3 occurs as a result of the rotation of the reference mat 5208a in physical space 5200.

[0203] 5B3 and 5B4, the device 100 has been moved closer to the reference mat 5208a. As a result, the displayed version 5208b of the reference mat and the virtual box 5210 have increased in size.

[0204] 5B5-5B41 show larger views of device 100, providing a full view of the user interface displayed on display 112 and not showing user's hand 5206. FIG.

[0205] FIG. 5B5 shows a user interface for creating and adjusting virtual user interface objects, displayed on display 112. The user interface includes an avatar 5212, a toggle 5214 (e.g., for toggling between virtual reality display mode and augmented reality display mode), a new object control 5216 (e.g., for adding a new object 5216 to the virtual three-dimensional space displayed by display 112), a color selection palette 5218 containing a number of controls corresponding to available colors (e.g., for selecting the color of the virtual object), and a delete control 5220 (e.g., for removing a virtual user interface object from the virtual three-dimensional space). In FIG. 5B5, toggle 5214 indicates that the current display mode is the augmented reality display mode (e.g., display 112 is displaying a virtual box 5210 and a view of physical space 5200 as captured by device 100's camera). FIGS. 5B37-5B39 show the virtual reality display mode. In Figures 5B37-5B39, the appearance of toggle 5214 has changed to indicate that virtual reality display mode is active (and that input on toggle 5214 will cause a transition from virtual reality display mode to augmented reality display mode).

[0206] 5B6-5B17 show inputs that cause the virtual box 5210 to move.

[0207] 5B6 , input (e.g., selection and movement input) by contact 5222 (e.g., contact with touch-sensitive display 112) is detected on first surface 5224 of virtual box 5210. When a surface of virtual box 5210 is selected, movement of virtual box 5210 is constrained to movement in a plane parallel to the selected surface. In response to detection of contact 5222 selecting first surface 5224 of virtual box 5210, movement projection 5226 is shown extending from virtual box 5210 to indicate the plane of movement of virtual box 5210 (e.g., a plane of movement parallel to the selected first surface 5224 of virtual box 5210).

[0208] 5B6 and 5B7 , contact 5222 is moving along the surface of touch-sensitive display 112 in the direction indicated by arrow 5228. In response to the movement of contact 5222, virtual box 5210 is moving in the plane indicated by moving projection 5226 in the direction indicated by arrow 5228. In Figures 5B7 and 5B8 , contact 5222 is moving along the surface of touch-sensitive display 112 in the direction indicated by arrow 5230. In response to the movement of contact 5222, virtual box 5210 is moving in the plane indicated by moving projection 5226 in the direction indicated by arrow 5230. In Figure 5B9 , contact 5222 has lifted off touch-sensitive display 112, and moving projection 5226 is no longer displayed.

[0209] 5B10, input by contact 5232 (e.g., selection and movement input) is detected on second surface 5234 of virtual box 5210. In response to detecting contact 5232 selecting second surface 5234 of virtual box 5210, movement projection 5236 is shown extending from virtual box 5210 to indicate the plane of movement of virtual box 5210 (e.g., a plane of movement parallel to selected second surface 5234 of virtual box 5210).

[0210] 5B10 and 5B11, contact 5232 is moving along the surface of touch-sensitive display 112 in the direction indicated by arrow 5238. In response to the movement of contact 5232, virtual box 5210 is moving in the plane indicated by motion projection 5236 in the direction indicated by arrow 5238. As virtual box 5210 moves upward, such that it is hovering over displayed reference mat 5208b, shadow 5240 of virtual box 5210 is displayed to indicate that virtual box 5210 is hovering.

[0211] 5B11 and 5B12, contact 5232 is moving along the surface of touch-sensitive display 112 in the direction indicated by arrow 5242. In response to the movement of contact 5232, virtual box 5210 is moving in the plane indicated by moving projection 5236 in the direction indicated by arrow 5242. In Figure 5B13, contact 5232 has lifted off touch-sensitive display 112, and moving projection 5236 is no longer displayed.

[0212] 5B14, input (e.g., selection and movement input) by contact 5233 is detected on first surface 5224 of virtual box 5210. In response to detecting contact 5233 selecting first surface 5224 of virtual box 5210, movement projection 5237 is shown extending from virtual box 5210 to indicate the plane of movement of virtual box 5210 (e.g., a plane of movement parallel to selected first surface 5224 of virtual box 5210).

[0213] 5B14 and 5B15, contact 5233 is moving along the surface of touch-sensitive display 112 in the direction indicated by arrow 5239. In response to the movement of contact 5233, virtual box 5210 is moving in the plane indicated by movement projection 5237 in the direction indicated by arrow 5238. The movement of contact 5232 shown in Figures 5B10 and 5B11 is in the same direction as the movement of contact 5233 shown in Figures 5B14 and 5B15. Because the movement of contact 5232 occurs while second surface 5234 of virtual box 5210 is selected, the plane of movement of virtual box 5210 in Figures 5B10 and 5B11 is different from the plane of movement of virtual box 5210 in Figures 5B14 and 5B15, where the movement of contact 5233 occurs while first surface 5224 of virtual box 5210 is selected. In this way, selection and movement inputs with the same movement direction of the input will cause different movements of the virtual box 5210 depending on which surface of the virtual box 5210 is selected.

[0214] 5B15 and 5B16, contact 5233 is moving along the surface of touch-sensitive display 112 in the direction indicated by arrow 5243. In response to the movement of contact 5233, virtual box 5210 is moving in the plane indicated by moving projection 5237 in the direction indicated by arrow 5243. In Figure 5B17, contact 5233 has lifted off touch-sensitive display 112, and moving projection 5237 is no longer displayed.

[0215] 5B18-5B21 show inputs that cause the virtual box 5210 to resize.

[0216] 5B18, input (e.g., a resize input) by contact 5244 is detected on first surface 5224 of virtual box 5210. In some embodiments, when the contact remains at a location corresponding to the surface of the virtual object for a period of time that increases above a resize time threshold, subsequent movement of the contact (and / or movement of device 100) causes a resize of the virtual object. In FIG. 5B19, contact 5244 remains in contact with first surface 5224 of virtual box 5210 for a period of time that increases above a resize time threshold, and resize projection 5246 is shown to indicate the axis (perpendicular to selected first surface 5224) along which virtual box 5210 will be resized in response to subsequent movement of contact 5244.

[0217] 5B19 and 5B20, contact 5244 is moving along the path indicated by arrow 5248. In response to the movement of contact 5244, the size of virtual box 5210 is increasing along the axis indicated by resize projection 5246 in the direction indicated by arrow 5248. In Figure 5B21, contact 5244 has lifted off touch-sensitive display 112, and projection 5246 is no longer displayed.

[0218] 5B22 to 5B27 show the placement of the object insertion cursor and the placement of the virtual box using the insertion cursor.

[0219] In Figure 5B22, input (e.g., a tap input) by contact 5250 is detected at a location corresponding to displayed version 5208b of physical reference mat 5208a. In response to detecting contact 5250, insertion cursor 5252 is displayed on display 112 at a location corresponding to contact 5250, and in Figure 5B23, contact 5250 has lifted off touch-sensitive display 112 and insertion cursor 5252 is shown. In some embodiments, insertion cursor 5252 ceases to be displayed after a predetermined period of time. In Figure 5B24, insertion cursor 5252 has ceased to be displayed and input (e.g., a tap input) by contact 5254 is detected at a location different from the location where insertion cursor 5252 was shown (as shown in Figure 5B23). In response to detecting contact 5254, a new insertion cursor 5256 is displayed on display 112 at a location corresponding to contact 5254. In FIG. 5B25, contact 5254 has lifted off touch-sensitive display 112, revealing insertion cursor 5256.

[0220] 5B26, insertion cursor 5256 has ceased to be displayed, and input (e.g., a tap input) by contact 5258 is detected at a location corresponding to the location where insertion cursor 5256 (as shown in FIG. 5B25) was displayed. In response to detecting contact 5258 at the location where the insertion cursor was located, a new virtual user interface object (virtual box 5260) is displayed on display 112 at a location corresponding to contact 5258. In FIG. 5B27, contact 5258 has lifted off touch-sensitive display 112.

[0221] 5B28 to 5B31 show the resizing of the virtual box 5260 due to the movement of the device 100. FIG.

[0222] 5B28, input (e.g., a resize input) through contact 5262 with touch-sensitive display 112 is detected on surface 5264 of virtual box 5260. In FIG. 5B29, contact 5262 remains in contact with surface 5264 of virtual box 5260 for an increased period of time above a resize time threshold, and resize projection 5266 is shown to indicate the axis (perpendicular to selected surface 5264) along which virtual box 5260 will be resized in response to subsequent movement of device 100. In FIGS. 5B29 and 5B30, device 100 moves along the path indicated by arrow 5268 while contact 5262 remains in contact with touch-sensitive display 112. In response to movement of device 100, the size of virtual box 5260 increases along the axis indicated by resize projection 5266, as shown in FIG. In FIG. 5B31, contact 5262 has lifted off touch-sensitive display 112 and resized projection 5266 is no longer displayed.

[0223] 5B32-5B35 illustrate the insertion of a new virtual object using the new object control 5216. FIG.

[0224] In Figure 5B32, input by contact 5270 (e.g., a tap input) is detected at a location on displayed version 5208b of physical reference mat 5208a. In response to detecting contact 5270, an insertion cursor 5272 is displayed on display 112 at a location corresponding to contact 5270. In Figure 5B33, contact 5270 has lifted off touch-sensitive display 112, and insertion cursor 5272 is shown. In Figure 5B34, insertion cursor 5272 has ceased to be displayed, and input by contact 5274 (e.g., a tap input) with touch-sensitive display 112 is detected at a location corresponding to new object control 5216. In Figure 5B35, in response to input at new object control 5216 (e.g., after placement of insert cursor 5272), a new virtual user interface object (virtual box 5276) is displayed on display 112 at a location corresponding to where insertion cursor 5272 was shown.

[0225] Figures 5B36 and 5B37 show a pinch-to-zoom input to cause a transition from augmented reality display mode to virtual reality display mode, and Figures 5B39 and 5B40 show an input at toggle 5214 to return from virtual reality display mode to augmented reality display mode.

[0226] 5B36, contacts 5278 and 5280 are detected simultaneously with touch-sensitive display 112. As shown in FIGS. 5B36 and 5B37, a pinch gesture is detected in which contacts 5278 and 5280 are simultaneously moved along the paths indicated by arrows 5282 and 5284, respectively. In response to detecting the pinch gesture, the display of virtual boxes 5210, 5260, and 5276 is zoomed (e.g., zoomed out, causing virtual boxes 5210, 5260, and 5276 to appear smaller in displayed size). In some embodiments, the gesture to zoom causes a transition from an augmented reality display mode to a virtual reality display mode (e.g., because the zoomed view of the boxes is no longer aligned with the field of view of the camera of device 100). In some embodiments, in the virtual reality display mode, physical objects (e.g., reference mat 5208) within the field of view of the camera of device 100 cease to be displayed, or virtual (rendered) versions of one or more of the physical objects are displayed.

[0227] In some embodiments, in virtual reality display mode, virtual objects displayed by device 100 are locked to the coordinate system of device 100. In Figures 5B37 and 5B38, the position of device 100 has changed. Because device 100 is in virtual reality display mode, the positions of virtual boxes 5210, 5260, and 5276 have not changed in response to the changed position of device 100.

[0228] In Figure 5B39, input by contact 5286 (e.g., a tap input) is detected at a location corresponding to toggle 5214. In response to the input by contact 5286, a transition from the virtual reality display mode to the augmented reality display mode is caused. Figure 5B40 shows the user interface displayed on display 112 after transitioning to the augmented reality display mode in response to the input by contact 5286. The transition includes redisplaying the field of view of the camera of device 100 (e.g., redisplaying the displayed view 5208b of the reference mat). In some embodiments, the transition includes zooming (e.g., zooming in) the display of virtual boxes 5210, 5260, and 5276 (e.g., to realign the boxes with the field of view of the camera of device 100).

[0229] In some embodiments, in the augmented reality display mode, the virtual objects displayed by device 100 are locked to physical space 5200 and / or to a physical object within physical space 5200 (e.g., reference mat 5208). In Figures 5B40 and 5B41, the position of device 100 has changed. Because device 100 is in the augmented reality display mode, virtual boxes 5210, 5260, and 5276 are locked to reference mat 5208a, and the positions of the virtual boxes on display 112 change according to the changed position of device 100.

[0230] 5C1-5C30 show example systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to some embodiments. The user interfaces in these figures are used to illustrate processes described below, including those in FIGS. 10A-10E. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. Similarly, similar operations are performed on a computer system having headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 5A2) in response to detecting contact on the touch-sensitive surface of input device 5010 while displaying the illustrated user interface on the display of headset 5008, optionally with a focus indicator.

[0231] 5C1 and 5C2 illustrate a situation in which the user interface described with respect to FIGS. 5C3 to 5C30 is used.

[0232] 5C1 shows a physical space 5200 in which a user and a table 5204 are located. The device 100 is held by the user in the user's hand 5206. A reference mat 5208 is positioned on the table 5204. A view of the simulated environment is displayed on the display 112 of the device 100. The reference mat 5208 is within the field of view of one or more cameras (e.g., light sensors 164) of the device 100 (hereinafter referred to as the "cameras") of the device 100. The display 112 shows a live view of the physical space 5200 as captured by the cameras, including a displayed version 5208b of the physical reference mat 5208a. Two virtual user interface objects (a first virtual box 5302 and a second virtual box 5304) are displayed within the simulated environment displayed on the display 112. In a first viewing mode (e.g., an augmented reality viewing mode), the virtual boxes 5302 and 5304 are fixed to the reference mat 5208b, such that the views of the virtual boxes 5302 and 5304 change as the displayed view 5208b of the reference mat changes in response to movement of the reference mat 5208a within the physical space 5200 (e.g., maintaining a fixed spatial relationship between the virtual boxes 5302 and 5304 and the physical environment including the reference mat 5208a). Similarly, in the first viewing mode, the views of the virtual boxes 5302 and 5304 change in response to movement of the device 100 relative to the reference mat 5208a.

[0233] In Figure 5C2, device 100 has been moved closer to reference mat 5208a. As a result, the displayed version 5208b of the reference mat and virtual boxes 5302 and 5304 have increased in size.

[0234] 5C3-5C30 show larger views of device 100, providing a full view of the user interface displayed on display 112 and not showing user's hand 5206. Features of the user interface are further described above with respect to FIG.

[0235] 5C4-5C6 illustrate input gestures (including swipes up and down) for moving virtual box 5302 while the virtual box is displayed in the augmented reality viewing mode. Because the input gestures described with respect to FIGS. 5C4-5C6 do not satisfy the mode change criteria (e.g., for changing the viewing mode from the augmented reality viewing mode to the virtual reality viewing mode), the view of virtual boxes 5302 and 5304 changes in response to subsequent movement of device 100 (e.g., thereby maintaining a fixed spatial relationship between virtual boxes 5302 and 5304 and the physical environment, including reference mat 5208a), as shown in FIGS.

[0236] Another example of a gesture that does not meet the mode change criteria is a resize gesture (eg, as described above with respect to FIGS. 5B18-5B21).

[0237] 5C4, contact 5306 input (e.g., selection and movement input) is detected on surface 5308 of virtual box 5302. In response to detecting contact 5306 selecting surface 5308 of virtual box 5302, movement projection 5310 is shown extending from virtual box 5302 to indicate the plane of movement of virtual box 5302 (e.g., a plane of movement parallel to the selected surface 5308 of virtual box 5302).

[0238] 5C4 and 5C5, contact 5306 moves along the surface of touch-sensitive display 112 in the direction indicated by arrow 5312. In response to the movement of contact 5306, virtual box 5302 moves in the plane indicated by motion projection 5310 in the direction indicated by arrow 5312. As virtual box 5302 moves upward, such that it is hovering over displayed reference mat 5208b, a shadow 5314 of virtual box 5302 is displayed to indicate that virtual box 5210 is hovering.

[0239] 5C5 and 5C6, contact 5306 moves along the surface of touch-sensitive display 112 in the direction indicated by arrow 5316. In response to the movement of contact 5306, virtual box 5302 moves in the plane indicated by moving projection 5310 in the direction indicated by arrow 5316. In Figure 5C7, contact 5306 has lifted off touch-sensitive display 112, and moving projection 5310 is no longer displayed.

[0240] 5C7 and 5C8 illustrate movement of device 100 along the path indicated by arrow 5318. As device 100 is moved, the positions of virtual boxes 5302 and 5304, which are displayed to be device 100, change on display 112 (e.g., thereby maintaining a fixed spatial relationship between virtual boxes 5302 and 5304 and reference mat 5208a in device 100's physical environment).

[0241] 5C9-5C10 illustrate an input gesture (a pinch gesture) that satisfies the mode change criteria (eg, causes a change in viewing mode from an augmented reality viewing mode to a virtual reality viewing mode).

[0242] In Figure 5C9, contacts 5320 and 5324 are detected on touch-sensitive display 112. In Figures 5C9-5C11, contact 5320 moves along the path indicated by arrow 5322, and contact 5324 moves along the path indicated by arrow 5324. In response to the simultaneous movement of contacts 5320 and 5324 decreasing the distance between contacts 5320 and 5324, the displayed view of the simulated environment, including virtual boxes 5302 and 5304, is zoomed out (e.g., causing virtual boxes 5302 and 5304 to increase in size on display 112). When the zoom input is received, a transition from the augmented reality viewing mode to the virtual reality viewing mode is triggered. The transition animation performed during the transition includes a gradual fade-out of the displayed view of the physical environment. For example, the displayed view of table 5204 and the displayed view 5208b of reference mat 5208a as captured by one or more cameras of device 100 gradually fade out (e.g., as shown in FIGS. 5C10 and 5C11). The transition animation includes a gradual fade-in of the virtual grid lines of virtual reference grid 5328 (e.g., as shown in FIGS. 5C11 and 5C12). During the transition, the appearance of toggle 5214 (e.g., for toggling between virtual reality display mode and augmented reality display mode) changes to indicate the current viewing mode (e.g., as shown in FIGS. 5C10 and 5C11). After liftoff of contacts 5320 and 5324, virtual boxes 5302 and 5304 in the simulated environment continue to move and decrease in size (e.g., the change in the simulated environment continues to have “momentum” that causes movement after the input gesture ends).

[0243] 5C12 and 5C13, device 100 is moved along the path indicated by arrow 5330. Because the pinch-zoom input gesture described with respect to FIGS. 5C9-5C11 caused a change from the augmented reality viewing mode to the virtual reality viewing mode, the positions of virtual boxes 5302 and 5304 do not change as device 100 moves (e.g., in the virtual reality viewing mode, a fixed spatial relationship is not maintained between virtual boxes 5302 and 5304 and the physical environment).

[0244] In FIGS. 5C13 and 5C14, device 100 is moved along the path indicated by arrow 5332. In FIGS.

[0245] 5C15-5C18 illustrate inputs for inserting a virtual box into a simulated environment displayed on device 100 while the simulated environment is displayed in virtual reality viewing mode.

[0246] In Figure 5C15, input (e.g., a tap input) by contact 5334 is detected on touch-sensitive display 112. In response to detecting contact 5334, insertion cursor 5336 is displayed on display 112 at a location corresponding to contact 5334, as shown in Figure 5C16. In Figure 5C17, insertion cursor 5336 ceases to be displayed, and input (e.g., a tap input) by contact 5338 is detected at a location corresponding to new object control 5216. In Figure 5C18, in response to input at new object control 5216 (e.g., after placing insert cursor 5336), a new virtual user interface object (virtual box 5340) is displayed at a location corresponding to where insertion cursor 5336 was previously located.

[0247] 5C19 and 5C20 illustrate inputs for manipulating virtual user interface objects in a simulated environment displayed on device 100 while the simulated environment is displayed in virtual reality viewing mode.

[0248] In Figure 5C19, input (e.g., selection and movement input) by contact 5342 is detected on surface 5344 of virtual box 5340. In response to detection of contact 5342 selecting surface 5344 of virtual box 5340, movement projection 5348 is shown extending from virtual box 5340 to indicate the plane of movement of virtual box 5340 (e.g., the plane of movement parallel to selected surface 5344 of virtual box 5340). In Figures 5194-5C20, contact 5342 moves along the surface of touch-sensitive display 112 in the direction indicated by arrow 5346. In response to the movement of contact 5342, virtual box 5340 moves within the plane indicated by movement projection 5348 in the direction indicated by arrow 5346.

[0249] In FIG. 5C21, contact 5342 has lifted off touch-sensitive display 112 and moving projection 5384 is no longer displayed.

[0250] 5C22 and 5C23 show input gestures (eg, rotation gestures) for changing the viewpoint of the simulated environment.

[0251] In Figure 5C22, contact 5350 is detected on touch-sensitive display 112. In Figures 5C22 and 5C23, contact 5350 moves along the path indicated by arrow 5352. As contact 5350 moves, the simulated environment rotates. In Figure 5C23, the positions of virtual reference grid 5328 and virtual boxes 5302, 5304, and 5340 have rotated in response to the input from contact 5350.

[0252] In Figures 5C24 and 5C25, device 100 is moved along the path indicated by arrow 5354. Because the simulated environment displayed on display 112 in Figures 5C24 and 5C25 is displayed in a virtual reality viewing mode, the positions of virtual boxes 5302 and 5304 on display 112 do not change as device 100 is moved.

[0253] 5C26 and 5C27 show an input gesture (de-pinch gesture) that causes a change of viewing mode from a virtual reality viewing mode to an augmented reality viewing mode.

[0254] In Figure 5C26, contacts 5356 and 5360 are detected on touch-sensitive display 112. In Figures 5C26 and 5C27, contact 5356 moves along the path indicated by arrow 5358, and contact 5360 moves along the path indicated by arrow 5362. In response to the simultaneous movement of contacts 5356 and 5360 increasing the distance between contact 5356 and contact 5360, the displayed view of the simulated environment, including virtual boxes 5302, 5304, and 5340, is zoomed in (e.g., causing virtual boxes 5302, 5304, and 5340 to increase in size on display 112). When the zoom input is received, a transition from the virtual reality viewing mode to the augmented reality viewing mode is triggered. The transition animation performed during the transition includes a gradual fade-out of virtual reference grid 5328 (e.g., as shown in Figures 5C26 and 5C27). The transition animation includes a gradual fade-in of the view of the physical environment. For example, table 5204 and reference mat 5208a as captured by one or more cameras of device 100 gradually become visible on display 112 (e.g., as shown in FIGS. 5C28-5C30). During the transition, the appearance of toggle 5214 changes to indicate the current viewing mode (e.g., as shown in FIGS. 5C27-5C28). After liftoff of contacts 5356 and 5360, virtual boxes 5302, 5304, and 5340 in the simulated environment continue to increase in size, move, and rotate (e.g., until the original space between virtual boxes 5302 and 5304 and reference mat 5208a is restored), as shown in FIGS. 5C28-5C30.

[0255] In some embodiments, a virtual box 5340 added while the virtual reality viewing mode was active is visible in the alternate reality viewing mode, as shown in FIG. 5C30.

[0256] In some embodiments, the change in viewing mode from the virtual reality viewing mode to the augmented reality viewing mode occurs in response to a contact input (e.g., a tap input) at a location corresponding to toggle 5214. For example, a transition from displaying the virtual reality viewing mode (e.g., as shown in FIG. 5C26) to the augmented reality viewing mode (e.g., as shown in FIG. 5C30) occurs in response to a tap input detected at a location corresponding to toggle 5214. In some embodiments, a transition animation the same as or similar to the animations shown in 5C26-5C30 is displayed during the transition.

[0257] 5D1-5D14 show example systems and user interfaces for updating an indication of a viewing viewpoint of a second computer system within a simulated environment displayed by a first computer system, according to some embodiments. The user interfaces in these figures are used to illustrate processes described below, including the processes in FIGS. 11A-11C. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. Similarly, similar operations are performed on a computer system having a headset 5008 and a separate input device 5010 with a touch-sensitive surface (e.g., as shown in FIG. 5A2) in response to detecting contact on the touch-sensitive surface of input device 5010 while displaying the illustrated user interface on the display of headset 5008, optionally with a focus indicator.

[0258] 5D1 and 5D2 illustrate situations in which the user interfaces described with respect to FIGS. 5D3-5D14 are used.

[0259] 5D1 shows a physical space 5400 in which two users 5402 and 5408 and a table 5414 are located. A first device 5406 (e.g., device 100) is held by first user 5402 in first user's hand 5404. A second device 5412 (e.g., device 100) is held by second user 5408 in second user's hand 5410. A reference mat 5416a is located on table 5414.

[0260] FIG. 5D2 shows a view of a virtual three-dimensional space displayed on a display 5148 (e.g., display 112) of device 5406. A reference mat 5416 is within the field of view of one or more cameras (e.g., light sensors 164) of device 5406 (hereinafter referred to as "cameras" to refer to the one or more cameras of device 5406). Display 5148 shows a live view of physical space 5400 as captured by the cameras, including a displayed version 5416b of a physical reference mat 5416a. A virtual user interface object (virtual box 5420) in the simulated environment is displayed on display 5418. In some embodiments, virtual box 5420 is anchored to reference mat 5416b, such that the view of virtual box 5420 changes as the view of displayed version 5416b of the reference mat changes in response to movement of device 100 relative to reference mat 5416a. User interface features are further described above with respect to FIG. 5B5.

[0261] 5D3-5D11 include subview "a" (e.g., as shown in FIG. 5D3a) illustrating the orientation in physical space 5400 of first device 5406 and second device 5412 relative to table 5414, subview "b" (e.g., as shown in FIG. 5D3b), and subview "c" (e.g., as shown in FIG. 5D3c) illustrating the user interface of second device 5412. In order to provide a full view of the user interfaces, the user interfaces of FIGS. 5D3-5D11 do not show the hands holding the devices. Also, for clarity, the user interfaces of FIGS. 5D3-5D11 do not show the bodies of users 5402 and 5408. It should be understood that any part of user 5408's body that is within the field of view of device 5406's camera will typically be visible in the user interface displayed on device 5406 (although the view of the user's body may be obstructed by virtual user interface objects or other user interface elements). For example, in FIG. 5D2, the body and hands of user 5408 are visible within the user interface displayed by device 5409.

[0262] 5D3 and 5D4 show the movement of the second device 5412. FIG.

[0263] In FIG. 5D3a, the second device 5412 is displayed in a first position relative to the table 5414 (eg, adjacent to the far left side of the table).

[0264] 5D3b, the user interface of device 5406 includes avatar key 5422, which includes key avatar 5424 corresponding to device 5406 and key avatar 5426 corresponding to device 5412. Avatar key 5422 includes a name ("Me") corresponding to key avatar 5424 and a name ("Zoe") corresponding to key avatar 5426. The key avatar shown in the avatar key provides a guide to the avatar (e.g., avatar 5428) shown in the visible environment to help the user of device 5406 understand, for example, that avatar 5428 in the simulated environment corresponds to device 5412 of user "Zoe" (e.g., because avatar 5428 in the simulated environment is a cat icon that matches key avatar 5426).

[0265] The simulated environment displayed on the user interface of device 5406 includes a virtual box 5420 shown from the perspective of device 5406 and a displayed view 5416b of a physical reference mat 5416a. The viewing perspective of device 5412 is indicated by a viewing perspective indicator 5432. Viewing perspective indicator 5432 is shown emanating from avatar 5428. A representation 5430 of device 5412 (e.g., a view of device 5412 as captured by a camera of device 5406 and / or a rendered version of device 5412) is shown in the simulated environment.

[0266] 5D3c, the user interface of device 5412 includes avatar key 5434, which includes key avatar 5436 corresponding to device 5412 and key avatar 5468 corresponding to device 5406. Avatar key 5434 includes a name corresponding to key avatar 5436 ("Me") and a name corresponding to key avatar 5438 ("Gabe"). The key avatar shown in the avatar key provides a guide to the avatar (e.g., avatar 5440) shown in the visible environment to help the user of device 5412 understand, for example, that avatar 5440 in the simulated environment corresponds to device 5406 of user "Gabe" (e.g., because avatar 5440 in the simulated environment is a smiling face icon that matches key avatar 5438).

[0267] In Figure 5D4a, the second device 5412 has moved from a first position relative to the table 5414 shown in Figure 5D3a to a second position relative to the table 5414 (e.g., a position adjacent to the front left side of the table). In Figure 5D4b, the user interface of device 5406 shows device 5412 (indicated by avatar 5428 and device representation 5430) in a changed position from Figure 5D3b. The change in viewing perspective of device 5412 is indicated by the different angle of viewing perspective indicator 5432 from Figure 5D3b to Figure 5D4b. The movement of device 5412 is also indicated by the displayed reference mat 5416b and virtual box 5420 of the changed view from within the user interface of device 5412 in Figure 5D3c to Figure 5D4c.

[0268] 5D5-5D7 illustrate the selection and movement of a virtual box 5420 by a device 5412. FIG.

[0269] 5D5c, contact 5446 input (e.g., selection and movement input) is detected on the touchscreen display of second device 5412 at a location corresponding to the surface of virtual box 5420. In response to detecting contact 5446 selecting the surface of virtual box 5420, movement projection 5448 is shown extending from virtual box 5420 to indicate the plane of movement of virtual box 5420 (e.g., a plane of movement parallel to the selected surface of virtual box 5420).

[0270] 5D5b, an interaction indicator 5452 is shown to indicate to the user of the first device 5406 that the second device 5412 is interacting with the virtual box 5420. The interaction indicator 5452 extends from a location corresponding to the avatar 5428 to a location corresponding to the virtual box 5420. A control handle 5454 is shown where the indication indicator 5452 intersects with the virtual box 5420.

[0271] 5D5c-5D6c, contact 5446 moves along the touch-sensitive display of device 5412 in the direction indicated by arrow 5450. In response to the movement of contact 5446, virtual box 5420 moves in the plane indicated by motion projection 5448 in the direction indicated by arrow 5450.

[0272] In Figures 5D5b-5D6b, when the virtual box 5420 is moved by a movement input detected on the second device 5412, the user interface of the first device 5406 shows movement of the interaction indicator 5452 and the control handle 5454 (e.g., to maintain a connection between the interaction indicator 5452 and the virtual box 5420).

[0273] In Figure 5D7c, contact 5446 has lifted off the touch-sensitive display device of device 5412 and movement projection 5448 is no longer displayed. In Figure 5D7b, interaction indicator 5452 and control handle 5454 are no longer displayed (because device 5412 is no longer interacting with virtual box 5420).

[0274] 5D8-5D11 show the resizing of the virtual box 5420 by the device 5412. FIG.

[0275] In FIG. 5D8c, input (eg, a resize input) by contact 5456 is detected on the touchscreen display of the second device 5412 at a location corresponding to the surface of the virtual box 5420.

[0276] In FIG. 5D8 b, an interaction indicator 5462 and a control handle 5464 are shown on the user interface of the first device 5406 to indicate that the second device 5412 is interacting with the virtual box 5420 .

[0277] In FIG. 5D9c, after contact 5456 remains at a position corresponding to the surface of virtual box 5420 for a period of time that increases above the resize time threshold, resize projection 5458 is shown to indicate the axis (perpendicular to the selected surface of virtual box 5420) along which virtual box 5420 will be resized in response to subsequent movement of contact 5456.

[0278] 5D9a-5D10a show second device 5412 moving upward (while contact 5456 is in contact with the touchscreen display of second device 5412) to resize virtual box 5420. As device 5412 moves, virtual box 5420 increases in size along the axis indicated by resize projection 5458 in the direction second device 5412 is moved.

[0279] In FIG. 5D11c, contact 5456 has lifted off touch-sensitive display 112 and projection 5458 is no longer displayed.

[0280] As shown in Figures 5D12-5D14, users who are not in the same physical space can view and collaboratively manipulate objects in a simulated environment. For example, a user in a first physical space views a virtual user interface object (e.g., virtual box 5420) anchored to a displayed version of a first physical reference mat (e.g., 5416a), while a different user at a remote location views the same virtual user interface object anchored to a displayed version of a second physical reference mat (e.g., 5476a).

[0281] Figure 5D12a shows a first physical space 5400 in which two users 5402 and 5408 and a table 5414 are located, as shown in Figure 5D1. Figure 5D12b shows a second physical space 5470, separate from the first physical space 5400, in which a third user 5472 and a table 5474 are located. A third device 5478 (e.g., device 100) is held by the third user 5472. A reference mat 5476a is located on the table 5474. The device 5478 of the third user 5472 displays the same simulated environment displayed by the device 5412 of the first user 5408 and the device 5404 of the second user 5402.

[0282] FIG. 5D13a shows a first physical space 5400 as described with respect to FIG. 5D12a, and FIG. 5D13b shows a second physical space 5470 as described with respect to FIG. 5D12b.

[0283] 5D13c, the user interface of first device 5406 includes avatar key 5422 that includes key avatar 5480 (for "Stan") corresponding to third device 5478. Avatar 5482 (as indicated by key avatar 5480) corresponding to third device 5478 is shown in the simulated environment displayed by device 5406 at a position relative to displayed version 5416b of physical reference mat 5416a that corresponds to the position of device 5478 relative to physical reference mat 5476a. The viewing viewpoint of device 5478 is indicated by viewing viewpoint indicator 5486. A representation 5484 of device 5478 (e.g., a rendered version of the device) is shown in the simulated environment displayed by device 5406.

[0284] As shown in FIG. 5D13d, the user interface of the second device 5412 also displays an avatar 5482 corresponding to the third device 5478, a viewing viewpoint indicator 5486 indicating the viewing viewpoint of the device 5478, and a representation 5484 of the device 5478.

[0285] FIG. 5D14a shows a first physical space 5400 as described with respect to FIG. 5D12a, and FIG. 5D14b shows a second physical space 5470 as described with respect to FIG. 5D12b.

[0286] Figure 5D14c shows the user interface of a third device 5478. In Figure 5D14c, a virtual box 5420 is shown anchored to a displayed view 5476b of a physical reference mat 5476a. An avatar 5488 corresponding to the first device 5406 is shown in the simulated environment displayed by the third device 5478 at a position relative to a displayed version 5476b of the physical reference mat 5476a that corresponds to the position of the first device 5406 relative to the physical reference mat 5416a. The viewing viewpoint of the first device 5406 is indicated by a viewing viewpoint indicator 5490. A representation 5490 of the first device 5406 (e.g., a rendered version of the first device) is shown in the simulated environment displayed by the third device 5476. An avatar 5494 corresponding to the second device 5412 is shown in the simulated environment at a position relative to a displayed version 5476b of the physical reference mat 5476a that corresponds to the position of the second device 5412 relative to the physical reference mat 5416a. The viewing viewpoint of the second device 5412 is indicated by a viewing viewpoint indicator 5498. A representation 5496 of the second device 5412 (e.g., a rendered version of the second device) is shown in the simulated environment displayed by the third device 5476.

[0287] 5E1-5E32 show example systems and user interfaces for placement of an insertion cursor, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes of FIGS. 12A-12D. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. Similarly, similar operations are performed on a computer system having headset 5008 and a separate input device 5010 with a touch-sensitive surface (e.g., as shown in FIG. 5A2) in response to detecting contact on the touch-sensitive surface of input device 5010 while displaying the user interface shown in the figures on the display of headset 5008, optionally with a focus indicator.

[0288] 5E1-5E3 illustrate situations in which the user interfaces described with respect to FIGS. 5E4-5E32 are used.

[0289] 5E1 shows a physical space 5200 in which a user 5202 and a table 5204 are located. The device 100 is held by the user 5202 in the user's hand 5206. A reference mat 5208 is located on the table 5204.

[0290] 5E2 shows a view of the virtual three-dimensional space displayed on display 112 of device 100. A reference mat 5208 is within the field of view of one or more cameras (e.g., light sensors 164) of device 100 (hereinafter referred to as "cameras" to refer to one or more cameras of device 100). Display 112 shows a live view of physical space 5200 as captured by the cameras, including a displayed version 5208b of physical reference mat 5208a.

[0291] In Figure 5E3, the device 100 has been moved closer to the reference mat 5208a. As a result, the displayed version 5208b of the reference mat has increased in size.

[0292] 5E4-5E32 show a larger view of device 100, providing a full view of the user interface displayed on display 112 and not showing user's hand 5206. FIG.

[0293] 5E5 and 5E6 show inputs that cause placement of the insertion cursor at a first location.

[0294] In Figure 5E5, input (e.g., a tap input) by contact 5502 is detected at a first location on displayed version 5208b of physical reference mat 5208a. In Figure 5E6, contact 5502 has lifted off touch-sensitive display 112, and insertion cursor 5504 is shown at the location where contact 5502 was detected.

[0295] Figures 5E7 and 5E8 show input that causes placement of an insertion cursor at a second location. In Figure 5E7, contact 5506 input (e.g., a tap input) is detected at a location different from the location where insertion cursor 5504 was displayed. In Figure 5E8, contact 5506 has lifted off touch-sensitive display 112, and insertion cursor 5508 is shown at the location where contact 5506 was detected.

[0296] 5E9 and 5E10 illustrate inputs that cause the insertion of a virtual user interface object. In FIG. 5E9, input by contact 5510 (e.g., a tap input) is detected at a location corresponding to the location of insertion cursor 5508. In response to detecting contact 5510 at the location where the insertion cursor was located, a virtual user interface object (first virtual box 5512) is displayed on display 112 at a location corresponding to contact 5510, and insertion cursor 5508 is moved from its previous location on displayed view 5208b of reference mat 5208a to surface 5514 of first virtual box 5512. In some embodiments, a shadow 5522 is displayed (e.g., a simulated light causes a shadow to be cast by first virtual box 5512).

[0297] 5E11 and 5E12 show input detected on the surface of a virtual user interface object (first virtual box 5512) that causes the insertion of an additional virtual user interface object. In FIG. 5E11, contact 5516 input (e.g., a tap input) is detected at a position on surface 5514 of first virtual box 5512 while insertion cursor 5516 is positioned on surface 5514. In response to detecting contact 5516 input, a new virtual user interface object (second virtual box 5518) is displayed on display 112 at a position corresponding to contact 5510, and insertion cursor 5508 is moved from surface 5514 of first virtual box 5512 to surface 5520 of second virtual box 5518. The length of shadow 5522 is increased (so that it appears to be cast by first virtual box 5512 and the newly added second virtual box 5518).

[0298] 5E12 and 5E13 illustrate the rotation of the physical reference mat 5208. For example, the user 5202 manually changes the position and / or orientation of the reference mat 5208. As the physical reference mat 5208a rotates, the virtual boxes 5512 and 5518 and the shadow 5522 rotate (because the virtual boxes 5512 and 5518 are fixed to the displayed view 5208b of the physical reference mat 5208a).

[0299] 5E14-5E16 illustrate movement of device 100. For example, user 5202 holding device 100 changes the device's position and / or orientation. In FIGS. 5E14 and 5E15, as device 100 moves, virtual boxes 5512 and 5518 and shadow 5522 move (because virtual boxes 5512 and 5518 are anchored to displayed view 5208b of physical reference mat 5208a). Similarly, in FIGS. 5E15 and 5E16, as device 100 moves, virtual boxes 5512 and 5518 and shadow 5522 move.

[0300] Figures 5E17 and 5E18 illustrate inputs that change the position of insertion cursor 5526 on virtual box 5518. In Figure 5E17, while insertion cursor 5526 is positioned on surface 5520 of virtual box 5518, input by contact 5524 (e.g., a tap input) is detected on surface 5528 of virtual box 5518. In Figure 5E18, contact 5524 has lifted off touch-sensitive display 112, and insertion cursor 5508 is moved from surface 5520 of virtual box 5518 to surface 5528 of virtual box 5518.

[0301] 5E19 and 5E20 show input detected on the surface of second virtual box 5518, causing the insertion of third virtual box 5532. In FIG. 5E19, contact 5530 input (e.g., a tap input) is detected at a position on surface 5528 of second virtual box 5518 while insertion cursor 5526 is positioned on surface 5268. In response to detecting contact 5530 input, third virtual box 5532 is displayed on display 112 at a position corresponding to contact 5530, and insertion cursor 5526 is moved from surface 5528 of second virtual box 5518 to surface 5526 of third virtual box 5532. The shape of shadow 5522 is changed (so that it appears to be cast by first virtual box 5512, second virtual box 5518, and newly added third virtual box 5532).

[0302] Figures 5E21 and 5E22 show inputs that change the position of insertion cursor 5538 on virtual box 5532. In Figure 5E21, while insertion cursor 5526 is positioned on surface 5534 of virtual box 5532, input by contact 5536 (e.g., a tap input) is detected on surface 5538 of virtual box 5532. In Figure 5E22, contact 5536 has lifted off touch-sensitive display 112, and insertion cursor 5526 is moved from surface 5534 of virtual box 5518 to surface 5538 of virtual box 5532.

[0303] 5E23 and 5E24 illustrate the insertion of a new virtual user interface object using the new object control 5216.

[0304] In Figure 5E23, while insertion cursor 5526 is on surface 5538 of virtual box 5532, input by contact 5542 (e.g., a tap input) is detected at a location on display 112 corresponding to new object control 5216. In Figure 5E24, in response to the input at the location corresponding to new object control 5216, a fourth virtual box 5546 is displayed on display 112 at a location corresponding to where insertion cursor 5526 was shown, and insertion cursor 5526 is moved from surface 5538 of virtual box 5532 to surface 5548 of fourth virtual box 5546.

[0305] 5E25-5E27 show inputs that cause the fourth virtual box 5546 to move.

[0306] In Figure 5E25, input by contact 5550 (e.g., selection and movement input) is detected on surface 5556 of fourth virtual box 5546. In response to detecting contact 5550 selecting surface 5556 of fourth virtual box 5546, movement projection 5552 is shown extending from virtual box 5546 to indicate the movement plane of fourth virtual box 5546 (e.g., a plane of movement parallel to selected surface 5556 of virtual box 5546).

[0307] 5E25 and 5E26, contact 5550 is moving along the surface of touch-sensitive display 112 in the direction indicated by arrow 5554. In response to the movement of contact 5550, fourth virtual box 5546 is moving in the plane indicated by moving projection 5552 in the direction indicated by arrow 5554. In Figure 5E27, contact 5550 has lifted off touch-sensitive display 112, and moving projection 5552 is no longer displayed.

[0308] 5E28-5E32 show inputs that cause the fourth virtual box 5546 to resize.

[0309] In FIG. 5E28 , input (eg, a resize input) by contact 5258 is detected on touchscreen display 112 at a location corresponding to surface 5556 of fourth virtual box 5546 .

[0310] In FIG. 5E29, after contact 5255 remains at a position corresponding to surface 5556 of fourth virtual box 5546 for a period of time that increases above the resize time threshold, resize projection 5560 is shown indicating the axis (perpendicular to the selected surface of virtual box 5546) along which virtual box 5546 will be resized in response to subsequent movement of contact 5558.

[0311] 5E30 and 5E31, contact 5558 moves across touchscreen display 112 along a path indicated by arrow 5562. In response to the movement of contact 5558, virtual box 5548 increases in size along an axis indicated by resize projection 5560 in the direction of the movement of contact 5558.

[0312] In FIG. 5E32, contact 5558 has lifted off touch-sensitive display 112 and projection 5560 is no longer displayed.

[0313] Figures 5F1-5F17 show example systems and user interfaces for displaying an augmented reality environment in a stable mode of operation, according to some embodiments. The user interfaces in these figures are used to illustrate processes described below, including those in Figures 13A-13E. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. Similarly, similar operations are performed on a computer system having headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in Figure 5A2) in response to detecting contact on the touch-sensitive surface of input device 5010 while displaying the illustrated user interface on the display of headset 5008, optionally with a focus indicator.

[0314] 5F1 and 5F2 illustrate a situation in which the user interface described with respect to FIGS. 5F3-5F17 is used.

[0315] 5F1 shows a physical space 5200 in which a user 5202 and a table 5204 are located. The device 100 is held by the user 5202 in the user's hand 5206. An object (physical box 5602) is located on the table 5204.

[0316] 5F2 shows an augmented reality environment displayed by display 112 of device 100. Table 5204 (referred to as 5204a when referring to the table in the physical space) and physical box 5602 are within the field of view of one or more cameras (e.g., light sensors 164) of device 100 (hereinafter referred to as the “cameras” to refer to the one or more cameras of device 100). Display 112 shows a live view of physical space 5200 as captured by the cameras, including a displayed version 5204b of table 5204a and a rendered virtual box 5604 displayed in a position in the simulated environment that corresponds to physical box 5602 as detected by the cameras of device 100.

[0317] 5F3-5F17 include subview "a" (e.g., as shown in FIG. 5F3a) showing the orientation of device 100 in physical space 5200 relative to table 5204a and physical box 5602, as well as subview "b" (e.g., as shown in FIG. 5F3b) showing the user interface of device 100. Also, for clarity, FIGS. 5F3-5F18 show a larger view of device 100 to provide a full view of the user interface displayed on display 112 and do not show user's hands 5206.

[0318] 5F3a-5F4a illustrate movement of device 100 relative to table 5204a and physical box 5602 while an augmented reality environment is displayed by device 100 in a non-stable mode of operation (as shown in FIGS. 5F3b-5F4b). When device 100 is in a first position relative to table 5204a, as shown in FIG. 5F3a, rendered version 5604 of physical object 5602 is fully visible in the user interface shown in FIG. 5F3b. In FIG. 5F4a, device 100 has been moved to a second position relative to table 5204a, and rendered version 5604 of physical object 5602 is only partially visible in the user interface shown in FIG. 5F4b. In the non-stable operating mode, as device 100 moves, the view of virtual box 5604 changes to maintain a fixed spatial relationship between virtual box 5604 and physical box 5602, and the displayed representation of the field of view of the camera of device 100 (e.g., including displayed table 5204b) is updated based on the movement of the device.

[0319] 5F5-5F8 illustrate inputs (eg, depinch zoom out inputs) that cause the device to display an augmented reality environment in stable operating mode.

[0320] In Figure 5F5, device 100 is in a first position relative to table 5204. In Figure 5F6, contacts 5606 and 5608 are detected on touch-sensitive display 112 (as shown in Figure 5F6b). As shown in Figures 5F6b-5F7b, contact 5606 moves along the path indicated by arrow 5610, and contact 5608 moves along the path indicated by arrow 5612. In response to the simultaneous movement of contacts 5606 and 5608 increasing the distance between contacts 5606 and 5608, the displayed augmented reality environment including virtual box 5604 is zoomed in (e.g., causing virtual box 5604 to increase in size on display 112). Virtual box 5604 is re-rendered in response to the zoom input (e.g., the larger virtual box 5604 in Figure 5F8b has the same resolution as the smaller virtual box 5604 in Figure 5F5b). In some embodiments, the field of view of the camera of device 100 displayed on display 112 (e.g., displayed view 5204b of table 5204a) does not change in response to the zoom input (as shown in Figures 5F5b-5F8b). When the zoom input is received, a transition from a non-stable to a stable mode of operation occurs. In Figure 5F8, contacts 5606 and 5608 have lifted off of touchscreen display 112.

[0321] In some embodiments, while the device is displaying an augmented reality environment in a stable operating mode, if movement of the device causes the virtual user interface object to extend beyond the field of view of the device camera, a portion of the virtual user interface object ceases to be displayed. Figures 5F8 and 5F9 illustrate movement of device 100 while device 100 is in a stable operating mode, causing a portion of virtual user interface object 5304 to cease to be displayed. Figures 5F8a-5F9a illustrate movement of device 100 relative to table 5204a and physical box 5602 (as shown in Figures 5F8b-5F9b) while an augmented reality environment is displayed by device 100 in a stable operating mode. As shown in Figure 5F8a, when device 100 is in a first position relative to table 5204a, a zoomed, rendered version 5604 of physical object 5602 is fully visible in the user interface shown in Figure 5F8b. In Figure 5F9a, device 100 has moved to a second position relative to table 5204a, thereby updating the view of virtual box 5604 to cause virtual box 5604 to extend beyond the field of view of the camera of device 100 by maintaining a fixed spatial relationship between virtual box 5604 and physical box 5602. As a result, the portion of virtual box 5604 that extends beyond the field of view of the camera of device 100 is not displayed.

[0322] In some embodiments, while a device is displaying an augmented reality environment in stable operating mode and movement of the device causes a virtual user interface object to extend beyond the field of view of the device camera, the augmented reality environment is zoomed out so that the virtual user interface object is fully displayed. For example, in Figures 5F9b through 5F10b, the displayed augmented reality environment including virtual box 5604 is zoomed out so that virtual box 5604 is fully displayed.

[0323] In some embodiments, in a stable mode of operation, updating the view of virtual box 5604 to maintain a fixed spatial relationship between virtual box 5604 and physical box 5602 causes virtual box 5604 to extend beyond the field of view of the device camera 100, so that virtual box 5604 is displayed with a placeholder image at a location corresponding to the portion of virtual box 5604 that extends beyond the field of view of the device camera. In Figure 5F10b, rendered version 5604 of physical object 5602 is displayed with placeholder image 5614 (white space) at a location corresponding to the portion of virtual box 5064 that extends beyond the field of view of the device camera. For example, placeholder image 5614 is displayed at a location in the augmented reality environment that is beyond the field of view of the camera, and therefore camera data that would be displayed within the space occupied by placeholder image 5614 is not available.

[0324] 5F10 and 5F11 show the movement of device 100 (back to the position of device 100 shown in FIGS. 5F8 and 5F3).

[0325] 5F11a-5F12a show movement of device 100 (e.g., moving back away from table 5204a and physical object 5602, causing device 100 to appear larger). In FIG. 5F12b, as a result of the movement shown in FIG. 5F11a-5F12a, the size of virtual object 5604 has decreased from the size of virtual object 5604 in FIG. 5F11b. This movement is shown in FIG. 5F11a-5F12a for illustrative purposes (whereby the size of virtual object 5604 in FIG. 5F12b in stable mode is the same as the size of virtual object 5604 in FIG. 5F3b in unstable mode) to provide a brief comparison of updates to the augmented reality environment in stable and unstable modes of operation.

[0326] 5F12a-5F13a illustrate movement of device 100 relative to table 5204a and physical box 5602 while an augmented reality environment is displayed by device 100 in a stable operating mode. As shown in FIG. 5F12a, when device 100 is in a first position relative to table 5204a, rendered version 5604 of physical object 5602 is fully visible in the user interface shown in FIG. 5F12b. In FIG. 5F13a, device 100 has been moved to a second position relative to table 5204a, and rendered version 5604 of physical object 5602 is only partially visible in the user interface shown in FIG. 5F13b. In some embodiments, in a stable operating mode, as device 100 moves, the view of virtual box 5604 changes to maintain a fixed spatial relationship between virtual box 5604 and physical box 5602, and the displayed representation of the field of view of the camera of device 100 (e.g., including displayed table 5204b) changes by an amount less than the amount of change that occurs in a non-stable mode (e.g., the amount of movement of displayed table 5204b from FIG. 5F12b to FIG. 5F13b while device 100 is in a stable operating mode is less than the amount of movement of displayed table 5204b from FIG. 5F4b to FIG. 5F5b while device 100 is in a non-stable operating mode).

[0327] Figures 5F14-5F16 illustrate an input at stabilization toggle 5616 to transition from an unstable to a stable mode of operation. In Figure 5F15b, an input by contact 5618 (e.g., a tap input) is detected at a location on touchscreen display 112 corresponding to stabilization toggle 5616. In response to the input by contact 5618, the appearance of stabilization toggle 5616 is changed (e.g., the toggle changes from a non-shaded state to a shaded state), as shown in Figure 5F16b, to indicate that a transition from an unstable to a stable mode of operation has occurred.

[0328] 5F16a-5F17a illustrate movements of device 100 relative to table 5204a and physical box 5602 (as shown in FIGS. 5F16a-5F17a) while an augmented reality environment is displayed by device 100 in a stable operating mode. As shown in FIG. 5F16a, when device 100 is in a first position relative to table 5204a, rendered version 5604 of physical object 5602 is fully visible in the user interface shown in FIG. 5F16b. In FIG. 5F17a, device 100 has been moved to a second position relative to table 5204a, and rendered version 5604 of physical object 5602 is only partially visible in the user interface shown in FIG. 5F17b. In stable operating mode, as device 100 moves, the view of virtual box 5604 changes to maintain a fixed spatial relationship between virtual box 5604 and physical box 5602, and the displayed representation of the field of view of the camera of device 100 (e.g., including displayed table 5204b) changes by an amount less than the amount of change that occurs in non-stable mode (e.g., the amount of movement of displayed table 5204b from Figure 5F15b to Figure 5F16b while device 100 is in stable operating mode is less than the amount of movement of displayed table 5204b from Figure 5F4b to Figure 5F5b while device 100 is in non-stable operating mode).

[0329] 6A-6D are flow diagrams illustrating a method 600 for adjusting the appearance of virtual user interface objects in an augmented reality environment, according to some embodiments. Method 600 is performed on a computer system (e.g., portable multifunction device 100, FIG. 1A , device 300, FIG. 3A , or a multi-component computer system including headset 5008 and input device 5010, FIG. 5A2) having a display generation component (e.g., a display, projector, heads-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of the content within the camera's field of view, and optionally generates a video output including one or more streams of image frames capturing the content within the camera's field of view), and an input device (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touchscreen display that also functions as a display generation component, a mouse, joystick, wand controller, and / or a camera that tracks the position of one or more features of a user, such as the user's hands). In some embodiments, the input device (e.g., having a touch-sensitive surface) and the display generation component are integrated into a touch-sensitive display. As described above with respect to Figures 3B and 3D, in some embodiments, method 600 is performed on a computer system 301 (e.g., computer system 301-a, 301-b, or 301-c) where each of the components, such as a display generation component, one or more cameras, one or more input devices, and optionally one or more orientation sensors, is either included in computer system 301 or is in communication with computer system 301, respectively.

[0330] In some embodiments, the display generation component is a touchscreen display, and the input device (e.g., having a touch-sensitive surface) is on or integrated with the display generation component. In some embodiments, the display generation component is separate from the input device (e.g., as shown in FIGS. 4B and 5A2). Some operations of method 600 are optionally combined, and / or the order of some operations is optionally changed.

[0331] For ease of explanation, some of the embodiments are described with reference to operations performed on a computer system having touch-sensitive display system 112 and one or more integrated cameras (e.g., on device 100 having touchscreen 112). However, similar operations are optionally performed on a computer system having headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 5A2 ) in response to detecting a contact on the touch-sensitive surface of input device 5010 while displaying the illustrated user interface on the display of headset 5008. Similarly, similar operations are optionally performed on a computer system having one or more cameras implemented separately (e.g., within a headset) from one or more other components (e.g., input devices) of the computer system, and in some such embodiments, "movement of the computer system" corresponds to movement of one or more cameras of the computer system or movement of one or more cameras in communication with the computer system.

[0332] As described below, method 600 relates to adjusting the appearance of virtual user interface objects (on a display of a computer system) within an augmented reality environment (e.g., reality is augmented with supplemental information that provides a user with additional information not available in the physical world) based on a combination of movement of the computer system (e.g., movement of one or more cameras of the computer system) and movement of a contact on an input device (e.g., a touchscreen display) of the computer system. In some embodiments, adjusting the appearance of the virtual user interface objects allows a user to access the supplemental information within the augmented reality environment. Adjusting the appearance of a virtual user interface object based on a combination of movement of a computer system and movement of a contact on an input device of the computer system provides an intuitive way for a user to adjust the appearance of a virtual user interface object (e.g., by allowing a user to adjust the appearance of a virtual user interface object by movement of the computer system alone, movement of a contact on an input device alone, or a combination of movement of the computer system and movement of the contact), allows a user to expand the range of adjustments available to the user (e.g., by allowing a user to continue adjusting the appearance of a virtual user interface object even when the contact or one or more cameras of the computer system cannot move further in a desired direction), thereby improving operability of the device and making the user-device interface more efficient (e.g., by reducing the number of steps required to achieve an intended result when operating the device and reducing user errors when operating / interacting with the device), and additionally reducing power usage and improving the battery life of the device by allowing a user to use the device more quickly and efficiently.

[0333] A computer system (e.g., device 100, FIG. 5A7) displays (602) an augmented reality environment (e.g., as shown in FIG. 5A7) via a display generation component (e.g., touchscreen 112, FIG. 5A7). Displaying the augmented reality environment includes a representation of at least a portion of the field of view of one or more cameras that includes corresponding physical objects (e.g., a three-dimensional model of a building, a piece of paper with a printed pattern, a poster on a wall or other physical object, a statue placed on a surface, etc.) (e.g., physical building model 5006, FIG. 5A1), where the representation is updated as the content of the field of view of the one or more cameras changes (e.g., the representation is a live preview of at least a portion of the field of view of the one or more cameras, where the corresponding physical objects are contained within and visible within the field of view of the cameras), and a corresponding virtual object at a corresponding position within the representation of the field of view of the one or more cameras. The method includes simultaneously displaying a user interface object (604) (e.g., a virtual roof of a three-dimensional model of a building, a virtual car parked on a surface represented by a piece of paper with a printed pattern, an interactive logo superimposed on a poster, a virtual three-dimensional mask covering the outline of a statue, etc.) (e.g., virtual building model 5012, FIG. 5A7), where the corresponding virtual user interface object (e.g., virtual building model 5012, FIG. 5A7) has a position determined based on a corresponding physical object (e.g., physical building model 5006) in the field of view of the one or more cameras. For example, in some embodiments, the corresponding virtual user interface object is a graphical object or a two-dimensional or three-dimensional virtual object that appears to be attached to or covers the corresponding physical object in the field of view of the one or more cameras (e.g., virtual building model 5012 is a three-dimensional virtual object that appears to cover physical building model 5006, FIG. 5A7). The position and / or orientation of the corresponding virtual user interface object is determined based on the position, shape, and / or orientation of the physical object within the field of view of one or more cameras (eg, as shown in Figures 5A3 and 5A5).While displaying the augmented reality environment (606), the computer system detects input (e.g., detects input on an input device, such as by detecting touch input through contact on a touchscreen display or touch-sensitive remote control) at a location corresponding to the corresponding virtual user interface object (e.g., a location on a touchscreen display or touch-sensitive remote control, or movement of a wand or the user's hand while a cursor is over the location of the corresponding virtual user interface object) (e.g., device 100 detects contact 5020-a on the virtual roof of virtual building model 5012, FIG. 5A8).

[0334] While continuing to detect 608 the input (e.g., while contact is maintained on the input device, such as while contact is maintained on a touchscreen display or a touch-sensitive remote control) (e.g., while contact 5020 is maintained on touchscreen 112, FIGS. 5A9-5A13), the computer system detects movement of the input relative to a corresponding physical object within the field of view of one or more cameras (e.g., as shown in FIGS. 5A9-5A13). In some embodiments, the movement of the input optionally includes movement of the contact across a touchscreen display or across a touch-sensitive surface of a touch-sensitive remote control while the computer system (e.g., device 100) is held substantially stationary in physical space (e.g., as shown in FIGS. 5A8-5A11). In some embodiments, the movement of the input optionally includes movement of the device, including the camera, in physical space while contact is maintained on the touchscreen display or touch-sensitive remote control and remains stationary (e.g., as shown in FIGS. 5A17 and 5A18). In some embodiments, the movement of the input includes, optionally simultaneously, movement of the contact across a touchscreen display or touch-sensitive remote control and movement of a device, including a camera, in physical space. In some embodiments, the movement of the computer system includes movement of a component of a multi-component computer system, such as movement of a virtual reality display headset or the like (e.g., as shown in FIG. 5A2). Additionally, while continuing to detect the input, and in response to detecting movement of the input relative to a corresponding physical object within the field of view of one or more cameras, the device adjusts the appearance of the corresponding virtual user interface object (e.g., by enlarging, shrinking, stretching, squeezing together, expanding, and / or squeezing together all or a portion(s) of the virtual user interface object) according to the magnitude of the movement of the input relative to the corresponding physical object.For example, when a contact is detected on the virtual roof of an architectural model and then moved across the touchscreen display, the virtual roof is lifted away from the architectural model in the live preview of the camera's field of view (e.g., as shown in Figures 5A8-5A11), and while the contact is maintained on the touchscreen display and the device as a whole is moved relative to the architectural model in physical space, the movement of the virtual roof is determined based on both the position of the contact on the touchscreen display and the position and orientation of the device relative to the corresponding physical object in physical space (e.g., as determined based on the position of the corresponding physical object shown in the live preview of the camera's field of view) (e.g., as shown in Figures 5A11-5A13).

[0335] As another example, in a block construction application (e.g., as described in further detail with respect to FIGS. 5B1-5B41 ), in which a virtual model is constructed on a corresponding physical object (e.g., a tabletop or a piece of paper with a printed pattern), when contact (e.g., contact 5262, FIG. 5B28 ) is detected on a block (e.g., in response to a long press input on the block), and the computer system displays a guide of how the block will scale (e.g., as shown using resize projection 5266, FIG. 5B29 ), while contact is maintained on the block and the device is moved relative to the block as a whole (e.g., as shown in FIGS. 5B28-5B30 ), the scaling of the block (e.g., stretching the block in the direction of the guide) is determined based on both the location of the contact on the touchscreen display (e.g., on a particular side or face of a block cube) and the position and orientation of the device relative to the corresponding physical object in physical space (e.g., as determined based on the position of the corresponding physical object shown in a live preview of the camera's field of view).

[0336] In some embodiments, adjusting the appearance of the corresponding virtual user interface object (e.g., virtual architectural model 5012, FIGS. 5A8-5A13) according to the magnitude of the movement of the input relative to the corresponding physical object includes: adjusting the appearance of the corresponding virtual user interface object by a first adjustment (e.g., a larger amount of relative movement causing a larger adjustment) (e.g., as shown in FIG. 5A10 compared to FIG. 5A9) according to a determination that the magnitude of the movement of the input relative to the corresponding physical object is a first magnitude (e.g., a relatively larger magnitude of movement); and adjusting the appearance of the corresponding virtual user interface object by a second adjustment (e.g., a smaller amount of relative movement causing a smaller adjustment) (e.g., as shown in FIG. 5A9 compared to FIG. 5A10) according to a determination that the magnitude of the movement of the input relative to the corresponding physical object is a second magnitude (e.g., a relatively smaller magnitude of movement) separate from the first magnitude (610). Adjusting the corresponding virtual user interface object by a first adjustment when the magnitude of the input movement is a first magnitude (e.g., a larger relative movement amount causes a larger adjustment), and adjusting the corresponding virtual user interface object by a second adjustment when the magnitude of the input movement is a second magnitude (e.g., a smaller relative movement amount causes a smaller adjustment), improves the visual feedback provided to the user (e.g., by making the computer system appear more responsive to user input), improves operability of the device, makes the user-device interface more efficient (e.g., by helping the user achieve the intended result with the required input and reducing user errors when operating / interacting with the device), and additionally reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

[0337] In some embodiments, the corresponding virtual user interface object (e.g., virtual building model 5012, FIGS. 5A3-5A6) is fixed (612) to the corresponding physical object (e.g., physical building model 5006) in the field of view of one or more cameras before and after the adjustment. For example, in some embodiments, the corresponding virtual user interface object appears to cover the corresponding physical object in the field of view of one or more cameras, and as the position and / or orientation of the physical object in the field of view of one or more cameras changes, the position and / or orientation of the corresponding virtual user interface object changes accordingly (e.g., as shown in FIGS. 5A3-5A6). In some embodiments, the corresponding virtual user interface object is fixed to the corresponding physical object in the field of view of one or more cameras during some or all of the adjustment (e.g., during the transition from FIG. 5A3 to FIG. 5A5). Anchoring the corresponding virtual user interface objects to corresponding physical objects improves the visual feedback provided to the user (e.g., by making the computer system appear more responsive to user input), improves usability of the device, makes the user-device interface more efficient (e.g., by helping the user achieve the intended result with the required input and reducing user errors when operating / interacting with the device), and additionally reduces power usage and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

[0338] In some embodiments, the appearance of the corresponding virtual user interface object is adjusted (614) in response to detecting movement of the input relative to the corresponding physical object within the field of view of one or more cameras, regardless of whether the movement of the input is due to movement of the input on the input device (e.g., movement of a contact across a touchscreen display or touch-sensitive surface of the input device while the input device is held substantially stationary in physical space) (e.g., as shown in FIGS. 5A9-5A11), movement of one or more cameras relative to the corresponding physical object (e.g., movement of a computer system including the cameras in physical space while a contact is maintained on a touchscreen display or touch-sensitive surface of the input device and remains stationary) (e.g., as shown in FIGS. 5A11-5A13), or a combination of movement of the input on the input device and movement of one or more cameras relative to the corresponding physical object (e.g., simultaneous movement of a contact across a touchscreen display or touch-sensitive surface of the input device and movement of a computer system including the cameras in physical space). Adjusting the appearance of virtual user interface objects regardless of the manner of input movement (e.g., by allowing a user to adjust the appearance of a virtual user interface object by only moving the input on an input device, by only moving the camera relative to a physical object, or by a combination of input and camera movement) provides an intuitive way for a user to adjust the appearance of a virtual user interface object, improves the visual feedback provided to the user (e.g., by making the computer system appear more responsive to user input), improves device usability and makes the user-device interface more efficient (e.g., by helping the user achieve the intended result with the required input and reducing user errors when operating / interacting with the device), and additionally reduces power usage and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

[0339] In some embodiments, the movement of the input relative to the corresponding physical object is based on movement of the field of view of one or more cameras relative to the corresponding physical object (e.g., as a result of movement of a computer system including the camera within physical space) (e.g., as shown in FIGS. 5A11-5A13) and movement of the input on the input device (e.g., movement of a contact across a touchscreen display or touch-sensitive surface of the input device) (e.g., as shown in FIGS. 5A8-5A11) (616). Enabling a user to move the input relative to the corresponding physical object through movement of the computer system and movement of the contact provides an intuitive way for the user to move the input, improves the feedback provided to the user (e.g., by making the computer system appear more responsive to user input), improves device usability, makes the user-device interface more efficient (e.g., by helping the user achieve intended results with required inputs and reducing user errors when operating / interacting with the device), and also reduces power usage and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

[0340] In some embodiments, movement of the input relative to the corresponding physical object is based on movement of the input on the input device (e.g., movement of a contact across a touchscreen display or touch-sensitive surface of the input device while the input device is held substantially stationary in physical space) (618) (e.g., as shown in Figures 5A8-5A11), and after the computer system adjusts the appearance of the corresponding virtual user interface object according to the magnitude of the movement of the input relative to the corresponding physical object (e.g., as shown in Figure 5A11), the computer system detects movement of the field of view of one or more cameras relative to the corresponding physical object (e.g., movement 5022, Figure 5A12), and in response to detecting the movement of the field of view of the one or more cameras relative to the corresponding physical object, continues to adjust the appearance of the corresponding virtual user interface object (e.g., in the same manner) according to the magnitude of the movement of the field of view of the one or more cameras relative to the corresponding physical object (e.g., as shown in Figure 5A13). In some embodiments, adjusting the appearance of the corresponding virtual user interface object includes moving a portion of the corresponding virtual user interface object, where the movement is initiated by a contact drag on the virtual user interface object and continued by moving the device as a whole (e.g., as shown in FIGS. 5A8-5A13). For example, when a contact is detected on a virtual roof of a three-dimensional model of a building and the contact is moved across the touchscreen display (e.g., in an upward direction), the virtual roof is lifted from the architectural model in the displayed augmented reality environment (e.g., as shown in FIGS. 5A8-5A11). After the virtual roof is lifted, as the device is moved as a whole (e.g., in an upward direction) relative to the architectural model in physical space, the virtual roof continues to lift (e.g., as shown in FIGS. 5A11-5A13) (and optionally, floors of the architectural model lift and extend).In some embodiments, adjusting the appearance of the corresponding virtual user interface object according to the magnitude of movement of the input, and then continuing to adjust the appearance of the corresponding virtual user interface object according to the magnitude of movement of the field of view of one or more cameras, allows the user to continue adjusting the appearance of the corresponding virtual user interface object even if the contact cannot move any further in the desired direction on the touchscreen display (e.g., because the touch is at or near the edge of the touchscreen display and further movement of the touch would move the touch off the edge of the touchscreen display). For example, in a virtual roof, if the contact approaches the top edge of the touchscreen display but the user still wants to continue raising the roof, the user can continue to do so by moving the device or camera (e.g., as shown in FIGS. 5A8-5A13), even if the contact cannot move any further up on the touchscreen display. By adjusting the appearance of the corresponding virtual user interface object according to the magnitude of movement of the input, and then continuing to adjust the appearance of the corresponding virtual user interface object according to the magnitude of movement of the field of view of one or more cameras, the user can expand the range of adjustments available to the user (e.g., allowing the user to continue adjusting the appearance of the corresponding virtual user interface object with movement of the computer system even when the contact cannot be moved further in the desired direction on the touchscreen display), thereby improving operability of the device and making the user-device interface more efficient (e.g., by reducing the number of steps required to achieve an intended result when operating the device and reducing user errors when operating / interacting with the device), and additionally reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

[0341] In some embodiments, movement of the input relative to the corresponding physical object is based on movement of the field of view of one or more cameras relative to the corresponding physical object (e.g., as a result of movement of one or more cameras of the computer system, or one or more cameras in communication with the computer system, in physical space while contact is maintained and remains stationary on the touchscreen display or touch-sensitive surface of the input device) (620) (e.g., as shown in FIGS. 5A17 and 5A18), and after the computer s...

Claims

1. 1. A computer system having a display generation component, one or more cameras, and an input device, comprising: Displaying an augmented reality environment via the display generation component, wherein displaying the augmented reality environment comprises: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; and detecting an input at a location corresponding to the corresponding virtual user interface object while displaying the augmented reality environment; While continuing to detect said input, Detecting movement of the input relative to the corresponding physical object within the field of view of the one or more cameras; responsive to detecting the movement of the input relative to the corresponding physical object within the field of view of the one or more cameras, adjusting an appearance of the corresponding virtual user interface object according to a magnitude of the movement of the input relative to the corresponding physical object; A method comprising:

2. adjusting the appearance of the corresponding virtual user interface object according to the magnitude of movement of the input relative to the corresponding physical object; adjusting the appearance of the corresponding virtual user interface object by a first adjustment in accordance with a determination that the magnitude of movement of the input relative to the corresponding physical object is a first magnitude; adjusting the appearance of the corresponding virtual user interface object by a second adjustment distinct from the first adjustment in accordance with a determination that the magnitude of movement of the input relative to the corresponding physical object is a second magnitude distinct from the first magnitude; The method of claim 1 , comprising:

3. The method of claim 1 or 2, wherein the corresponding virtual user interface object is fixed to the corresponding physical object in the field of view of the one or more cameras before and after the adjusting.

4. In response to detecting the movement of the input relative to the corresponding physical object within the field of view of the one or more cameras, the appearance of the corresponding virtual user interface object is adjusted, and the movement of the input is moving the input on the input device; and moving the one or more cameras relative to the corresponding physical object; or combining the movement of the input on the input device with the movement of the one or more cameras relative to the corresponding physical object; 4. The method according to claim 1 , wherein the method is based on the above-mentioned method.

5. said movement of said input relative to said corresponding physical object; moving the field of view of the one or more cameras relative to the corresponding physical object; and moving the input on the input device; The method according to any one of claims 1 to 4, which is based on

6. the movement of the input relative to the corresponding physical object is based on a movement of the input on the input device, the method comprising: adjusting the appearance of the corresponding virtual user interface object according to the magnitude of movement of the input relative to the corresponding physical object; detecting movement of the field of view of the one or more cameras relative to the corresponding physical object; in response to detecting the movement of the field of view of the one or more cameras relative to the corresponding physical object, continuing to adjust the appearance of the corresponding virtual user interface object according to a magnitude of the movement of the field of view of the one or more cameras relative to the corresponding physical object; The method of any one of claims 1 to 4, comprising:

7. the movement of the input relative to the corresponding physical object is based on movement of the field of view of the one or more cameras relative to the corresponding physical object, the method comprising: adjusting the appearance of the corresponding virtual user interface object according to the magnitude of movement of the input relative to the corresponding physical object; Detecting movement of the input on the input device; responsive to detecting the movement of the input on the input device, continuing to adjust the appearance of the corresponding virtual user interface object according to a magnitude of the movement of the input on the input device; The method of any one of claims 1 to 4, comprising:

8. detecting the input at the location corresponding to the corresponding virtual user interface object includes detecting the input at a first contact point on the corresponding virtual user interface object; the computer system updates the display of the corresponding virtual user interface object to maintain a display of the first contact point on the corresponding virtual user interface object at a position corresponding to a position of the input.

8. The method according to any one of claims 1 to 7.

9. the movement of the input relative to the corresponding physical object includes a movement of the computer system; the movement of the computer system is derived from image analysis that indicates one or more reference points within the field of view of the one or more cameras are changing between successive images captured by the one or more cameras.

9. The method according to any one of claims 1 to 8.

10. 10. The method of claim 1, wherein adjusting the appearance of the corresponding virtual user interface object includes moving at least a portion of the corresponding virtual user interface object, the movement of the corresponding virtual user interface object being based on a physical shape of the corresponding physical object.

11. 11. The method of claim 1, wherein adjusting the appearance of the corresponding virtual user interface object includes moving at least a portion of the corresponding virtual user interface object, the movement of the corresponding virtual user interface object being based on a simultaneous movement of one or more touch inputs and a movement of the computer system.

12. adjusting the appearance of the corresponding virtual user interface object includes moving at least a portion of the corresponding virtual user interface object beyond a maximum resting limit of the corresponding virtual user interface object, the method comprising: while continuing to detect the input, displaying the corresponding virtual user interface object at a position beyond the maximum limit of the rest state of the corresponding virtual user interface object according to the magnitude of movement of the input relative to the corresponding physical object; ceasing to detect the input; and displaying the corresponding virtual user interface object at a position corresponding to the maximum limit of the rest state of the corresponding virtual user interface object in response to ceasing to detect the input; The method of any one of claims 1 to 11, comprising:

13. the displayed augmented reality environment, one or more virtual objects that do not correspond to physical objects in the field of view of the one or more cameras; one or more physical objects within the field of view of the one or more cameras; one or more three-dimensional virtual models of the one or more physical objects within the field of view of the one or more cameras that replace at least a portion of the corresponding one or more physical objects; 13. The method of any one of claims 1 to 12, comprising:

14. 14. The method of claim 1, wherein the corresponding physical object is a three-dimensional marker that is recognizable from different angles, and the corresponding virtual user interface object is a three-dimensional virtual model that is superimposed on the corresponding physical object based on the camera angles of the one or more cameras.

15. 1. A computer system comprising: A display generating component; One or more cameras; An input device; one or more processors; a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: Displaying an augmented reality environment via the display generation component includes: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; including simultaneously displaying Display an augmented reality environment, detecting an input at a location corresponding to the corresponding virtual user interface object while displaying the augmented reality environment; While continuing to detect said input, Detecting movement of the input relative to the corresponding physical object within the field of view of the one or more cameras; responsive to detecting the movement of the input relative to the corresponding physical object within the field of view of the one or more cameras, adjusting an appearance of the corresponding virtual user interface object according to a magnitude of the movement of the input relative to the corresponding physical object. Including instructions, Computer system.

16. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more cameras, and an input device, cause the computer system to: Displaying an augmented reality environment via the display generation component includes: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; including simultaneously displaying Display an augmented reality environment, detecting an input at a location corresponding to the corresponding virtual user interface object while displaying the augmented reality environment; While continuing to detect said input, detecting movement of the input relative to the corresponding physical object within the field of view of the one or more cameras; responsive to detecting the movement of the input relative to the corresponding physical object within the field of view of the one or more cameras, adjusting an appearance of the corresponding virtual user interface object according to a magnitude of the movement of the input relative to the corresponding physical object. A computer-readable storage medium.

17. A display generating component; One or more cameras; An input device; A means for displaying an augmented reality environment via the display generation component, the means for displaying the augmented reality environment comprising: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; means for simultaneously displaying Means, means, enabled while displaying the augmented reality environment, for detecting an input at a location corresponding to the corresponding virtual user interface object; a means for detecting said input, the means being enabled while continuing to detect said input, means for detecting movement of said input relative to said corresponding physical object within the field of view of said one or more cameras; means, activated in response to detecting the movement of the input relative to the corresponding physical object within the field of view of the one or more cameras, for adjusting an appearance of the corresponding virtual user interface object according to a magnitude of the movement of the input relative to the corresponding physical object; and A computer system comprising:

18. 1. An information processing device for use in a computer system having a display generation component, one or more cameras, and an input device, comprising: A means for displaying an augmented reality environment via the display generation component, the means for displaying the augmented reality environment comprising: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; means for simultaneously displaying the means, enabled while displaying the augmented reality environment, for detecting an input at a location corresponding to the corresponding virtual user interface object; a means for detecting said input, the means being enabled while continuing to detect said input, means for detecting movement of said input relative to said corresponding physical object within the field of view of said one or more cameras; means, activated in response to detecting the movement of the input relative to the corresponding physical object within the field of view of the one or more cameras, for adjusting an appearance of the corresponding virtual user interface object according to a magnitude of the movement of the input relative to the corresponding physical object; and An information processing device comprising:

19. 1. A computer system comprising: A display generating component; One or more cameras; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 14. Computer system.

20. 15. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more cameras, and an input device, cause the computer system to perform the method of any one of claims 1 to 14.

21. A graphical user interface for a computer system having a display generation component, one or more cameras, an input device, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 1 to 14.

22. A display generating component; One or more cameras; An input device; Means for carrying out the method according to any one of claims 1 to 14; A computer system comprising:

23. 1. An information processing device for use in a computer system having a display generation component, one or more cameras, and an input device, comprising: Means for carrying out the method according to any one of claims 1 to 14, Information processing device.

24. 1. A computer system having a display generation component, one or more cameras, and an input device, comprising: Displaying an augmented reality environment via the display generation component, wherein displaying the augmented reality environment comprises: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; and detecting an input that modifies a virtual environment setting of the augmented reality environment while displaying the augmented reality environment; In response to detecting the input that changes the virtual environment setting, adjusting an appearance of the corresponding virtual user interface object according to the changes made to the virtual environment setting of the augmented reality environment; applying a filter to at least a portion of the representation of the field of view of the one or more cameras, the filter being selected based on the changes made to the virtual environment setting; A method comprising:

25. 25. The method of claim 24, wherein applying the filter to at least a portion of the representation of the field of view of the one or more cameras causes an appearance adjustment of the augmented reality environment in addition to the appearance adjustment of the corresponding virtual user interface object.

26. The virtual environment setting is changed to night mode, applying the filter to at least a portion of the representation of the field of view of the one or more cameras; reducing brightness of images captured by the one or more cameras; applying a color filter to the images captured by the one or more cameras; 26. The method of claim 24 or 25, comprising:

27. 27. The method of any one of claims 24 to 26, wherein the input that changes the virtual environment setting is a swipe input that navigates through time within the augmented reality environment.

28. detecting the input to change the virtual environment setting includes detecting a movement of the input to change the virtual environment setting; adjusting the appearance of the corresponding virtual user interface object in accordance with the change made to the virtual environment setting of the augmented reality environment includes progressively adjusting the appearance of the corresponding virtual user interface object in accordance with the movement of the input to change the virtual environment setting; applying the filter to at least a portion of the representation of the field of view of the one or more cameras includes progressively applying the filter in accordance with the movement of the input to modify the virtual environment setting.

28. The method of any one of claims 24 to 27.

29. 29. The method of claim 24, wherein the corresponding virtual user interface object casts a shadow on the corresponding physical object in the augmented reality environment.

30. 30. The method of claim 24, wherein the corresponding physical object casts a shadow on the corresponding virtual user interface object in the augmented reality environment.

31. 31. The method of claim 24, wherein movement of the corresponding physical object causes one or more changes in the appearance of the corresponding virtual user interface object in the augmented reality environment.

32. 32. The method of any one of claims 24 to 31, wherein movement of the computer system causes one or more changes in visual effects applied to the representation of at least a portion of the field of view of the one or more cameras and the appearance of the corresponding virtual user interface objects.

33. applying the filter to at least a portion of the representation of the field of view of the one or more cameras; applying the filter to images captured by the one or more cameras before the images are transmitted to the display generation component.

33. The method of any one of claims 24 to 32.

34. 34. The method of claim 24, wherein the input for changing the virtual environment setting is an input for switching between different virtual environments for the virtual user interface object, the different virtual environments being associated with different interactions for exploring the virtual user interface object.

35. displaying a first set of virtual objects within the augmented reality environment in accordance with a determination that a first virtual environment setting is selected; displaying, in accordance with determining that a second virtual environment setting is selected, a second set of virtual objects within the augmented reality environment, the second set of virtual objects being distinct from the first set of virtual objects; 35. The method of any one of claims 24 to 34, comprising:

36. 1. A computer system comprising: A display generating component; One or more cameras; An input device; one or more processors; a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: Displaying an augmented reality environment via the display generation component includes: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; including simultaneously displaying Display an augmented reality environment, detecting an input that modifies a virtual environment setting of the augmented reality environment while displaying the augmented reality environment; In response to detecting the input that changes the virtual environment setting, adjusting an appearance of the corresponding virtual user interface object according to the changes made to the virtual environment setting of the augmented reality environment; applying a filter to at least a portion of the representation of the field of view of the one or more cameras, the filter being selected based on the changes made to the virtual environment setting. Including instructions, Computer system.

37. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more cameras, and an input device, cause the computer system to: Displaying an augmented reality environment via the display generation component includes: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; including simultaneously displaying Display an augmented reality environment, detecting an input that modifies a virtual environment setting of the augmented reality environment while displaying the augmented reality environment; In response to detecting the input that changes the virtual environment setting, adjusting an appearance of the corresponding virtual user interface object in accordance with the changes made to the virtual environment setting of the augmented reality environment; applying a filter to at least a portion of the representation of the field of view of the one or more cameras, the filter being selected based on the changes made to the virtual environment setting. A computer-readable storage medium.

38. A display generating component; One or more cameras; An input device; A means for displaying an augmented reality environment via the display generation component, the means for displaying the augmented reality environment comprising: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; means for simultaneously displaying Means, means for detecting an input that is activated while displaying the augmented reality environment and that changes a virtual environment setting of the augmented reality environment; a means for enabling said input for modifying said virtual environment setting, said means comprising: means for adjusting an appearance of the corresponding virtual user interface object in accordance with the changes made to the virtual environment setting of the augmented reality environment; means for applying a filter to at least a portion of the representation of the field of view of the one or more cameras, the filter being selected based on the changes made to the virtual environment setting; and A computer system comprising:

39. 1. An information processing device for use in a computer system having a display generation component, one or more cameras, and an input device, comprising: A means for displaying an augmented reality environment via the display generation component, the means for displaying the augmented reality environment comprising: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; corresponding virtual user interface objects at corresponding positions within the representation of the field of view of the one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects within the field of view of the one or more cameras; means for simultaneously displaying Means, means for detecting an input that is activated while displaying the augmented reality environment and that changes a virtual environment setting of the augmented reality environment; a means for enabling said input for modifying said virtual environment setting, said means comprising: means for adjusting an appearance of the corresponding virtual user interface object in accordance with the changes made to the virtual environment setting of the augmented reality environment; means for applying a filter to at least a portion of the representation of the field of view of the one or more cameras, the filter being selected based on the changes made to the virtual environment setting; and An information processing device comprising:

40. 1. A computer system comprising: A display generating component; One or more cameras; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 24 to 35. Computer system.

41. 36. A computer readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more cameras, and an input device, cause the computer system to perform the method of any one of claims 24 to 35.

42. A graphical user interface for a computer system having a display generation component, one or more cameras, an input device, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 24 to 35.

43. A display generating component; One or more cameras; An input device; Means for carrying out the method according to any one of claims 24 to 35; A computer system comprising:

44. 1. An information processing device for use in a computer system having a display generation component, one or more cameras, and an input device, comprising: Means for carrying out the method according to any one of claims 24 to 35, Information processing device.

45. 1. A computer system having a display generation component, one or more cameras, and an input device, comprising: Displaying an augmented reality environment via the display generation component, wherein displaying the augmented reality environment comprises: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; a first virtual user interface object in a virtual model displayed at a corresponding position in the representation of the field of view of the one or more cameras, the first virtual user interface object having a position determined based on the corresponding physical object in the field of view of the one or more cameras; and detecting a first input corresponding to a selection of the first virtual user interface object while displaying the augmented reality environment; in response to detecting the first input corresponding to a selection of the first virtual user interface object; displaying a simulated view of the virtual model from a perspective of the first virtual user interface object in the virtual model; A method comprising:

46. 46. ​​The method of claim 45, comprising ceasing to display the representation of the field of view of the one or more cameras in response to detecting the first input corresponding to a selection of the first virtual user interface object.

47. 47. A method according to claim 45 or 46, comprising updating the representation of the field of view of the one or more cameras in response to detecting a movement of at least a portion of the computer system that changes the field of view of the one or more cameras while displaying the augmented reality environment.

48. 48. A method according to any one of claims 45 to 47, comprising, in response to detecting a movement of at least a portion of the computer system that changes a viewpoint of the content of the field of view of the one or more cameras while displaying the augmented reality environment, updating the representation of the field of view of the one or more cameras and the virtual model in accordance with the change in viewpoint of the content of the field of view.

49. 49. The method of any one of claims 45 to 48, comprising, in response to detecting the first input corresponding to a selection of the first virtual user interface object, displaying an animated transition from the augmented reality environment to the simulated field of view of the virtual model from the viewpoint of the first virtual user interface object in the virtual model.

50. while displaying the simulated view of the virtual model from the viewpoint of the first virtual user interface object in the virtual model; detecting a second input corresponding to a request to display the augmented reality environment; in response to the second input corresponding to the request to display the augmented reality environment, displaying an animated transition from the simulated view of the virtual model to the augmented reality environment; and displaying the augmented reality environment; and 50. The method of any one of claims 45 to 49, comprising:

51. 51. The method of claim 50, wherein displaying the augmented reality environment in response to the second input comprises displaying the augmented reality environment according to the field of view of the one or more cameras after detecting the second input.

52. 52. The method of claim 51, wherein the field of view of the one or more cameras after detecting the second input is different from the field of view of the one or more cameras when the first input is detected.

53. 53. The method of any one of claims 45 to 52, wherein the first virtual user interface object moves within the virtual model independent of input from a user of the computer system.

54. 54. The method of any one of claims 45 to 53, wherein the first virtual user interface object moves within the virtual model in response to one or more inputs from a user of the computer system while displaying the simulated view of the virtual model from the viewpoint of the first virtual user interface object within the virtual model.

55. 55. The method of any one of claims 45 to 54, comprising: detecting a movement of at least a portion of the computer system while displaying the simulated field of view; and in response to detecting the movement of the computer system, modifying the simulated field of view of the virtual model from the viewpoint of the first virtual user interface object in accordance with the movement of the computer system.

56. detecting a third input corresponding to a selection of a second virtual user interface object within the virtual model while displaying the simulated view of the virtual model from the viewpoint of the first virtual user interface object within the virtual model; in response to detecting the third input corresponding to a selection of the second virtual user interface object; displaying a second simulated view of the virtual model from a perspective of the second virtual user interface object in the virtual model; 56. The method of any one of claims 45 to 55, comprising:

57. detecting a fourth input corresponding to a selection of a location within the virtual model other than a virtual user interface object capable of displaying an associated simulated view while displaying the simulated view of the virtual model from the viewpoint of the first virtual user interface object within the virtual model; In response to detecting the fourth input, re-displaying the augmented reality environment; and 57. The method of any one of claims 45 to 56, comprising:

58. 1. A computer system comprising: A display generating component; One or more cameras; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: Displaying an augmented reality environment via the display generation component includes: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; a first virtual user interface object in a virtual model displayed at a corresponding position in the representation of the field of view of the one or more cameras, the first virtual user interface object having a position determined based on the corresponding physical object in the field of view of the one or more cameras; including simultaneously displaying Display an augmented reality environment, detecting a first input corresponding to a selection of the first virtual user interface object while displaying the augmented reality environment; in response to detecting the first input corresponding to a selection of the first virtual user interface object; displaying a simulated view of the virtual model from a perspective of the first virtual user interface object in the virtual model. A computer system including instructions.

59. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more cameras, and an input device, cause the computer system to: Displaying an augmented reality environment via the display generation component includes: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; a first virtual user interface object in a virtual model displayed at a corresponding position in the representation of the field of view of the one or more cameras, the first virtual user interface object having a position determined based on the corresponding physical object in the field of view of the one or more cameras; including simultaneously displaying Display an augmented reality environment, detecting a first input corresponding to a selection of the first virtual user interface object while displaying the augmented reality environment; in response to detecting the first input corresponding to a selection of the first virtual user interface object; displaying a simulated view of the virtual model from a perspective of the first virtual user interface object in the virtual model. A computer-readable storage medium.

60. 1. A computer system comprising: A display generating component; One or more cameras; An input device; A means for displaying an augmented reality environment via the display generation component, the means for displaying the augmented reality environment comprising: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; a first virtual user interface object in a virtual model displayed at a corresponding position in the representation of the field of view of the one or more cameras, the first virtual user interface object having a position determined based on the corresponding physical object in the field of view of the one or more cameras; means for simultaneously displaying Means, means for detecting a first input, enabled while displaying the augmented reality environment, corresponding to a selection of the first virtual user interface object; a means for detecting the first input corresponding to a selection of the first virtual user interface object, the means being enabled in response to detecting the first input corresponding to a selection of the first virtual user interface object, the means comprising: means for displaying a simulated view of the virtual model from a perspective of the first virtual user interface object in the virtual model. Means, A computer system comprising:

61. 1. An information processing device for use in an electronic device having a display generation component, one or more cameras, and an input device, comprising: A means for displaying an augmented reality environment via the display generation component, the means for displaying the augmented reality environment comprising: a representation of at least a portion of the field of view of the one or more cameras including a corresponding physical object, the representation being updated as the content of the field of view of the one or more cameras changes; a first virtual user interface object in a virtual model displayed at a corresponding position in the representation of the field of view of the one or more cameras, the first virtual user interface object having a position determined based on the corresponding physical object in the field of view of the one or more cameras; means for simultaneously displaying Means, means for detecting a first input, enabled while displaying the augmented reality environment, corresponding to a selection of the first virtual user interface object; a means for detecting the first input corresponding to a selection of the first virtual user interface object, the means being enabled in response to detecting the first input corresponding to a selection of the first virtual user interface object, the means comprising: means for displaying a simulated view of the virtual model from a perspective of the first virtual user interface object in the virtual model. Means, An information processing device comprising:

62. 1. A computer system comprising: A display generating component; One or more cameras; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 45 to 57. Computer system.

63. 58. A computer readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more cameras, and an input device, cause the computer system to perform the method of any one of claims 45 to 57.

64. 58. A graphical user interface for a computer system having a display generation component, one or more cameras, an input device, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 45 to 57.

65. A display generating component; One or more cameras; An input device; Means for carrying out the method according to any one of claims 45 to 57; A computer system comprising:

66. 1. An information processing device for use in a computer system having a display generation component, one or more cameras, and an input device, comprising:

58. A method for performing a method according to any one of claims 45 to 57, Information processing device.

67. 1. A computer system having a display generation component and an input device, comprising: displaying a first virtual user interface object in a virtual three-dimensional space via the display generation component; detecting, while displaying the first virtual user interface object in the virtual three-dimensional space, via the input device, the first input including a selection of a corresponding portion of the first virtual user interface object and a movement of the first input in two dimensions; in response to detecting the first input comprising a movement of the first input in two dimensions; adjusting an appearance of the first virtual user interface object in a first direction determined based on the movement of the first input in two dimensions and the first portion of the selected first virtual user interface object pursuant to a determination that the corresponding portion of the first virtual user interface object is a first portion of the first virtual user interface object, wherein the adjustment of the first virtual user interface object in the first direction is constrained to movement within a first set of two dimensions of the virtual three-dimensional space; adjusting the appearance of the first virtual user interface object in a second direction different from the first direction pursuant to a determination that the corresponding portion of the first virtual user interface object is a second portion of the first virtual user interface object separate from the first portion of the first virtual user interface object, the second direction being determined based on the movement of the first input in two dimensions and the second portion of the selected first virtual user interface object, and the adjusting of the first virtual user interface object in the second direction being constrained to movement in a second set of two dimensions of the virtual three-dimensional space different from the first set of two dimensions of the virtual three-dimensional space; A method comprising:

68. the first portion is a first side of the first virtual user interface object; the second portion is a second side of the first virtual user interface object, and the first side is not parallel to the second side.

68. The method of claim 67.

69. Adjusting the appearance of the first virtual user interface object comprises adjusting the appearance of the first virtual user interface object, a position of the first virtual user interface object is locked to a plane parallel to the selected corresponding portion of the virtual user interface object; the two-dimensional movement of the first input corresponds to a two-dimensional movement of the first virtual user interface object on the plane parallel to the selected corresponding portion of the virtual user interface object. Including becoming, 69. The method of claim 67 or 68.

70. 70. The method of claim 69, wherein adjusting the appearance of the first virtual user interface object includes displaying a movement plane indicator including a visual indication of the plane that is parallel to the selected corresponding portion of the virtual user interface object.

71. 71. The method of claim 70, wherein the movement plane indicator includes one or more projections extending from the first virtual user interface object along the plane that is parallel to the selected corresponding portion of the virtual user interface object.

72. In response to detecting the first input including a movement of the first input, determining whether the first input satisfies a resizing criterion; adjusting the appearance of the first virtual user interface object in accordance with a determination that the first input satisfies the size adjustment criterion; a position of the first virtual user interface object is locked to an anchor point in the virtual three-dimensional space; a size of the first virtual user interface object is adjusted along an axis perpendicular to the selected corresponding portion of the first virtual user interface object; and 69. The method of claim 67 or 68.

73. 73. The method of claim 72, wherein the anchor point is located on a portion of the first virtual user interface object opposite the selected corresponding portion of the first virtual user interface object.

74. 73. The method of claim 72, wherein the anchor point is a center of gravity of the first virtual user interface object.

75. 75. The method of any one of claims 72 to 74, wherein adjusting the appearance of the first virtual user interface object includes displaying a movement axis indicator, the movement axis indicator including a visual indication of an axis perpendicular to the selected corresponding portion of the first virtual user interface object.

76. 76. The method of claim 75, wherein the movement axis indicator includes one or more projections parallel to the axis perpendicular to the corresponding portion of the first virtual user interface object, the one or more projections extending from the first virtual user interface object.

77. The computer system includes one or more tactile output generators for generating tactile outputs, and the method further comprises: determining that the movement of the first input causes a corresponding portion of the first virtual user interface object to collide with a virtual element present in the virtual three-dimensional space while adjusting an appearance of the first virtual user interface object in a corresponding direction; generating a tactile output using the one or more tactile output generators in accordance with a determination that the movement of the first input causes the corresponding portion of the first virtual user interface object to collide with the virtual element; 77. The method of any one of claims 67 to 76, comprising:

78. detecting, while displaying the virtual three-dimensional space, a second input directed toward a first location within the virtual three-dimensional space via the input device; in response to detecting the second input, displaying an insertion cursor at the first position within the virtual three-dimensional space in accordance with a determination that the second input has a first input format; detecting a third input via the input device while the insertion cursor is displayed at the first position; In response to detecting the third input, inserting a second virtual user interface object at the first location in response to a determination that the third input has the first input format and is directed to the first location corresponding to the displayed insertion cursor; displaying the insertion cursor at the second location in response to a determination that the third input has the first input format and is directed to a second location that does not correspond to the displayed insertion cursor; 78. The method of any one of claims 67 to 77, comprising:

79. detecting, while displaying the virtual three-dimensional space, a fourth input directed toward a third location within the virtual three-dimensional space via the input device; in response to detecting the fourth input directed at the third location within the virtual three-dimensional space, displaying an insertion cursor at the third location in accordance with a determination that the fourth input has the first input format; detecting, while the insertion cursor is displayed at the third location, a fifth input via the input device at a location corresponding to a new object control that, when activated, causes insertion of a new virtual user interface object at the third location; responsive to detecting the fifth input, inserting the new virtual user interface object at the third position; 79. The method of any one of claims 67 to 78, comprising:

80. Detecting, via the input device, a gesture corresponding to an interaction with the virtual three-dimensional space; In response to detecting the gesture corresponding to the interaction with the virtual three-dimensional space, performing an action in the virtual three-dimensional space corresponding to the gesture; 80. The method of any one of claims 67 to 79, comprising:

81. the computer system includes one or more cameras; The displayed virtual three-dimensional space is one or more physical objects within the field of view of the one or more cameras; one or more virtual three-dimensional models of the one or more physical objects within the field of view of the one or more cameras; 81. The method of any one of claims 67 to 80, comprising:

82. In response to detecting the movement of the first input relative to the corresponding physical object within the field of view of the one or more cameras, the appearance of the first virtual user interface object is adjusted, and the movement of the first input is moving the first input on the input device; moving the one or more cameras relative to a corresponding physical object; or combining the movement of the first input on the input device with the movement of the one or more cameras relative to the corresponding physical object; 82. The method of claim 81, regardless of whether the condition is caused by

83. 1. A computer system comprising: A display generating component; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying a first virtual user interface object in a virtual three-dimensional space via the display generation component; detecting, while displaying the first virtual user interface object within the virtual three-dimensional space, via the input device, the first input including a selection of a corresponding portion of the first virtual user interface object and a movement of the first input in two dimensions; in response to detecting the first input comprising a movement of the first input in two dimensions; pursuant to determining that the corresponding portion of the first virtual user interface object is a first portion of the first virtual user interface object, adjusting an appearance of the first virtual user interface object in a first direction determined based on the movement of the first input in two dimensions and the first portion of the selected first virtual user interface object, wherein the adjustment of the first virtual user interface object in the first direction is constrained to movement within a first set of two dimensions of the virtual three-dimensional space; adjusting the appearance of the first virtual user interface object in a second direction different from the first direction in accordance with a determination that the corresponding portion of the first virtual user interface object is a second portion of the first virtual user interface object separate from the first portion of the first virtual user interface object, wherein the second direction is determined based on the movement of the first input in two dimensions and the second portion of the first virtual user interface object selected, and the adjusting of the first virtual user interface object in the second direction is constrained to movement in a second set of two dimensions of the virtual three-dimensional space different from the first set of two dimensions of the virtual three-dimensional space. A computer system including instructions.

84. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component and an input device, cause the computer system to: displaying a first virtual user interface object in a virtual three-dimensional space via the display generation component; detecting, while displaying the first virtual user interface object in the virtual three-dimensional space, via the input device, the first input including a selection of a corresponding portion of the first virtual user interface object and a movement of the first input in two dimensions; in response to detecting the first input comprising a movement of the first input in two dimensions; adjusting an appearance of the first virtual user interface object in a first direction determined based on the movement of the first input in two dimensions and the first portion of the selected first virtual user interface object in accordance with a determination that the corresponding portion of the first virtual user interface object is a first portion of the first virtual user interface object, wherein the adjustment of the first virtual user interface object in the first direction is constrained to movement within a first set of two dimensions of the virtual three-dimensional space; adjusting the appearance of the first virtual user interface object in a second direction different from the first direction in accordance with a determination that the corresponding portion of the first virtual user interface object is a second portion of the first virtual user interface object separate from the first portion of the first virtual user interface object, the second direction being determined based on the movement of the first input in two dimensions and the second portion of the first virtual user interface object selected, and the adjusting of the first virtual user interface object in the second direction being constrained to movement in a second set of two dimensions of the virtual three-dimensional space different from the first set of two dimensions of the virtual three-dimensional space. A computer-readable storage medium.

85. 1. A computer system comprising: A display generating component; An input device; means for displaying a first virtual user interface object in a virtual three-dimensional space via the display generation component; means, activated while displaying the first virtual user interface object within the virtual three-dimensional space, for detecting the first input via the input device, the first input comprising a selection of a corresponding portion of the first virtual user interface object and a movement of the first input in two dimensions; a means enabled in response to detecting the first input comprising a movement of the first input in two dimensions, the means comprising: means for adjusting an appearance of the first virtual user interface object in a first direction determined based on the movement of the first input in two dimensions and the first portion of the first virtual user interface object selected, the adjustment of the first virtual user interface object in the first direction being constrained to movement within a first set of two dimensions of the virtual three-dimensional space; and means for enabling the corresponding portion of the first virtual user interface object in accordance with a determination that the corresponding portion of the first virtual user interface object is a second portion of the first virtual user interface object distinct from the first portion of the first virtual user interface object and adjusting the appearance of the first virtual user interface object in a second direction different from the first direction, the second direction being determined based on the movement of the first input in two dimensions and the second portion of the first virtual user interface object selected, and the adjustment of the first virtual user interface object in the second direction being constrained to movement in a second set of two dimensions of the virtual three-dimensional space different from the first set of two dimensions of the virtual three-dimensional space; and A computer system comprising:

86. 1. An information processing device for use in a computer system having a display generation component and an input device, comprising: means for displaying a first virtual user interface object in a virtual three-dimensional space via the display generation component; means, activated while displaying the first virtual user interface object within the virtual three-dimensional space, for detecting the first input via the input device, the first input comprising a selection of a corresponding portion of the first virtual user interface object and a movement of the first input in two dimensions; a means enabled in response to detecting the first input comprising a movement of the first input in two dimensions, the means comprising: means for adjusting an appearance of the first virtual user interface object in a first direction determined based on the movement of the first input in two dimensions and the first portion of the first virtual user interface object selected, the adjustment of the first virtual user interface object in the first direction being constrained to movement within a first set of two dimensions of the virtual three-dimensional space; and means for enabling the corresponding portion of the first virtual user interface object in accordance with a determination that the corresponding portion of the first virtual user interface object is a second portion of the first virtual user interface object distinct from the first portion of the first virtual user interface object and adjusting the appearance of the first virtual user interface object in a second direction different from the first direction, the second direction being determined based on the movement of the first input in two dimensions and the second portion of the first virtual user interface object selected, and the adjustment of the first virtual user interface object in the second direction being constrained to movement in a second set of two dimensions of the virtual three-dimensional space different from the first set of two dimensions of the virtual three-dimensional space; and An information processing device comprising:

87. 1. A computer system comprising: A display generating component; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 67 to 82. Computer system.

88. 83. A computer readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component and an input device, cause the computer system to perform the method of any one of claims 67 to 82.

89. A graphical user interface for a computer system having a display generation component, an input device, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 67 to 82.

90. A display generating component; An input device; Means for carrying out the method according to any one of claims 67 to 82; A computer system comprising:

91. 1. An information processing device for use in a computer system having a display generation component and an input device, comprising: Means for carrying out the method according to any one of claims 67 to 82, Information processing device.

92. 1. A computer system having a display generation component, one or more orientation sensors, and an input device, comprising: displaying, via the display generation component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, where displaying the simulated environment in the first viewing mode includes displaying a first virtual user interface object in a virtual model that is displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system; detecting, while displaying the simulated environment, a first change in an attitude of at least a portion of the computer system relative to the physical environment via the one or more attitude sensors; in response to detecting the first change in the pose of the portion of the computer system, modifying an appearance of the first virtual user interface object in the virtual model to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; detecting, via the input device, a first gesture corresponding to an interaction with the simulated environment after modifying the appearance of the first virtual user interface object based on the first change in pose of the portion of the computer system; in response to detecting the first gesture corresponding to the interaction with the simulated environment, performing an action within the simulated environment corresponding to the first gesture; detecting, after performing the action corresponding to the first gesture, a second change in an attitude of the portion of the computer system relative to the physical environment via the one or more attitude sensors; in response to detecting the second change in the attitude of the portion of the computer system, transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode in accordance with a determination that the first gesture satisfies mode change criteria, the mode change criteria including a requirement that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, wherein displaying the virtual model within the simulated environment in the second viewing mode includes refraining from modifying the appearance of the first virtual user interface object to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; continuing to display the first virtual model in the simulated environment in the first viewing mode pursuant to a determination that the first gesture did not satisfy the mode change criterion, wherein displaying the virtual model in the first viewing mode includes altering an appearance of the first virtual user interface object in the virtual model in response to the second change in attitude of the portion of the computer system relative to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; A method comprising:

93. the computer system includes one or more cameras; displaying the simulated environment in the first viewing mode includes displaying a representation of at least a portion of a field of view of the one or more cameras, the field of view of the one or more cameras including a representation of physical objects within the physical environment.

93. The method of claim 92.

94. detecting the first gesture corresponding to the interaction with the simulated environment; Detecting a plurality of contacts with a touch-sensitive surface of the input device; detecting a movement of a first contact of the plurality of contacts relative to a movement of a second contact of the plurality of contacts while the plurality of contacts with the touch-sensitive surface is being detected; Including, performing the action within the simulated environment corresponding to the first gesture includes resizing the first virtual user interface object by an amount corresponding to the movement of the first contact relative to the movement of the second contact.

94. The method of claim 92 or 93.

95. while displaying the first virtual user interface object within the simulated environment in the second viewing mode; detecting a second gesture via the input device corresponding to an interaction with the simulated environment, the second gesture including an input for changing a viewpoint of the simulated environment; in response to detecting the second gesture corresponding to the interaction with the simulated environment, updating a displayed viewpoint of the simulated environment according to the input for changing the viewpoint of the simulated environment; 95. The method of any one of claims 92 to 94, comprising:

96. while displaying the simulated environment in the second viewing mode; detecting an insertion input via the input device to insert a second virtual user interface object at a second corresponding location in the simulated environment; in response to detecting the insertion input for inserting the second virtual user interface object, displaying the second virtual user interface object at the second corresponding location in the simulated environment while maintaining the fixed spatial relationship between the first virtual user interface object and the physical environment; 96. The method of any one of claims 92 to 95, comprising:

97. while displaying the simulated environment in the second viewing mode; detecting, via the input device, the input of a movement including a selection of a corresponding aspect of a corresponding virtual user interface object of the virtual model and a movement of the input in two dimensions; responsive to detecting the movement, moving the corresponding virtual user interface object in a plane parallel to the selected corresponding side of the corresponding virtual user interface object in a first direction determined based on the movement of the second input while maintaining the fixed spatial relationship between the first virtual user interface object and the physical environment; 97. The method of any one of claims 92 to 96, comprising:

98. 98. The method of any one of claims 92 to 97, comprising displaying a transition animation during a transition from displaying the simulated environment in the first viewing mode to displaying the simulated environment in the second viewing mode to provide a visual indication of the transition.

99. 99. The method of claim 98, wherein displaying the transition animation comprises gradually ceasing to display at least one visual element that is displayed in the first viewing mode and not displayed in the second viewing mode.

100. 100. The method of claim 98 or 99, wherein displaying the transition animation comprises progressively displaying at least one visual element of the second viewing mode that is not displayed in the first viewing mode.

101. 101. A method according to any one of claims 92 to 100, comprising, in response to detecting the first gesture corresponding to the interaction with the simulated environment, changing a viewpoint from which the virtual model within the simulated environment is displayed in accordance with the change to the spatial parameters due to the input corresponding to the first gesture.

102. 102. The method of claim 101, including, after detecting an end of the first gesture, continuing to change the viewpoint from which the virtual model in the simulated environment is displayed to indicate the transition from displaying the simulated environment in the first viewing mode to displaying the simulated environment in the second viewing mode.

103. detecting, while displaying the simulated environment in the second viewing mode, a third gesture via the input device corresponding to an input for transitioning from the second viewing mode to the first viewing mode; transitioning from displaying the simulated environment in the second viewing mode to displaying the simulated environment in the first viewing mode in response to detecting the third gesture; 102. The method of any one of claims 92 to 101, comprising:

104. the input device includes a touch-sensitive surface; detecting the third gesture corresponding to the input for transitioning from the second viewing mode to the first viewing mode; Detecting the plurality of contacts with the touch-sensitive surface of the input device; detecting a movement of the first contact of the plurality of contacts relative to a movement of the second contact of the plurality of contacts while the plurality of contacts with the touch-sensitive surface is being detected; Including, transitioning from displaying the simulated environment in the second viewing mode to displaying the simulated environment in the first viewing mode includes resizing the virtual model in the simulated environment back to a size of the virtual model prior to the transition from the first viewing mode to the second viewing mode. The method of claim 103.

105. 104. The method of claim 103, wherein the third gesture comprises an input at a location on the input device that corresponds to a control that, when activated, causes the transition from the second viewing mode to the first viewing mode.

106. in response to detecting the third gesture, transitioning the position of the first virtual user interface object from a current position relative to the physical environment to a previous position relative to the physical environment to return to the fixed spatial relationship between the first virtual user interface object and the physical environment.

106. The method of any one of claims 103 to 105.

107. 107. The method of any one of claims 103 to 106, comprising, after detecting an end of the third gesture, continuing to change the viewpoint from which the virtual model in the simulated environment is displayed to indicate the transition from displaying the simulated environment in the second viewing mode to displaying the simulated environment in the first viewing mode.

108. 1. A computer system comprising: A display generating component; one or more orientation sensors; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, via the display generation component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, wherein displaying the simulated environment in the first viewing mode includes displaying a first virtual user interface object in a virtual model that is displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system; detecting, while displaying the simulated environment, a first change in an attitude of at least a portion of the computer system relative to the physical environment via the one or more attitude sensors; modifying an appearance of the first virtual user interface object in the virtual model to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment in response to detecting the first change in the pose of the portion of the computer system; detecting a first gesture via the input device corresponding to an interaction with the simulated environment after modifying the appearance of the first virtual user interface object based on the first change in pose of the portion of the computer system; in response to detecting the first gesture corresponding to the interaction with the simulated environment, performing an action within the simulated environment corresponding to the first gesture; detecting, after performing the action corresponding to the first gesture, a second change in an attitude of the portion of the computer system relative to the physical environment via the one or more attitude sensors; in response to detecting the second change in the attitude of the portion of the computer system, transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode in accordance with a determination that the first gesture satisfies mode change criteria, the mode change criteria including a requirement that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, wherein displaying the virtual model within the simulated environment in the second viewing mode includes refraining from modifying the appearance of the first virtual user interface object to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; continuing to display the first virtual model in the simulated environment in the first viewing mode in accordance with a determination that the first gesture did not satisfy the mode change criterion, wherein displaying the virtual model in the first viewing mode includes modifying an appearance of the first virtual user interface object in the virtual model in response to the second change in attitude of the portion of the computer system relative to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment. A computer system including instructions.

109. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more orientation sensors, and an input device, cause the computer system to: displaying, via the display generation component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, where displaying the simulated environment in the first viewing mode includes displaying a first virtual user interface object in a virtual model that is displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system; detecting, while displaying the simulated environment, a first change in an attitude of at least a portion of the computer system relative to the physical environment via the one or more attitude sensors; in response to detecting the first change in the pose of the portion of the computer system, modifying an appearance of the first virtual user interface object in the virtual model to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; detecting, via the input device, a first gesture corresponding to an interaction with the simulated environment after modifying the appearance of the first virtual user interface object based on the first change in pose of the portion of the computer system; In response to detecting the first gesture corresponding to the interaction with the simulated environment, performing an action within the simulated environment corresponding to the first gesture; detecting, after performing the action corresponding to the first gesture, a second change in an attitude of the portion of the computer system relative to the physical environment via the one or more attitude sensors; in response to detecting the second change in the attitude of the portion of the computer system, transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode in accordance with a determination that the first gesture satisfies mode change criteria, the mode change criteria including a requirement that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, wherein displaying the virtual model within the simulated environment in the second viewing mode includes refraining from modifying the appearance of the first virtual user interface object to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; continuing to display the first virtual model in the simulated environment in the first viewing mode in accordance with a determination that the first gesture did not satisfy the mode change criterion, wherein displaying the virtual model in the first viewing mode includes modifying an appearance of the first virtual user interface object in the virtual model in response to the second change in attitude of the portion of the computer system with respect to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment. A computer-readable storage medium.

110. 1. A computer system comprising: A display generating component; one or more orientation sensors; An input device; means for displaying, via the display generation component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, the means for displaying the simulated environment in the first viewing mode including means for displaying a first virtual user interface object in a virtual model that is displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system; means, enabled while displaying the simulated environment, for detecting a first change in an attitude of at least a portion of the computer system relative to the physical environment via the one or more attitude sensors; means, activated in response to detecting the first change in the pose of the portion of the computer system, for modifying an appearance of the first virtual user interface object in the virtual model so as to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; means for detecting a first gesture, via the input device, corresponding to an interaction with the simulated environment, the first gesture being activated after modifying the appearance of the first virtual user interface object based on the first change in pose of the portion of the computer system; and means enabled in response to detecting the first gesture corresponding to the interaction with the simulated environment for performing an action within the simulated environment corresponding to the first gesture; means, enabled after performing the action corresponding to the first gesture, for detecting a second change in an attitude of the portion of the computer system relative to the physical environment via the one or more attitude sensors; a means, enabled in response to detecting the second change in the attitude of the portion of the computer system, means for transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode, the mode change criteria being enabled pursuant to a determination that the first gesture satisfies a mode change criteria including a requirement that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, the mode change criteria including a requirement that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment; and means for transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode, the displaying the virtual model within the simulated environment in the second viewing mode including refraining from modifying the appearance of the first virtual user interface object to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; means for continuing to display the first virtual model within the simulated environment in the first viewing mode, enabled pursuant to a determination that the first gesture did not satisfy the mode change criterion, wherein displaying the virtual model in the first viewing mode includes altering an appearance of the first virtual user interface object in the virtual model in response to the second change in attitude of the portion of the computer system relative to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; and A computer system comprising:

111. 1. An information processing device for use in an electronic device comprising a display generation component, one or more orientation sensors, and an input device, comprising: means for displaying, via the display generation component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, the means for displaying the simulated environment in the first viewing mode including means for displaying a first virtual user interface object in a virtual model that is displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system; means, enabled while displaying the simulated environment, for detecting a first change in an attitude of at least a portion of the computer system relative to the physical environment via the one or more attitude sensors; means, activated in response to detecting the first change in the pose of the portion of the computer system, for modifying an appearance of the first virtual user interface object in the virtual model so as to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; means for detecting a first gesture, via the input device, corresponding to an interaction with the simulated environment, the first gesture being activated after modifying the appearance of the first virtual user interface object based on the first change in pose of the portion of the computer system; and means enabled in response to detecting the first gesture corresponding to the interaction with the simulated environment for performing an action within the simulated environment corresponding to the first gesture; means, enabled after performing the action corresponding to the first gesture, for detecting a second change in an attitude of the portion of the computer system relative to the physical environment via the one or more attitude sensors; a means, enabled in response to detecting the second change in the attitude of the portion of the computer system, means for transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode, the mode change criteria being enabled pursuant to a determination that the first gesture satisfies a mode change criteria including a requirement that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, the mode change criteria including a requirement that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment; and means for transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode, the displaying the virtual model within the simulated environment in the second viewing mode including refraining from modifying the appearance of the first virtual user interface object to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; means for continuing to display the first virtual model within the simulated environment in the first viewing mode, enabled pursuant to a determination that the first gesture did not satisfy the mode change criterion, wherein displaying the virtual model in the first viewing mode includes altering an appearance of the first virtual user interface object in the virtual model in response to the second change in attitude of the portion of the computer system relative to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; and An information processing device comprising:

112. 1. A computer system comprising: A display generating component; one or more orientation sensors; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 92 to 107. Computer system.

113. 108. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more orientation sensors, and an input device, cause the computer system to perform the method of any one of claims 92 to 107.

114. A graphical user interface for a computer system having a display generation component, one or more orientation sensors, an input device, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 92 to 107.

115. A display generating component; one or more orientation sensors; An input device; Means for carrying out the method according to any one of claims 92 to 107; A computer system comprising:

116. 1. An information processing device for use in a computer system having a display generation component, one or more orientation sensors, and an input device, comprising: Means for carrying out the method according to any one of claims 92 to 107, Information processing device.

117. A first computer system having a first display generation component, one or more first orientation sensors, and a first input device, comprising: displaying, via the first display generating component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, wherein displaying the simulated environment includes: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; a visual indication of a second computer system's viewing perspective of the simulated environment, the second computer system having a second display generation component, one or more second orientation sensors, and a second input device, the second computer system being a computer system displaying, via the second display generation component of the second computer system, a view of the simulated environment oriented relative to a second physical environment of the second computer system; and including simultaneously displaying And, detecting a change in the viewing viewpoint of the second computer system of the simulated environment based on a change in the attitude of a portion of the second computer system relative to the second physical environment of the second computer system while displaying the simulated environment via the first display generation component of the first computer system; in response to detecting a change in the viewing viewpoint of the second computer system of the simulated environment based on the change in the attitude of the portion of the second computer system relative to the physical environment of the second computer system, updating the visual indication of the viewing viewpoint of the second computer system of the simulated environment displayed via the first display generation component of the first computer system according to the change in the viewing viewpoint of the second computer system of the simulated environment; A method comprising:

118. 118. The method of claim 117, wherein the visual indication of the viewing viewpoint of the second computer system includes a representation of the second computer system displayed at a location within the simulated environment corresponding to the second computer system.

119. 119. The method of claim 118, wherein the representation of the second computer system includes an identification indicator corresponding to the second computer system.

120. 120. A method according to any one of claims 117 to 119, wherein the visual indication of the viewing viewpoint of the second computer system includes an indicator emanating from a location in the simulated environment corresponding to the second computer system and indicating the line of sight of the second computer system.

121. Displaying the simulated environment displaying, via the first display generation component of the first computer system, an interaction indicator visually associated with the first virtual user interface object in accordance with determining that the second computer system in the simulated environment is interacting with the first virtual user interface object.

121. The method of any one of claims 117 to 120.

122. Displaying the simulated environment in response to determining that the interaction of the second computer system with the first virtual user interface object includes an object manipulation input, modifying an appearance of the first virtual user interface object in accordance with the object manipulation input. The method of claim 121.

123. modifying the appearance of the first virtual user interface object in accordance with the object manipulation input includes displaying a movement of the interaction indicator visually associated with the first virtual user interface object; the movement of the interaction indicator corresponds to the object manipulation input. The method of claim 122.

124. 122. The method of claim 121, wherein the interaction indicator comprises a visual indication of a connection between a location corresponding to the second computer system in the simulated environment and the first virtual user interface object.

125. 122. The method of claim 121, wherein the interaction indicator comprises a visual indication of an interaction point with the first user interface object.

126. Displaying the simulated environment Detecting, via the first computer system, a first physical reference object within the first physical environment; displaying the first virtual user interface object in the simulated environment displayed via the first display generation component of the first computer system at a position relative to the first physical reference object; Including, and updating the position of the interaction indicator relative to the first physical reference object in response to detecting the change in the viewing viewpoint of the second computer system. The method of claim 121.

127. the second physical environment of the second computer system is separate from the first physical environment of the first computer system; the second computer system detects a second physical reference object in the second physical environment; within the simulated environment displayed via the second display generation component of the second computer system, the first virtual user interface object is displayed at a position relative to the second physical reference object that corresponds to a position of the first virtual user interface object relative to the first physical reference object. The method of claim 126.

128. the first physical environment of the first computer system comprises at least a portion of the second physical environment of the second computer system; the second computer system is visible within the simulated environment displayed via the first display generation component; 127. The method of any one of claims 117 to 126.

129. Detecting a remote device viewpoint input by the first input device; in response to detecting the remote device viewpoint input, replacing a representation of the simulated environment oriented relative to the first physical environment of the first computer system with a representation of the simulated environment oriented relative to the second physical environment of the second computer system; 129. The method of any one of claims 117 to 128, comprising:

130. 1. A first computer system, comprising: A first display generation component; one or more first attitude sensors; A first input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, via the first display generating component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, wherein displaying the simulated environment includes: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; a visual indication of a second computer system's viewing perspective of the simulated environment, the second computer system having a second display generation component, one or more second orientation sensors, and a second input device, the second computer system being a computer system displaying, via the second display generation component of the second computer system, a view of the simulated environment oriented relative to a second physical environment of the second computer system; and including simultaneously displaying Doing things, detect a change in the viewing viewpoint of the second computer system of the simulated environment based on a change in the attitude of a portion of the second computer system relative to the second physical environment of the second computer system while displaying the simulated environment via the first display generation component of the first computer system; in response to detecting the change in the viewing viewpoint of the second computer system of the simulated environment based on the change in the attitude of the portion of the second computer system relative to the physical environment of the second computer system, updating the visual indication of the viewing viewpoint of the second computer system of the simulated environment displayed via the first display generation component of the first computer system according to the change in the viewing viewpoint of the second computer system of the simulated environment. Including instructions, A first computer system.

131. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a first computer system having a first display generating component, one or more first orientation sensors, and a first input device, cause the first computer system to: displaying, via the first display generating component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, wherein displaying the simulated environment includes: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; a visual indication of a second computer system's viewing perspective of the simulated environment, the second computer system having a second display generation component, one or more second orientation sensors, and a second input device, the second computer system being a computer system displaying, via the second display generation component of the second computer system, a view of the simulated environment oriented relative to a second physical environment of the second computer system; and including simultaneously displaying Let them do that. detecting a change in the viewing viewpoint of the second computer system of the simulated environment based on a change in the attitude of a portion of the second computer system relative to the second physical environment of the second computer system while displaying the simulated environment via the first display generation component of the first computer system; in response to detecting the change in the viewing viewpoint of the second computer system of the simulated environment based on the change in the attitude of the portion of the second computer system relative to the physical environment of the second computer system, updating the visual indication of the viewing viewpoint of the second computer system of the simulated environment displayed via the first display generation component of the first computer system according to the change in the viewing viewpoint of the second computer system of the simulated environment. A computer-readable storage medium.

132. 1. A first computer system, comprising: A first display generation component; one or more first attitude sensors; A first input device; means for displaying, via the first display generating component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, the means for displaying the simulated environment comprising: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; a visual indication of a second computer system's viewing perspective of the simulated environment, the second computer system having a second display generation component, one or more second orientation sensors, and a second input device, the second computer system being a computer system displaying, via the second display generation component of the second computer system, a view of the simulated environment oriented relative to a second physical environment of the second computer system; and means for simultaneously displaying Means, means, enabled while displaying the simulated environment via the first display generation component of the first computer system, for detecting a change in the viewing viewpoint of the second computer system of the simulated environment based on a change in the attitude of a portion of the second computer system relative to the second physical environment of the second computer system; means for updating the visual indication of the viewing viewpoint of the second computer system of the simulated environment displayed via the first display generation component of the first computer system in accordance with the change in the viewing viewpoint of the second computer system of the simulated environment, the visual indication being enabled in response to detecting a change in the viewing viewpoint of the second computer system of the simulated environment based on the change in the attitude of the portion of the second computer system relative to the physical environment of the second computer system; A first computer system comprising:

133. An information processing apparatus for use in an electronic device comprising a first display generating component, one or more first orientation sensors, and a first input device, comprising: means for displaying, via the first display generating component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, the means for displaying the simulated environment comprising: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; a visual indication of a second computer system's viewing perspective of the simulated environment, the second computer system having a second display generation component, one or more second orientation sensors, and a second input device, the second computer system being a computer system displaying, via the second display generation component of the second computer system, a view of the simulated environment oriented relative to a second physical environment of the second computer system; and means for simultaneously displaying Means, means, enabled while displaying the simulated environment via the first display generation component of the first computer system, for detecting a change in the viewing viewpoint of the second computer system of the simulated environment based on a change in the attitude of a portion of the second computer system relative to the second physical environment of the second computer system; means for updating the visual indication of the viewing viewpoint of the second computer system of the simulated environment displayed via the first display generation component of the first computer system in accordance with the change in the viewing viewpoint of the second computer system of the simulated environment, the visual indication being enabled in response to detecting a change in the viewing viewpoint of the second computer system of the simulated environment based on the change in the attitude of the portion of the second computer system relative to the physical environment of the second computer system; An information processing device comprising:

134. 1. A first computer system, comprising: A first display generation component; one or more first attitude sensors; A first input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 117 to 129. A first computer system.

135. 130. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a first computer system having a first display generating component, one or more first orientation sensors, and a first input device, cause the first computer system to perform the method of any one of claims 117 to 129.

136. A graphical user interface of a first computer system having a first display generation component, one or more first orientation sensors, a first input device, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 117 to 129.

137. A first display generation component; one or more first attitude sensors; A first input device; Means for carrying out the method of any one of claims 117 to 129; A first computer system comprising:

138. 1. An information processing apparatus for use in a first computer system, the information processing apparatus comprising a first display generating component, one or more first orientation sensors, and a first input device, the information processing apparatus comprising: Means for carrying out the method according to any one of claims 117 to 129, Information processing device.

139. 1. A computer system having a display generation component, one or more orientation sensors, and an input device, comprising: displaying a simulated environment via said display generation component; and detecting, while displaying the simulated environment, a first input via the input device directed to a corresponding location within the simulated environment; in response to detecting the first input directed to the corresponding location within the simulated environment; displaying the insertion cursor at a first location in accordance with a determination that the first input is in a first input format and that the first input is detected at a first location in the simulated environment other than a current location of an insertion cursor in the simulated environment; pursuant to a determination that the first input is in the first input format and that the first input is detected at a second location in the simulated environment corresponding to the current location of the insertion cursor, inserting a first object at the second location and moving the insertion cursor to a third location over the first object; A method comprising:

140. repeatedly performing the method over a number of successive iterations; in a first one of the successive iterations, the first input is of the first type and is detected at the first location within the simulated environment, and in response, the insertion cursor is displayed at the first location; in a second one of the successive iterations, the first input is of the first type and is detected at the second location in the simulated environment corresponding to the current location of the insertion point, and in response, the first object is inserted at the second location and the insertion cursor is moved to the third location over the first object. The method of claim 139.

141. the first object has a number of non-adjacent sides that are not adjacent to the second position that corresponds to the current position of the insertion cursor; the third location on the first object is on a corresponding non-adjacent side of the plurality of non-adjacent sides that is not adjacent to the second location.

141. The method of claim 139 or 140.

142. 142. The method of any one of claims 139 to 141, wherein, in accordance with a determination that the current position of an insertion cursor is located on a corresponding side of an existing object, the third position is on a first corresponding side of the first object that is parallel to the corresponding side of the existing object.

143. 143. The method of claim 142, wherein the first location is on a first side of the existing object and the second location is on a second side of the existing object that is different from the first side of the existing object.

144. the simulated environment is oriented relative to a physical environment of the computer system; inserting the first object at the second location includes inserting the first object in the simulated environment at a location in the simulated environment associated with a corresponding location of a corresponding reference physical object in the physical environment of the computer system.

144. The method of any one of claims 139 to 143.

145. in response to detecting the first input directed to the corresponding location within the simulated environment, inserting the first object at the second location within the simulated environment in accordance with a determination that the first input is a second form of input and the insertion cursor is displayed at the second location within the simulated environment and moving the insertion cursor to the third location on the first object.

145. The method of any one of claims 139 to 144.

146. detecting a second input comprising a selection of a second corresponding side of the first object and a movement of the second input in two dimensions; in response to detecting the second input comprising a movement of the second input in two dimensions; pursuant to a determination that the second input satisfies a movement criterion, moving the first object in a first direction determined based on the movement of the second input within a first plane parallel to the selected second corresponding side of the first object; refraining from moving the first object in accordance with a determination that the second input does not satisfy a movement criterion; 146. The method of any one of claims 139 to 145, comprising:

147. detecting a third input comprising a selection of a third corresponding side of the first object and a movement of the third input in two dimensions; in response to detecting the third input comprising a movement of the third input in two dimensions; adjusting a size of the first object along an axis perpendicular to the selected third corresponding side of the first object based on the movement of the third input in accordance with a determination that the third input satisfies a resizing criterion; refraining from adjusting the size of the first object in accordance with a determination that the third input does not satisfy a resizing criterion; and 147. The method of any one of claims 139 to 146, comprising:

148. detecting a fourth input comprising a selection of a fourth corresponding side of the first object and a movement of the fourth input in two dimensions; in response to detecting the fourth input comprising a movement of the second input in two dimensions; adjusting a size of the first object based on the movement of the fourth input pursuant to a determination that the contact satisfies a resizing criterion; moving the first object based on the movement of the fourth input in accordance with a determination that the contact does not satisfy a resize criterion.

148. The method of any one of claims 139 to 147.

149. 149. The method of claim 148, wherein adjusting the size of the first object based on the movement of the fourth input comprises adjusting the size of the first object along an axis perpendicular to the selected third corresponding side of the first object.

150. 150. The method of claim 148 or 149, wherein moving the first object based on the movement of the fourth input comprises moving the first object in a first direction determined based on the movement of the second input, in a first plane parallel to the selected second corresponding side of the first object.

151. detecting a fifth input on a corresponding portion of the first object that does not correspond to the third position on the first object while the first object is displayed; In response to detecting the fifth input, moving the insertion cursor from the third position to a position corresponding to the corresponding portion of the first object; 151. The method of any one of claims 139 to 150, comprising:

152. 1. A computer system comprising: A display generating component; one or more orientation sensors; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying a simulated environment via said display generation component; detecting, while displaying the simulated environment, a first input via the input device directed to a corresponding location within the simulated environment; in response to detecting the first input directed to the corresponding location within the simulated environment; pursuant to a determination that the first input is in a first input format and that the first input is detected at a first location within the simulated environment other than a current location of an insertion cursor within the simulated environment, displaying the insertion cursor at the first location; pursuant to a determination that the first input is in the first input format and that the first input is detected at a second location in the simulated environment corresponding to the current location of the insertion cursor, inserting a first object at the second location and moving the insertion cursor to a third location over the first object. Including instructions, Computer system.

153. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more orientation sensors, and an input device, cause the computer system to: displaying a simulated environment via said display generation component; detecting, while displaying the simulated environment, a first input via the input device directed to a corresponding location within the simulated environment; in response to detecting the first input directed to the corresponding location within the simulated environment; responsive to a determination that the first input is in a first input format and that the first input is detected at a first location within the simulated environment other than a current location of an insertion cursor within the simulated environment, displaying the insertion cursor at the first location; pursuant to a determination that the first input is in the first input format and that the first input is detected at a second location in the simulated environment corresponding to the current location of the insertion cursor, causing a first object to be inserted at the second location and moving the insertion cursor to a third location over the first object. A computer-readable storage medium.

154. 1. A computer system comprising: A display generating component; one or more orientation sensors; An input device; means for displaying a simulated environment via said display generation component; means enabled to detect, while displaying the simulated environment, a first input via the input device directed to a corresponding location within the simulated environment; in response to detecting the first input directed to the corresponding location within the simulated environment; pursuant to a determination that the first input is in a first input format and that the first input is detected at a first location within the simulated environment other than a current location of an insertion cursor within the simulated environment, displaying the insertion cursor at the first location; pursuant to a determination that the first input is in the first input format and that the first input is detected at a second location in the simulated environment corresponding to the current location of the insertion cursor, inserting a first object at the second location and moving the insertion cursor to a third location over the first object. and a means for enabling said A computer system comprising:

155. 1. An information processing device for use in an electronic device comprising a display generation component, one or more orientation sensors, and an input device, comprising: means for displaying a simulated environment via said display generation component; means enabled to detect, while displaying the simulated environment, a first input via the input device directed to a corresponding location within the simulated environment; in response to detecting the first input directed to the corresponding location within the simulated environment; pursuant to a determination that the first input is in a first input format and that the first input is detected at a first location within the simulated environment other than a current location of an insertion cursor within the simulated environment, displaying the insertion cursor at the first location; pursuant to a determination that the first input is in the first input format and that the first input is detected at a second location in the simulated environment corresponding to the current location of the insertion cursor, inserting a first object at the second location and moving the insertion cursor to a third location over the first object. and a means for enabling said An information processing device comprising:

156. 1. A computer system comprising: A display generating component; one or more orientation sensors; An input device; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for carrying out the method of any one of claims 139 to 151. Computer system.

157. 152. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more orientation sensors, and an input device, cause the computer system to perform the method of any one of claims 139 to 151.

158. A graphical user interface for a computer system having a display generation component, one or more orientation sensors, an input device, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 139 to 151.

159. A display generating component; one or more orientation sensors; An input device; Means for carrying out the method according to any one of claims 139 to 151; Equipped with Computer system.

160. 1. An information processing device for use in a computer system having a display generation component, one or more orientation sensors, and an input device, comprising: Means for carrying out the method according to any one of claims 139 to 151, Information processing device.

161. 1. A computer system having a display generation component, one or more cameras, and one or more pose sensors, comprising: Displaying an augmented reality environment via the display generation component, wherein displaying the augmented reality environment comprises: a representation of at least a portion of the field of view of the one or more cameras, the representation including physical objects, and updated as the content of the field of view of the one or more cameras changes; a virtual user interface object at a corresponding position within the representation of the field of view of the one or more cameras, the corresponding position of the virtual user interface object within the representation of the field of view of the one or more cameras being determined based on a fixed spatial relationship between the virtual user interface object and the physical object included in the representation of the field of view of the one or more cameras; including simultaneously displaying And, detecting, while displaying the augmented reality environment, a first change in an attitude of at least a portion of the computer system relative to a physical environment of the computer system via the one or more attitude sensors; responsive to detecting the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system, updating the augmented reality environment according to the first change in attitude of the portion of the computer system; updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a non-stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras by a first adjustment amount that is based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system; updating the corresponding positions of the virtual user interface objects to positions selected to maintain the fixed space between the virtual user interface objects and the physical objects included in the representation of the field of view of the one or more cameras; Including, updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system and by a second adjustment amount that is less than the first adjustment amount; updating the corresponding positions of the virtual user interface objects to positions selected to maintain the fixed spatial relationship between the virtual user interface objects and the physical objects included in the representation of the field of view of the one or more cameras; and A method comprising:

162. when the first change in attitude of the portion of the computer system is detected, and when the augmented reality environment was being displayed in the non-stable mode of operation, updating the augmented reality environment in accordance with the first change in attitude of the portion of the computer system; receiving a request to stabilize the virtual user interface object on the display; in response to the request to stabilize the virtual user interface object on the display, entering a stabilized mode of operation for the augmented reality environment; While in the stable mode of operation for the augmented reality environment, detecting a second change in an attitude of the portion of the computer system relative to the physical environment via the one or more orientation sensors; responsive to detecting the second change in an attitude of the portion of the computer system relative to the physical environment, updating the augmented reality environment according to the second change in attitude of the portion of the computer system; updating the representation of the portion of the field of view of the one or more cameras by less than the amount of the second change in pose of the portion of the computer system relative to the physical environment; and updating the virtual user interface objects to positions selected to maintain the fixed spatial relationship between the virtual user interface objects and the representations of the physical objects contained within the fields of view of the one or more cameras; Including, To do something, The method of claim 161.

163. the computer system includes an input device; and wherein the request to stabilize the virtual user interface object on the display includes an input received via the input device to zoom at least a portion of the augmented reality environment. The method of claim 162.

164. re-rendering the virtual user interface object in accordance with the magnitude of the input to zoom the portion of the displayed augmented reality environment in response to the request to stabilize the virtual user interface object on the display, the request including the input to zoom the portion of the displayed augmented reality environment. The method of claim 163.

165. 165. The method of any one of claims 161 to 164, wherein the physical object is replaced by the virtual user interface object in the displayed augmented reality environment.

166. detecting a first corresponding change in an attitude of the portion of the computer system relative to the physical environment of the computer system while the augmented reality environment is displayed in the stable operating mode; in response to detecting the first corresponding change in an attitude of the portion of the computer system relative to the physical environment of the computer system while the augmented reality environment is displayed in the stable operating mode, updating the augmented reality environment according to the corresponding change in attitude of the portion of the computer system, wherein updating the representation of the portion of the field of view of the one or more cameras in response to determining that the updated corresponding position of the virtual user interface object extends beyond the field of view of the one or more cameras includes displaying a placeholder image at a corresponding position in the augmented reality environment that corresponds to the portion of the virtual user interface object that extends beyond the field of view of the one or more cameras; 166. The method of any one of claims 161 to 165, comprising:

167. detecting a second corresponding change in an attitude of the portion of the computer system relative to the physical environment of the computer system while the augmented reality environment is displayed in the stable operating mode; updating the augmented reality environment in accordance with the corresponding change in attitude of the portion of the computer system in response to detecting the second corresponding change in attitude of the portion of the computer system relative to the physical environment of the computer system while the augmented reality environment is displayed in the stable operating mode, comprising ceasing to display at least a portion of the virtual user interface object in response to determining that the updated corresponding position of the virtual user interface object extends beyond the field of view of the one or more cameras; 166. The method of any one of claims 161 to 165, comprising:

168. updating the augmented reality environment in accordance with the corresponding change in attitude of the portion of the computer system relative to the physical environment of the computer system while the augmented reality environment is displayed in the stable operating mode; zooming the displayed augmented reality environment to augment a portion of the displayed virtual user interface object in accordance with a determination that the updated corresponding position of the virtual user interface object extends beyond the field of view of the one or more cameras; moving the virtual user interface object without zooming the displayed augmented reality environment in accordance with a determination that the updated corresponding position of the virtual user interface object does not extend beyond the field of view of the one or more cameras; Including, 166. The method of any one of claims 161 to 165, comprising:

169. 1. A computer system comprising: A display generating component; One or more cameras; one or more orientation sensors; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: Displaying an augmented reality environment via the display generation component, wherein displaying the augmented reality environment comprises: a representation of at least a portion of the field of view of the one or more cameras, the representation including physical objects, and updated as the content of the field of view of the one or more cameras changes; a virtual user interface object at a corresponding position within the representation of the field of view of the one or more cameras, the corresponding position of the virtual user interface object within the representation of the field of view of the one or more cameras being determined based on a fixed spatial relationship between the virtual user interface object and the physical object included in the representation of the field of view of the one or more cameras; including simultaneously displaying Doing things, detecting, while displaying the augmented reality environment, a first change in an attitude of at least a portion of the computer system relative to a physical environment of the computer system via the one or more attitude sensors; responsive to detecting the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system, updating the augmented reality environment according to the first change in attitude of the portion of the computer system; updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a non-stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras by a first adjustment amount that is based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system; updating the corresponding positions of the virtual user interface objects to positions selected to maintain the fixed space between the virtual user interface objects and the physical objects included in the representation of the field of view of the one or more cameras; Including, updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system and by a second adjustment amount that is less than the first adjustment amount; updating the corresponding position of the virtual user interface object to the selected position; Including, To do something, Including instructions, Computer system.

170. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more cameras, and one or more orientation sensors, cause the computer system to: Displaying an augmented reality environment via the display generation component, wherein displaying the augmented reality environment comprises: a representation of at least a portion of the field of view of the one or more cameras, the representation including physical objects, and updated as the content of the field of view of the one or more cameras changes; a virtual user interface object at a corresponding position within the representation of the field of view of the one or more cameras, the corresponding position of the virtual user interface object within the representation of the field of view of the one or more cameras being determined based on a fixed spatial relationship between the virtual user interface object and the physical object included in the representation of the field of view of the one or more cameras; including simultaneously displaying Let them do that. detecting, while displaying the augmented reality environment, a first change in an attitude of at least a portion of the computer system relative to a physical environment of the computer system via the one or more attitude sensors; responsive to detecting the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system, updating the augmented reality environment according to the first change in attitude of the portion of the computer system; updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a non-stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras by a first adjustment amount that is based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system; updating the corresponding positions of the virtual user interface objects to positions selected to maintain the fixed space between the virtual user interface objects and the physical objects included in the representation of the field of view of the one or more cameras; Including, updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system and by a second adjustment amount that is less than the first adjustment amount; updating the corresponding position of the virtual user interface object to the selected position; Including, To make A computer-readable storage medium.

171. 1. A computer system comprising: A display generating component; One or more cameras; one or more orientation sensors; A means for displaying an augmented reality environment via the display generation component, the displaying the augmented reality environment comprising: a representation of at least a portion of the field of view of the one or more cameras, the representation including physical objects, and updated as the content of the field of view of the one or more cameras changes; a virtual user interface object at a corresponding position within the representation of the field of view of the one or more cameras, the corresponding position of the virtual user interface object within the representation of the field of view of the one or more cameras being determined based on a fixed spatial relationship between the virtual user interface object and the physical object included in the representation of the field of view of the one or more cameras; including simultaneously displaying Means, means enabled to detect, while displaying the augmented reality environment, a first change in an attitude of at least a portion of the computer system relative to a physical environment of the computer system via the one or more attitude sensors; a means for detecting a first change in an attitude of the portion of the computer system relative to the physical environment of the computer system, the means being enabled to update the augmented reality environment according to the first change in attitude of the portion of the computer system, the means comprising: updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a non-stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras by a first adjustment amount that is based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system; updating the corresponding positions of the virtual user interface objects to positions selected to maintain the fixed space between the virtual user interface objects and the physical objects included in the representation of the field of view of the one or more cameras; Including, updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system and by a second adjustment amount that is less than the first adjustment amount; updating the corresponding position of the virtual user interface object to the selected position; Including, Means, Equipped with Computer system.

172. 1. An information processing device for use in an electronic device having a display generation component, one or more cameras, and one or more orientation sensors, comprising: A means for displaying an augmented reality environment via the display generation component, the displaying the augmented reality environment comprising: a representation of at least a portion of the field of view of the one or more cameras, the representation including physical objects, and updated as the content of the field of view of the one or more cameras changes; a virtual user interface object at a corresponding position within the representation of the field of view of the one or more cameras, the corresponding position of the virtual user interface object within the representation of the field of view of the one or more cameras being determined based on a fixed spatial relationship between the virtual user interface object and the physical object included in the representation of the field of view of the one or more cameras; including simultaneously displaying Means, means enabled to detect, while displaying the augmented reality environment, a first change in an attitude of at least a portion of the computer system relative to a physical environment of the computer system via the one or more attitude sensors; a means for detecting a first change in an attitude of the portion of the computer system relative to the physical environment of the computer system, the means being enabled to update the augmented reality environment according to the first change in attitude of the portion of the computer system, the means comprising: updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a non-stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras by a first adjustment amount that is based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system; updating the corresponding positions of the virtual user interface objects to positions selected to maintain the fixed space between the virtual user interface objects and the physical objects included in the representation of the field of view of the one or more cameras; Including, updating the augmented reality environment according to the first change in attitude of the portion of the computer system in accordance with a determination that the augmented reality environment is being displayed in a stable mode of operation; updating the representation of the portion of the field of view of the one or more cameras based on the first change in an attitude of the portion of the computer system relative to the physical environment of the computer system and by a second adjustment amount that is less than the first adjustment amount; updating the corresponding position of the virtual user interface object to the selected position; Including, Means, An information processing device comprising:

173. 1. A computer system comprising: A display generating component; One or more cameras; one or more orientation sensors; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 161 to 168. Computer system.

174. 169. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system having a display generating component, one or more cameras, and one or more orientation sensors, cause the computer system to perform the method of any one of claims 161 to 168.

175. A graphical user interface for a computer system having a display generation component, one or more cameras, one or more orientation sensors, memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 161 to 168.

176. A display generating component; One or more cameras; one or more orientation sensors; Means for carrying out the method according to any one of claims 161 to 168; A computer system comprising:

177. 1. An information processing device for use in a computer system having a display generation component, one or more cameras, and one or more orientation sensors, comprising: Means for carrying out the method according to any one of claims 161 to 168, Information processing device.

Images