Systems, methods, and graphical user interfaces for interacting with augmented reality and virtual reality environments.

Abstract

To provide mutual interaction with augmented reality environment and virtual reality environment.SOLUTION: A computer system simultaneously displays, in an augmented reality environment, at least a part of representation of visual fields of one or more cameras having corresponding physical objects and a corresponding virtual user interface object at corresponding positions in the virtual user interface determined based on the positions of the corresponding physical objects in the visual field. While detecting any input at the position corresponding to the displayed corresponding virtual user interface objects, in response to detection of movement of an input to the corresponding physical objects in the one or more visual fields of the cameras, the system adjusts appearance of the corresponding virtual user interface objects in accordance with magnitude of the movement of the input to the corresponding physical objects.SELECTED DRAWING: Figure 5A1

Description

Technical Field

[0001] This application generally relates to computer systems for virtual / augmented reality, including, but not limited to, electronic devices for interacting with augmented reality environments and virtual reality environments.

Background Art

[0002] The development of computer systems for virtual / augmented reality has advanced significantly in recent years. Exemplary virtual / augmented reality environments include at least some virtual elements that replace or augment the physical world. To interact with virtual / augmented reality environments, input devices such as the touch-sensitive surfaces of computer systems and other electronic computing devices are used. Exemplary touch-sensitive surfaces include touch pads, touch-sensitive remote control devices, and touch screen displays. These surfaces are used to manipulate user interfaces and objects therein on the display. Exemplary user interface objects include digital images, videos, text, icons, and control elements such as buttons and other graphics.

[0003] However, the methods and interfaces for interacting with environments that include at least some virtual elements (e.g., augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, using a series of inputs to select one or more user interface objects (e.g., one or more virtual elements within a virtual / augmented reality environment) and perform one or more actions regarding the selected user interface objects is redundant, imposes a significant cognitive burden on the user, and impairs the experience in the virtual / augmented reality environment. In addition, those methods are time-consuming more than necessary, thereby wasting energy. The latter problem is particularly critical in battery-operated devices.

Summary of the Invention

[0004] Therefore, there is a need for computer systems with improved methods and interfaces for interacting with augmented reality and virtual reality environments. Such methods and interfaces optionally complement or replace conventional methods for interacting with augmented reality and virtual reality environments. Such methods and interfaces reduce the number, range, and / or types of user input, resulting in a more efficient human-machine interface. With respect to battery-operated devices, such methods and interfaces conserve power and extend the time between battery charges.

[0005] The drawbacks and other problems associated with user interfaces for virtual / augmented reality described above are mitigated or eliminated by the disclosed computer systems. In some embodiments, the computer system includes a desktop computer. In some embodiments, the computer system is portable (e.g., a notebook computer, tablet computer, or handheld device). In some embodiments, the computer system includes a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the computer system has (and / or communicates with) a touchpad. In some embodiments, the computer system has (and / or communicates with) a touch-sensitive display (also known as a “touchscreen” or “touchscreen display”). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in memory for performing multiple functions. In some embodiments, the user interacts with the GUI in part through touch and gestures with a stylus and / or finger on a touch-sensitive surface. In some embodiments, the functions optionally include gameplay, image editing, drawing, presentation, word processing, spreadsheet creation, telephone, video conferencing, email, instant messaging, training support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. The executable instructions for performing those functions optionally include non-temporary computer-readable storage media or other computer program products configured to be executed by one or more processors.

[0006] According to some embodiments, the method is performed on a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying a representation of at least a portion of the field of view of one or more cameras, which includes corresponding physical objects, the representation being updated as the content of the field of view of one or more cameras changes, and corresponding virtual user interface objects located at corresponding positions within the representation of the field of view of one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects in the field of view of one or more cameras. The method also includes detecting input at corresponding positions on the corresponding virtual user interface objects while displaying the augmented reality environment. The method further includes detecting movement of input relative to the corresponding physical objects in the field of view of one or more cameras while continuing to detect input, and adjusting the appearance of the corresponding virtual user interface objects according to the magnitude of the movement of input relative to the corresponding physical objects in response to the detection of movement of input relative to the corresponding physical objects in the field of view of one or more cameras.

[0007] According to some embodiments, the method is performed on a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying a representation of at least a portion of the field of view of one or more cameras, which includes corresponding physical objects, the representation being updated as the content of the field of view of one or more cameras changes, and corresponding virtual user interface objects located at corresponding positions within the representation of the field of view of one or more cameras, the corresponding virtual user interface objects having positions determined based on the corresponding physical objects in the field of view of one or more cameras. The method also includes detecting inputs that change the virtual environment settings of the augmented reality environment while displaying the augmented reality environment. The method further includes, in response to detecting inputs that change the virtual environment settings, adjusting the appearance of the corresponding virtual user interface objects in accordance with the changes made to the virtual environment settings of the augmented reality environment, and filtering at least a portion of the representation of the field of view of one or more cameras, the filter being selected based on the changes made to the virtual environment settings.

[0008] According to some embodiments, the method is performed on a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying a representation of at least a portion of the field of view of one or more cameras, which includes corresponding physical objects, the representation being updated as the content of the field of view of one or more cameras changes, and a first virtual user interface object in a virtual model displayed at a corresponding position in the representation of the field of view of one or more cameras, the first virtual user interface object having a position determined based on the corresponding physical objects in the field of view of one or more cameras. The method also includes detecting a first input corresponding to a selection of the first virtual user interface object while displaying the augmented reality environment, and displaying a simulated field of view of the virtual model from the viewpoint of the first virtual user interface object in the virtual model in response to detecting the first input corresponding to a selection of the first virtual user interface object.

[0009] According to some embodiments, the method is performed on a computer system having a display generation component and an input device. The method includes displaying a first virtual user interface object in a virtual three-dimensional space via the display generation component. The method also includes detecting a first input via the input device, including the selection of a corresponding portion of the first virtual user interface object and the movement of the first input in two dimensions, while the first virtual user interface object is being displayed in the virtual three-dimensional space. In response to detecting the first input, including the movement of the first input in two dimensions, the method adjusts the appearance of the first virtual user interface object in a first direction determined based on the movement of the first input in two dimensions and the selected first portion of the first virtual user interface object, according to the determination that the corresponding portion of the first virtual user interface object is a first portion of the first virtual user interface object, wherein the adjustment of the first virtual user interface object in the first direction is constrained to movement within a first set of two dimensions of the virtual three-dimensional space, and the first virtual user interface object The method further includes adjusting the appearance of the first virtual user interface object in a second direction different from the first direction, based on the determination that the corresponding part of the input is a second part of the first virtual user interface object, separate from the first part of the first virtual user interface object, wherein the second direction is determined based on the movement of the first input in two dimensions and the selected second part of the first virtual user interface object, and the adjustment of the first virtual user interface object in the second direction is constrained to movement in a second set of two dimensions of a virtual three-dimensional space different from a first set of two dimensions of a virtual three-dimensional space.

[0010] According to some embodiments, the method is performed on a computer system having a display generation component, one or more attitude sensors, and an input device. The method includes displaying a simulated environment directed to the physical environment of the computer system in a first viewing mode via the display generation component, and displaying the simulated environment in the first viewing mode includes displaying a first virtual user interface object in a virtual model displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system. The method also includes detecting a first change in the attitude of at least a portion of the computer system relative to the physical environment via one or more attitude sensors while displaying the simulated environment, and changing the appearance of the first virtual user interface object in the virtual model in response to detecting the first change in the attitude of the portion of the computer system, so as to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment. The method further includes: changing the appearance of a first virtual user interface object based on a first change in the orientation of a part of a computer system; detecting a first gesture corresponding to interaction with a simulated environment via an input device; and, in response to detecting the first gesture corresponding to interaction with the simulated environment, performing an action within the simulated environment corresponding to the first gesture.In addition, this method involves detecting a second change in the orientation of a portion of the computer system relative to the physical environment via one or more orientation sensors after performing an action corresponding to a first gesture, and, in response to detecting a second change in the orientation of a portion of the computer system, transitioning from displaying a simulated environment including a virtual model in a first browsing mode to displaying a simulated environment including a virtual model in a second browsing mode, in which a virtual model is displayed within the simulated environment in the second browsing mode. The virtual model is then displayed within the simulated environment in the second browsing mode. Displaying a virtual model includes maintaining a fixed spatial relationship between the first virtual user interface object and the physical environment without altering the appearance of the first virtual user interface object, and continuing to display the first virtual model in the simulated environment in the first browsing mode in accordance with the determination that the first gesture did not satisfy the mode change criterion, and displaying a virtual model in the first browsing mode includes altering the appearance of the first virtual user interface object in the virtual model in accordance with a second change in the orientation of a part of the computer system relative to the physical environment, so as to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment.

[0011] According to some embodiments, the method is performed on a first computer system having a first display generation component, one or more first attitude sensors, and a first input device. The method includes displaying a simulated environment directed toward a first physical environment of the first computer system via the first display generation component of the first computer system, wherein displaying the simulated environment includes simultaneously displaying a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system, and a visual indication of a viewing viewpoint of the simulated environment of a second computer system, wherein the second computer system is a computer system having a second display generation component, one or more second attitude sensors, and a second input device, and displays a view of the simulated environment directed toward a second physical environment of the second computer system via the second display generation component of the second computer system. This method also includes detecting a change in the viewing viewpoint of the second computer system in the simulated environment based on a change in the orientation of a portion of the second computer system relative to the second physical environment while the simulated environment is being displayed via a first display generation component of the first computer system. The method further includes updating the visual indication of the viewing viewpoint of the second computer system in the simulated environment, as displayed via the first display generation component of the first computer system, in accordance with the change in the viewing viewpoint of the second computer system in the simulated environment, based on the detection of a change in the viewing viewpoint of the second computer system in the simulated environment based on a change in the orientation of a portion of the second computer system relative to the physical environment of the second computer system.

[0012] According to some embodiments, the method is performed on a computer system having a display generation component, one or more attitude sensors, and an input device. The method includes displaying a simulated environment via the display generation component. The method also includes detecting a first input directed to a corresponding location in the simulated environment via the input device while the simulated environment is being displayed. The method also includes, in response to detecting a first input directed to a corresponding location in the simulated environment, displaying an insertion cursor at a first location according to a determination that the first input is of a first input format and that the first input was detected at a first location in the simulated environment other than the current location of the insertion cursor in the simulated environment; and inserting a first object at a second location and moving the insertion cursor to a third location on the first object according to a determination that the first input is of a first input format and that the first input was detected at a second location in the simulated environment corresponding to the current location of the insertion cursor.

[0013] According to some embodiments, the method is performed on a computer system having a display generation component, one or more cameras, and one or more attitude sensors. The method includes displaying an augmented reality environment via the display generation component, which includes simultaneously displaying a representation of at least a portion of the field of view of one or more cameras, which includes physical objects and is updated as the content of the field of view of one or more cameras changes, and a virtual user interface object located at a corresponding position within the representation of the field of view of one or more cameras, wherein the corresponding position of the virtual user interface object within the representation of the field of view of one or more cameras is determined based on a fixed spatial relationship between the virtual user interface object and the physical objects included in the representation of the field of view of one or more cameras. The method also includes detecting a first change in the attitude of at least a portion of the computer system relative to the physical environment via one or more attitude sensors while the augmented reality environment is being displayed.The method also includes updating the augmented reality environment in accordance with a first change in the orientation of a portion of the computer system relative to the physical environment of the computer system, in response to the detection of a first change in the orientation of a portion of the computer system relative to the physical environment of the computer system, and updating the augmented reality environment in accordance with a first change in the orientation of a portion of the computer system relative to the physical environment of the computer system includes updating the representation of a portion of the field of view of one or more cameras by a first adjustment amount based on the first change in the orientation of a portion of the computer system relative to the physical environment of the computer system, and updating the corresponding position of a virtual user interface object to a position selected to maintain a fixed spatial relationship between the virtual user interface object and a physical object included in the representation of the field of view of one or more cameras, and updating the augmented reality environment in accordance with a first change in the orientation of a portion of the computer system relative to the physical environment of the computer system by a second adjustment amount less than the first adjustment amount, and updating the corresponding position of a virtual user interface object to a position selected to maintain a fixed spatial relationship between the virtual user interface object and a physical object included in the representation of the field of view of one or more cameras, in accordance with the determination that the augmented reality environment is displayed in stable operation mode.

[0014] According to some embodiments, the computer system includes (and / or communicates with) a display generation component (e.g., a display, projector, head-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of the content within the camera's field of view and optionally generates a video output including one or more streams of image frames capturing the content within the camera's field of view), one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a camera that tracks the position of one or more features of a user, such as a touchscreen display, mouse, joystick, wand controller, and / or the user's hand, which also functions as a display generation component), optionally one or more attitude sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, one or more processors, and memory for storing one or more programs, the one or more programs being configured to be executed by the one or more processors, and the one or more programs including instructions to perform or cause to perform any of the actions described herein. According to some embodiments, a computer-readable storage medium stores instructions internally, and when executed by a computer system including (and / or communicating with) a display generation component, one or more cameras, one or more input devices, optionally one or more attitude sensors, optionally one or more sensors for detecting the intensity of contact with a touch-sensing surface, and optionally one or more tactile output generators, the computer system is caused to perform any of the operations described herein.According to some embodiments, a graphical user interface on a computer system including (and / or communicating with) a display generation component, one or more cameras, one or more input devices, optionally one or more attitude sensors, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more tactile output generators, memory, and one or more processors for executing one or more programs stored in memory, includes one or more elements that are displayed in any of the methods described herein, and the elements are updated in response to input as described in any of the methods described herein. According to some embodiments, the computer system includes (and / or communicates with) a display generation component, one or more cameras, one or more input devices, optionally one or more attitude sensors, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more tactile output generators, and means for performing or causing to perform any of the operations described herein. According to some embodiments, an information processing device used in a computer system including (and / or communicating with) a display generation component, one or more cameras, one or more input devices, optionally one or more attitude sensors, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, and optionally one or more tactile output generators, includes means for performing or causing to perform any of the operations described herein.

[0015] Accordingly, a computer system having (and / or communicating with) a display generation component, one or more cameras, one or more input devices, optionally one or more attitude sensors, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, and optionally one or more tactile output generators provides improved methods and interfaces for interacting with augmented reality and virtual reality environments, thereby enhancing the effectiveness, efficiency, and user satisfaction of such a computer system. Such methods and interfaces can complement or replace conventional methods for interacting with augmented reality and virtual reality environments. [Brief explanation of the drawing]

[0016] To better understand the various embodiments described, the following “Modes for Carrying Out the Invention” should be referenced in conjunction with the following drawings, and similar reference numbers throughout the following drawings refer to the corresponding parts.

[0017] [Figure 1A] This is a block diagram showing a portable multifunctional device having a touch-sensitive display, according to several embodiments.

[0018] [Figure 1B] This is a block diagram showing exemplary components for event handling according to several embodiments.

[0019] [Figure 2] Several embodiments of a portable multifunctional device having a touchscreen are shown.

[0020] [Figure 3A] This is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, according to several embodiments.

[0021] [Figure 3B]A block diagram of an exemplary computer system according to some embodiments. [Figure 3C] A block diagram of an exemplary computer system according to some embodiments.

[0022] [Figure 4A] Shows an exemplary user interface for a menu of an application on a portable multifunctional device according to some embodiments.

[0023] [Figure 4B] Shows an exemplary user interface for a multifunctional device having a touch-sensitive surface separate from a display according to some embodiments.

[0024] [Figure 4C] Shows examples of dynamic intensity thresholds according to some embodiments. [Figure 4D] Shows examples of dynamic intensity thresholds according to some embodiments. [Figure 4E] Shows examples of dynamic intensity thresholds according to some embodiments.

[0025] [Figure 5A1] Shows an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model according to some embodiments. [Figure 5A2] Shows an exemplary user interface for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object within the virtual model according to some embodiments. [Figure 5A3] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A4] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A5] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A6] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A7] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A8]The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A9] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A10] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A11] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A12] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A13]The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A14] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A15] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A16] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A17] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A18]The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A19] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A20] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A21] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A22] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A23]The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A24] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A25] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A26] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A27] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A28]The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A29] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A30] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A31] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A32] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A33]The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A34] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A35] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A36] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A37] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A38]The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A39] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model. [Figure 5A40] The present invention provides exemplary user interfaces for displaying an augmented reality environment according to several embodiments, for adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between viewing a virtual model within the augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model.

[0026] [Figure 5B1] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B2] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B3] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B4] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B5]Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B6] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B7] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B8] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B9] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B10] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B11] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B12] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B13] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B14] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B15] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B16]Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B17] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B18] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B19] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B20] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B21] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B22] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B23] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B24] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B25] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B26] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B27]Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B28] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B29] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B30] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B31] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B32] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B33] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B34] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B35] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B36] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B37] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B38]Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B39] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B40] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown. [Figure 5B41] Examples of systems and user interfaces for 3D manipulation of virtual user interface objects, according to several embodiments, are shown.

[0027] [Figure 5C1] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C2] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C3] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C4] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C5] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C6] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C7]Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C8] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C9] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C10] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C11] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C12] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C13] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C14] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C15] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C16] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C17] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C18] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C19] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C20] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C21] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C22] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C23] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C24] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C25] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C26] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C27] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C28] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C29] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown. [Figure 5C30] Examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment, according to several embodiments, are shown.

[0028] [Figure 5D1] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D2] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D3a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D3b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D3c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D4a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D4b]Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D4c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D5a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D5b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D5c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D6a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D6b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D6c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D7a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D7b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D7c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D8a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D8b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D8c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D9a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D9b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D9c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D10a]Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D10b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D10c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D11a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D11b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D11c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D12a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D12b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D13a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D13b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D13c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D13d] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D14a] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D14b] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments. [Figure 5D14c] Examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment are shown according to several embodiments.

[0029] [Figure 5E1] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E2] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E3] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E4]Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E5] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E6] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E7] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E8] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E9] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E10] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E11] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E12] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E13] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E14] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E15] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E16] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E17]Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E18] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E19] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E20] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E21] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E22] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E23] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E24] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E25] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E26] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E27] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E28] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E29] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E30]Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E31] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown. [Figure 5E32] Examples of systems and user interfaces for positioning an insertion cursor, according to several embodiments, are shown.

[0030] [Figure 5F1] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F2] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F3a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F3b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F4a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F4b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F5a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F5b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F6a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F6b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F7a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F7b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F8a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F8b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F9a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F9b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F10a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F10b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F11a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F11b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F12a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F12b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F13a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F13b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F14a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F14b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F15a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F15b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F16a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F16b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F17a] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown. [Figure 5F17b] Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode, according to several embodiments, are shown.

[0031] [Figure 6A]This is a flowchart of a process for adjusting the appearance of virtual user interface objects in an augmented reality environment, according to several embodiments. [Figure 6B] This is a flowchart of a process for adjusting the appearance of virtual user interface objects in an augmented reality environment, according to several embodiments. [Figure 6C] This is a flowchart of a process for adjusting the appearance of virtual user interface objects in an augmented reality environment, according to several embodiments. [Figure 6D] This is a flowchart of a process for adjusting the appearance of virtual user interface objects in an augmented reality environment, according to several embodiments.

[0032] [Figure 7A] This is a flowchart of a process for applying filters to live images captured by one or more cameras in a computer system within an augmented reality environment, according to several embodiments. [Figure 7B] This is a flowchart of a process for applying filters to live images captured by one or more cameras in a computer system within an augmented reality environment, according to several embodiments. [Figure 7C] This is a flowchart of a process for applying filters to live images captured by one or more cameras in a computer system within an augmented reality environment, according to several embodiments.

[0033] [Figure 8A] This is a flowchart of the process for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model, according to several embodiments. [Figure 8B] This is a flowchart of the process for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model, according to several embodiments. [Figure 8C] This is a flowchart of the process for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model, according to several embodiments.

[0034] [Figure 9A] This is a flowchart of a process for 3D manipulation of a virtual user interface object according to several embodiments. [Figure 9B] This is a flowchart of a process for 3D manipulation of a virtual user interface object according to several embodiments. [Figure 9C] This is a flowchart of a process for 3D manipulation of a virtual user interface object according to several embodiments. [Figure 9D] This is a flowchart of a process for 3D manipulation of a virtual user interface object according to several embodiments. [Figure 9E] This is a flowchart of a process for 3D manipulation of a virtual user interface object according to several embodiments.

[0035] [Figure 10A] This is a flowchart of the process for transitioning between viewing modes of a displayed simulated environment, according to several embodiments. [Figure 10B] This is a flowchart of the process for transitioning between viewing modes of a displayed simulated environment, according to several embodiments. [Figure 10C] This is a flowchart of the process for transitioning between viewing modes of a displayed simulated environment, according to several embodiments. [Figure 10D] This is a flowchart of the process for transitioning between viewing modes of a displayed simulated environment, according to several embodiments. [Figure 10E]This is a flowchart of the process for transitioning between viewing modes of a displayed simulated environment, according to several embodiments.

[0036] [Figure 11A] This is a flowchart of a process for updating the viewing viewpoint indication of a second computer system within a simulated environment displayed by a first computer system, according to several embodiments. [Figure 11B] This is a flowchart of a process for updating the viewing viewpoint indication of a second computer system within a simulated environment displayed by a first computer system, according to several embodiments. [Figure 11C] This is a flowchart of a process for updating the viewing viewpoint indication of a second computer system within a simulated environment displayed by a first computer system, according to several embodiments.

[0037] [Figure 12A] This is a flowchart of the process for positioning an insertion cursor according to several embodiments. [Figure 12B] This is a flowchart of the process for positioning an insertion cursor according to several embodiments. [Figure 12C] This is a flowchart of the process for positioning an insertion cursor according to several embodiments. [Figure 12D] This is a flowchart of the process for positioning an insertion cursor according to several embodiments.

[0038] [Figure 13A] This is a flowchart of a process for displaying an augmented reality environment in a stable operating mode, according to several embodiments. [Figure 13B] This is a flowchart of a process for displaying an augmented reality environment in a stable operating mode, according to several embodiments. [Figure 13C]This is a flowchart of a process for displaying an augmented reality environment in a stable operating mode, according to several embodiments. [Figure 13D] This is a flowchart of a process for displaying an augmented reality environment in a stable operating mode, according to several embodiments. [Figure 13E] This is a flowchart of a process for displaying an augmented reality environment in a stable operating mode, according to several embodiments. [Modes for carrying out the invention]

[0039] An augmented reality environment is an environment in which reality is augmented with supplemental information that provides the user with additional information not available in the physical world. Conventional methods of interacting with an augmented reality environment (e.g., to access supplemental information) often require a number of separate inputs (e.g., a series of gestures and button presses) to achieve the intended result. Furthermore, conventional input methods are often limited in scope (e.g., by the size of the computer system's touch-sensitive display). Embodiments herein provide an intuitive method for the user to interact with an augmented reality environment (e.g., by adjusting the appearance of virtual user interface objects based on a combination of the movement of the computer system and the movement of touch on the computer system's input device (e.g., a touchscreen display), and by applying filters selected in real time based on the virtual environment settings of the augmented reality environment to live images captured by one or more cameras of the computer system).

[0040] In addition, conventional interactions with virtual / augmented reality environments are generally limited to a single viewpoint (e.g., from the viewpoint of the user wearing / holding the device). Embodiments herein provide a more immersive and intuitive way to experience virtual / augmented reality environments by presenting simulated views of virtual models (e.g., physical objects) within a virtual reality environment from the viewpoint of a virtual user interface object (e.g., from the viewpoint of a car or person in the augmented reality environment).

[0041] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways. For example, they facilitate displaying augmented reality environments, adjusting the appearance of the augmented reality environment and / or objects within it in response to different inputs, transitioning between browsing a virtual model within the augmented reality environment and browsing a simulated view of the virtual model from the viewpoint of objects within the virtual model, and facilitating three-dimensional manipulation of virtual user interface objects.

[0042] Figures 1A-1B, 2, and 3A-3C below provide a description of exemplary devices. Figures 4A-4B, 5A1-5A40, 5B1-5B41, 5C1-5C30, 5D1-5D14, 5E1-5E32, and 5F1-5F17 show examples of systems and user interfaces for multiple users interacting with virtual user interface objects in a displayed simulated environment, according to several embodiments, and show exemplary user interfaces for interacting with an augmented reality environment and a virtual reality environment, including, according to several embodiments, displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, transitioning between browsing a virtual model in the augmented reality environment and browsing a simulated view of the virtual model from the viewpoint of objects within the virtual model, and 3D manipulation of virtual user interface objects. Figures 6A-6D show flowcharts of methods for adjusting the appearance of virtual user interface objects in an augmented reality environment, according to several embodiments. Figures 7A to 7C show flowcharts of methods for applying filters to live images captured by one or more cameras of a computer system in an augmented reality environment, according to several embodiments. Figures 8A to 8C show flowcharts of methods for transitioning between browsing a virtual model in an augmented reality environment and browsing a simulated view of the virtual model from the viewpoint of an object within the virtual model, according to several embodiments. Figures 9A to 9E show flowcharts of methods for three-dimensional manipulation of a virtual user interface object, according to several embodiments. Figures 10A to 10E show flowcharts of methods for transitioning between browsing modes of a displayed simulated environment, according to several embodiments. Figures 11A to 11C show flowcharts of methods for updating the browsing viewpoint indication of a second computer system in a simulated environment displayed by a first computer system, according to several embodiments. Figures 12A to 12D show flowcharts of methods for positioning an insertion cursor, according to several embodiments.Figures 13A to 13E show flowcharts of methods for displaying an augmented reality environment in a stable operating mode according to several embodiments.

[0043] The processes shown in Figures 6A to 6D, 7A to 7C, 8A to 8C, 9A to 9E, 10A to 10E, 11A to 11C, 12A to 12D, and 13A to 13E are illustrated using the user interfaces in Figures 5A1 to 5A40, 5B1 to 5B41, 5C1 to 5C30, 5D1 to 5D14, 5E1 to 5E32, and 5F1 to 5F17. Example device

[0044] Herein, a detailed reference is made to embodiments shown in the accompanying drawings. The following detailed description includes numerous specific details to provide a complete understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other examples, well-known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the aspects of the embodiments.

[0045] In this specification, terms such as "first," "second," etc., are used to describe various elements in several embodiments, but it will be understood that these elements should not be limited by those terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the various embodiments described, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first and second contacts are contacts, but they are not the same contact unless the context explicitly indicates otherwise.

[0046] The terminology used in the descriptions of the various embodiments described herein is intended solely to describe specific embodiments and is not intended to be limiting. In the descriptions of the various embodiments and the accompanying claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless otherwise explicitly stated in the context. Furthermore, it should be understood that, as used herein, the term “and / or” refers to and includes any and all possible combinations of one or more of the enumerated items relating to the description. It will be further understood that, as used herein, the terms “includes,” “comprises,” and / or “comprising” specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0047] Where used herein, the term "if" is interpreted, at its discretion and in context, to mean "when," "upon," "in response to determining," or "in response to detecting." Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" are interpreted, at its discretion and in context, to mean "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0048] Computer systems for virtual / augmented reality include electronic devices that generate virtual / augmented reality environments. Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, which also includes other functions such as PDA functionality and / or music player functionality. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. Other portable electronic devices, such as laptops or tablet computers having touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads), are optionally used. In some embodiments, it should also be understood that the device is not a portable communication device but a desktop computer having touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads) and also including or communicating with one or more cameras.

[0049] The following description refers to computer systems that include electronic devices having (and / or communicating with) displays and touch-sensitive surfaces. However, it should be understood that computer systems optionally include one or more other physical user interface devices, such as a physical keyboard, mouse, joystick, wand controller, and / or a camera that tracks the position of one or more features of the user, such as the user's hands.

[0050] The device typically supports a variety of applications, including one or more of the following: gaming applications, note-taking applications, drawing applications, presentation applications, word processing applications, spreadsheet applications, phone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0051] Various applications running on this device optionally utilize at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed by the device, are optionally adjusted and / or modified on an application-by-application basis and / or within each application. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally supports a variety of applications with intuitive and transparent user interfaces for the user.

[0052] Here, we turn our attention to embodiments of portable devices having a touch-sensitive display. Figure 1A is a block diagram of a portable multifunction device 100 having a touch-sensitive display system 112 according to several embodiments. The touch-sensitive display system 112 may be referred to as a “touchscreen” for convenience, or simply as a touch-sensitive display. The device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164 (for example, as part of one or more cameras). The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (for example, a touch-sensitive surface such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 that generate tactile output on Device 100 (for example, on a touch-sensitive surface such as the touch-sensitive display system 112 of Device 100 or the touchpad 355 of Device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0053] As used herein and in the claims, the term “tactile output” means a physical displacement of the device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by the user through the user’s sense of touch. For example, in a situation where the device or a component of the device is in contact with the touch-sensitive surface of the user (e.g., the user’s fingers, palm, or other part of their hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or a component of the device. For example, the movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may be optionally interpreted by the user as a “down-click” or “up-click” of a physical actuator button. In some cases, the user may feel a tactile sensation such as a “down-click” or “up-click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user’s action. As another example, movement of a touch-sensitive surface may be interpreted or perceived by the user as "roughness" of that surface, even if there is no change in the smoothness of the touch-sensitive surface. Such user interpretations of touch depend on the user's personal sensory perception, but there are many touch sensory perceptions common to the majority of users. Therefore, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "up-click," "down-click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that produce the described sensory perception of a typical (or average) user.By providing haptic feedback to the user using tactile output, device usability is improved, the user device interface becomes more efficient (for example, by helping the user provide appropriate input when operating / interacting with the device and reducing user errors), and in addition, power consumption is reduced and the device's battery life is improved by enabling the user to use the device more quickly and efficiently.

[0054] It should be understood that device 100 is merely an example of a portable multifunction device, and that device 100 may optionally have more or fewer components than those shown, may optionally be a combination of two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in Figure 1A are implemented in hardware, software, firmware, or a combination thereof, including one or more signal processing circuits and / or application-specific integrated circuits.

[0055] Memory 102 optionally includes high-speed random-access memory and optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by CPU(s) 120 and other components of device 100 such as peripheral interface 118 is optionally controlled by memory controller 122.

[0056] A peripheral device interface 118 is used to connect the device's input and output peripherals to the CPU(s) 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and process data.

[0057] In some embodiments, the peripheral interface 118, the CPU(s) 120, and the memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0058] The RF (radio frequency) circuit 108 transmits and receives RF signals, also known as electromagnetic signals. The RF circuit 108 converts electrical signals to electromagnetic signals, or electromagnetic signals to electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, which include, but are not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, and memory. The RF circuit 108 optionally communicates wirelessly with networks such as the Internet, also known as the World Wide Web (WWW), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), as well as with other devices. Wireless communication is optional and includes Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long-Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), and Code Division Multiple Access (W-CDMA).Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), Instant Messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE)), Instant Messaging and Presence Services Using any of several communication standards, communication protocols, and communication technologies, including, but not limited to, a Service (IMPS), and / or Short Message Service (SMS), or any other suitable communication protocol, including a communication protocol not yet developed as of the filing date of this document.

[0059] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral interface 118, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. The audio circuit 110 also receives the electrical signal converted from the sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or RF circuit 108 by the peripheral interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212, Figure 2). The headset jack provides an interface between the audio circuit 110 and detachable audio input / output peripherals such as output-only headphones or headsets that have both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0060] The I / O subsystem 106 connects input / output peripherals on device 100, such as the touch-sensitive display system 112 and other input or control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from and transmit electrical signals to the other input or control devices 116. The other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller(s) 160 are optionally connected to (or not connected to) any of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., 208, Figure 2) optionally include up / down buttons for controlling the volume of speaker 111 and / or microphone 113. One or more buttons optionally include push buttons (e.g., 206, Figure 2).

[0061] The touch-sensitive display system 112 provides input and output interfaces between the device and the user. The display controller 156 receives electrical signals from and / or transmits electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays a visual output to the user. This visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively, “graphics”). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term “affordance” refers to a user-interactive graphical user interface object (for example, a graphical user interface object configured to respond to input directed toward the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0062] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accept user input based on tactile and / or haptic contact. The touch-sensitive display system 112 and the display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact on the touch-sensitive display system 112 (and any movement or interruption of contact) and translate the detected contact into interaction with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0063] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light-emitting polymer display) technology, or LED (light-emitting diode) technology, but other display technologies are used in other embodiments. The touch-sensitive display system 112 and the display controller 156 optionally use any of several currently known or future-developed touch-sensing technologies, including but not limited to capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements for determining one or more contact points with the touch-sensitive display system 112, to detect contact and any movement or interruption thereof. In some embodiments, projected mutual capacitance sensing technology is used, such as that found in iPhone®, iPod Touch®, and iPad® from Apple Inc. of Cupertino, California.

[0064] The touch-sensitive display system 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the video resolution of the touchscreen is greater than 400 dpi (e.g., 500 dpi, 800 dpi, or higher). The user optionally touches the touch-sensitive display system 112 using any suitable object or attachment such as a stylus or finger. In some embodiments, the user interface is designed to function with finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​a finger on the touchscreen than that of a stylus. In some embodiments, the device translates coarse finger-based input into a precise pointer / cursor position or command to perform an action desired by the user.

[0065] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0066] Device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with generating, managing, and distributing power within the portable device.

[0067] Device 100 also optionally includes one or more optical sensors 164 (for example, as part of one or more cameras). Figure 1A shows an optical sensor coupled with an optical sensor controller 158 in the I / O subsystem 106. The optical sensor(s) 164 optionally includes a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensor(s) 164 receives light from the environment, projected through one or more lenses, and converts the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), the optical sensor(s) 164 optionally captures still images and / or video. In some embodiments, the optical sensors are located on the back of device 100, opposite the touch-sensitive display system 112 on the front of the device, so that a touchscreen can be used as a viewfinder for acquiring still images and / or video images. In some embodiments, a separate light sensor is positioned on the front of the device so that an image of the user can be captured (for example, for selfies, or for video conferencing while the user is viewing other video conference participants on the touchscreen).

[0068] The device 100 also optionally includes one or more contact strength sensors 165. Figure 1A shows a contact strength sensor coupled with a strength sensor controller 159 in the I / O subsystem 106. The contact strength sensor(s) 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, pressure-power sensors, optical force sensors, capacitive touch-sensing surfaces, or other strength sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch-sensing surface). The contact strength sensor(s) 165 receives contact strength information (e.g., pressure information or a proxy for pressure information) from the environment. In some embodiments, at least one contact strength sensor is positioned juxtaposed with or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112). In some embodiments, at least one contact strength sensor is located on the back of the device 100, opposite the touchscreen display system 112 which is located on the front of the device 100.

[0069] The device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows a proximity sensor 166 coupled to a peripheral interface 118. Alternatively, the proximity sensor 166 is coupled to an input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is positioned near the user's ear (for example, when the user is making a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0070] The device 100 also optionally includes one or more tactile output generators 163. Figure 1A shows a tactile output generator coupled with a tactile feedback controller 161 in the I / O subsystem 106. In some embodiments, the tactile output generator(s) 163 includes one or more electroacoustic devices such as speakers or other audio components, and / or electromechanical devices that convert energy into linear motion, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile outputs on the device). The tactile output generator(s) 163 receives a tactile feedback generation command from the tactile feedback module 133 and generates a tactile output on the device 100 that can be sensed by the user of the device 100. In some embodiments, at least one tactile output generator is located on or near a touch-sensitive surface (e.g., a touch-sensitive display system 112) and optionally generates a tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward from the surface of device 100) or horizontally (e.g., forward / backward in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100, opposite the touch-sensitive display system 112 which is located on the front of device 100.

[0071] Device 100 also optionally includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (e.g., as part of an inertial measurement unit, IMU) for obtaining information about the device's position (e.g., attitude). Figure 1A shows sensors 167, 168, and 169 coupled to a peripheral interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to an input controller 160 in an I / O subsystem 106. In some embodiments, the information is displayed on a touchscreen display in a portrait or landscape view based on an analysis of data received from one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the device's location.

[0072] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a touch / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, as shown in Figures 1A and 3, memory 102 stores device / global internal state 157. The device / global internal state 157 includes one or more of the following: active application state, which indicates which application is active if there is an application currently active; display state, which indicates which applications, views, or other information occupy various areas of the touch-sensitive display system 112; sensor state, which includes information obtained from various sensors and other input or control devices 116 of the device; and position and / or orientation information, which relates to the position and / or orientation of the device.

[0073] An operating system (for example, an embedded operating system such as iOS, Android, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) includes various software components and / or drivers for controlling and managing overall system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0074] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire®, etc.) are adapted to connect to other devices directly or indirectly via a network (e.g., the Internet, Wi-Fi, etc.). In some embodiments, the external ports are multi-pin (e.g., 30-pin) connectors identical or similar to and / or compatible with the 30-pin connectors used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external ports are Lightning connectors identical or similar to and / or compatible with the Lightning connectors used in some iPhone®, iPod Touch®, and iPad® devices from Apple Inc. of Cupertino, California. In some embodiments, the external port is a USB Type-C connector identical, similar to, and / or compatible with the USB Type-C connector used in some electronic devices from Apple Inc. in Cupertino, California.

[0075] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in cooperation with the display controller 156) and contact with other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger down event), determining the intensity of contact (e.g., the force or pressure of contact, or an alternative to the force or pressure of contact), determining whether there is movement of contact and tracking movement across the touch-sensitive surface (e.g., detecting a drag event of one or more fingers), and determining whether contact has stopped (e.g., detecting a finger up event or interruption of contact). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions can optionally be applied to a single contact (e.g., a single finger or stylus contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0076] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motion, timing, and / or intensity of the detected contact). Therefore, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture involves detecting a finger down event, followed by a finger up (lift-off) event at the same location (or substantially the same location) as the finger down event (e.g., at the icon location). In another embodiment, detecting a finger swipe gesture on the touch-sensitive surface involves detecting a finger down event, followed by a drag event of one or more fingers, and then a finger up (lift-off) event. Similarly, taps, swipes, drags, and other gestures are optionally detected with respect to the stylus by detecting a specific contact pattern with respect to the stylus.

[0077] In some embodiments, detecting a finger tap gesture depends on the length of time between detecting a finger down event and detecting a finger up event, but not on the intensity of finger contact between the two events. In some embodiments, a tap gesture is detected according to the determination that the length of time between the finger down event and the finger up event is shorter than a predetermined value (e.g., shorter than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the intensity of finger contact between taps meets a given intensity threshold (greater than a nominal contact detection intensity threshold), such as a light or deep pressure intensity threshold. Thus, a finger tap gesture can satisfy certain input criteria that do not require the characteristic intensity of contact to meet a given intensity threshold in order for certain input criteria to be met. To clarify, finger contact in a tap gesture generally needs to meet a nominal contact detection intensity threshold below which contact is not detected in order to detect a finger down event. A similar analysis applies to detecting tap gestures or other contacts by a stylus. In cases where the device can detect contact from a finger or stylus hovering over the touch-sensitive surface, the nominal contact detection intensity threshold is optional and does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0078] In a similar manner, the same concept applies to other types of gestures. For example, swipe gestures, pinch gestures, de-pinch gestures, and / or long press gestures are optional and are detected based on whether they meet criteria that are either unrelated to the intensity of the contacts involved in the gesture, or do not require the contacts performing the gesture to reach an intensity threshold in order to be recognized. For example, swipe gestures are detected based on the amount of movement of one or more contacts, pinch gestures are detected based on the movement of two or more contacts toward each other, de-pinch gestures are detected based on the movement of two or more contacts toward each other, and long press gestures are detected based on the duration of contact on the touch-sensitive surface that is less than a threshold amount of movement. Therefore, the statement that a particular gesture recognition criterion does not require the intensity of one or more contacts to meet a corresponding intensity threshold for that criterion to be satisfied means that a particular gesture recognition criterion can be satisfied when the contact(s) in the gesture do not reach the corresponding intensity threshold, and can also be satisfied when one or more of the contacts in the gesture reach or exceed the corresponding intensity threshold. In some embodiments, a tap gesture is detected based on the determination that a finger-down event and a finger-up event were detected within a predetermined period, regardless of whether the contacts are above or below their respective intensity thresholds during a predetermined period, and a swipe gesture is detected based on the determination that the movement of the contact is greater than a predetermined magnitude, even if the contacts exceed their respective intensity thresholds at the end of the movement of the contacts. Even in embodiments where gesture detection is influenced by the intensity of the contact performing the gesture (for example, the device detects longer presses more quickly when the contact intensity exceeds an intensity threshold, or the device is slower to detect tap inputs when the contact intensity is higher), the detection of those gestures does not require the contact to reach a specific intensity threshold, as long as the criteria for recognizing the gesture can be met in situations where the contact does not reach a specific intensity threshold (for example, even if the amount of time required to recognize the gesture changes).

[0079] Contact intensity thresholds, duration thresholds, and movement thresholds are combined in various different combinations to create heuristics for distinguishing two or more different gestures directed towards the same input element or region in certain situations, thereby enabling multiple different interactions with the same input element to provide a richer set of user interactions and responses. A description that a particular set of gesture recognition criteria does not require that the intensity of a contact(s) meets a corresponding intensity threshold for a particular gesture recognition criterion to be satisfied does not preclude the simultaneous evaluation of other intensity-dependent gesture recognition criteria for identifying other gestures whose criteria are satisfied when the gesture includes a contact with an intensity exceeding a corresponding intensity threshold. For example, in some situations, a first gesture recognition criterion for a first gesture that does not require that the intensity of a contact(s) meet a corresponding intensity threshold for the first gesture recognition criterion to be satisfied is in competition with a second gesture recognition criterion for a second gesture that depends on the contact(s) reaching a corresponding intensity threshold. In such competition, a gesture is optional and will not be recognized as satisfying the first gesture recognition criterion for the first gesture if the second gesture recognition criterion for the second gesture is first satisfied. For example, if a contact reaches the corresponding intensity threshold before it moves a predetermined amount, a deep press gesture will be detected instead of a swipe gesture. Conversely, if a contact moves a predetermined amount before it reaches the corresponding intensity threshold, a swipe gesture will be detected instead of a deep press gesture. Even in such situations, the first gesture recognition criterion for the first gesture still does not require the intensity of the contact(s) to meet the corresponding intensity threshold for the first gesture recognition criterion to be satisfied, because if the contact remains below the corresponding intensity threshold until the end of the gesture (e.g., a swipe gesture with a contact that does not increase to an intensity above the corresponding intensity threshold), the gesture is recognized as a swipe gesture by the first gesture recognition criterion.In this way, a particular gesture recognition criterion that does not require the intensity of a contact(s) to meet a corresponding intensity threshold for that particular gesture recognition criterion to be satisfied still depends on the intensity of the contact(s) with respect to the intensity threshold in some situations, and / or (B) in some situations, the particular gesture recognition criterion (e.g., for a long press gesture) does not function if a competing set of intensity-dependent gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-dependent gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input (e.g., for a long press gesture competing with a deep press gesture for recognition).

[0080] In conjunction with the accelerometer 167, gyroscope 168, and / or magnetometer 169, the attitude module 131 selectively detects attitude information about the device, such as the device's attitude (e.g., roll, pitch, and / or yaw) within a specific coordinate system. The attitude module 131 includes software components for performing various actions related to detecting the device's position and changes in the device's attitude.

[0081] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch-sensitive display system 112 or other display, including components for modifying the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual characteristics). As used herein, the term “graphics” includes, but is not limited to, any object that can be displayed to a user, including, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, and animations.

[0082] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like, as needed, along with coordinate data and other graphic characteristic data, specifying the graphics to be displayed, and then generates screen image data to output to the display controller 156.

[0083] The haptic feedback module 133 includes various software components that generate commands (for example, commands used by the haptic feedback controller 161) that create haptic outputs at one or more locations on the device 100 using a haptic output generator (one or more) 163 in response to user interaction with the device 100.

[0084] The text input module 134 is optionally a component of the graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other applications that require text input).

[0085] The GPS module 135 determines the device's location and provides this information for use in various applications (for example, to the phone 138 for location-based phone calls, to the camera 143 as metadata for photos / videos, and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0086] The virtual / augmented reality module 145 provides virtual reality logic and / or augmented reality logic to an application 136 that implements augmented reality functionality, and in some embodiments, virtual reality functionality. The virtual / augmented reality module 145 facilitates the overlaying of virtual content, such as virtual user interface objects, onto representations of at least a portion of the field of view of one or more cameras. For example, with assistance from the virtual / augmented reality module 145, representations of at least a portion of the field of view of one or more cameras may include corresponding physical objects, and virtual user interface objects may be displayed in a position within the virtual reality environment determined based on the corresponding physical objects in the field of view of one or more cameras, or based on the orientation of at least a portion of the computer system (e.g., the orientation of a display device used to display the user interface to a user of the computer system).

[0087] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof: ●Contact module 137 (sometimes also called the address book or contact list), ●Telephone module 138, ●Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ●Training support module 142, ● Camera module 143 for still images and / or video images, ●Image management module 144, ● Browser module 147, ● Calendar module 148, ● A widget module 149 that optionally includes one or more of the following: weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, other widgets obtained by the user, and user-created widgets 149-6. ●Widget creator module 150 for creating user-created widget 149-6, ●Search module 151, ● A video and music player module 152, which optionally consists of a video player module and a music player module. ●Memo Module 153, ● Map module 154 and / or ● Online video module 155.

[0088] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java®-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0089] Together with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contact module 137 includes executable instructions for managing an address book or contact list (stored, for example, in the application internal state 192 of the contact module 137 in memory 102 or memory 370), which include adding names(s) to the address book, removing names(s) from the address book, associating telephone numbers(s) to names, email addresses(s) to names, addresses(s) to names, or other information, associating images to names, categorizing and sorting names, providing telephone numbers and / or email addresses to initiate and / or facilitate communication by telephone 138, video conferencing 139, email 140, or IM 141, and so on.

[0090] In conjunction with the RF circuit 108, voice circuit 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, and text input module 134, the telephone module 138 includes executable commands to input a series of characters corresponding to a telephone number, access one or more telephone numbers in the address book 137, change the entered telephone number, dial each telephone number, conduct a conversation, and disconnect or hang up when the conversation is complete. As described above, the wireless communication may optionally use any of several communication standards, protocols, and technologies.

[0091] In conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, the video conferencing module 139 includes executable commands to start, conduct, and end a video conference between the user and one or more other participants, in accordance with the user's commands.

[0092] In conjunction with the RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, and text input module 134, the email client module 140 includes executable commands for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with the image management module 144, the email client module 140 makes it extremely easy to create and send emails containing still or video images captured by the camera module 143.

[0093] In conjunction with the RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, and text input module 134, the instant messaging module 141 includes executable instructions for inputting a series of characters corresponding to an instant message, modifying previously entered characters, sending each instant message (for example, using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone-based instant messaging, or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, “instant messaging” refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0094] In conjunction with the RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, the training support module 142 includes executable commands for creating training (e.g., having time, distance, and / or calorie consumption targets), communicating with training sensors (in sports devices and smartwatches), receiving training sensor data, calibrating sensors used to monitor training, selecting and playing music for training, and displaying, storing, and transmitting training data.

[0095] Together with the touch-sensitive display system 112, display controller 156, optical sensor(s) 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, the camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0096] Together with the touch-sensitive display system 112, display controller 156, contact module 130, graphic module 132, text input module 134, and camera module 143, the image management module 144 includes executable commands for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0097] Together with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, the browser module 147 includes executable commands for browsing the internet in accordance with user commands, including searching, linking, receiving, and displaying web pages or parts thereof, as well as attachments and other files linked to web pages.

[0098] Together with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, the calendar module 148 includes executable instructions that, in accordance with user commands, create, display, modify, and store a calendar and data associated with the calendar (e.g., calendar items, to-do lists, etc.).

[0099] Along with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, the widget module 149 is optionally a mini-application downloaded and used by the user (e.g., weather widget 149-1, stock price widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or a mini-application created by the user (e.g., user-created widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript® file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! Widget).

[0100] In conjunction with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, the widget creator module 150 includes executable commands for creating widgets (for example, converting user-specified portions of a web page into widgets).

[0101] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions to search for text, music, sounds, images, videos, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to user commands.

[0102] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, the RF circuit 108, and the browser module 147, the video and music player module 152 includes executable commands that enable the user to download and play recorded music or other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable commands to display, present, or otherwise play video (for example, on the touch-sensitive display system 112, or on an external display connected wirelessly or via the external port 124). In some embodiments, the device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0103] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the memo module 153 includes executable commands that create and manage notes, to-do lists, and the like, according to user commands.

[0104] In conjunction with the RF circuit 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, the map module 154 includes executable commands that, in accordance with user commands, receive, display, modify, and store maps and map-associated data (e.g., driving directions, data about stores and other points of interest at or near a particular location, and other location-based data).

[0105] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, the RF circuit 108, the text input module 134, the email client module 140, and the browser module 147, the online video module 155 includes executable instructions that enable the user to access, view, receive (e.g., by streaming and / or downloading), play (e.g., on the touchscreen 112, or on an external display connected wirelessly or via the external port 124), send emails containing links to specific online videos, and otherwise manage them. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send links to specific online videos.

[0106] Each of the modules and applications identified above corresponds to one or more of the functions described above, as well as an executable instruction set for performing the methods described in this application (e.g., methods performed by a computer and other information processing methods described herein). These modules (i.e., instruction sets) do not need to be implemented as separate software programs, procedures, or modules; therefore, various subsets of these modules can be optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0107] In some embodiments, device 100 is a device in which the operation of a predetermined set of functions in the device is performed solely through a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for device 100 to operate, the number of physical input control devices (push buttons, dials, etc.) on device 100 is optionally reduced.

[0108] A predetermined set of functions performed only through a touchscreen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, home menu, or root menu. In such embodiments, a “menu button” is implemented using a touch-sensitive surface. In some other embodiments, the menu button is a physical push button or other physical input control device instead of a touch-sensitive surface.

[0109] Figure 1B is a block diagram showing exemplary components for event processing according to several embodiments. In some embodiments, memory 102 (in Figure 1A) or 370 (in Figure 3A) includes an event sorting unit 170 (e.g., within the operating system 126) and each application 136-1 (e.g., any of the applications 136, 137-155, 380-390 described above).

[0110] The event sorting unit 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is distributed. The event sorting unit 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates the current application view(s) displayed on the touch-sensitive display system 112 when the application is active or running. In some embodiments, a device / global internal state 157 is used by the event sorting unit 170 to determine which application(s) are currently active, and the application internal state 192 is used by the event sorting unit 170 to determine the application view(s) to which the event information is distributed.

[0111] In some embodiments, the application internal state 192 includes additional information such as resume information used when the application 136-1 resumes execution, user interface state information indicating information displayed or ready to be displayed by the application 136-1, state queues that allow the user to return to a previous state or view of the application 136-1, and redo / undo queues of previous actions taken by the user.

[0112] The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about sub-events (for example, a user touch on the touch-sensitive display system 112 as part of a multi-touch gesture). The peripheral interface 118 transmits information received from the I / O subsystem 106, or from sensors such as the proximity sensor 166, one or more accelerometers 167, and / or the microphone 113 (via the audio circuit 110). The information received by the peripheral interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display system 112 or the touch-sensitive surface.

[0113] In some embodiments, the event monitor 171 sends requests to the peripheral interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when a significant event occurs (e.g., receiving input that exceeds a predetermined noise threshold and / or for a longer period than predetermined).

[0114] In some embodiments, the event sorting unit 170 also includes a hit view determination module 172 and / or an active event recognition determination module 173.

[0115] The hit view determination module 172 provides software procedures for determining where in one or more views a sub-event occurred when the touch-sensitive display system 112 displays two or more views. A view consists of control devices and other elements that the user can see on the display.

[0116] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of each application) in which a touch is detected optionally corresponds to the program level within the application's program hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected optionally refers to the hit view, and the set of events recognized as appropriate input is optionally determined at least in part based on the hit view of the initial touch that initiates the touch gesture.

[0117] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views arranged in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchy from which the sub-events should be processed. In most situations, the hit view is the lowest-level view from which the initiating sub-event (i.e., the first sub-event in a series of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source from which it was identified as a hit view.

[0118] The active event recognition unit determination module 173 determines which view(s) within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition unit determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition unit determination module 173 determines that all views including the physical location of the sub-event are views that are actively involved, and thus, determines that all views that are actively involved should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely limited to an area related to one particular view, the upper-level views within the hierarchy still remain views that are actively involved.

[0119] The event dispatcher module 174 dispatches event information to an event recognition unit (e.g., event recognition unit 180). In embodiments including the active event recognition unit determination module 173, the event dispatcher module 174 dispatches event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information obtained by each event receiver module 182 in an event queue.

[0120] In some embodiments, the operating system 126 includes an event sorter 170. Alternatively, the application 136-1 includes the event sorter 170. In still other embodiments, the event sorter 170 is an independent module or part of another module stored in the memory 102, such as the touch / motion module 130.

[0121] In some embodiments, application 136-1 includes a plurality of event processing units 190 and one or more application views 191, each including instructions for processing touch events that occur within respective views of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, each application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other characteristics. In some embodiments, the corresponding event processing unit 190 includes one or more of event data 179 received from data update unit 176, object update unit 177, GUI update unit 178, and / or event sorting unit 170. The event processing unit 190 optionally utilizes or calls data update unit 176, object update unit 177, or GUI update unit 178 to update the internal state 192 of the application. Instead, one or more of the application views 191 include one or more respective event processing units 190. Also, in some embodiments, one or more of data update unit 176, object update unit 177, and GUI update unit 178 are included in the corresponding application view 191.

[0122] Each event recognition unit 180 receives event information (e.g., event data 179) from event sorting unit 170 and identifies an event from the event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparing unit 184. In some embodiments, the event recognition unit 180 also includes at least a subset of metadata 183 and event distribution instructions 188 (optionally including sub-event distribution instructions).

[0123] The event receiving unit 182 receives event information from the event sorting unit 170. The event information includes information about sub-events, such as information about touches or the movement of touches. Depending on the sub-event, the event information also includes additional information such as the position of the sub-event. When the sub-event involves a touch movement, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, an event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation of the device (also called the device's orientation).

[0124] The event comparison unit 184 compares event information with a predefined definition of an event or sub-event, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes definitions of events (e.g., a predetermined sequence of sub-events), such as event 1 (187-1) and event 2 (187-2). In some embodiments, sub-events in event 187 include, for example, the start of a touch, the end of a touch, a move of a touch, a stop of a touch, and multiple touches. In one example, the definition for event 1 (187-1) is a double tap on a displayed object. A double tap includes, for example, a first touch (start of touch) for a predetermined stage on the displayed object, a first lift-off (end of touch) for a predetermined stage, a second touch (start of touch) for a predetermined stage on the displayed object, and a second lift-off (end of touch) for a predetermined stage. In another example, the definition of event 2(187-2) is a drag on a displayed object. The drag includes, for example, touching (or contacting) a predetermined number of stages on the displayed object, moving the touch across the touch-sensitive display system 112, and lifting off the touch (end of the touch). In some embodiments, the event also includes information about one or more associated event processing units 190.

[0125] In some embodiments, the event definition 187 includes an event definition for each user interface object. In some embodiments, the event comparison unit 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display system 112, when a touch is detected on the touch-sensitive display system 112, the event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event processing unit 190, the event comparison unit uses the results of the hit test to determine which event processing unit 190 should be activated. For example, the event comparison unit 184 selects the sub-event and the event processing unit associated with the object that triggers the hit test.

[0126] In some embodiments, the definition of each event 187 also includes a delay action that delays the delivery of event information until it is determined whether a series of sub-events correspond to the event type of the event recognition unit.

[0127] If each event recognition unit 180 determines that a series of sub-events does not match any of the events in the event definition 186, each event recognition unit 180 enters an event impossible, event failed, or event terminated state, and thereafter ignores subsequent sub-events of the touch gesture. In this situation, if there are other event recognition units that remain active for the hit view, those event recognition units continue to track and process the sub-events of the ongoing touch gesture.

[0128] In some embodiments, the corresponding event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or lists that indicate to the actively involved event recognition units how the event distribution system should perform sub-event distribution. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how the event recognition units interact with each other, or how they can interact with each other. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how sub-events are distributed to various levels in the view hierarchy or program hierarchy.

[0129] In some embodiments, each event recognition unit 180 activates an event processing unit 190 associated with an event when one or more specific sub-events of an event are recognized. In some embodiments, each event recognition unit 180 delivers event information associated with the event to the event processing unit 190. Activating the event processing unit 190 is separate from sending (and delaying sending) sub-events to the respective hit views. In some embodiments, the event recognition unit 180 sets a flag associated with the recognized event, and the event processing unit 190 associated with that flag captures the flag and executes a default process.

[0130] In some embodiments, the event distribution command 188 includes a sub-event distribution command that distributes event information about a sub-event without activating an event processing unit. Instead, the sub-event distribution command distributes event information to an event processing unit associated with a set of sub-events, or to a view that is actively involved. The event processing unit associated with the set of sub-events or the view that is actively involved receives the event information and performs predetermined processing.

[0131] In some embodiments, the data update unit 176 creates and updates data used in application 136-1. For example, the data update unit 176 updates telephone numbers used in contact module 137 or stores video files used in video and music player module 152. In some embodiments, the object update unit 177 creates and updates objects used in application 136-1. For example, the object update unit 177 creates new user interface objects or updates the position of user interface objects. The GUI update unit 178 updates the GUI. For example, the GUI update unit 178 prepares display information and sends it to graphics module 132 for display on touch-sensitive display.

[0132] In some embodiments, the event processing unit(s) 190 includes or has access to a data update unit 176, an object update unit 177, and a GUI update unit 178. In some embodiments, the data update unit 176, the object update unit 177, and the GUI update unit 178 are included in a single module of their respective applications 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0133] The foregoing description regarding the handling of user touch events on a touch-sensitive display also applies to other forms of user input for operating the multifunction device 100 using input devices, but it should be understood that not all of these begin on the touchscreen. For example, mouse movement and mouse button presses, touch movements such as taps, drags, and scrolls on a touchpad, which are optionally linked to single or multiple presses or holds on a keyboard, pen stylus input, input based on real-time analysis of video images acquired by one or more cameras, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof may be optionally used as inputs corresponding to sub-events that define the events to be recognized.

[0134] Figure 2 shows a portable multifunctional device 100 having a touchscreen (e.g., a touch-sensitive display system 112, Figure 1A) according to several embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In these embodiments, and in embodiments described later, the user can select one or more graphics by making gestures on the graphics using, for example, one or more fingers 202 (not shown in the figure to an exact scale) or one or more styluses 203 (not shown in the figure to an exact scale). In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with that one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and / or downward), and / or rolling of a finger in contact with the device 100 (from right to left, left to right, upward and / or downward). In some implementations or situations, accidental contact with a graphic does not constitute a selection of that graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon does not arbitrarily select the corresponding application.

[0135] Device 100 optionally also includes one or more physical buttons, such as a "Home" or menu button 204. As previously mentioned, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally run on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on a touchscreen display.

[0136] In some embodiments, device 100 includes a touchscreen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for turning the device on / off and locking the device, one or more volume buttons 208, a subscriber identification module (SIM) card slot 210, a headset jack 212, and an external docking / charging port 124. The push button 206 is optionally used to turn the device on / off by pressing down and holding the button down for a predetermined period of time, to lock the device, and / or to unlock the device or initiate an unlocking process by pressing down and releasing the button before the predetermined period has elapsed. In some embodiments, device 100 also accepts verbal input through a microphone 113 to activate or deactivate certain functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output to the user of device 100.

[0137] Figure 3A is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to several embodiments. Device 300 does not need to be portable. In some embodiments, device 300 is a game system, laptop computer, desktop computer, tablet computer, multimedia playback device, navigation device, educational device (such as a children's learning toy), game system, or control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. The communication buses 320 optionally include circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 300 includes an input / output (I / O) interface 330, which optionally includes a display 340, which is a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, a touchpad 355, a touch output generator 357 for generating tactile output on a device 300 (similar to the tactile output generator(s) 163 described above with reference to Figure 1A, for example), and a sensor 359 (e.g., an optical sensor, an accelerometer, a proximity sensor, a touch sensor, and / or a touch intensity sensor similar to the touch intensity sensor(s) 165 described above with reference to Figure 1A). The memory 370 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU 310.In some embodiments, memory 370 stores programs, modules, and data structures similar to, or subsets thereof, that are stored in memory 102 of the portable multifunction device 100 (Figure 1A). Furthermore, memory 370 optionally stores additional programs, modules, and data structures that are not present in memory 102 of the portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, while memory 102 of the portable multifunction device 100 (Figure 1A) optionally does not store those modules.

[0138] Each of the elements identified in Figure 3A above is optionally stored in one or more of the previously mentioned memory devices. Each of the modules identified above corresponds to an instruction set that performs the function described above. The modules or programs (e.g., instruction sets) identified above do not need to be implemented as separate software programs, procedures, or modules; therefore, various subsets of these modules can be optionally combined or otherwise reconfigured in various embodiments. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0139] Figures 3B to 3D are block diagrams of exemplary computer systems 301 according to several embodiments.

[0140] In some embodiments, the computer system 301 includes and / or communicates with the following: ● Input device(s) (302 and / or 307, e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touch screen display that also functions as a display generation component, a mouse, a joystick, a wand controller, and / or a camera that tracks the position of one or more features of the user such as the user's hand), ● Virtual / augmented reality logic 303 (e.g., virtual / augmented reality module 145), ● Display generation component(s) (304 and / or 308, e.g., a display, a projector, a head-up display, etc.) for displaying virtual user interface elements to the user, ● Camera(s) (e.g., 305 and / or 311) for capturing an image of the device's field of view, e.g., to determine the placement of virtual user interface elements, to determine the device's pose, and / or to display a portion of the physical environment in which the camera(s) is / are located, ● Pose sensor(s) (e.g., 306 and / or 311) for determining the pose of the device with respect to the physical environment and / or changes in the pose of the device.

[0141] In some computer systems (e.g., 301-a of FIG. 3B), the input device(s) 302, the virtual / augmented reality logic 303, the display generation component(s) 304, the camera(s) 305, and the pose sensor(s) 306 are all integrated into a computer system (e.g., the portable multifunctional device 100 of FIGS. 1A - 1B or the device 300 of FIG. 3 such as a smartphone or a tablet).

[0142] In some computer systems (e.g., 301-b), in addition to an integrated input device(s) 302, virtual / augmented reality logic 303, display generation component(s) 304, camera(s) 305, and attitude sensor(s) 306, the computer system also communicates with additional devices separate from the computer system, such as a separate input device(s) 307, such as a touch-sensitive surface, wand, remote control, and / or a separate display generation component(s) 308, such as a virtual reality headset or augmented reality glasses that overlay virtual objects onto the physical environment.

[0143] In some computer systems (e.g., 301-c in Figure 3C), the input device(s) 307, display generation component(s) 309, camera(s) 311, and / or attitude sensor(s) 312 are separate from the computer system and communicate with the computer system. In some embodiments, other combinations of components within the computer system 301 and components communicating with the computer system are used. For example, in some embodiments, the display generation component(s) 309, camera(s) 311, and attitude sensor(s) 312 are incorporated into a headset that is either integrated with the computer system or communicates with the computer system.

[0144] In some embodiments, all operations described later with reference to Figures 5A1-5A40 and 5B1-5B41 are performed on a single computing device having the virtual / augmented reality logic 303 (e.g., computer system 301-a described later with reference to Figure 3B). However, it should be understood that often multiple different computing devices are linked together to perform operations described later with reference to Figures 5A1-5A40 and 5B1-5B41 (e.g., a computing device having the virtual / augmented reality logic 303 communicates with a separate computing device having a display 450 and / or a separate computing device having a touch-sensitive surface 451). In any of these embodiments, the computing device described later with reference to Figures 5A1-5A40 and 5B1-5B41 is one or more computing devices including the virtual / augmented reality logic 303. In addition, it should be understood that in various embodiments, the virtual / augmented reality logic 303 may be divided among multiple separate modules or computing devices. However, for the purposes of this description, the virtual / augmented reality logic 303 will be referred to primarily as residing within a single computing device, so as not to unnecessarily obscure other aspects of the embodiment.

[0145] In some embodiments, the virtual / augmented reality logic 303 includes one or more modules (e.g., one or more event processing units 190 including one or more object update units 177 and one or more GUI update units 178, as described above in more detail with reference to Figure 1B) that receive interpreted inputs and generate instructions for updating the graphical user interface according to these interpreted inputs, which are then used to update the graphical user interface on the display. In some embodiments, interpreted inputs for inputs detected (e.g., by the contact motion module 130 in Figures 1A and 3), recognized (e.g., by the event recognition unit 180 in Figure 1B), and / or delivered (e.g., by the event sorting unit 170 in Figure 1B) are used to update the graphical user interface on the display. In some embodiments, the interpreted inputs are generated by modules in a computing device (e.g., the computing device receives raw contact input data to identify gestures from raw contact input data). In some embodiments, some or all of the interpreted input is received by a computing device as interpreted input (for example, a computing device including a touch-sensitive surface 451 processes the raw contact input data to identify a gesture from the raw contact input data and transmits gesture-indicating information to a computing device including virtual / augmented reality logic 303).

[0146] In some embodiments, both the display and the touch-sensitive surface are integrated with a computer system (e.g., 301-a in Figure 3B) that includes a virtual / augmented reality logic 303. For example, the computer system may be a desktop or laptop computer having an integrated display (e.g., 340 in Figure 3) and a touchpad (e.g., 355 in Figure 3). In another example, the computing device may be a portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) having a touchscreen (e.g., 112 in Figure 2).

[0147] In some embodiments, the touch-sensitive surface is integrated with the computer system, while the display is not integrated with the computer system, including the virtual / augmented reality logic 303. For example, the computer system may be a device 300 (e.g., a desktop computer or a laptop computer) in which an integrated touchpad (e.g., 355 in Figure 3) is connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, television, etc.). In another example, the computer system may be a portable multifunction device 100 (e.g., a smartphone, PDA, tablet computer, etc.) in which a touchscreen (e.g., 112 in Figure 2) is connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, television, etc.).

[0148] In some embodiments, the display is integrated with the computer system, while the touch-sensitive surface is not integrated with the computer system, including the virtual / augmented reality logic 303. For example, the computer system may be a device 300 (e.g., a desktop computer, a laptop computer, a television with an all-in-one set-top box) in which an integrated touch display (e.g., 340 in Figure 3) is connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device, etc.). In another example, the computer system may be a portable multifunction device 100 (e.g., a smartphone, a PDA, a tablet computer, etc.) in which a touchscreen (e.g., 112 in Figure 2) is connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, another portable multifunction device having a touchscreen that functions as a remote touchpad, etc.).

[0149] In some embodiments, neither the display nor the touch-sensitive surface is integrated with a computer system including the virtual / augmented reality logic 303 (e.g., 301-c in Figure 3C). For example, the computer system may be a standalone computing device 300 (e.g., a set-top box, a game console, etc.) connected (via wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device, etc.) and a separate display (e.g., a computer monitor, a television, etc.).

[0150] In some embodiments, the computer system has an integrated audio system (e.g., an audio circuit 110 and speaker 111 in a portable multifunction device 100). In some embodiments, the computing device communicates with an audio system separate from the computing device. In some embodiments, the audio system (e.g., an audio system integrated in a television unit) is integrated with a separate display. In some embodiments, the audio system (e.g., a stereo system) is a standalone system separate from the computer system and the display.

[0151] Here, we focus on embodiments of a user interface ("UI") that are optionally implemented on the portable multi-functional device 100.

[0152] Figure 4A shows an exemplary user interface for an application menu on a portable multifunction device 100 according to several embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof. ● Signal strength indicators (single or multiple) for wireless communication (single or multiple) such as cellular signals and Wi-Fi signals. ●Time, ●Bluetooth (registered trademark) indicator, ●Battery status indicator, ●Tray 408 contains icons for frequently used applications, as shown below. ○Optionally including an indicator 414 for the number of missed calls or voicemail messages, an icon 416 of the telephone module 138 labeled "Telephone", ○Optionally including an indicator 410 for the number of unread emails, an icon 418 of the email client module 140 labeled "Mail", ○ Icon 420 of browser module 147, labeled "Browser," and, ○ Icon 422 of the video and music player module 152 labeled "Music," and, ● Icons of other applications, such as the following: ○ Icon 424 of IM module 141, labeled "Message", ○ Icon 426 of calendar module 148, labeled "Calendar", ○ Icon 428 of image management module 144, labeled "Photo" ○ Icon 430 of camera module 143, labeled "Camera" ○ Icon 432 of online video module 155, labeled "online video" ○ Icon 434 of stock price widget 149-2, labeled "Stock Price" ○ Icon 436 of map module 154, labeled "Map" ○ Icon 438 of weather widget 149-1, labeled "Weather" ○ Icon 440 of the alarm clock widget 149-4, labeled "Clock" ○ Icon 442 of training support module 142, labeled "Training Support" ○ Icon 444 of memo module 153, labeled "Memo," and, ○ An icon 446 for a settings application or module labeled "Settings" that provides access to settings for device 100 and its various applications 136.

[0153] Please note that the icon labels shown in Figure 4A are merely examples. For example, other labels may be used selectively for various application icons. In some embodiments, the label for each application icon includes the name of the application to which that application icon corresponds. In some embodiments, the label for a particular application icon is different from the name of the application to which that particular application icon corresponds.

[0154] Figure 4B shows an exemplary user interface on a device (e.g., device 300, Figure 3A) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355, Figure 3A) separate from the display 450. Many of the following embodiments are given by reference to input on the touchscreen display 112 (when the touch-sensitive surface and the display are combined), but in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as shown in Figure 4B. In some embodiments, the touch-sensitive surface (e.g., 451 in Figure 4B) has a principal axis (e.g., 452 in Figure 4B) corresponding to a principal axis (e.g., 453 in Figure 4B) on the display (e.g., 450). According to those embodiments, the device detects contact with the touch-sensitive surface 451 (e.g., 460 and 462 in Figure 4B) at positions corresponding to each of the positions on the display (e.g., 460 corresponds to 468 and 462 corresponds to 470 in Figure 4B). Thus, when the touch-sensitive surface is separated from the display, user input detected by the device on the touch-sensitive surface (e.g., 451 in Figure 4B) (e.g., touches 460 and 462, and their movement) is used by the device to operate the user interface on the display of the multifunction device (e.g., 450 in Figure 4B). It should be understood that a similar method may be optionally used for other user interfaces described herein.

[0155] In addition, while the following embodiments are given primarily with reference to finger input (e.g., finger touch, finger tap gesture, finger swipe gesture, etc.), it should be understood that in some embodiments, one or more of these finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may be optionally replaced by cursor movement along the swipe path (e.g., instead of touch movement) followed by a mouse click (e.g., instead of touch movement). As another example, a tap gesture may be optionally replaced by a mouse click while the cursor is positioned over the tap gesture location (e.g., instead of detecting touch and then ceasing touch detection). Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice may be optionally used simultaneously, or a mouse and finger touch may be optionally used simultaneously.

[0156] As used herein, the term “focus selector” refers to an input element that indicates the current portion of a user interface with which a user is interacting. In some implementations, including a cursor or other location marker, the cursor functions as a “focus selector” such that when input (e.g., press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in Figure 3A, or touch-sensitive surface 451 in Figure 4B) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, including a touchscreen display (e.g., touch-sensitive display system 112 in Figure 1A, or touchscreen in Figure 4A) that enables direct interaction with user interface elements on the touchscreen display, contact detected on the touchscreen functions as a “focus selector” such that when input (e.g., press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of ​​the user interface to another without the movement of a corresponding cursor or touch on the touchscreen display (for example, by moving focus from one button to another using the tab key or arrow keys). In these implementations, the focus selector moves in accordance with the movement of focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user-controlled user interface element (or touch on the touchscreen display) to communicate the user's intended interaction with the user interface (for example, by indicating to the device the user interface element that the user intends to interact with).For example, the position of a focus selector (e.g., cursor, touch, or selection box) over a corresponding button while a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen) indicates that the user intends to activate that corresponding button (rather than other user interface elements displayed on the device's display). In some embodiments, the focus indicator (e.g., cursor or selection indicator) is displayed via a display device to indicate the current portion of the user interface that will be affected by input received from one or more input devices.

[0157] In some embodiments, the device's response to an input detected by the device depends on a criterion based on the contact intensity during the input. For example, for some "light press" inputs, a contact intensity exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the device's response to an input detected by the device depends on a criterion that includes both the contact intensity during the input and a time-based criterion. For example, for some "deep press" inputs, a contact intensity exceeding a second intensity threshold during an input greater than a first intensity threshold for light presses triggers a second response only if a delay time has elapsed between satisfying the first and second intensity thresholds. This delay time is typically shorter than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, and the delay time increases as the second intensity threshold increases). This delay time helps avoid accidental recognition of deep press inputs. As another example, for some "deep press" inputs, there is a period of reduced sensitivity that occurs after the time the first intensity threshold is satisfied. During the period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps to avoid accidental deep press inputs. For other deep press inputs, the response to the detection of deep press inputs is independent of time-based criteria.

[0158] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors, such as user settings, contact motion, input timing, the application being run, the rate at which intensity is applied, the number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), and the position of the focus selector. Exemplary factors are described in U.S. Patent Publications 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0159] For example, Figure 4C shows a dynamic intensity threshold 480 that changes over time, partially based on the intensity of the touch input 476 over time. The dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predetermined delay time p1 from when the touch input 476 is first detected, and a second component 478 that tracks the intensity of the touch input 476 over time. The initial high intensity threshold of the first component 474 reduces accidental triggering of a "deep press" response while further allowing for an immediate "deep press" response when the touch input 476 provides sufficient intensity. The second component 478 reduces unintended triggering of a "deep press" response through gradual intensity fluctuations of the touch input. In some embodiments, a "deep press" response is triggered when the touch input 476 satisfies the dynamic intensity threshold 480 (for example, at point 481 in Figure 4C).

[0160] Figure 4D shows another dynamic intensity threshold 486 (e.g., intensity threshold ITD). Figure 4D also shows two other intensity thresholds, a first intensity threshold ITH and a second intensity threshold ITL. In Figure 4D, a touch input 484 satisfies the first intensity threshold ITH and the second intensity threshold ITL before time p2, but no response is provided until the delay time p2 has elapsed at time 482. Also in Figure 4D, the dynamic intensity threshold 486 decays over time, with decay starting at time 488 after a predefined delay time p1 has elapsed from time 482 (when the response associated with the second intensity threshold ITL is triggered). This type of dynamic intensity threshold reduces the accidental triggering of a response associated with the dynamic intensity threshold ITD immediately after or simultaneously with the triggering of a response associated with a lower intensity threshold, such as the first intensity threshold ITH or the second intensity threshold ITL.

[0161] Figure 4E shows yet another dynamic intensity threshold 492 (e.g., intensity threshold ITD). In Figure 4E, the response associated with the intensity threshold ITL is triggered after a delay time p2 has elapsed from the time the touch input 490 is first detected. Simultaneously, the dynamic intensity threshold 492 decays after a predetermined delay time p1 has elapsed from the time the touch input 490 is first detected. Therefore, without releasing the touch input 490, a decrease in the intensity of the touch input 490 after triggering the response associated with the intensity threshold ITL, followed by an increase in the intensity of the touch input 490, can trigger the response associated with the intensity threshold ITD (e.g., at time 494) even when the intensity of the touch input 490 falls below another intensity threshold, e.g., intensity threshold ITL. User interface and related processes

[0162] Here, we draw our attention to embodiments of a user interface ("UI") and associated processes that can be implemented on a computer system (e.g., portable multifunction device 100 or device 300) which includes (and / or communicates with) a display generation component (e.g., a display, projector, head-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of the content within the camera's field of view and optionally generates a video output that includes one or more streams of image frames capturing the content within the camera's field of view), one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a camera that tracks the position of one or more features of the user such as a touchscreen display, mouse, joystick, wand controller, and / or the user's hand, which also functions as a display generation component), optionally one or more attitude sensors, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators.

[0163] Figures 5A1 to 5A40 illustrate exemplary user interfaces for displaying an augmented reality environment, adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, and for transitioning between browsing a virtual model within the augmented reality environment and browsing a simulated view of the virtual model from the viewpoint of objects within the virtual model, according to several embodiments. The user interfaces in these figures are used to illustrate processes described later, including the processes in Figures 6A to 6D, 7A to 7C, and 8A to 8C. For convenience of explanation, some embodiments are discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, similar operations are optionally performed on a computer system (e.g., as shown in Figure 5A2) having a headset 5008 and a separate input device 5010 having a touch-sensitive surface, in response to detection of contact on the touch-sensitive surface of the input device 5010 while the user interface shown in the figures is displayed on the display of the headset 5008, along with a focus indicator.

[0164] Figures 5A1 to 5A27 show exemplary user interfaces for displaying an augmented reality environment and adjusting the appearance of the augmented reality environment and / or the appearance of objects within the augmented reality environment in response to different inputs, according to several embodiments.

[0165] Figures 5A1 and 5A2 show the situations in which the user interface described in Figures 5A3 to 5A40 is used.

[0166] Figure 5A1 shows the physical space in which user 5002, table 5004, and physical building model 5006 are located. User 5002 holds device 100 and views the physical building model 5006 through the display of device 100 (for example, on the touch-sensitive display system 112 of device 100, which may be called the “touchscreen display 112,” “touchscreen 112,” “display 112,” or “touch-sensitive display 112,” as shown in Figures 1A, 4A, and 5A4). One or more cameras of device 100 (sometimes called the “cameras” of device 100) continuously provide a live preview of content within the camera’s field of view, including one or more physical objects in the physical space (for example, the wallpaper 5007 of the room in the physical space, table 5004, etc.). Device 100 displays an augmented reality environment that includes a representation of at least a portion of the camera's field of view, including physical objects (e.g., a physical architectural model 5006) and one or more virtual objects (e.g., a virtual model of a building covering the physical architectural model 5006, a virtual tree, etc.), and user 5002 interacts with the augmented reality environment using the touchscreen display of device 100.

[0167] Figure 5A2 illustrates an alternative method by which user 5002 views a physical architectural model 5006 using a computer system that includes a headset 5008 and a separate input device 5010 having a touch-sensitive surface. In this embodiment, the headset 5008 displays an augmented reality environment, and user 5002 interacts with the augmented reality environment using the separate input device 5010. In some embodiments, device 100 is used as the separate input device 5010. In some embodiments, the separate input device 5010 is a touch-sensitive remote control, mouse, joystick, wand controller, etc. In some embodiments, the separate input device 5010 includes one or more cameras that track the position and movement of one or more features of user 5002, such as the user's hand.

[0168] Figures 5A3 and 5A4 show views of the augmented reality environment displayed on the touchscreen 112 of device 100. Figure 5A3 shows the position of device 100 relative to table 5004 and physical building model 5006 from the viewpoint of user 5002. Figure 5A4 shows a closer view of device 100 from Figure 5A3. Device 100 displays an augmented reality environment, including a live view of the physical space as captured by the camera of device 100 and virtual user interface objects (virtual building model 5012). Here, the virtual building model 5012 is a three-dimensional virtual model of the physical building model 5006 that appears to be mounted on or cover the physical building model 5006 within the camera's field of view (for example, replacing the physical building model 5006 in the augmented reality environment). The displayed augmented reality environment also includes virtual objects that do not correspond to physical objects within the camera's field of view (e.g., virtual trees, virtual bushes, virtual people, and virtual cars), and physical objects within the camera's field of view (e.g., wallpaper 5007). In some embodiments, device 100 displays one or more buttons (e.g., buttons 5014, 5016, and 5018, sometimes referred to as virtual buttons or displayed buttons) for interacting with the augmented reality environment (e.g., as described later with respect to Figures 5A25 to 5A27).

[0169] Figures 5A5 and 5A6 show different views of the augmented reality environment displayed on the touchscreen 112 of device 100 after user 5002 moves from the front of table 5004 (for example, as shown in Figure 5A3) to the side of table 5004 (for example, as shown in Figure 5A5). Figure 5A5 shows the position of device 100 relative to table 5004 and physical building model 5006 from user 5002's viewpoint. Figure 5A6 shows a closer view of device 100 from Figure 5A5. As shown in Figures 5A5 and 5A6, the virtual building model 5012 remains fixed to the physical building model 5006, and the view of the virtual building model 5012 changes as the position, shape, and / or orientation of the physical building model 5006 changes within the camera's field of view.

[0170] Figures 5A7 to 5A14 illustrate how the appearance of the virtual building model 5012 in the augmented reality environment is adjusted based on a combination of touch movement on the touchscreen 112 and movement of device 100. Reference box 5019 shows the position of device 100 relative to table 5004 and physical building model 5006 from the user 5002's viewpoint.

[0171] In Figure 5A7, device 100 displays the augmented reality environment when device 100 is in a first position relative to table 5004 and physical building model 5006 (for example, as shown in reference box 5019). In Figure 5A8, device 100 detects input on virtual building model 5012 (for example, by detecting touch input by contact 5020-a on the roof of virtual building model 5012). In Figures 5A9 to 5A11, while continuing to detect input (for example, while contact 5020 is maintained on touchscreen 112), device 100 detects movement of input relative to physical building model 5006 (for example, a drag gesture by contact 5020) and adjusts the appearance of virtual building model 5012 according to the magnitude of the movement of input relative to physical building model 5006 (for example, lifting the virtual roof 5012-a from the virtual building model). In Figure 5A9, when contact 5020-b moves a relatively small amount, the virtual roof 5012-a is lifted by a corresponding small amount. In Figure 5A10, when contact 5020-c moves a larger amount, the virtual roof 5012-a is lifted by a corresponding larger amount. In some embodiments, as shown in Figure 5A11, as the virtual roof 5012-a continues to lift, the floors of the virtual building model 5012 lift and extend (e.g., showing a virtual first floor 5012-d, a virtual second floor 5012-c, and a virtual third floor 5012-b). As contact 5020 moves upward, as shown in Figures 5A9 to 5A11, the device 100 updates the display of the virtual building model 5012 to maintain the display of the initial contact point on the virtual roof 5012-a at the position of contact 5020.

[0172] In Figures 5A12 and 5A13, while the contact 5020-d is maintained and stationary on the touchscreen 112, device 100 detects the movement of device 100 in physical space (for example, movement from a first position lower to the physical building model 5006, as shown in reference box 5019 in Figure 5A12, to a second position higher to the physical building model 5006, as shown in reference box 5019 in Figure 5A13). In response to the movement of the input (from the movement of device 100 in physical space), device 100 adjusts the appearance of the virtual building model 5012 by further raising the virtual roof 5012-a according to the magnitude of the movement. In some embodiments, as shown in Figure 5A13, when the appearance of the virtual model 5012 is adjusted according to the magnitude of the movement, the virtual roof 5012-a is displayed in a position beyond the maximum limit of the stationary state of the virtual roof 5012-a.

[0173] In Figure 5A14, device 100 stops detecting input (e.g., contact 5020 lifts off) and displays the virtual roof 5012-a at a position corresponding to the maximum limit of the stationary state. In some embodiments, device 100 displays an animated transition (e.g., from Figure 5A13 to Figure 5A14) from the virtual roof 5012-a at a position beyond the maximum limit of the stationary state (e.g., Figure 5A13) to the position corresponding to the maximum limit of the stationary state (e.g., Figure 5A14).

[0174] Figures 5A15 and 5A16 show the movement of device 100 in physical space (e.g., movement 5024) when no input is detected on the touchscreen 112 (e.g., no touch input is detected by contact on the touchscreen 112). Because no input is detected, the movement of device 100 changes the field of view of the device 100's camera from a first position lower to the physical building model 5006 (e.g., as shown in reference box 5019 in Figure 5A15) to a second position higher to the physical building model 5006 (e.g., as shown in reference box 5019 in Figure 5A16), without adjusting the appearance of the virtual building model 5012.

[0175] In contrast to Figures 5A15 and 5A16, Figures 5A17 and 5A18 show the movement of device 100 in physical space (e.g., movement 5028) when input is detected on the touchscreen 112 (e.g., touch input by contact 5026-a is detected on the touchscreen 112). While input continues to be detected (e.g., while contact 5026-a is maintained and stationary on the touchscreen 112), device 100 detects the movement of device 100 in physical space (from a first position lower to the physical building model 5006, as shown in reference box 5019 in Figure 5A17, to a second position higher to the physical building model 5006, as shown in reference box 5019 in Figure 5A18). In response to the movement of input (from the movement of device 100 in physical space), device 100 adjusts the appearance of the virtual building model 5012 by raising the virtual roof 5012-a according to the magnitude of the movement.

[0176] In Figures 5A19 and 5A20, while input is being continuously detected (for example, while contact 5026 is maintained on the touchscreen 112), device 100 detects movement of the input relative to the physical building model 5006 (e.g., a drag gesture by contact 5026) and adjusts the appearance of the virtual building model 5012 according to the magnitude of the movement of the input relative to the physical building model 5006 (e.g., further lifting the virtual roof 5012-a from the virtual building model 5012). In some embodiments, as the virtual roof 5012-a continues to lift, as shown in Figure 5A20, the floors of the virtual building model 5012 lift and extend (e.g., showing a first floor 5012-d, a second floor 5012-c, and a third floor 5012-b).

[0177] As shown in Figures 5A17 to 5A20, as the input moves upward (whether the input movement is due to the movement of device 100 while the contact (e.g., contact 5026-a) is maintained and stationary on the touchscreen 112, or whether the input movement is due to the movement of the contact across the touchscreen 112 while device 100 is held substantially stationary in physical space), device 100 updates the display of the virtual building model 5012 to maintain the display of the initial contact point on the virtual roof 5012-a at the position of contact 5026.

[0178] Figures 5A21 to 5A24 illustrate how the virtual environment settings (e.g., time) of the augmented reality environment are changed in response to inputs navigating through time within the augmented reality environment. In Figures 5A21 to 5A24, device 100 detects an input that changes the virtual environment settings (e.g., a left-to-right swipe gesture by touch 5030) and, accordingly, changes the time within the augmented reality environment (e.g., by adjusting the appearance of the virtual building model 5012 and applying a filter to the portion of the camera's field of view that is not obscured by the virtual building model 5012). In Figure 5A21, the time within the augmented reality environment is AM, as indicated by the shadows of the virtual building model 5012 and the shadows of virtual objects (e.g., a virtual tree, a virtual bush, a virtual person, and a virtual car) to the right of the object. As contact 5030 moves from left to right, the time in the augmented reality environment changes from AM to night (for example, according to the speed and / or distance of the input's movement) (for example, from AM in Figure 5A21 to noon in Figure 5A22, afternoon in Figure 5A23, and night in Figure 5A24). In some embodiments, in addition to adjusting the appearance of the virtual scene, device 100 applies a filter to the portion of the live view that is not obscured by the virtual scene (for example, to wallpaper 5007). For example, in Figure 5A24 (for example, when the virtual environment setting is changed to night mode), in addition to adjusting the appearance of the virtual scene in night mode (for example, indicated by a second shading pattern), a different filter is applied to wallpaper 5007 (for example, indicated by a first shading pattern).

[0179] Figures 5A25 to 5A27 illustrate how to change the virtual environment settings of an augmented reality environment in response to inputs (e.g., tap input on a displayed button) that switch between different virtual environments (e.g., default virtual environments such as landscape view, interior view, and day / night view) for a virtual user interface object (e.g., a virtual architectural model 5012), where different virtual environments are associated with different interactions for exploring the virtual user interface object. In Figure 5A25, the landscape button 5014 is selected, and the landscape view of the virtual architectural model 5012 is displayed (e.g., with virtual trees, virtual bushes, virtual people, and virtual cars). In Figures 5A26 and 5A27, device 100 detects input on an internal button 5016, such as a tap gesture by contact 5032, and accordingly displays an internal view of the virtual building model 5012 (for example, an unfolded view of the virtual building model 5012, which has a virtual first floor 5012-d, a virtual second floor 5012-c, a virtual third floor 5012-b, and a virtual roof 5012-a, but no virtual trees, virtual bushes, virtual people, and virtual cars). In some embodiments, when the virtual environment setting is changed (for example to an internal view), the surrounding physical environment is blurred (for example, using a filter). For example, although not shown in Figure 5A27, in some embodiments, when the virtual environment setting is changed to an internal view, the wallpaper 5007 is blurred.

[0180] Figures 5A28 to 5A40 show exemplary user interfaces for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the viewpoint of an object within the virtual model, according to several embodiments.

[0181] Figure 5A28, similar to Figure 5A4, shows a view of the augmented reality environment displayed on the touchscreen 112 of device 100, including a live view of the physical space as captured by the camera of device 100, a virtual building model 5012, a virtual vehicle 5050, and a virtual person 5060. In addition, the reference box 5019 in Figure 5A28 shows the position of device 100 relative to the table 5004 and the physical building model 5006 from the viewpoint of user 5002 (for example, as shown in Figures 5A1 and 5A2).

[0182] Figures 5A29 to 5A31 show the transition from Figure 5A28. Specifically, Figures 5A29 to 5A31 show the transition from a view of the augmented reality environment (for example, shown in Figure 5A28) to a simulated view of the virtual model from the viewpoint of the virtual vehicle 5050 within the virtual model.

[0183] Figure 5A29 shows the input 5052 detected at a location corresponding to the vehicle 5050 (for example, a tap gesture on the touchscreen 112 of device 100, or a selection using a separate input device along with a focus indicator).

[0184] Figures 5A30 and 5A31 show the transition from a view of the augmented reality environment to a simulated view of the virtual model from the viewpoint of the vehicle 5050, as displayed in response to the detection of input 5052. Specifically, Figure 5A30 shows the view displayed on device 100 during an animated transition from the view shown in Figure 5A29 to a simulated viewpoint view from the vehicle 5050 (for example, from the viewpoint of a person such as a driver or occupant inside the vehicle 5050), and Figure 5A31 shows the simulated viewpoint view from the vehicle 5050.

[0185] In some embodiments, the transition from a view of the augmented reality environment to a simulated viewpoint view includes an animated transition. Optionally, the transition includes an animation of flight from the position viewing the augmented reality environment to the position of the vehicle 5050 (e.g., the position of a person inside the vehicle 5050). For example, Figure 5A30 shows a view of the virtual model from a position between the user's position and the vehicle 5050's position in Figure 5A29 (e.g., midway through the animated transition), even if the user has not moved the device 100 (for example, the position of the device 100 relative to the physical building model 5006, as shown in reference box 5019 in Figure 5A30, is the same as in Figure 5A29).

[0186] In some embodiments, a portion of the field of view of device 100 (e.g., the camera of device 100) remains visible during the animated transition to a viewpoint view from vehicle 5050. For example, as shown in Figure 5A30, the edges of wallpaper 5007 and table 5004 are visible during the animated transition to the simulated viewpoint view (e.g., as if viewed from a position corresponding to the view shown in Figure 5A30, between the user's position in Figure 5A29 and the position of vehicle 5050). In some embodiments, the camera's field of view ceases to be visible during the animated transition to the viewpoint view of vehicle 5050 (e.g., the edges of wallpaper 5007 and table 5004 are not visible during the animated transition, and optionally, the corresponding portion of the virtual model is displayed instead).

[0187] In Figure 5A31, the simulated viewpoint view from the vehicle 5050 also shows controls 5054, including directional arrows (up, down, left, and right) for controlling the movement (e.g., direction of movement) of the vehicle 5050 (e.g., the virtual object on which the simulated viewpoint view is displayed). In the embodiment shown in Figure 5A31, the up arrow 5056 controls the forward movement of the vehicle 5050. Thus, in some embodiments, the user can control the movement of a corresponding virtual object (e.g., the vehicle 5050) while a simulated view from the viewpoint of that virtual object is displayed. In some embodiments, the user cannot control the movement of a corresponding virtual object (e.g., the virtual vehicle 5050 and / or the virtual person 5060) within the virtual model while the augmented reality environment view is displayed. For example, in some embodiments, the user cannot control the movement of the vehicle 5050 in the augmented reality environment view of Figure 5A28. In some embodiments, the vehicle 5050 moves autonomously within the virtual model while the augmented reality environment view (e.g., Figure 5A28) is displayed.

[0188] Figures 5A32 and 5A33 show the transition from Figure 5A31. Specifically, Figures 5A32 and 5A33 show the user-controlled movement of vehicle 5050 in the virtual model. Figure 5A32 shows input 5058 detected at the position corresponding to the up arrow 5056 (shown in Figure 5A31) of control 5054. In response to input 5058 on the up arrow 5056, vehicle 5050 moves forward in the virtual model. Thus, Figure 5A33 shows an updated simulated viewpoint view of the virtual model corresponding to the forward movement of vehicle 5050 in the virtual model. For example, in the updated simulated viewpoint view of Figure 5A33, the virtual building model 5012 is less visible, and person 5060 appears closer than in Figure 5A32.

[0189] Figures 5A34 and 5A35 show the transition from Figure 5A33. Specifically, Figures 5A34 and 5A35 show the transition from a simulated view of the virtual model from the viewpoint of the vehicle 5050 to a simulated view of the virtual model from the viewpoint of a virtual person 5060. Figure 5A34 shows the input 5062 detected at the location corresponding to person 5060. Figure 5A35 shows the simulated view of the virtual model from the viewpoint of person 5060, displayed in response to the detection of input 5062. In some embodiments, device 100 displays an animated transition between the simulated viewpoint view from vehicle 5050 and the simulated viewpoint view from person 5060 (for example, as if the user is moving from a location in vehicle 5050 (e.g., within vehicle 5050) to the location of person 5060).

[0190] Figures 5A36 and 5A37 show the transition from Figure 5A35. Specifically, Figures 5A36 and 5A37 show how the view of the virtual model from the perspective of person 5060 (e.g., a selected virtual object) changes in response to the movement of device 100 (e.g., in physical space).

[0191] Figure 5A36 shows arrow 5064 indicating the movement of device 100 to the left and the rotation of device 100 around the z-axis (for example, this causes the right edge of device 100 to move closer to the user and the left edge of device 100 to move further away from the user). Figure 5A37 shows an updated simulated viewpoint view of the virtual model from the viewpoint of person 5060, displayed in response to the detection of the movement of device 100. The updated simulated viewpoint view in Figure 5A37 corresponds to the view of the virtual model when person 5060 moves to the left and rotates their head slightly to the right relative to the person's position in Figure 5A36. The reference box 5019 in Figure 5A37 shows the new position of device 100 relative to the physical building model 5006 after device 100 has moved as indicated by arrow 5064.

[0192] In some embodiments, a control 5054 (for example, shown in Figure 5A31, but not in Figures 5A35 and 5A36) is displayed while a simulated view from the viewpoint of person 5060 is displayed, thereby allowing the user to control the movement of person 5060 in the virtual model using arrows on the control 5054 while the simulated view from the viewpoint of person 5060 is displayed (for example, in Figure 5A35).

[0193] Figures 5A38 to 5A40 show the transition from Figure 5A37. Specifically, Figures 5A38 to 5A40 show the transition from the simulated viewpoint view shown in Figure 5A37 back to the augmented reality environment view.

[0194] Figure 5A38 shows input 5066. In the embodiment shown in 5A38, input 5066 is a pinch gesture (e.g., from the minimum zoom level of the simulated viewpoint view of the virtual model). In some embodiments, input 5066 is a gesture (e.g., a tap) on an "empty" location in the virtual model (e.g., a location where the simulated viewpoint view is unavailable, such as a patch of grass). In some embodiments, input 5066 is a gesture (e.g., a tap) on an affordance (e.g., an icon such as an "X" to exit the simulated viewpoint view) to display or redisplay the augmented reality environment.

[0195] Figures 5A39 and 5A40 show the transition from a simulated view of a virtual model from the viewpoint of person 5060 to a view of the augmented reality environment, which is displayed in response to the detection of input 5066. Specifically, Figure 5A39 shows the view shown on device 100 during the animated transition from the view shown in Figure 5A38 to the view of the augmented reality environment, and Figure 5A40 shows the view of the augmented reality environment. In some embodiments, the transition from the simulated viewpoint view to the view of the augmented reality environment includes an animated transition which optionally includes an animation of flight from the position of the virtual object (where the simulated viewpoint view is shown) to the position for viewing the augmented reality environment.

[0196] Because device 100 is in a different position relative to the physical building model 5006 in Figures 5A38 to 5A40 than in Figures 5A28 to 5A30, the augmented reality view shown in Figure 5A40 corresponds to the new position of device 100 and is different from that shown in Figure 5A28. Similarly, Figure 5A39 shows a view of the virtual model from a position between the position of person 5060 in Figure 5A38 and the user's position in Figure 5A40 (e.g., midway through an animated transition), even if the user has not moved device 100 (for example, the position of device 100 relative to the physical building model 5006, as shown in reference box 5019, is the same in Figures 5A38 to 5A40).

[0197] Similar to the animated transition to the simulated viewpoint view described above with reference to Figures 5A29 to 5A31, in some embodiments, a portion of the field of view of device 100 (e.g., the camera of device 100) is visible during the animated transition from the simulated viewpoint view to the view of the augmented reality environment. For example, as shown in Figure 5A39, the wallpaper 5007 is displayed during the animated transition from the simulated viewpoint view (e.g., as if viewed from a position corresponding to the view shown in Figure 5A39, between the position of person 5060 and the user's position in Figure 5A40). In some embodiments, the camera's field of view is not displayed during the animated transition to the view of the augmented reality environment (e.g., the wallpaper 5007 is not displayed during the animated transition, and optionally, a corresponding portion of the virtual model is displayed instead).

[0198] Figures 5B1 to 5B41 show examples of systems and user interfaces for three-dimensional manipulation of virtual user interface objects according to several embodiments. The user interfaces in these figures are used to illustrate processes described later, including the processes in Figures 6A to 6D, 7A to 7C, and 8A to 8C. For convenience of explanation, some embodiments are discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, similar operations are optionally performed on a computer system (e.g., as shown in Figure 5B2) having a headset 5008 and a separate input device 5010 having a touch-sensitive surface, in response to detection of contact on the touch-sensitive surface of the input device 5010 while the user interface shown in the figures is displayed on the display of the headset 5008, along with a focus indicator.

[0199] Figures 5B1 to 5B4 show the situations in which the user interface described in Figures 5B5 to 5B41 is used.

[0200] Figure 5B1 shows the physical space 5200 where user 5202 and table 5204 are located. Device 100 is held in user 5206 by user 5202. Reference mat 5208 is located on table 5204.

[0201] Figure 5B2 shows a view of the virtual three-dimensional space displayed on the display 112 of device 100. The reference mat 5208 is within the field of view of one or more cameras of device 100 (e.g., light sensor 164) (hereinafter referred to as "cameras" and indicating one or more cameras of device 100). The display 112 shows a live view of the physical space 5200 as captured by the cameras, including a displayed version 5208b of the physical reference mat 5208a. A virtual user interface object (virtual box 5210) is displayed in the virtual three-dimensional space shown on the display 112. In some embodiments, the virtual box 5210 is fixed to the reference mat 5208b, so that the view of the virtual box 5210 changes as the displayed view 5208b of the reference mat changes in response to the movement of the reference mat 5208a in the physical space 5200 (for example, as shown in Figures 5B2 to 5B3). Similarly, the view of virtual box 5210 changes as the displayed version 5208b view changes in response to the movement of device 100 relative to reference mat 5208a.

[0202] In Figure 5B3, the reference mat 5208 has been rotated so that the longer side of the reference mat 5208a is adjacent to the device 100 (in Figure 5B2, the shorter side of the reference mat 5208a was adjacent to the device 100). The rotation of the displayed version 5208b of the reference mat from Figure 5B2 to Figure 5B3 results from the rotation of the reference mat 5208a in physical space 5200.

[0203] In Figures 5B3 and 5B4, device 100 is moving closer to the reference mat 5208a. As a result, the displayed version 5208b of the reference mat and the size of the virtual box 5210 are increasing.

[0204] Figures 5B5 to 5B41 show a larger view of device 100, providing a full view of the user interface displayed on display 112, and do not show the user's hands 5206.

[0205] Figure 5B5 shows a user interface for creating and adjusting virtual user interface objects, displayed on display 112. The user interface includes an avatar 5212, a toggle 5214 (for, for example, to toggle between virtual reality display mode and augmented reality display mode), a new object control 5216 (for, for example, to add a new object 5216 to the virtual three-dimensional space displayed by display 112), a color selection palette 5218 containing numerous controls corresponding to available colors (for, for example, to select the color of a virtual object), and a delete control 5220 (for, for, to remove a virtual user interface object from the virtual three-dimensional space). In Figure 5B5, the toggle 5214 indicates that the current display mode is augmented reality display mode (for example, display 112 is displaying a virtual box 5210 and a view of the physical space 5200 as captured by the camera of device 100). Figures 5B37 to 5B39 show the virtual reality display mode. Figures 5B37 to 5B39 show that the appearance of toggle 5214 has been changed to indicate that the virtual reality display mode is active (and that input on toggle 5214 will trigger a transition from virtual reality display mode to augmented reality display mode).

[0206] Figures 5B6 to 5B17 show the inputs that cause the movement of virtual box 5210.

[0207] In Figure 5B6, input via contact 5222 (e.g., contact with the touch-sensitive display 112) (e.g., selection and movement input) is detected on the first surface 5224 of the virtual box 5210. Once a surface of the virtual box 5210 is selected, the movement of the virtual box 5210 is restricted to movement in a plane parallel to the selected surface. In response to the detection of contact 5222 selecting the first surface 5224 of the virtual box 5210, a movement projection 5226 is shown extending from the virtual box 5210 to indicate the movement plane of the virtual box 5210 (e.g., the plane of movement parallel to the selected first surface 5224 of the virtual box 5210).

[0208] In Figures 5B6 and 5B7, contact 5222 moves along the surface of the touch-sensitive display 112 in the direction indicated by arrow 5228. In response to the movement of contact 5222, the virtual box 5210 moves in the plane indicated by the moving projection 5226 in the direction indicated by arrow 5228. In Figures 5B7 and 5B8, contact 5222 moves along the surface of the touch-sensitive display 112 in the direction indicated by arrow 5230. In response to the movement of contact 5222, the virtual box 5210 moves in the plane indicated by the moving projection 5226 in the direction indicated by arrow 5230. In Figure 5B9, contact 5222 has lifted off the touch-sensitive display 112, and the moving projection 5226 is no longer visible.

[0209] In Figure 5B10, input by contact 5232 (e.g., selection and movement input) is detected on the second surface 5234 of the virtual box 5210. In response to the detection of contact 5232 selecting the second surface 5234 of the virtual box 5210, a movement projection 5236 is shown extending from the virtual box 5210 to indicate the movement plane of the virtual box 5210 (e.g., the plane of movement parallel to the selected second surface 5234 of the virtual box 5210).

[0210] In Figures 5B10 and 5B11, the contact 5232 moves along the surface of the touch-sensitive display 112 in the direction indicated by the arrow 5238. In response to the movement of the contact 5232, the virtual box 5210 moves in the plane indicated by the movement projection 5236 in the direction indicated by the arrow 5238. When the virtual box 5210 moves upward so as to hover over the displayed reference mat 5208b, the shadow 5240 of the virtual box 5210 is displayed to indicate that the virtual box 5210 is hovering.

[0211] In Figures 5B11 and 5B12, contact 5232 moves along the surface of the touch-sensitive display 112 in the direction indicated by arrow 5242. As contact 5232 moves, the virtual box 5210 moves in the plane indicated by moving projection 5236 in the direction indicated by arrow 5242. In Figure 5B13, contact 5232 has lifted off the touch-sensitive display 112, and moving projection 5236 is no longer visible.

[0212] In Figure 5B14, input by contact 5233 (e.g., selection and movement input) is detected on the first surface 5224 of the virtual box 5210. In response to the detection of contact 5233 selecting the first surface 5224 of the virtual box 5210, a movement projection 5237 is shown extending from the virtual box 5210 to indicate the movement plane of the virtual box 5210 (e.g., the plane of movement parallel to the selected first surface 5224 of the virtual box 5210).

[0213] In Figures 5B14 and 5B15, contact 5233 moves along the surface of the touch-sensitive display 112 in the direction indicated by arrow 5239. In response to the movement of contact 5233, the virtual box 5210 moves in the plane indicated by the movement projection 5237 in the direction indicated by arrow 5238. The movement of contact 5232 shown in Figures 5B10 and 5B11 is in the same direction as the movement of contact 5233 shown in Figures 5B14 and 5B15. The movement plane of the virtual box 5210 in Figures 5B10 and 5B11 is different from the movement plane of the virtual box 5210 in Figures 5B14 and 5B15, where the movement of contact 5233 occurs while the first surface 5224 of the virtual box 5210 is selected, because the movement of contact 5232 occurs while the second surface 5234 of the virtual box 5210 is selected. Thus, selection and movement inputs having the same direction of movement cause different movements of the virtual box 5210 depending on the surface of the selected virtual box 5210.

[0214] In Figures 5B15 and 5B16, the contact 5233 moves along the surface of the touch-sensitive display 112 in the direction indicated by arrow 5243. In response to the movement of the contact 5233, the virtual box 5210 moves in the plane indicated by the moving projection 5237 in the direction indicated by arrow 5243. In Figure 5B17, the contact 5233 has lifted off the touch-sensitive display 112, and the moving projection 5237 is no longer visible.

[0215] Figures 5B18 to 5B21 show the inputs that cause the virtual box 5210 to resize.

[0216] In Figure 5B18, an input by contact 5244 (e.g., a resize input) is detected on the first surface 5224 of the virtual box 5210. In some embodiments, when the contact remains in a position corresponding to the surface of the virtual object for a period of time increasing above a resize time threshold, subsequent movement of the contact (and / or movement of device 100) causes the virtual object to resize. In Figure 5B19, the contact 5244 remains in contact with the first surface 5224 of the virtual box 5210 for a period of time increasing above a resize time threshold, and a resize projection 5246 is shown indicating the axis (perpendicular to the selected first surface 5224) on which the virtual box 5210 will be resized in response to the subsequent movement of the contact 5244.

[0217] In Figures 5B19 and 5B20, contact 5244 is moving along the path indicated by arrow 5248. As contact 5244 moves, the size of the virtual box 5210 increases along the axis indicated by resizing projection 5246 in the direction indicated by arrow 5248. In Figure 5B21, contact 5244 has lifted off from the touch-sensitive display 112, and projection 5246 is no longer visible.

[0218] Figures 5B22 to 5B27 show the placement of the object insertion cursor and the virtual box using the insertion cursor.

[0219] In Figure 5B22, input from contact 5250 (e.g., a tap input) is detected at a position on the physical reference mat 5208a corresponding to the displayed version 5208b. In response to the detection of contact 5250, an insertion cursor 5252 is displayed at the position on the display 112 corresponding to contact 5250. In Figure 5B23, contact 5250 has lifted off the touch-sensitive display 112, and the insertion cursor 5252 is indicated. In some embodiments, the insertion cursor 5252 stops being displayed after a predetermined period of time. In Figure 5B24, the insertion cursor 5252 has stopped being displayed, and input from contact 5254 (e.g., a tap input) is detected at a different position than where the insertion cursor 5252 was indicated (as shown in Figure 5B23). In response to the detection of contact 5254, a new insertion cursor 5256 is displayed at the position on the display 112 corresponding to contact 5254. In Figure 5B25, contact 5254 has lifted off from the touch-sensitive display 112, and the insertion cursor 5256 is shown.

[0220] In Figure 5B26, the insertion cursor 5256 has stopped being displayed, and input (e.g., tap input) from contact 5258 is detected at the position where the insertion cursor 5256 was located (as shown in Figure 5B25). In response to the detection of contact 5258 at the position where the insertion cursor was located, a new virtual user interface object (virtual box 5260) is displayed on the display 112 at the position corresponding to contact 5258. In Figure 5B27, contact 5258 has lifted off the touch-sensitive display 112.

[0221] Figures 5B28 to 5B31 show the resizing of the virtual box 5260 due to the movement of device 100.

[0222] In Figure 5B28, an input (e.g., a resize input) via contact 5262 with the touch-sensitive display 112 is detected on the surface 5264 of the virtual box 5260. In Figure 5B29, the contact 5262 remains in contact with the surface 5264 of the virtual box 5260 for an increased period exceeding the resize time threshold, and a resize projection 5266 is shown, indicating the axis (perpendicular to the selected surface 5264) along which the virtual box 5260 will be resized in response to subsequent movement of the device 100. In Figures 5B29 and 5B30, the device 100 moves along the path indicated by arrow 5268 while the contact 5262 remains in contact with the touch-sensitive display 112. In response to the movement of the device 100, the size of the virtual box 5260 increases along the axis indicated by the resize projection 5266, as shown in Figure 5B30. In Figure 5B31, the contact 5262 has lifted off from the touch-sensitive display 112, and the resizing projection 5266 is no longer displayed.

[0223] Figures 5B32 to 5B35 show the insertion of a new virtual object using the new object control 5216.

[0224] In Figure 5B32, input from contact 5270 (e.g., a tap input) is detected at a position on the displayed version 5208b of the physical reference mat 5208a. In response to the detection of contact 5270, the insertion cursor 5272 is displayed at the position on the display 112 corresponding to contact 5270. In Figure 5B33, contact 5270 has lifted off the touch-sensitive display 112, and the insertion cursor 5272 is indicated. In Figure 5B34, the insertion cursor 5272 has stopped being displayed, and input from contact 5274 with the touch-sensitive display 112 (e.g., a tap input) is detected at a position corresponding to the new object control 5216. In Figure 5B35, in response to input on the new object control 5216 (e.g., after the placement of the insertion cursor 5272), a new virtual user interface object (virtual box 5276) is displayed on the display 112 at a position corresponding to the position where the insertion cursor 5272 was indicated.

[0225] Figures 5B36 and 5B37 show the pinch-zoom input that triggers the transition from augmented reality display mode to virtual reality display mode. Figures 5B39 and 5B40 show the input at toggle 5214 for returning from virtual reality display mode to augmented reality display mode.

[0226] In Figure 5B36, contacts 5278 and 5280 with the touch-sensitive display 112 are detected simultaneously. As shown in Figures 5B36 and 5B37, a pinch gesture is detected in which contacts 5278 and 5280 move simultaneously along the paths indicated by arrows 5282 and 5284, respectively. In response to the detection of the pinch gesture, the display of virtual boxes 5210, 5260, and 5276 is zoomed (e.g., zoomed out, thereby making the displayed size of virtual boxes 5210, 5260, and 5276 smaller). In some embodiments, the gesture to zoom triggers a transition from augmented reality display mode to virtual reality display mode (e.g., because the zoomed view of the boxes no longer aligns with the field of view and position of the camera of device 100). In some embodiments, in virtual reality display mode, physical objects within the field of view of the camera of device 100 (e.g., reference mat 5208) stop being displayed, or one or more virtual (rendered) versions of the physical objects are displayed.

[0227] In some embodiments, in virtual reality display mode, virtual objects displayed by device 100 are locked to the coordinate system of device 100. In Figures 5B37 and 5B38, the position of device 100 is changing. Because device 100 is in virtual reality display mode, the positions of virtual boxes 5210, 5260, and 5276 have not changed in accordance with the changed position of device 100.

[0228] In Figure 5B39, input via contact 5286 (e.g., tap input) is detected at the position corresponding to the toggle 5214. The input via contact 5286 triggers a transition from virtual reality display mode to augmented reality display mode. Figure 5B40 shows the user interface displayed on display 112 after the transition to augmented reality display mode in response to the input via contact 5286. The transition includes redisplaying the camera field of view of device 100 (e.g., redisplaying the displayed view 5208b of the reference mat). In some embodiments, the transition includes zooming in on the display of virtual boxes 5210, 5260, and 5276 (e.g., zooming in) (e.g., to realign the boxes with the camera field of view of device 100).

[0229] In some embodiments, in augmented reality display mode, virtual objects displayed by device 100 are locked to physical space 5200 and / or physical objects within physical space 5200 (e.g., reference mat 5208). In Figures 5B40 and 5B41, the position of device 100 is changing. Because device 100 is in augmented reality display mode, virtual boxes 5210, 5260, and 5276 are locked to reference mat 5208a, and the positions of the virtual boxes on display 112 are changed according to the changed position of device 100.

[0230] Figures 5C1 to 5C30 show examples of systems and user interfaces for transitioning between viewing modes of a displayed simulated environment according to several embodiments. The user interfaces in these figures are used to illustrate processes described later, including the processes in Figures 10A to 10E. For convenience of explanation, some embodiments are discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, similar operations are optionally performed on a computer system (e.g., as shown in Figure 5A2) having a headset 5008 and a separate input device 5010 having a touch-sensitive surface, in response to detection of contact on the touch-sensitive surface of the input device 5010 while the user interface shown in the figures is displayed on the display of the headset 5008, along with a focus indicator.

[0231] Figures 5C1 and 5C2 show the situations in which the user interface described in Figures 5C3 to 5C30 is used.

[0232] Figure 5C1 shows the physical space 5200 in which the user and table 5204 are located. Device 100 is held by the user in the user's hand 5206. A reference mat 5208 is located on table 5204. A view of the simulated environment is displayed on the display 112 of device 100. The reference mat 5208 is within the field of view of one or more cameras of device 100 (e.g., light sensor 164) (hereinafter referred to as "cameras," and one or more cameras of device 100 are shown). The display 112 shows a live view of the physical space 5200 as captured by the cameras, including a displayed version 5208b of the physical reference mat 5208a. Two virtual user interface objects (a first virtual box 5302 and a second virtual box 5304) are displayed within the simulated environment shown on the display 112. In the first viewing mode (e.g., augmented reality viewing mode), virtual boxes 5302 and 5304 are fixed to a reference mat 5208b, so that as the displayed view 5208b of the reference mat changes in response to the movement of the reference mat 5208a in the physical space 5200, the views of virtual boxes 5302 and 5304 change (e.g., a fixed spatial relationship is maintained between virtual boxes 5302 and 5304 and the physical environment including the reference mat 5208a). Similarly, in the first viewing mode, the views of virtual boxes 5302 and 5304 change in response to the movement of device 100 relative to the reference mat 5208a.

[0233] In Figure 5C2, device 100 is moving closer to the reference mat 5208a. As a result, the displayed version 5208b of the reference mat, as well as the virtual boxes 5302 and 5304, are increasing in size.

[0234] Figures 5C3 to 5C30 show a larger view of device 100, providing a complete view of the user interface displayed on display 112, without showing the user's hands 5206. Features of the user interface are further described in relation to Figure 5B5.

[0235] Figures 5C4 to 5C6 show input gestures (including upward and downward swipes) for moving virtual box 5302 while the virtual box is displayed in augmented reality viewing mode. Since the input gestures described with respect to Figures 5C4 to 5C6 are not gestures that satisfy the mode change criterion (e.g., for changing the viewing mode from augmented reality viewing mode to virtual reality viewing mode), the views of virtual boxes 5302 and 5304 change in response to subsequent movement of device 100, as shown in Figures 5C7 and 5C8 (e.g., thereby maintaining a fixed spatial relationship between virtual boxes 5302 and 5304 and the physical environment including the reference mat 5208a).

[0236] Another example of a gesture that does not meet the mode change criteria is a resize gesture (for example, as described above with respect to Figures 5B18 to 5B21).

[0237] In Figure 5C4, input by contact 5306 (e.g., selection and movement input) is detected on the surface 5308 of the virtual box 5302. In response to the detection of contact 5306 selecting the surface 5308 of the virtual box 5302, a movement projection 5310 is shown extending from the virtual box 5302 to indicate the movement plane of the virtual box 5302 (e.g., the plane of movement parallel to the selected surface 5308 of the virtual box 5302).

[0238] In Figures 5C4 and 5C5, the contact 5306 moves along the surface of the touch-sensitive display 112 in the direction indicated by the arrow 5312. In response to the movement of the contact 5306, the virtual box 5302 moves in the plane indicated by the movement projection 5310 in the direction indicated by the arrow 5312. When the virtual box 5302 moves upward so as to hover over the displayed reference mat 5208b, the shadow 5314 of the virtual box 5302 is displayed to indicate that the virtual box 5210 is hovering.

[0239] In Figures 5C5 and 5C6, the contact 5306 moves along the surface of the touch-sensitive display 112 in the direction indicated by the arrow 5316. As the contact 5306 moves, the virtual box 5302 moves in the plane indicated by the moving projection 5310 in the direction indicated by the arrow 5316. In Figure 5C7, the contact 5306 has lifted off the touch-sensitive display 112, and the moving projection 5310 is no longer visible.

[0240] Figures 5C7 and 5C8 show the movement of device 100 along the path indicated by arrow 5318. As device 100 moves, the positions of the virtual boxes 5302 and 5304, which are displayed as device 100, change on the display 112 (for example, thereby maintaining a fixed spatial relationship between the virtual boxes 5302 and 5304 and the reference mat 5208a in the physical environment of device 100).

[0241] Figures 5C9 to 5C10 show input gestures (pinch gestures) that satisfy the mode change criteria (for example, triggering a change in viewing mode from augmented reality viewing mode to virtual reality viewing mode).

[0242] In Figure 5C9, touches 5320 and 5324 are detected on the touch-sensitive display 112. In Figures 5C9 to 5C11, touch 5320 moves along the path indicated by arrow 5322, and touch 5324 moves along the path indicated by arrow 5324. In response to the simultaneous movement of touches 5320 and 5324, which reduces the distance between them, the displayed view of the simulated environment, including virtual boxes 5302 and 5304, zooms out (for example, thereby increasing the size of virtual boxes 5302 and 5304 on the display 112). When zoom input is received, a transition from augmented reality viewing mode to virtual reality viewing mode is triggered. The transition animation performed during the transition includes a gradual fade-out of the displayed view of the physical environment. For example, the displayed view of table 5204 and the displayed view 5208b of reference mat 5208a, when captured by one or more cameras of device 100, gradually fade out (e.g., as shown in Figures 5C10 and 5C11). The transition animation includes a gradual fade-in of the virtual grid lines of virtual reference grid 5328 (e.g., as shown in Figures 5C11 and 5C12). During the transition, the appearance of toggle 5214 (e.g., for toggling between virtual reality display mode and augmented reality display mode) changes to indicate the current viewing mode (e.g., as shown in Figures 5C10 and 5C11). After the lift-off of contacts 5320 and 5324, the virtual boxes 5302 and 5304 in the simulated environment continue to move and decrease in size (e.g., changes in the simulated environment continue to have “momentum” causing movement after the end of the input gesture).

[0243] In Figures 5C12 and 5C13, device 100 is moved along the path indicated by arrow 5330. Because the pinch-zoom input gesture described in Figures 5C9 to 5C11 triggered a change from augmented reality browsing mode to virtual reality browsing mode, the positions of virtual boxes 5302 and 5304 do not change in response to the movement of device 100 (for example, in virtual reality browsing mode, a fixed spatial relationship between virtual boxes 5302 and 5304 and the physical environment is not maintained).

[0244] In Figures 5C13 and 5C14, device 100 is moved along the path indicated by arrow 5332.

[0245] Figures 5C15 to 5C18 show inputs for inserting a virtual box into the simulated environment displayed on device 100 while the simulated environment is displayed in virtual reality viewing mode.

[0246] In Figure 5C15, input via contact 5334 (e.g., tap input) is detected on the touch-sensitive display 112. In response to the detection of contact 5334, an insertion cursor 5336 is displayed at the position on the display 112 corresponding to contact 5334, as shown in Figure 5C16. In Figure 5C17, the insertion cursor 5336 has stopped being displayed, and input via contact 5338 (e.g., tap input) is detected at the position corresponding to the new object control 5216. In Figure 5C18, in response to input on the new object control 5216 (e.g., after the insertion cursor 5336 is placed), a new virtual user interface object (virtual box 5340) is displayed at the position corresponding to where the insertion cursor 5336 was previously located.

[0247] Figures 5C19 and 5C20 show inputs for manipulating virtual user interface objects within the simulated environment displayed on device 100 while the simulated environment is shown in virtual reality viewing mode.

[0248] In Figure 5C19, input from contact 5342 (e.g., selection and movement input) is detected on the surface 5344 of the virtual box 5340. In response to the detection of contact 5342 selecting the surface 5344 of the virtual box 5340, a movement projection 5348 is shown extending from the virtual box 5340 to indicate the plane of movement of the virtual box 5340 (e.g., the plane of movement parallel to the selected surface 5344 of the virtual box 5340). In Figures 5194 to 5C20, contact 5342 moves along the surface of the touch-sensitive display 112 in the direction indicated by arrow 5346. In response to the movement of contact 5342, the virtual box 5340 moves in the plane indicated by movement projection 5348 in the direction indicated by arrow 5346.

[0249] In Figure 5C21, the contact 5342 has lifted off from the touch-sensitive display 112, and the moving projection 5384 is no longer displayed.

[0250] Figures 5C22 and 5C23 show input gestures (e.g., rotation gestures) for changing the viewpoint of the simulated environment.

[0251] In Figure 5C22, a contact 5350 is detected by the touch-sensitive display 112. In Figures 5C22 and 5C23, the contact 5350 moves along the path indicated by the arrow 5352. As the contact 5350 moves, the simulated environment rotates. In Figure 5C23, the positions of the virtual reference grid 5328 and the virtual boxes 5302, 5304, and 5340 rotate in response to the input from the contact 5350.

[0252] In Figures 5C24 and 5C25, device 100 is moved along the path indicated by arrow 5354. Because the simulated environment displayed on display 112 in Figures 5C24 and 5C25 is displayed in virtual reality viewing mode, the positions of virtual boxes 5302 and 5304 on display 112 do not change as device 100 moves.

[0253] Figures 5C26 and 5C27 show the input gesture (de-pinch gesture) that triggers a change in viewing mode from virtual reality viewing mode to augmented reality viewing mode.

[0254] In Figure 5C26, touches 5356 and 5360 are detected on the touch-sensitive display 112. In Figures 5C26 and 5C27, touch 5356 moves along the path indicated by arrow 5358, and touch 5360 moves along the path indicated by arrow 5362. In response to the simultaneous movement of touches 5356 and 5360, which increases the distance between them, the displayed view of the simulated environment, including virtual boxes 5302, 5304, and 5340, zooms in (for example, thereby increasing the size of virtual boxes 5302, 5304, and 5340 on the display 112). When zoom input is received, a transition from virtual reality viewing mode to augmented reality viewing mode is triggered. The transition animation performed during the transition includes a gradual fade-out of the virtual reference grid 5328 (for example, as shown in Figures 5C26 and 5C27). The transition animation includes a gradual fade-in of the view of the physical environment. For example, the table 5204 and reference mat 5208a, when captured by one or more cameras of device 100, gradually become visible on display 112 (for example, as shown in Figures 5C28–5C30). During the transition, the appearance of the toggle 5214 changes to indicate the current viewing mode (for example, as shown in Figures 5C27 and 5C28). After the lift-off of contacts 5356 and 5360, the virtual boxes 5302, 5304, and 5340 in the simulated environment continue to increase in size, move, and rotate (for example, until the original space between virtual boxes 5302 and 5304 and reference mat 5208a is restored), as shown in Figures 5C28–5C30.

[0255] In some embodiments, the virtual box 5340 added while the virtual reality browsing mode was active is visible in alternate reality browsing mode, as shown in Figure 5C30.

[0256] In some embodiments, the change in viewing mode from virtual reality viewing mode to augmented reality viewing mode is performed in response to input by touch (e.g., tap input) at the location corresponding to the toggle 5214. For example, in response to a tap input detected at the location corresponding to the toggle 5214, a transition occurs from displaying virtual reality viewing mode (e.g., as shown in Figure 5C26) to augmented reality viewing mode (e.g., as shown in Figure 5C30). In some embodiments, a transition animation identical or similar to the animations shown in 5C26-5C30 is displayed during the transition.

[0257] Figures 5D1 to 5D14 show examples of systems and user interfaces for updating the viewing viewpoint indication of a second computer system in a simulated environment displayed by a first computer system, according to several embodiments. The user interfaces in these figures are used to illustrate processes described later, including the processes in Figures 11A to 11C. For convenience of explanation, some embodiments are discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, similar operations are optionally performed on a computer system (e.g., as shown in Figure 5A2) having a headset 5008 and a separate input device 5010 having a touch-sensitive surface, in response to detection of contact on the touch-sensitive surface of the input device 5010 while the user interface shown in the figures is displayed on the display of the headset 5008, along with a focus indicator.

[0258] Figures 5D1 and 5D2 show the situations in which the user interface described in Figures 5D3 to 5D14 is used.

[0259] Figure 5D1 shows the physical space 5400 in which two users 5402 and 5408 and a table 5414 are located. A first device 5406 (e.g., device 100) is held in the first user's hand 5404 by the first user 5402. A second device 5412 (e.g., device 100) is held in the second user's hand 5410 by the second user 5408. A reference mat 5416a is located on the table 5414.

[0260] Figure 5D2 shows a view of a virtual three-dimensional space displayed on the display 5148 (e.g., display 112) of device 5406. The reference mat 5416 is within the field of view of one or more cameras of device 5406 (e.g., light sensor 164) (hereinafter referred to as "cameras," and referring to one or more cameras of device 5406). Display 5148 shows a live view of the physical space 5400 as captured by the cameras, including a displayed version 5416b of the physical reference mat 5416a. A virtual user interface object (virtual box 5420) in the simulated environment is displayed on display 5418. In some embodiments, the virtual box 5420 is fixed to the reference mat 5416b, thereby changing the view of the virtual box 5420 as the view of the displayed version 5416b of the reference mat changes in response to the movement of device 100 relative to the reference mat 5416a. User interface features are further described with respect to Figure 5B5.

[0261] Figures 5D3–5D11 include sub-figures "a" (e.g., as shown in Figure 5D3a) showing the orientation of the first device 5406 and the second device 5412 in physical space 5400 relative to table 5414, sub-figure "b" (e.g., as shown in Figure 5D3b) showing the user interface of the first device 5412, and sub-figure "c" (e.g., as shown in Figure 5D3c) showing the user interface of the second device 5412. To provide a full view of the user interface, the user interfaces in Figures 5D3–5D11 do not show the hands holding the devices. Also, for clarity, the user interfaces in Figures 5D3–5D11 do not show the bodies of users 5402 and 5408. It should be understood that any part of user 5408's body within the field of view of the camera of device 5406 will typically be visible within the user interface displayed on device 5406 (however, the view of the user's body may be obstructed by virtual user interface objects or other user interface elements). For example, in Figure 5D2, the body and hands of user 5408 are visible within the user interface displayed by device 5409.

[0262] Figures 5D3 and 5D4 show the movement of the second device 5412.

[0263] In Figure 5D3a, the second device 5412 is displayed in a first position relative to the table 5414 (for example, adjacent to the left rear side of the table).

[0264] In Figure 5D3b, the user interface of device 5406 includes an avatar key 5422, which includes a key avatar 5424 corresponding to device 5406 and a key avatar 5426 corresponding to device 5412. The avatar key 5422 includes a name ("Me") corresponding to key avatar 5424 and a name ("Zoe") corresponding to key avatar 5426. The key avatars indicated by the avatar key provide a guide to the avatars shown in the visible environment (e.g., avatar 5428) to help the user of device 5406 understand that, for example, avatar 5428 in the simulated environment corresponds to the user "Zoe" on device 5412 (for example, because avatar 5428 in the simulated environment is a cat icon that matches key avatar 5426).

[0265] The simulated environment displayed on the user interface of device 5406 includes a virtual box 5420 and a displayed view 5416b of the physical reference mat 5416a, shown from the viewpoint of device 5406. The viewing viewpoint of device 5412 is indicated by the viewing viewpoint indicator 5432, which is shown emanating from avatar 5428. The simulated environment displays a representation 5430 of device 5412 (for example, a view of device 5412 as captured by the camera of device 5406 and / or a rendered version of device 5412).

[0266] In Figure 5D3c, the user interface of device 5412 includes an avatar key 5434, which includes a key avatar 5436 corresponding to device 5412 and a key avatar 5468 corresponding to device 5406. The avatar key 5434 includes a name ("Me") corresponding to key avatar 5436 and a name ("Gabe") corresponding to key avatar 5438. The key avatars indicated by the avatar key provide a guide to the avatar shown in the visible environment (e.g., avatar 5440) to help the user of device 5412 understand that avatar 5440 in the simulated environment corresponds to the user "Gabe"'s device 5406 (e.g., avatar 5440 in the simulated environment is a smiling icon that matches key avatar 5438).

[0267] In Figure 5D4a, the second device 5412 has moved from a first position relative to the table 5414 shown in Figure 5D3a to a second position relative to the table 5414 (for example, adjacent to the left front side of the table). In Figure 5D4b, the user interface of device 5406 shows device 5412 (indicated by the avatar 5428 and the device representation 5430) in the changed position from Figure 5D3b. The change in the viewing viewpoint of device 5412 is indicated by the different angles of the viewing viewpoint indicator 5432 from Figure 5D3b to Figure 5D4b. The movement of device 5412 is also indicated by the reference mat 5416b and virtual box 5420 showing the changed view from within the user interface of device 5412 in Figure 5D3c to Figure 5D4c.

[0268] Figures 5D5 to 5D7 show the selection and movement of the virtual box 5420 by device 5412.

[0269] In Figure 5D5c, input by contact 5446 (e.g., selection and movement input) is detected on the touchscreen display of the second device 5412 at a position corresponding to the surface of the virtual box 5420. In response to the detection of contact 5446 selecting the surface of the virtual box 5420, a movement projection 5448 is shown extending from the virtual box 5420, indicating the movement plane of the virtual box 5420 (e.g., the plane of movement parallel to the selected surface of the virtual box 5420).

[0270] In Figure 5D5b, an interaction indicator 5452 is shown to indicate to the user of the first device 5406 that the second device 5412 is interacting with the virtual box 5420. The interaction indicator 5452 extends from a position corresponding to the avatar 5428 to a position corresponding to the virtual box 5420. A control handle 5454 is shown at the position where the indication indicator 5452 intersects with the virtual box 5420.

[0271] In Figures 5D5c to 5D6c, contact 5446 moves along the touch-sensitive display of device 5412 in the direction indicated by arrow 5450. In response to the movement of contact 5446, the virtual box 5420 moves in the plane indicated by the movement projection 5448 in the direction indicated by arrow 5450.

[0272] In Figures 5D5b to 5D6b, when the virtual box 5420 is moved by a move input detected by the second device 5412, the user interface of the first device 5406 indicates the movement of the interaction indicator 5452 and the control handle 5454 (for example, to maintain the connection between the interaction indicator 5452 and the virtual box 5420).

[0273] In Figure 5D7c, contact 5446 has lifted off from the touch-sensitive display device of device 5412, and the moving projection 5448 is no longer displayed. In Figure 5D7b, the interaction indicator 5452 and control handle 5454 are no longer displayed (because device 5412 is no longer interacting with the virtual box 5420).

[0274] Figures 5D8 to 5D11 show the resizing of the virtual box 5420 by device 5412.

[0275] In Figure 5D8c, a contact input (e.g., a resize input) from 5456 is detected on the touchscreen display of the second device 5412 at a location corresponding to the surface of the virtual box 5420.

[0276] In Figure 5D8b, an interaction indicator 5462 and a control handle 5464 are shown on the user interface of the first device 5406, indicating that the second device 5412 is interacting with the virtual box 5420.

[0277] In Figure 5D9c, after contact 5456 remains in a position corresponding to the surface of the virtual box 5420 for a period of time that exceeds the resizing time threshold, a resizing projection 5458 is shown, indicating the axis (perpendicular to the selected surface of the virtual box 5420) along which the virtual box 5420 will be resized in response to the subsequent movement of contact 5456.

[0278] Figures 5D9a to 5D10a show the second device 5412 moving upward (while contact 5456 is in contact with the touchscreen display of the second device 5412) in order to resize the virtual box 5420. As the device 5412 moves, the size of the virtual box 5420 increases along the axis indicated by the resizing projection 5458 in the direction in which the second device 5412 moved.

[0279] In Figure 5D11c, the contact 5456 has lifted off from the touch-sensitive display 112, and the projection 5458 is no longer displayed.

[0280] As shown in Figures 5D12 to 5D14, users who are not in the same physical space can view and collaboratively interact with objects in a simulated environment. For example, a user in a first physical space can view a virtual user interface object (e.g., virtual box 5420) fixed to a displayed version of a first physical reference mat (e.g., 5416a), while a user in a different remote location can view the same virtual user interface object fixed to a displayed version of a second physical reference mat (e.g., 5476a).

[0281] Figure 5D12a shows a first physical space 5400 where two users 5402 and 5408 and table 5414 are located, as shown in Figure 5D1. Figure 5D12b shows a second physical space 5470, separate from the first physical space 5400, where a third user 5472 and table 5474 are located. A third device 5478 (e.g., device 100) is held by the third user 5472. A reference mat 5476a is located on table 5474. The third user 5472's device 5478 displays the same simulated environment as displayed by the first user 5408's device 5412 and the second user 5402's device 5404.

[0282] Figure 5D13a shows a first physical space 5400 as described in relation to Figure 5D12a, and Figure 5D13b shows a second physical space 5470 as described in relation to Figure 5D12b.

[0283] In Figure 5D13c, the user interface of the first device 5406 includes an avatar key 5422, which includes a key avatar 5480 (for "Stan") corresponding to the third device 5478. The avatar 5482 (as shown by the key avatar 5480) corresponding to the third device 5478 is shown in the simulated environment displayed by 5406, in position relative to the displayed version 5416b of the physical reference mat 5416a, which corresponds to the position of the device 5478 relative to the physical reference mat 5476a. The viewing viewpoint of the device 5478 is indicated by the viewing viewpoint indicator 5486. A representation of the device 5478 (e.g., a rendered version of the device) is shown in the simulated environment displayed by the device 5406.

[0284] As shown in Figure 5D13d, the user interface of the second device 5412 also displays an avatar 5482 corresponding to the third device 5478, a viewing viewpoint indicator 5486 showing the viewing viewpoint of device 5478, and a representation 5484 of device 5478.

[0285] Figure 5D14a shows a first physical space 5400 as described in relation to Figure 5D12a, and Figure 5D14b shows a second physical space 5470 as described in relation to Figure 5D12b.

[0286] Figure 5D14c shows the user interface of the third device 5478. In Figure 5D14c, the virtual box 5420 is shown fixed to the displayed view 5476b of the physical reference mat 5476a. The avatar 5488 corresponding to the first device 5406 is shown in a simulated environment displayed by the third device 5478, in a position relative to the displayed version 5476b of the physical reference mat 5476a, which corresponds to the position of the first device 5406 relative to the physical reference mat 5416a. The viewing viewpoint of the first device 5406 is indicated by the viewing viewpoint indicator 5490. The representation 5490 of the first device 5406 (e.g., a rendered version of the first device) is shown in a simulated environment displayed by the third device 5476. The avatar 5494 corresponding to the second device 5412 is shown in the simulated environment at a position relative to the displayed version 5476b of the physical reference mat 5476a, which corresponds to the position of the second device 5412 relative to the physical reference mat 5416a. The viewing viewpoint of the second device 5412 is indicated by the viewing viewpoint indicator 5498. The representation 5496 of the second device 5412 (e.g., a rendered version of the second device) is shown in the simulated environment displayed by the third device 5476.

[0287] Figures 5E1 to 5E32 show examples of systems and user interfaces for inserting a cursor according to several embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in Figures 12A to 12D. For convenience of explanation, some embodiments are discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, similar operations are optionally performed on a computer system (e.g., as shown in Figure 5A2) having a headset 5008 and a separate input device 5010 having a touch-sensitive surface, in response to detection of contact on the touch-sensitive surface of the input device 5010 while the user interface shown in the figures is displayed on the display of the headset 5008, along with a focus indicator.

[0288] Figures 5E1 to 5E3 illustrate the situations in which the user interface described in Figures 5E4 to 5E32 is used.

[0289] Figure 5E1 shows the physical space 5200 where user 5202 and table 5204 are located. Device 100 is held in user 5206 by user 5202. Reference mat 5208 is located on table 5204.

[0290] Figure 5E2 shows a view of the virtual three-dimensional space displayed on the display 112 of device 100. The reference mat 5208 is within the field of view of one or more cameras of device 100 (e.g., the light sensor 164) (hereinafter referred to as "cameras," and one or more cameras of device 100 are shown). The display 112 shows a live view of the physical space 5200 as captured by the cameras, including a displayed version 5208b of the physical reference mat 5208a.

[0291] In Figure 5E3, device 100 is moving closer to the reference mat 5208a. As a result, the size of the displayed version 5208b of the reference mat is increasing.

[0292] Figures 5E4 to 5E32 show a larger view of device 100, providing a full view of the user interface displayed on display 112, and do not show the user's hands 5206.

[0293] Figures 5E5 and 5E6 show the inputs that cause the insertion cursor to be placed at the first position.

[0294] In Figure 5E5, input by contact 5502 (e.g., tap input) is detected at a first position on the displayed version 5208b of the physical reference mat 5208a. In Figure 5E6, contact 5502 has lifted off from the touch-sensitive display 112, and the insertion cursor 5504 is indicated at the position where contact 5502 was detected.

[0295] Figures 5E7 and 5E8 show inputs that cause the insertion cursor to be placed at a second position. In Figure 5E7, input by contact 5506 (e.g., a tap input) is detected at a different position from where the insertion cursor 5504 is displayed. In Figure 5E8, contact 5506 has lifted off the touch-sensitive display 112, and the insertion cursor 5508 is indicated at the position where contact 5506 was detected.

[0296] Figures 5E9 and 5E10 illustrate inputs that trigger the insertion of virtual user interface objects. In Figure 5E9, an input by contact 5510 (e.g., a tap input) is detected at a position corresponding to the position of the insertion cursor 5508. In response to the detection of contact 5510 at the position where the insertion cursor was located, the virtual user interface object (first virtual box 5512) is displayed on the display 112 at the position corresponding to the contact 5510, and the insertion cursor 5508 is moved from its previous position on the displayed view 5208b of the reference mat 5208a to the surface 5514 of the first virtual box 5512. In some embodiments, a shadow 5522 is displayed (e.g., simulated light is caused by the first virtual box 5512 casting a shadow).

[0297] Figures 5E11 and 5E12 show inputs detected on the surface of a virtual user interface object (first virtual box 5512) that trigger the insertion of an additional virtual user interface object. In Figure 5E11, an input by contact 5516 (e.g., a tap input) is detected at a position on the surface 5514 of the first virtual box 5512 while the insertion cursor 5516 is positioned on the surface 5514. In response to the detection of the input by contact 5516, a new virtual user interface object (second virtual box 5518) is displayed on the display 112 at the position corresponding to the contact 5510, and the insertion cursor 5508 is moved from the surface 5514 of the first virtual box 5512 to the surface 5520 of the second virtual box 5518. The length of the shadow 5522 is increased (so that the shadow appears to be cast by the first virtual box 5512 and the newly added second virtual box 5518).

[0298] Figures 5E12 and 5E13 illustrate the rotation of the physical reference mat 5208. For example, user 5202 manually changes the position and / or orientation of the reference mat 5208. When the physical reference mat 5208a rotates, the virtual boxes 5512 and 5518 and the shadow 5522 also rotate (because the virtual boxes 5512 and 5518 are fixed to the displayed view 5208b of the physical reference mat 5208a).

[0299] Figures 5E14 to 5E16 illustrate the movement of device 100. For example, user 5202, holding device 100, changes the device's position and / or orientation. In Figures 5E14 and 5E15, as device 100 moves, virtual boxes 5512 and 5518 and shadow 5522 also move (because virtual boxes 5512 and 5518 are fixed to the displayed view 5208b of the physical reference mat 5208a). Similarly, in Figures 5E15 and 5E16, as device 100 moves, virtual boxes 5512 and 5518 and shadow 5522 also move.

[0300] Figures 5E17 and 5E18 show inputs that change the position of the insertion cursor 5526 on the virtual box 5518. In Figure 5E17, while the insertion cursor 5526 is positioned on the surface 5520 of the virtual box 55182, an input by contact 5524 (e.g., a tap input) is detected on the surface 5528 of the virtual box 5518. In Figure 5E18, contact 5524 lifts off from the touch-sensitive display 112, and the insertion cursor 5508 is moved from the surface 5520 of the virtual box 5518 to the surface 5528 of the virtual box 5518.

[0301] Figures 5E19 and 5E20 show inputs detected on the surface of the second virtual box 5518 that trigger the insertion of the third virtual box 5532. In Figure 5E19, an input by contact 5530 (e.g., a tap input) is detected at a position on the surface 5528 of the second virtual box 5518 while the insertion cursor 5526 is positioned on the surface 5268. In response to the detection of the input by contact 5530, the third virtual box 5532 is displayed on the display 112 at the position corresponding to the contact 5530, and the insertion cursor 5526 is moved from the surface 5528 of the second virtual box 5518 to the surface 5526 of the third virtual box 5532. The shape of the shadow 5522 is changed (so that the shadow appears to be cast by the first virtual box 5512, the second virtual box 5518, and the newly added third virtual box 5532).

[0302] Figures 5E21 and 5E22 show inputs that change the position of the insertion cursor 5538 on the virtual box 5532. In Figure 5E21, while the insertion cursor 5526 is positioned on the surface 5534 of the virtual box 5532, an input by contact 5536 (e.g., a tap input) is detected on the surface 5538 of the virtual box 5532. In Figure 5E22, the contact 5536 lifts off from the touch-sensitive display 112, and the insertion cursor 5526 moves from the surface 5534 of the virtual box 5518 to the surface 5538 of the virtual box 5532.

[0303] Figures 5E23 and 5E24 show the insertion of a new virtual user interface object using the new object control 5216.

[0304] In Figure 5E23, while the insertion cursor 5526 is on the surface 5538 of the virtual box 5532, input by contact 5542 (e.g., tap input) is detected at the position on the display 112 corresponding to the new object control 5216. In Figure 5E24, in response to the input at the position corresponding to the new object control 5216, the fourth virtual box 5546 is displayed on the display 112 at the position where the insertion cursor 5526 was indicated, and the insertion cursor 5526 is moved from the surface 5538 of the virtual box 5532 to the surface 5548 of the fourth virtual box 5546.

[0305] Figures 5E25 to 5E27 show the inputs that cause the movement of the fourth virtual box 5546.

[0306] In Figure 5E25, input by contact 5550 (e.g., selection and movement input) is detected on the surface 5556 of the fourth virtual box 5546. In response to the detection of contact 5550 selecting the surface 5556 of the fourth virtual box 5546, a movement projection 5552 is shown extending from the virtual box 5546 to indicate the movement plane of the fourth virtual box 5546 (e.g., the plane of movement parallel to the selected surface 5556 of the virtual box 5546).

[0307] In Figures 5E25 and 5E26, the contact 5550 moves along the surface of the touch-sensitive display 112 in the direction indicated by the arrow 5554. In response to the movement of the contact 5550, the fourth virtual box 5546 moves in the plane indicated by the moving projection 5552 in the direction indicated by the arrow 5554. In Figure 5E27, the contact 5550 has lifted off the touch-sensitive display 112, and the moving projection 5552 is no longer visible.

[0308] Figures 5E28 to 5E32 show the inputs that cause the resizing of the fourth virtual box 5546.

[0309] In Figure 5E28, a contact input (e.g., a size change input) is detected on the touchscreen display 112 at a position corresponding to the surface 5556 of the fourth virtual box 5546.

[0310] In Figure 5E29, after contact 5255 remains in a position corresponding to the surface 5556 of the fourth virtual box 5546 for a period of time that increases above the resizing time threshold, a resizing projection 5560 is shown, indicating the axis (perpendicular to the selected surface of the virtual box 5546) along which the virtual box 5546 will be resized in response to the subsequent movement of contact 5558.

[0311] In Figures 5E30 and 5E31, contact 5558 moves across the touchscreen display 112 along the path indicated by arrow 5562. As contact 5558 moves, the size of the virtual box 5548 increases along the axis indicated by the resizing projection 5560 in the direction of contact 5558's movement.

[0312] In Figure 5E32, the contact 5558 has lifted off from the touch-sensitive display 112, and the projection 5560 is no longer displayed.

[0313] Figures 5F1 to 5F17 show examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode according to several embodiments. The user interfaces in these figures are used to illustrate processes described later, including the processes in Figures 13A to 13E. For convenience of explanation, some embodiments are discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, similar operations are optionally performed on a computer system (e.g., as shown in Figure 5A2) having a headset 5008 and a separate input device 5010 having a touch-sensitive surface, in response to detection of contact on the touch-sensitive surface of the input device 5010 while the user interface shown in the figures is displayed on the display of the headset 5008, along with a focus indicator.

[0314] Figures 5F1 and 5F2 show the situations in which the user interface described in Figures 5F3 to 5F17 is used.

[0315] Figure 5F1 shows the physical space 5200 where user 5202 and table 5204 are located. Device 100 is held in the user's hand 5206 by user 5202. The object (physical box 5602) is located on table 5204.

[0316] Figure 5F2 shows the augmented reality environment as displayed by the display 112 of device 100. Table 5204 (referred to as 5204a when referring to a table in physical space) and physical box 5602 are within the field of view of one or more cameras of device 100 (e.g., light sensor 164) (hereinafter referred to as "cameras," and one or more cameras of device 100 are shown). Display 112 shows a live view of physical space 5200 as captured by the camera, including a displayed version 5204b of table 5204a and a rendered virtual box 5604 displayed in a simulated environment location corresponding to physical box 5602 as detected by the camera of device 100.

[0317] Figures 5F3 to 5F17 include sub-figures "a" (e.g., as shown in Figure 5F3a) showing the orientation of device 100 within the physical space 5200 relative to table 5204a and physical box 5602, and sub-figures "b" (e.g., as shown in Figure 5F3b) showing the user interface of device 100. Also, for clarity, Figures 5F3 to 5F18 show a larger view of device 100, providing a full view of the user interface displayed on display 112 and not showing the user's hands 5206.

[0318] Figures 5F3a to 5F4a show the movement of device 100 relative to table 5204a and physical box 5602 while the augmented reality environment is displayed by device 100 in an unstable operating mode (as shown in Figures 5F3b to 5F4b). As shown in Figure 5F3a, when device 100 is in a first position relative to table 5204a, the rendered version 5604 of physical object 5602 is fully visible within the user interface shown in Figure 5F3b. In Figure 5F4a, device 100 has been moved to a second position relative to table 5204a, and the rendered version 5604 of physical object 5602 is only partially visible within the user interface shown in Figure 5F4b. In the unstable operation mode, as device 100 moves, the view of virtual box 5604 changes to maintain a fixed spatial relationship between virtual box 5604 and physical box 5602, and the displayed representation of the camera field of view of device 100 (including, for example, the displayed table 5204b) is updated based on the device's movement.

[0319] Figures 5F5 to 5F8 show inputs that cause the device to display an augmented reality environment in stable operating mode (e.g., pinch-to-zoom input).

[0320] In Figure 5F5, device 100 is in a first position relative to table 5204. In Figure 5F6, touches 5606 and 5608 are detected on the touch-sensitive display 112 (as shown in Figure 5F6b). As shown in Figures 5F6b to 5F7b, touch 5606 moves along the path indicated by arrow 5610, and touch 5608 moves along the path indicated by arrow 5612. In response to the simultaneous movement of touches 5606 and 5608, which increases the distance between them, the displayed augmented reality environment, including the virtual box 5604, is zoomed in (for example, thereby increasing the size of the virtual box 5604 on the display 112). The virtual box 5604 is re-rendered in response to the zoom input (for example, the larger virtual box 5604 in Figure 5F8b has the same resolution as the smaller virtual box 5604 in Figure 5F5b). In some embodiments, the camera field of view of the device 100 displayed on the display 112 (e.g., the displayed view 5204b of table 5204a) does not change in response to zoom input (as shown in Figures 5F5b to 5F8b). When zoom input is received, a transition occurs from an unstable operating mode to a stable operating mode. In Figure 5F8, contacts 5606 and 5608 are lifted off the touchscreen display 112.

[0321] In some embodiments, while the device is displaying an augmented reality environment in stable operation mode, if the device moves, causing a virtual user interface object to extend beyond the field of view of the device camera, a portion of the virtual user interface object will cease to be displayed. Figures 5F8 and 5F9 show the movement of device 100 while device 100 is in stable operation mode, causing a portion of the virtual user interface object 5304 to cease to be displayed. Figures 5F8a to 5F9a show the movement of device 100 relative to table 5204a and physical box 5602, which occurs while the augmented reality environment is being displayed by device 100 in stable operation mode (as shown in Figures 5F8b to 5F9b). As shown in Figure 5F8a, when device 100 is in a first position relative to table 5204a, a zoomed-in, rendered version 5604 of physical object 5602 is fully visible within the user interface shown in Figure 5F8b. In Figure 5F9a, device 100 has moved to a second position relative to table 5204a, thereby updating the view of virtual box 5604 and allowing virtual box 5604 to extend beyond the field of view of device 100's camera while maintaining a fixed spatial relationship between virtual box 5604 and physical box 5602. As a result, the portion of virtual box 5064 that extends beyond the field of view of device 100's camera is not displayed.

[0322] In some embodiments, when the device is displaying an augmented reality environment in stable operation mode, and the device's movement causes the virtual user interface objects to extend beyond the device camera's field of view, the augmented reality environment is zoomed out so that the virtual user interface objects are fully displayed. For example, in Figures 5F9b to 5F10b, the displayed augmented reality environment, including the virtual box 5604, is zoomed out so that the virtual box 5604 is fully displayed.

[0323] In some embodiments, in stable operation mode, the view of the virtual box 5604 is updated to maintain a fixed spatial relationship between the virtual box 5604 and the physical box 5602, thereby causing the virtual box 5604 to extend beyond the field of view of the device 100's camera. The virtual box 5604 is then displayed with placeholder images of the positions corresponding to the portion of the virtual box 5064 that extends beyond the device camera's field of view. In Figure 5F10b, the rendered version 5604 of the physical object 5602 is displayed with placeholder image 5614 (margin) of the positions corresponding to the portion of the virtual box 5064 that extends beyond the device camera's field of view. For example, placeholder image 5614 is displayed at a position in the augmented reality environment that extends beyond the camera's field of view, and therefore camera data that would be displayed within the space occupied by placeholder image 5614 is unavailable.

[0324] Figures 5F10 and 5F11 show the movement of device 100 (returning to the position of device 100 shown in Figures 5F8 and 5F3).

[0325] Figures 5F11a to 5F12a show the movement of device 100 (for example, moving away from table 5204a and physical object 5602, thereby making device 100 appear larger). In Figure 5F12b, as a result of the movement shown in Figures 5F11a to 5F12a, the size of virtual object 5604 is reduced from the size of virtual object 5604 in Figure 5F11b. For illustrative purposes, this movement is shown in Figures 5F11a to 5F12a (therefore, the size of virtual object 5604 in Figure 5F12b in stable mode is the same as the size of virtual object 5604 in Figure 5F3b in unstable mode) to provide a concise comparison of the augmented reality environment updates in stable and unstable operating modes.

[0326] Figures 5F12a to 5F13a show the movement of device 100 relative to table 5204a and physical box 5602 while the augmented reality environment is displayed by device 100 in stable operation mode. As shown in Figure 5F12a, when device 100 is in a first position relative to table 5204a, the rendered version 5604 of physical object 5602 is fully visible within the user interface shown in Figure 5F12b. In Figure 5F13a, device 100 has been moved to a second position relative to table 5204a, and the rendered version 5604 of physical object 5602 is only partially visible within the user interface shown in Figure 5F13b. In some embodiments, in stable operation mode, as device 100 moves, the view of virtual box 5604 changes to maintain a fixed spatial relationship between virtual box 5604 and physical box 5602, and the displayed representation of the camera field of view of device 100 (including, for example, the displayed table 5204b) changes by an amount less than the amount of change that occurs in unstable mode (for example, the amount of movement of the displayed table 5204b from Figure 5F12b to Figure 5F13b while device 100 is in stable operation mode is less than the amount of movement of the displayed table 5204b from Figure 5F4b to Figure 5F5b while device 100 is in unstable operation mode).

[0327] Figures 5F14 to 5F16 show the inputs to the stabilization toggle 5616 for transitioning from an unstable operating mode to a stable operating mode. In Figure 5F15b, input via contact 5618 (e.g., tap input) is detected at a position on the touchscreen display 112 corresponding to the stabilization toggle 5616. In response to the input via contact 5618, the appearance of the stabilization toggle 5616 changes (e.g., the toggle changes from an unshaded state to a shaded state), as shown in Figure 5F16b, indicating that a transition from an unstable operating mode to a stable operating mode has occurred.

[0328] Figures 5F16a to 5F17a show the movement of device 100 relative to table 5204a and physical box 5602 while the augmented reality environment is displayed by device 100 in stable operation mode (as shown in Figures 5F16a to 5F17a). As shown in Figure 5F16a, when device 100 is in a first position relative to table 5204a, the rendered version 5604 of physical object 5602 is fully visible within the user interface shown in Figure 5F16b. In Figure 5F17a, device 100 has been moved to a second position relative to table 5204a, and the rendered version 5604 of physical object 5602 is only partially visible within the user interface shown in Figure 5F17b. In stable operation mode, as device 100 moves, the view of virtual box 5604 changes to maintain a fixed spatial relationship between virtual box 5604 and physical box 5602, and the displayed representation of the camera field of view of device 100 (including, for example, the displayed table 5204b) changes by less than the amount of change that occurs in unstable mode (for example, the amount of movement of the displayed table 5204b from Figure 5F15b to Figure 5F16b while device 100 is in stable operation mode is less than the amount of movement of the displayed table 5204b from Figure 5F4b to Figure 5F5b while device 100 is in unstable operation mode).

[0329] Figures 6A to 6D are flowcharts illustrating a method 600 for adjusting the appearance of virtual user interface objects in an augmented reality environment according to several embodiments. The method 600 is performed in a computer system (e.g., portable multifunction device 100, Figure 1A; device 300, Figure 3A; or a multi-component computer system including a headset 5008 and input device 5010, Figure 5A2) having a display generation component (e.g., a display, projector, head-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of the content within the camera's field of view and optionally generates a video output including one or more streams of image frames capturing the content within the camera's field of view), and an input device (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touchscreen display that also functions as a display generation component, a mouse, joystick, wand controller, and / or a camera that tracks the position of one or more features of the user such as the user's hand). In some embodiments, the input device (e.g., having a touch-sensitive surface) and the display generation component are integrated into a touch-sensitive display. As described above with respect to Figures 3B and 3D, in some embodiments, the method 600 is performed on a computer system 301 (e.g., computer systems 301-a, 301-b, or 301-c), where each component, such as a display generation component, one or more cameras, one or more input devices, and optionally one or more attitude sensors, is either included in the computer system 301 or communicates with the computer system 301.

[0330] In some embodiments, the display generation component is a touchscreen display, and the input device (e.g., having a touch-sensitive surface) is on or integrated with the display generation component. In some embodiments, the display generation component is separate from the input device (e.g., as shown in Figures 4B and 5A2). Some operations of Method 600 are optionally combined and / or the order of some operations is optionally changed.

[0331] For the sake of explanation, some embodiments will be described with reference to operations performed on a computer system having a touch-sensitive display system 112 and one or more integrated cameras (e.g., on a device 100 having a touchscreen 112). However, similar operations are optionally performed on a computer system having a headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in Figure 5A2) in response to detection of contact on the touch-sensitive surface of the input device 5010 while the user interface shown in the figure is displayed on the display of the headset 5008. Similarly, similar operations are optionally performed on a computer system having one or more cameras implemented separately (e.g., in the headset) from one or more other components of the computer system (e.g., input devices), and in some such embodiments, “movement of the computer system” corresponds to movement of one or more cameras of the computer system, or movement of one or more cameras communicating with the computer system.

[0332] As described later, Method 600 relates to adjusting the appearance of virtual user interface objects (on the computer system's display) in an augmented reality environment (where reality is augmented with supplemental information that provides additional information to the user that is not available in the physical world) based on a combination of movement of the computer system (e.g., movement of one or more cameras of the computer system) and movement of touch on an input device of the computer system (e.g., touchscreen display). In some embodiments, adjusting the appearance of virtual user interface objects enables the user to access supplemental information in the augmented reality environment. By adjusting the appearance of virtual user interface objects based on a combination of computer system movement and contact movement on the computer system's input devices, an intuitive method for users to adjust the appearance of virtual user interface objects is provided (e.g., by allowing users to adjust the appearance of virtual user interface objects based on computer system movement only, contact movement on input devices only, or a combination of computer system movement and contact movement), enabling users to expand the range of adjustments available to them (e.g., by allowing users to continue adjusting the appearance of virtual user interface objects even if contact or one or more cameras of the computer system cannot move further in the desired direction), thereby improving device usability and making the user-device interface more efficient (e.g., by reducing the number of steps required to achieve the intended result when operating the device and reducing user errors when operating / interacting with the device), and in addition, by enabling users to use the device more quickly and efficiently, power consumption is reduced and the device's battery life is improved.

[0333] A computer system (e.g., device 100, Figure 5A7) displays an augmented reality environment (602) via a display generation component (e.g., touchscreen 112, Figure 5A7) (as shown in Figure 5A7). Displaying an augmented reality environment is a representation of at least a portion of the field of view of one or more cameras, including corresponding physical objects (e.g., a three-dimensional model of a building, a sheet of paper with a printed pattern, a poster on a wall or other physical object, an image placed on a surface, etc.) (e.g., a physical architectural model 5006, Figure 5A1), the representation being updated as the content of the field of view of one or more cameras changes (e.g., the representation is a live preview of at least a portion of the field of view of one or more cameras, and the corresponding physical objects are included in and visible within the field of view of the cameras), and corresponding virtual objects located at corresponding positions within the representation of the field of view of one or more cameras. The system includes displaying a user interface object (e.g., a virtual roof of a three-dimensional model of a building, a virtual car parked on a surface represented by a sheet of paper with a printed pattern, an interactive logo superimposed on a poster, a virtual three-dimensional mask covering the outline of an image) (e.g., a virtual architectural model 5012, Figure 5A7), where the corresponding virtual user interface object (e.g., a virtual architectural model 5012, Figure 5A7) has a position determined based on the corresponding physical object (e.g., a physical architectural model 5006) in the field of view of one or more cameras, and the corresponding virtual user interface object (604). For example, in some embodiments, the corresponding virtual user interface object is a graphical object or a two-dimensional or three-dimensional virtual object that appears to be attached to or covering the corresponding physical object in the field of view of one or more cameras (e.g., the virtual architectural model 5012 is a three-dimensional virtual object that appears to cover the physical architectural model 5006, Figure 5A7). The position and / or orientation of the corresponding virtual user interface object is determined based on the position, shape, and / or orientation of the physical object within the field of view of one or more cameras (for example, as shown in Figures 5A3 and 5A5).While displaying the augmented reality environment (606), the computer system detects input at a location corresponding to the corresponding virtual user interface object (e.g., a location on a touchscreen display or touch-sensitive remote control, or movement of a wand or user's hand while the cursor is at the location of the corresponding virtual user interface object) (e.g., by detecting touch input by contact on a touchscreen display or touch-sensitive remote control, etc.) (e.g., device 100 detects contact 5020-a on the virtual roof of the virtual building model 5012, Figure 5A8).

[0334] While the input continues to be detected (608) (for example, while the contact is maintained on the input device, such as while the contact is maintained on the touchscreen display or touch-sensitive remote control) (for example, while the contact 5020 is maintained on the touchscreen 112, as shown in Figures 5A9 to 5A13), the computer system detects the movement of the input relative to the corresponding physical object within the field of view of one or more cameras (for example, as shown in Figures 5A9 to 5A13). In some embodiments, the movement of the input optionally includes the movement of the contact across the touchscreen display or across the touch-sensitive surface of the touch-sensitive remote control while the computer system (e.g., device 100) is held substantially stationary in physical space (for example, as shown in Figures 5A8 to 5A11). In some embodiments, the movement of the input optionally includes the movement of a device including a camera in physical space while the contact is maintained on the touchscreen display or touch-sensitive remote control and remains stationary (for example, as shown in Figures 5A17 and 5A18). In some embodiments, input movement may optionally include simultaneous, simultaneous movements of contact across a touchscreen display or touch-sensitive remote control, and movement of a device including a camera in physical space. In some embodiments, computer system movement may include movement of components of a multi-component computer system, such as the movement of a virtual reality display headset (for example, as shown in Figure 5A2). In addition, while continuously detecting input, and in response to detecting input movement relative to a corresponding physical object within the field of view of one or more cameras, the device adjusts the appearance of the corresponding virtual user interface object according to the magnitude of the input movement relative to the corresponding physical object (for example, by enlarging, shrinking, stretching, pushing together, expanding, and / or pressing together all or some(s) of the virtual user interface objects).For example, when contact is detected on the virtual roof of an architectural model and then moves across the touchscreen display, the virtual roof is lifted away from the architectural model in the live preview of the camera's field of view (e.g., as shown in Figures 5A8 to 5A11), and while contact is maintained on the touchscreen display and the device as a whole is moving relative to the architectural model in physical space, the movement of the virtual roof is determined based on both the location of the contact on the touchscreen display and the position and orientation of the device relative to the corresponding physical object in physical space (e.g., as determined based on the position of the corresponding physical object shown in the live preview of the camera's field of view) (e.g., as shown in Figures 5A11 to 5A13).

[0335] As another example, in a block-building application where a virtual model is constructed on a corresponding physical object (e.g., a table surface or a sheet of paper with a printed pattern) (as described in more detail, e.g., with respect to Figures 5B1 to 5B41), when contact on the block (e.g., contact 5262, Figure 5B28) is detected (e.g., in response to a long press input on the block), and the computer system displays a guide on how the block should scale (e.g., as shown using resizing projection 5266, Figure 5B29), while the contact on the block is maintained and the device as a whole is moving relative to the block (e.g., as shown in Figures 5B28 to 5B30), the scaling of the block (e.g., stretching of the block in the direction of the guide) is determined based on both the location of the contact on the touchscreen display (e.g., on a particular side or face of the block cube), as well as the position and orientation of the device relative to the corresponding physical object in physical space (e.g., as determined based on the position of the corresponding physical object shown in a live preview of the camera's field of view).

[0336] In some embodiments, adjusting the appearance of a corresponding virtual user interface object (e.g., a virtual architectural model 5012, Figures 5A8 to 5A13) according to the magnitude of the input movement relative to the corresponding physical object includes: adjusting the appearance of the corresponding virtual user interface object by a first adjustment according to the determination that the magnitude of the input movement relative to the corresponding physical object is a first magnitude (e.g., a relatively larger magnitude of movement) (e.g., a larger relative movement results in a larger adjustment) (e.g., as shown in Figure 5A10 compared to Figure 5A9); and adjusting the appearance of the corresponding virtual user interface object by a second adjustment separate from the first adjustment according to the determination that the magnitude of the input movement relative to the corresponding physical object is a second magnitude separate from the first magnitude (e.g., a relatively smaller magnitude of movement) (e.g., a smaller relative movement results in a smaller adjustment) (e.g., as shown in Figure 5A9 compared to Figure 5A10) (610). By adjusting the corresponding virtual user interface object by a first adjustment when the magnitude of input movement is a first magnitude (e.g., a larger relative movement causes a larger adjustment), and by adjusting the corresponding virtual user interface object by a second adjustment when the magnitude of input movement is a second magnitude (e.g., a smaller relative movement causes a smaller adjustment), the visual feedback provided to the user is improved (e.g., by making the computer system appear more responsive to user input), the usability of the device is enhanced, the user-device interface is made more efficient (e.g., by helping the user achieve the intended result with the required input and reducing user errors when operating / interacting with the device), and in addition, power consumption is reduced and the battery life of the device is improved by enabling the user to use the device more quickly and efficiently.

[0337] In some embodiments, a corresponding virtual user interface object (e.g., a virtual building model 5012, Figures 5A3 to 5A6) is fixed to a corresponding physical object (e.g., a physical building model 5006) in the field of view of one or more cameras before and after adjustment (612). For example, in some embodiments, a corresponding virtual user interface object appears to cover a corresponding physical object in the field of view of one or more cameras, and as the position and / or orientation of the physical object in the field of view of one or more cameras changes, the position and / or orientation of the corresponding virtual user interface object changes accordingly (e.g., as shown in Figures 5A3 to 5A6). In some embodiments, a corresponding virtual user interface object is fixed to a corresponding physical object in the field of view of one or more cameras during some or all of the adjustment (e.g., during the transition from Figure 5A3 to Figure 5A5). By fixing corresponding virtual user interface objects to corresponding physical objects, the visual feedback provided to the user is improved (e.g., by making the computer system appear more responsive to user input), device usability is enhanced, the user-device interface is made more efficient (e.g., by helping the user achieve the intended result with the required input and reducing user errors when operating / interacting with the device), and in addition, power consumption is reduced and device battery life is improved by enabling the user to use the device more quickly and efficiently.

[0338] In some embodiments, the appearance of a corresponding virtual user interface object is adjusted in response to the detection of movement of input to a corresponding physical object within the field of view of one or more cameras, regardless of whether the movement of input is due to movement of input on an input device (e.g., movement of contact across the touchscreen display or touch-sensitive surface of an input device while the input device is held substantially stationary in physical space) (e.g., as shown in Figures 5A9 to 5A11), movement of one or more cameras relative to a corresponding physical object (e.g., movement of a computer system including cameras in physical space while the contact is maintained and stationary on the touchscreen display or touch-sensitive surface of an input device) (e.g., as shown in Figures 5A11 to 5A13), or a combination of movement of input on an input device and movement of one or more cameras relative to a corresponding physical object (e.g., simultaneous movement of contact across the touchscreen display or touch-sensitive surface of an input device and movement of a computer system including cameras in physical space) (614). By adjusting the appearance of virtual user interface objects regardless of how input is moved (for example, by allowing users to adjust the appearance of virtual user interface objects by moving input on an input device only, moving the camera relative to a physical object only, or a combination of input and camera movement), we provide an intuitive way for users to adjust the appearance of virtual user interface objects, improve the visual feedback provided to the user (for example, by making the computer system appear more responsive to user input), improve device usability, make the user-device interface more efficient (for example, by helping users achieve the intended results with the required input and reducing user errors when operating / interacting with the device), and in addition, reduce power consumption and improve device battery life by enabling users to use the device more quickly and efficiently. 【03...

Claims

1. In a computer system having a display generation component, one or more cameras, and an input device, Displaying an augmented reality environment via the aforementioned display generation component, and displaying the augmented reality environment is, A representation of at least a portion of the field of view of one or more cameras, including a corresponding physical object, wherein the representation is updated as the content of the field of view of the one or more cameras changes. A first virtual user interface object displayed at a corresponding position within the representation of the field of view of one or more cameras, wherein the first virtual user interface object has a position determined based on the corresponding physical object within the field of view of one or more cameras, This includes displaying them simultaneously, While the augmented reality environment is being displayed, a first input corresponding to the selection of the first virtual user interface object is detected, In response to detecting the first input corresponding to the selection of the first virtual user interface object, To stop displaying at least a portion of the representation of the field of view of one or more cameras, Displaying a view of a virtual model including 3D virtual content, wherein the first virtual user interface object has a corresponding position within the 3D virtual content of the virtual model, Methods that include...

2. The method according to claim 1, comprising stopping the display of the representation of the field of view of one or more cameras in response to detecting the first input corresponding to the selection of the first virtual user interface object.

3. The method according to claim 1, comprising updating the representation of the field of view of one or more cameras in response to detecting movement of at least a portion of the computer system that changes the field of view of one or more cameras while the augmented reality environment is being displayed.

4. The method according to claim 1, comprising updating the representation of the field of view of the one or more cameras and the first virtual user interface object in accordance with the change in the viewpoint of the content in the field of view, in response to detecting movement of at least a portion of the computer system that changes the viewpoint of the content in the field of view of the one or more cameras while the augmented reality environment is being displayed.

5. The method according to claim 1, comprising displaying an animated transition from the augmented reality environment to the view of the virtual model in response to detecting the first input corresponding to the selection of the first virtual user interface object.

6. While the view of the virtual model is being displayed, a second input corresponding to a request to display the augmented reality environment is detected. In response to the second input corresponding to the request to display the augmented reality environment, Displaying an animated transition from the view of the virtual model to the augmented reality environment, Displaying the aforementioned augmented reality environment, The method according to claim 1, including the method described in claim 1.

7. The method according to claim 6, wherein displaying the augmented reality environment in response to the second input includes displaying the augmented reality environment according to the field of view of one or more cameras after detecting the second input, wherein the field of view of one or more cameras after detecting the second input is different from the field of view of one or more cameras when the first input was detected.

8. The method according to any one of claims 1 to 7, wherein while the view of the virtual model is being displayed, the first virtual user interface object moves within the virtual model in response to one or more inputs from a user of the computer system.

9. The method according to any one of claims 1 to 7, comprising: detecting movement of at least a portion of the computer system while displaying the view of the virtual model; and, in response to detecting the movement of the computer system, modifying the view of the virtual model in accordance with the movement of the computer system.

10. While the view of the virtual model is being displayed, a third input corresponding to a request to exit the view of the virtual model is detected. In response to detecting the third input, Redisplaying the aforementioned augmented reality environment, To stop displaying the view of the virtual model, The method according to any one of claims 1 to 7, including the method described in any one of claims 1 to 7.

11. A computer system, Display generation component, One or more cameras, Input device and, One or more processors, A memory storing one or more programs is provided, and the one or more programs are configured to be executed by the one or more processors, The display generation component enables the display of an augmented reality environment, which is the augmented reality environment. A representation of at least a portion of the field of view of one or more cameras, including a corresponding physical object, wherein the representation is updated as the content of the field of view of the one or more cameras changes. A first virtual user interface object displayed at a corresponding position within the representation of the field of view of one or more cameras, wherein the first virtual user interface object has a position determined based on the corresponding physical object within the field of view of one or more cameras, Including displaying simultaneously, Display an augmented reality environment, While the augmented reality environment is being displayed, a first input corresponding to the selection of the first virtual user interface object is detected. In response to detecting the first input corresponding to the selection of the first virtual user interface object, The camera stops displaying at least a portion of the representation of the field of view of one or more cameras. A view of a virtual model containing 3D virtual content is displayed, and the first virtual user interface object has a corresponding position within the 3D virtual content of the virtual model. A computer system, including instructions.

12. The computer system according to claim 11, wherein one or more programs include instructions for performing the method described in any one of claims 2 to 10.

13. When executed by a computer system comprising a display generation component, one or more cameras, and an input device, the computer system, The display generation component enables the display of an augmented reality environment, which is the augmented reality environment. A representation of at least a portion of the field of view of one or more cameras, including a corresponding physical object, wherein the representation is updated as the content of the field of view of the one or more cameras changes. A first virtual user interface object displayed at a corresponding position within the representation of the field of view of one or more cameras, wherein the first virtual user interface object has a position determined based on the corresponding physical object within the field of view of one or more cameras, Including displaying simultaneously, Display an augmented reality environment, While the augmented reality environment is being displayed, a first input corresponding to the selection of the first virtual user interface object is detected. In response to detecting the first input corresponding to the selection of the first virtual user interface object, To stop displaying at least a portion of the representation of the field of view of one or more cameras, A view of a virtual model containing 3D virtual content is displayed, and the first virtual user interface object has a corresponding position within the 3D virtual content of the virtual model. A computer program that contains instructions.

14. The computer program according to claim 13, which, when executed by the computer system, includes an instruction causing the computer system to execute the method described in any one of claims 2 to 10.

Images