Systems, methods, and graphical user interfaces for interacting with augmented and virtual reality environments

Abstract

The disclosure relates to systems, methods, and graphical user interfaces for interacting with augmented and virtual reality environments. The method includes, at a computer system having a display generation component, one or more cameras, and an input device: displaying an augmented reality environment via the display generation component, wherein displaying the augmented reality environment includes simultaneously displaying: a representation of at least a portion of a field of view of the one or more cameras including respective physical objects; and a first virtual user interface object displayed at respective locations in a representation of a field of view of the one or more cameras; detecting a first input corresponding to a selection of a first virtual user interface object while displaying the augmented reality environment; and in response to detecting a first input corresponding to the selection of the first virtual user interface object: stopping displaying at least a portion of the representation of the field of view of the one or more cameras; and displaying a view of the virtual model, wherein the first virtual user interface object has a corresponding location in the virtual model.

Description

[0001] This application is a divisional application of the invention patent application with the application date of August 30, 2018, application number 201880055301.0, and invention name “Systems, methods and graphical user interfaces for interacting with augmented and virtual reality environments”. Technical Field

[0002] The present invention generally relates to computer systems for virtual / augmented reality, including but not limited to electronic devices for interacting with augmented reality environments and virtual reality environments. Background Art

[0003] In recent years, the development of computer systems for virtual / augmented reality has increased significantly. Example virtual / augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as touch-sensitive surfaces for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Example touch-sensitive surfaces include touchpads, touch-sensitive remote controls, and touchscreen displays. Such surfaces are used to manipulate user interfaces on displays and objects therein. Exemplary user interface objects include digital images, videos, text, icons, and control elements (such as buttons), as well as other graphics.

[0004] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, using an input sequence to select one or more user interface objects (e.g., one or more virtual elements in a virtual / augmented reality environment) and perform one or more actions on the selected user interface objects is cumbersome, places a significant cognitive burden on the user, and detracts from the experience with the virtual / augmented reality environment. Furthermore, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the Invention

[0005] Therefore, there is a need for computer systems with improved methods and interfaces for interacting with augmented reality and virtual reality environments. Such methods and interfaces optionally supplement or replace conventional methods for interacting with augmented reality and virtual reality environments. Such methods and interfaces reduce the amount, extent, and / or nature of input from the user and produce a more efficient human-computer interface. For battery-powered devices, such methods and interfaces can conserve power and increase the time between battery charges.

[0006] The above-mentioned defects and other problems associated with the user interface for virtual / augmented reality are reduced or eliminated using the disclosed computer system. In some embodiments, the computer system includes a desktop computer. In some embodiments, the computer system is portable (e.g., a laptop computer, a tablet computer, or a handheld device). In some embodiments, the computer system includes a personal electronic device (e.g., a wearable electronic device, such as a watch). In some embodiments, the computer system has a touchpad (and / or communicates with a touchpad). In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display") (and / or communicates with a touch-sensitive display). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set for performing multiple functions stored in the memory. In some embodiments, the user interacts with the GUI in part by stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include playing games, image editing, drawing, presentations, word processing, spreadsheet creation, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note taking and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0007] According to some embodiments, a method is performed at a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying: a representation of at least a portion of the field of view of one or more cameras including a corresponding physical object, wherein the representation is updated as the content of the field of view of the one or more cameras changes; and a corresponding virtual user interface object at a corresponding position in the field of view representation of the one or more cameras, wherein the corresponding virtual user interface object has a position determined based on the corresponding physical object in the field of view of the one or more cameras. The method also includes, while displaying the augmented reality environment, detecting input at a position corresponding to the corresponding virtual user interface object. The method also includes, while continuing to detect input: detecting movement of the input relative to the corresponding physical object in the field of view of the one or more cameras; and in response to detecting movement of the input relative to the corresponding physical object in the field of view of the one or more cameras, adjusting the appearance of the corresponding virtual user interface object according to the amount of movement of the input relative to the corresponding physical object.

[0008] According to some embodiments, a method is performed at a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying: a representation of at least a portion of the field of view of one or more cameras including corresponding physical objects, wherein the representation is updated as the content of the field of view of the one or more cameras changes; and corresponding virtual user interface objects at corresponding positions in the field of view representation of the one or more cameras, wherein the corresponding virtual user interface objects have positions determined based on the corresponding physical objects in the field of view of the one or more cameras. The method also includes, while displaying the augmented reality environment, detecting input that changes a virtual environment setting for the augmented reality environment. The method also includes, in response to detecting input that changes the virtual environment setting: adjusting the appearance of the corresponding virtual user interface object according to the change made to the virtual environment setting of the augmented reality environment; and applying a filter to at least a portion of the field of view representation of the one or more cameras, wherein the filter is selected based on the change made to the virtual environment setting.

[0009] According to some embodiments, a method is performed at a computer system having a display generation component, one or more cameras, and an input device. The method includes displaying an augmented reality environment via the display generation component. Displaying the augmented reality environment includes simultaneously displaying: a representation of at least a portion of the field of view of one or more cameras including corresponding physical objects, wherein the representation is updated as the content of the field of view of the one or more cameras changes; and a first virtual user interface object in a virtual model displayed at a corresponding position in the field of view representation of the one or more cameras, wherein the first virtual user interface object has a position determined based on the corresponding physical object in the field of view of the one or more cameras. The method also includes, while displaying the augmented reality environment, detecting a first input corresponding to a selection of the first virtual user interface object; and in response to detecting the first input corresponding to the selection of the first virtual user interface object, displaying a simulated field of view of the virtual model from the perspective of the first virtual user interface object in the virtual model.

[0010] According to some embodiments, a method is performed at a computer system having a display generation component and an input device. The method includes displaying, via the display generation component, a first virtual user interface object in a virtual three-dimensional space. The method also includes, while displaying the first virtual user interface object in the virtual three-dimensional space, detecting, via the input device, a first input, the first input including a selection of a corresponding portion of the first virtual user interface object and movement of the first input in two dimensions. The method also includes, in response to detecting a first input comprising movement of the first input in two dimensions: adjusting the appearance of the first virtual user interface object in a first direction determined based on the movement of the first input in two dimensions and the selected first portion of the first virtual user interface object based on determining that the corresponding portion of the first virtual user interface object is the first portion of the first virtual user interface object, wherein the adjustment of the first virtual user interface object in the first direction is constrained to movement in a first set of two dimensions of the virtual three-dimensional space; and adjusting the appearance of the first virtual user interface object in a second direction different from the first direction based on determining that the corresponding portion of the first virtual user interface object is a second portion of the first virtual user interface object different from the first portion of the first virtual user interface object, wherein the second direction is determined based on the movement of the first input in two dimensions and the selected second portion of the first virtual user interface object, wherein the adjustment of the first virtual user interface object in the second direction is constrained to movement in a second set of two dimensions of the virtual three-dimensional space different from the first set of two dimensions of the virtual three-dimensional space.

[0011] According to some embodiments, a method is performed at a computer system having a display generation component, one or more posture sensors, and an input device. The method includes displaying a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode via the display generation component, wherein displaying the simulated environment in the first viewing mode includes displaying a first virtual user interface object in a virtual model, the virtual model being displayed at a first corresponding position in the simulated environment associated with the physical environment of the computer system. The method also includes, while displaying the simulated environment, detecting a first change in the posture of at least a portion of the computer system relative to the physical environment via the one or more posture sensors; and in response to detecting the first change in the posture of the portion of the computer system, changing the appearance of the first virtual user interface object in the virtual model so as to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment. The method also includes, after changing the appearance of the first virtual user interface object based on the first change in the posture of the portion of the computer system, detecting a first gesture corresponding to an interaction with the simulated environment via the input device; and in response to detecting the first gesture corresponding to an interaction with the simulated environment, performing an operation corresponding to the first gesture in the simulated environment. In addition, the method includes, after performing an operation corresponding to the first gesture, detecting a second change in the posture of the portion of the computer system relative to the physical environment via one or more posture sensors; and in response to detecting the second change in the posture of the portion of the computer system: transitioning from displaying the simulated environment (including the virtual model) in a first viewing mode to displaying the simulated environment (including the virtual model) in a second viewing mode based on determining that the first gesture satisfies a mode change criterion, wherein the mode change criterion includes requiring that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, wherein displaying the virtual model in the simulated environment in the second viewing mode includes abandoning changing the appearance of the first virtual user interface object to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; and continuing to display the first virtual model in the simulated environment in the first viewing mode based on determining that the first gesture does not satisfy the mode change criterion, wherein displaying the virtual model in the first viewing mode includes changing the appearance of the first virtual user interface object in the virtual model in response to the second change in the posture of the portion of the computer system relative to the physical environment to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment.

[0012] According to some embodiments, a method is performed at a first computer system having a first display generation component, one or more first posture sensors, and a first input device. The method includes displaying a simulated environment oriented relative to a first physical environment of the first computer system via the first display generation component of the first computer system, wherein displaying the simulated environment includes simultaneously displaying: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; and a visual indication of a viewing angle of a second computer system of the simulated environment, wherein the second computer system is a computer system having a second display generation component, one or more second posture sensors, and a second input device, and the second computer system is displaying a view of the simulated environment oriented relative to a second physical environment of the second computer system via the second display generation component of the second computer system. The method also includes: while displaying the simulated environment via the first display generation component of the first computer system, detecting a change in the viewing angle of the second computer system of the simulated environment based on a change in a posture of a portion of the second computer system relative to the second physical environment of the second computer system. The method also includes, in response to detecting a change in the viewing perspective of the second computer system of the simulated environment based on a change in the posture of the portion of the second computer system relative to the physical environment of the second computer system, updating a visual indication of the viewing perspective of the second computer system of the simulated environment displayed via the first display generating component of the first computer system in accordance with the change in the viewing perspective of the second computer system of the simulated environment.

[0013] According to some embodiments, a method is performed at a computer system having a display generation component, one or more gesture sensors, and an input device. The method includes displaying a simulated environment via the display generation component. The method also includes detecting a first input pointing to a corresponding position in the simulated environment via the input device while displaying the simulated environment. The method also includes, in response to detecting the first input pointing to the corresponding position in the simulated environment: displaying an insertion cursor at the first position based on determining that the first input is a first input type and the first input is detected at a first position in the simulated environment that is different from the current position of the insertion cursor in the simulated environment; and inserting a first object at the second position and moving the insertion cursor to a third position on the first object based on determining that the first input is a first input type and the first input is detected at a second position in the simulated environment that corresponds to the current position of the insertion cursor.

[0014] According to some embodiments, a method is performed at a computer system having a display generation component, one or more cameras, and one or more posture sensors. The method includes displaying an augmented reality environment via the display generation component, wherein displaying the augmented reality environment includes simultaneously displaying: a representation of at least a portion of the field of view of one or more cameras including physical objects, and the representation is updated as the content of the field of view of the one or more cameras changes; and a virtual user interface object at a corresponding position in the field of view representation of the one or more cameras, wherein the corresponding position of the virtual user interface object in the field of view representation of the one or more cameras is determined based on a fixed spatial relationship between the physical objects and the virtual user interface objects included in the field of view representation of the one or more cameras. The method also includes, while displaying the augmented reality environment, detecting a first change in the posture of at least a portion of the computer system relative to the physical environment of the computer system via the one or more posture sensors. The method also includes, in response to detecting a first change in the posture of the portion of the computer system relative to the physical environment of the computer system, updating the augmented reality environment based on the first change in the posture of the portion of the computer system, wherein: based on determining that the augmented display environment is displayed in an unstable operating mode, updating the augmented reality environment based on the first change in the posture of the portion of the computer system includes: updating the representation of the portion of the field of view of one or more cameras by a first adjustment amount, the first adjustment amount being based on the first change in the posture of the portion of the computer system relative to the physical environment of the computer system; and updating the corresponding position of the virtual user interface object to a position selected to maintain a fixed spatial relationship between the physical objects and the virtual user interface object included in the one or more camera field of view representations; and based on determining that the augmented reality environment is displayed in a stable operating mode, updating the augmented reality environment based on the first change in the posture of the portion of the computer system includes: updating the representation of the portion of the field of view of the one or more cameras by a second adjustment amount, the second adjustment amount being based on the first change in the posture of the portion of the computer system relative to the physical environment of the computer system and being less than the first adjustment amount; and updating the corresponding position of the virtual user interface object to a position selected to maintain a fixed spatial relationship between the physical objects and the virtual user interface object included in the one or more camera field of view representations.

[0015] According to some embodiments, a computer system includes a display generating component (e.g., a display, a projector, a head-up display, etc.), one or more cameras (e.g., a camera that continuously provides a real-time preview of at least a portion of the content within the camera's field of view and optionally generates a video output including one or more image frame streams capturing the content within the camera's field of view), and one or more input devices (e.g., a touch-sensitive surface, such as a touch-sensitive remote control, or a touch screen display that also serves as a display generating component, a mouse, a joystick, a stylus controller, and / or a camera that tracks one or more features of a user, such as the position of the user's hands), optionally one or more gesture sensors, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, optionally one or more tactile output generators, one or more processors, and a memory that stores one or more programs (and / or communicates with these components); the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the performance of operations of any of the methods described herein. According to some embodiments, a computer-readable storage medium has stored therein instructions that, when executed by a computer system comprising a display generating component, one or more cameras, one or more input devices, optionally one or more gesture sensors, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, and optionally one or more tactile output generators (and / or in communication with these components), causes the computer system to perform the operations of any method described herein or causes the operations of any method described herein to be performed. According to some embodiments, a graphical user interface on a computer system comprising a display generating component, one or more cameras, one or more input devices, optionally one or more gesture sensors, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more tactile output generators, a memory, and one or more processors for executing one or more programs stored in the memory (and / or in communication with these components) includes one or more elements displayed in any method described herein, and the one or more elements are updated in response to input, as described in any method described herein. According to some embodiments, the computer system includes a display generating component, one or more cameras, one or more input devices, optionally one or more gesture sensors, optionally one or more sensors for detecting intensity of contact with a touch-sensitive surface, optionally one or more tactile output generators, and a device for performing or causing the operations of any method described herein to be performed (and / or communicating with these components).In accordance with some embodiments, an information processing device for use in a computer system that includes display generating components, one or more cameras, one or more input devices, optionally one or more gesture sensors, optionally one or more sensors for detecting intensity of contact with a touch-sensitive surface, and optionally one or more tactile output generators (and / or communicates with these components) includes a device for performing the operations of any method described herein or causing the operations of any method described herein to be performed.

[0016] Thus, a computer system having a display generation component, one or more cameras, one or more input devices, optionally one or more gesture sensors, optionally one or more sensors for detecting intensity of contact with a touch-sensitive surface, and optionally one or more tactile output generators (and / or communicating with these components) provides improved methods and interfaces for interacting with augmented and virtual reality environments, thereby increasing the effectiveness, efficiency, and user satisfaction of such computer systems. Such methods and interfaces can supplement or replace conventional methods for interacting with augmented and virtual reality environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0018] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0019] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0020] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0021] Figure 3A is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0022] Figure 3B-3C is a block diagram of an exemplary computer system according to some embodiments.

[0023] Figure 4A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0024] Figure 4B An example user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0025] Figure 4C-4E An example of a dynamic intensity threshold according to some embodiments is shown.

[0026] Figures 5A1-5A40 An example user interface according to some embodiments is shown for displaying an augmented reality environment and adjusting the appearance of the augmented reality environment and / or the appearance of objects in the augmented reality environment in response to different inputs, and switching between viewing a virtual model in the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model.

[0027] Figure 5B1-Figure 5B41 Examples of systems and user interfaces for three-dimensional manipulation of virtual user interface objects are shown, according to some embodiments.

[0028] Figure 5C1-Figure 5C30 An example of a system and user interface for switching between viewing modes of a displayed simulated environment is shown, according to some embodiments.

[0029] Figure 5D1-Figure 5D1 4C illustrates an example of a system and user interface for multiple users to interact with virtual user interface objects in a displayed simulated environment, according to some embodiments.

[0030] Figure 5E1-Figure 5E32 An example of a system and user interface for placing an insertion cursor is shown, according to some embodiments.

[0031] Figure 5F1-5F17b Examples of systems and user interfaces for displaying an augmented reality environment in a stable operating mode are shown, according to some embodiments.

[0032] Figures 6A-6D is a flowchart of a process for adjusting the appearance of a virtual user interface object in an augmented reality environment according to some embodiments.

[0033] Figures 7A-7C is a flowchart of a process for applying a filter to real-time images captured by one or more cameras of a computer system in an augmented reality environment, according to some embodiments.

[0034] Figures 8A-8C is a flow chart of a process for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments.

[0035] Figures 9A-9E is a flowchart of a process for three-dimensional manipulation of a virtual user interface object according to some embodiments.

[0036] Figures 10A-10Eis a flowchart of a process for switching between viewing modes of a displayed simulated environment according to some embodiments.

[0037] Figures 11A-11C is a flow diagram of a process for updating an indication of a viewing perspective of a second computer system in a simulated environment displayed by a first computer system, according to some embodiments.

[0038] Figures 12A-12D is a flowchart of a process for placing an insertion cursor according to some embodiments.

[0039] Figures 13A-13E is a flowchart of a process for displaying an augmented reality environment in a stable operating mode according to some embodiments. DETAILED DESCRIPTION

[0040] An augmented reality environment is an environment in which supplementary information that provides additional information not available in the physical world is used to enhance reality. Conventional methods for interacting with an augmented reality environment (e.g., accessing supplementary information) typically require multiple separate inputs (e.g., a sequence of gestures and button presses, etc.) to achieve the desired result. In addition, the scope of conventional input methods is typically limited (e.g., by the size of a touch-sensitive display of a computer system). Embodiments herein provide an intuitive way for a user to interact with an augmented reality environment (e.g., by adjusting the appearance of a virtual user interface object based on a combination of movement of a computer system and movement of an input device (e.g., a touch screen display) of the computer system, and by applying a filter in real time to a real-time image captured by one or more cameras of the computer system, wherein the filter is selected based on the virtual environment settings for the augmented reality environment).

[0041] Additionally, conventional interactions with a virtual / augmented reality environment are typically limited to a single perspective (e.g., from the perspective of a user wearing / holding the device). Embodiments herein provide a more immersive and intuitive way to experience a virtual / augmented reality environment by presenting a simulated view of a virtual model (e.g., of a physical object) in a virtual reality environment from the perspective of a virtual user interface object (e.g., from the perspective of a car or a person in an augmented reality environment).

[0042] The systems, methods, and GUIs described herein improve user interface interactions with virtual / augmented reality environments in a variety of ways. For example, they facilitate: displaying an augmented reality environment and adjusting the appearance of the augmented reality environment and / or objects therein in response to various inputs; switching between viewing a virtual model in the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model; and three-dimensional manipulation of virtual user interface objects.

[0043] under, Figure 1A-1B 、 Figure 2 as well as Figure 3A-3B A description of an exemplary apparatus is provided. According to some embodiments, Figure 4A-4B 、 Figures 5A1-5A40 、 Figure 5B1-Figure 5B41 、 Figure 5C1-Figure 5C30 、 Figure 5D1-Figure 5D1 4. Figure 5E1-Figure 5E32 as well as Figure 5F1-Figure 5F1 7 shows an example of a system and user interface for multiple users to interact with virtual user interface objects in a displayed simulated environment according to some embodiments, showing an example user interface for interacting with augmented and virtual reality environments, including displaying an augmented reality environment and, in response to different inputs, adjusting the appearance of the augmented reality environment and / or the appearance of objects in the augmented reality environment, switching between viewing a virtual model in the augmented reality environment and viewing a simulated view of the virtual model from the perspective of the objects in the virtual model, and three-dimensional manipulation of virtual user interface objects. Figures 6A-6D A flowchart of a method for adjusting the appearance of a virtual user interface object in an augmented reality environment is shown according to some embodiments. Figures 7A-7C A flowchart illustrating a method of applying a filter to real-time images captured by one or more cameras of a computer system in an augmented reality environment, according to some embodiments. Figures 8A-8C A flow chart illustrating a method for transitioning between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments. Figures 9A-9E A flowchart of a method for three-dimensional manipulation of a virtual user interface object according to some embodiments is shown. Figures 10A-10E A flow chart illustrating a method of switching between viewing modes of a displayed simulated environment according to some embodiments is shown. Figures 11A-11C A flowchart illustrating a method of updating an indication of a viewing perspective of a second computer system in a simulated environment displayed by a first computer system according to some embodiments. Figures 12A-12D A flowchart of a method of placing an insertion cursor according to some embodiments is shown. Figures 13A-13E A flowchart of a method for displaying an augmented reality environment in a stable operating mode according to some embodiments is shown.

[0044] Figures 5A1-5A40 、 Figure 5B1-Figure 5B41 、 Figure 5C1-Figure 5C30 、 Figure 5D1-Figure 5D1 4. Figure 5E1-Figure 5E32 as well as Figure 5F1-Figure 5F1 The user interface in 7 is used to show Figures 6A-6D 、 Figures 7A-7C 、 Figures 8A-8C 、 Figures 9A-9E 、 Figures 10A-10E 、 Figures 11A-11C 、 Figures 12A-12D as well as Figures 13A-13E in the process.

[0045] Exemplary devices

[0046] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure various aspects of the embodiments.

[0047] It will also be understood that, although in some cases, the terms "first," "second," etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are simply used to distinguish one element from another. For example, a first contact can be named a second contact, and similarly, a second contact can be named a first contact without departing from the scope of the various described embodiments. Both a first contact and a second contact are contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0048] The terms used in the description of the various embodiments described herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a" and "the" are intended to also include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "includes," "including," "comprises," and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0049] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [stated condition or event] is detected" are optionally interpreted to mean "upon determining" or "in response to determining" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

[0050] A computer system for virtual / augmented reality includes an electronic device that generates a virtual / augmented reality environment. Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices, such as laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads) that also includes or communicates with one or more cameras.

[0051] In the following discussion, a computer system is described that includes an electronic device having a display and a touch-sensitive surface (and / or communicating with these components). However, it should be understood that the computer system can optionally include one or more other physical user interface devices, such as a physical keyboard, a mouse, a joystick, a stylus controller, and / or a camera that tracks one or more characteristics of a user, such as the position of the user's hand.

[0052] The device typically supports a variety of applications, such as one or more of the following: a gaming application, a note-taking application, a drawing application, a presentation application, a word processing application, a spreadsheet application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0053] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed by the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0054] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and sometimes simply as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and external ports 124. The device 100 optionally includes one or more optical sensors 164 (e.g., as part of one or more cameras). The device 100 optionally includes one or more intensity sensors 165 for detecting intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0055] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that will be detected by a user using the user's sense of touch. For example, when a device or a component of the device is in contact with a surface that is touch-sensitive to a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such interpretation of touch by the user will be limited by the user's individualized sensory perception, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the tactile output generated corresponds to a physical displacement of the device or a component thereof that would generate the sensory perception of a typical (or average) user. Providing tactile feedback to the user using tactile output enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0056] It should be understood that device 100 is merely one example of a portable multifunction device and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0057] Memory 102 optionally includes high-speed random access memory and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100 (such as one or more CPUs 120 and peripheral device interface 118) is optionally controlled by a memory controller 122.

[0058] Peripherals interface 118 may be used to couple the device's input and output peripherals to memory 102 and CPU(s) 120. Processor(s) 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0059] In some embodiments, peripherals interface 118, one or more CPUs 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0060] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed on the date of filing of this document.

[0061] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0062] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from or send electrical signals to the other input or control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. The one or more buttons (e.g., Figure 2 208) optionally includes up / down buttons for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in ).

[0063] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some visual outputs or all of the visual outputs correspond to user interface objects. As used herein, the term "indicator" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0064] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from the user based on tactile and / or haptic contact. The touch-sensitive display system 112 and the display controller 156 (together with any associated modules and / or instruction sets in the memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0065] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In some embodiments, projected mutual capacitance sensing technology is used, such as the ATmega 2000 from Apple Inc. (Cupertino, California). iPod and Technology found in.

[0066] The touch-sensitive display system 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or greater). The user optionally uses any suitable object or appendage, such as a stylus, finger, or the like, to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the action desired by the user.

[0067] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0068] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0069] Device 100 optionally also includes one or more optical sensors 164 (eg, as part of one or more cameras). Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The one or more optical sensors 164 receive light projected from the environment through one or more lenses and convert the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the one or more optical sensors 164 optionally capture still images and / or video. In some embodiments, the optical sensor is located on the rear portion of the device 100 opposite the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for selfies, for video conferencing when the user views other video conference participants on the touch screen, etc.).

[0070] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to a force sensor controller 159 in the I / O subsystem 106. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front of the device 100.

[0071] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0072] Device 100 optionally also includes one or more tactile output generators 163 . Figure 1AA tactile output generator is shown coupled to a tactile feedback controller 161 in the I / O subsystem 106. In some embodiments, one or more tactile output generators 163 include one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile outputs on the device). One or more tactile output generators 163 receive tactile feedback generation instructions from the tactile feedback module 133 and generate tactile outputs on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of the device 100) or laterally (e.g., back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100 opposite the touch-sensitive display system 112 located on the front of the device 100.

[0073] Device 100 optionally also includes one or more accelerometers 167 , gyroscopes 168 , and / or magnetometers 169 (eg, as part of an inertial measurement unit (IMU)) for obtaining information about the position (eg, attitude) of the device. Figure 1A Sensors 167, 168, and 169 are shown coupled to peripherals interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on the touch screen display in a portrait view or a landscape view based on analysis of data received from the one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location of device 100.

[0074] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and applications (or instruction sets) 136. In addition, in some embodiments, memory 102 stores a device / global internal state 157, as shown in Figures 1A and 3. The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, which includes information obtained from various sensors and other input or control devices 116 of the device; and position and / or orientation information about the position and / or posture of the device.

[0075] The operating system 126 (e.g., iOS, Android, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0076] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device that is compatible with some of Apple Inc. (Cupertino, California). iPod and In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in some Apple Inc. (Cupertino, California) iPod and In some embodiments, the external port is a USB Type-C connector that is the same as, similar to, and / or compatible with the Lightning connector used in some electronic devices from Apple Inc. (Cupertino, California).

[0077] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or stylus contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multi-finger contact). In some embodiments, contact / motion module 130 and display controller 156 detect contact on a touchpad.

[0078] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0079] In some embodiments, detecting a finger tap gesture depends on detecting the length of time between a finger press event and a finger lift event, but is independent of detecting the intensity of the finger contact between the finger press event and the finger lift event. In some embodiments, a tap gesture is detected based on determining that the length of time between the finger press event and the finger lift event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the intensity of the finger contact during the tap reaches a given intensity threshold (greater than a nominal contact detection intensity threshold), such as a light press or deep press intensity threshold. Thus, a finger tap gesture can meet a specific input criterion that does not require the characteristic intensity of the contact to meet a given intensity threshold to meet the specific input criterion. For clarity, the finger contact in a tap gesture is generally required to meet a nominal contact detection intensity threshold to detect a finger press event, below which the contact is not detected. A similar analysis applies to detecting a tap gesture by a stylus or other contact. In cases where the device is capable of detecting contact by a finger or stylus hovering over the touch-sensitive surface, the nominal contact detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0080] The same concepts apply in a similar manner to other types of gestures. For example, a swipe gesture, a pinch gesture, an expand gesture, and / or a long press gesture may be optionally detected based on satisfying criteria that are independent of the intensity of the contacts included in the gesture or that do not require one or more contacts performing the gesture to reach an intensity threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; a zoom gesture is detected based on the movement of two or more contacts toward each other; a zoom gesture is detected based on the movement of two or more contacts away from each other; and a long press gesture is detected based on the duration of a contact on the touch-sensitive surface having less than a threshold amount of movement. Thus, the statement that a particular gesture recognition criterion does not require the intensity of the contacts to meet the corresponding intensity threshold in order to satisfy the particular gesture recognition criterion means that the particular gesture recognition criterion can be satisfied when the contacts in the gesture do not reach the corresponding intensity threshold, and can also be satisfied when one or more contacts in the gesture reaches or exceeds the corresponding intensity threshold. In some embodiments, a tap gesture is detected based on determining that a finger down event and a finger up event are detected within a predefined time period, regardless of whether the contact is above or below a corresponding intensity threshold during the predefined time period, and a swipe gesture is detected based on determining that the contact moves by more than a predefined amount, even if the contact is above the corresponding intensity threshold at the end of the contact movement. Even in specific implementations where detection of gestures is affected by the intensity of the contact performing the gesture (e.g., the device detects a long press more quickly when the intensity of the contact is above an intensity threshold, or the device delays detection of a tap input when the intensity of the contact is higher), detection of these gestures does not require the contact to reach a particular intensity threshold (e.g., even if the amount of time required to recognize the gesture varies), as long as the criteria for recognizing the gesture can be met without the contact reaching the particular intensity threshold.

[0081] In some cases, contact intensity thresholds, duration thresholds, and movement thresholds are combined in various combinations to create heuristic algorithms that distinguish between two or more different gestures directed to the same input element or area, enabling multiple different interactions with the same input element to provide a richer set of user interactions and responses. The statement that a particular set of gesture recognition criteria does not require the intensity of a contact to meet the corresponding intensity threshold to meet the particular gesture recognition criteria does not preclude the simultaneous evaluation of other intensity-related gesture recognition criteria to identify other gestures whose criteria are met when the gesture includes a contact with an intensity above the corresponding intensity threshold. For example, in some cases, a first gesture recognition criterion for a first gesture (which does not require the intensity of the contact to meet the corresponding intensity threshold to meet the first gesture recognition criterion) competes with a second gesture recognition criterion for a second gesture (which depends on the contact meeting the corresponding intensity threshold). In such a competition, if the second gesture recognition criterion for the second gesture is met first, the gesture is optionally not recognized as meeting the first gesture recognition criterion for the first gesture. For example, if the contact reaches the corresponding intensity threshold before the contact moves a predefined amount, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contact moves a predefined amount before the contact reaches the corresponding intensity threshold, a swipe gesture is detected instead of a deep press gesture. Even in such cases, the first gesture recognition criterion for the first gesture still does not require the intensity of the contact to meet the corresponding intensity threshold in order to satisfy the first gesture recognition criterion because, if the contact remains below the corresponding intensity threshold until the gesture ends (e.g., a swipe gesture with an intensity that does not increase above the corresponding intensity threshold), the gesture will be recognized as a swipe gesture by the first gesture recognition criterion. Thus, a particular gesture recognition criterion that does not require the intensity of the contact to meet the corresponding intensity threshold in order to satisfy the particular gesture recognition criterion may (A) in some cases ignore the intensity of the contact relative to the intensity threshold (e.g., for a tap gesture) and / or (B) in some cases fail to satisfy the particular gesture recognition criterion (e.g., for a long press gesture) if a set of competing intensity-related gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-related gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input, and in this sense still depends on the intensity of the contact relative to the intensity threshold (e.g., for a long press gesture competing with a deep press gesture for recognition).

[0082] In conjunction with accelerometer 167, gyroscope 168, and / or magnetometer 169, attitude module 131 optionally detects attitude information about the device, such as the attitude of the device in a particular reference frame (e.g., roll, pitch, and / or yaw). Attitude module 131 includes software components for performing various operations related to detecting device position and detecting changes in device attitude.

[0083] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch-sensitive display system 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0084] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application program or the like, along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

[0085] The tactile feedback module 133 includes various software components for generating instructions (e.g., instructions used by the tactile feedback controller 161) to produce tactile output at one or more locations on the device 100 using one or more tactile output generators 163 in response to user interaction with the device 100.

[0086] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0087] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as a weather widget, local pages widget, and map / navigation widget).

[0088] The virtual / augmented reality module 145 provides virtual and / or augmented reality logic to the application 136, which implements augmented reality features, and in some embodiments, virtual reality features. The virtual / augmented reality module 145 facilitates the overlay of virtual content, such as virtual user interface objects, on a representation of at least a portion of the field of view of one or more cameras. For example, with the assistance of the virtual / augmented reality module 145, the representation of at least a portion of the field of view of one or more cameras can include corresponding physical objects, and the virtual user interface objects can be displayed in a displayed augmented reality environment at a location determined based on the corresponding physical objects in the field of view of the one or more cameras, or in a virtual reality environment determined based on a posture of at least a portion of the computer system (e.g., a posture of a display device used to display a user interface to a user of the computer system).

[0089] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0090] Contacts module 137 (sometimes called address book or contact list);

[0091] Telephone module 138;

[0092] Video conferencing module 139;

[0093] Email client module 140;

[0094] Instant messaging (IM) module 141;

[0095] Fitness support module 142;

[0096] A camera module 143 for still and / or video images;

[0097] Image management module 144;

[0098] Browser module 147;

[0099] Calendar module 148;

[0100] Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6;

[0101] A widget creator module 150 for forming user-created widgets 149-6;

[0102] Search module 151;

[0103] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0104] Notepad module 153;

[0105] Map module 154; and / or

[0106] Online video module 155.

[0107] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0108] In combination with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and so on.

[0109] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing the corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0110] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and phone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0111] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0112] In conjunction with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, transmitting the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0113] In combination with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the video and music player module 152, the fitness support module 142 includes executable instructions for creating a workout (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring a workout; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0114] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0115] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0116] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching for, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0117] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0118] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., the Yahoo! desktop widget).

[0119] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page into a desktop widget).

[0120] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0121] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0122] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notes module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0123] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving directions; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0124] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0125] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0126] In some embodiments, device 100 is a device in which operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touchpad. By using a touch screen and / or a touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0127] A predefined set of functions that are exclusively performed through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, a home screen menu, or a root menu. In such embodiments, a touch-sensitive surface is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touch-sensitive surface.

[0128] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3A ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0129] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0130] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a redo / undo queue of previous actions taken by the user.

[0131] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0132] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0133] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0134] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0135] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0136] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0137] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0138] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0139] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or part of another module stored in memory 102, such as contact / motion module 130.

[0140] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 is part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application's internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0141] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0142] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0143] Event comparator 184 compares event information with predefined event or sub-event definitions and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, the sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one embodiment, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another embodiment, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0144] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler that is associated with the sub-event and the object that triggered the hit test.

[0145] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0146] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0147] In some embodiments, corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0148] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating an event handler 190 is different from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag obtains the flag and performs a predefined process.

[0149] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0150] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0151] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0152] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a trackpad, such as taps, drags, scrolls, etc.; stylus input; input based on real-time analysis of video images obtained by one or more cameras; movement of the device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0153] Figure 2 A device with a touch screen (e.g., Figure 1AIn some embodiments, the portable multifunction device 100 includes a touch-sensitive display system 112 (e.g., a touch screen). The touch screen optionally displays one or more graphics within a user interface (UI) 200. In these embodiments, and in other embodiments described below, a user can select one or more of the graphics by, for example, making a gesture on the graphics using one or more fingers 202 (not drawn to scale in the figures) or one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, from left to right, up and / or down). In some implementations or in some cases, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0154] The device 100 optionally also includes one or more physical buttons, such as a "home" button, or a menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0155] In some embodiments, the device 100 includes a touch screen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for powering the device on / off and for locking the device, a volume adjustment button 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and holding the button in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing the button before the predefined time interval has passed; and / or to unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through the microphone 113. The device 100 also optionally includes one or more contact force sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output for the user of the device 100.

[0156] Figure 3A3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a gaming system, a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 having a display 340, which is optionally a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above referenced devices). Figure 1A The one or more tactile output generators 163 described above), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or sensors similar to those described above) Figure 1A Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDRRAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310. In some embodiments, memory 370 stores data related to portable multifunction device 100 ( Figure 1A ), or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100( Figure 1A )'s memory 102 optionally does not store these modules.

[0157] Figure 3AEach element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device.Each module in the above-mentioned identified module corresponds to the instruction set for performing the functions described above.The above-mentioned identified module or program (that is, instruction set) need not be implemented as an independent software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments.In some embodiments, memory 370 optionally stores the subgroup of above-mentioned modules and data structure.In addition, memory 370 optionally stores other modules and data structure not described above.

[0158] Figure 3B-3C is a block diagram of an exemplary computer system 301 according to some embodiments.

[0159] In some embodiments, computer system 301 includes and / or is in communication with the following components:

[0160] Input devices (302 and / or 307, e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touch screen display that also serves as a display generating component, a mouse, a joystick, a stylus controller, and / or a camera that tracks one or more characteristics of a user, such as the position of a user's hand;

[0161] Virtual / augmented reality logic 303 (e.g., virtual / augmented reality module 145);

[0162] Display generation components (304 and / or 308, e.g., displays, projectors, heads-up displays, etc.) for displaying virtual user interface elements to a user;

[0163] A camera (e.g., 305 and / or 311 ) for capturing images of the device's field of view, e.g., for determining placement of virtual user interface elements, determining the device's pose, and / or displaying an image of a portion of the physical environment in which the camera is located; and

[0164] • A posture sensor (eg, 306 and / or 311 ) for determining the posture of the device relative to the physical environment and / or changes in the posture of the device.

[0165] On some computer systems (e.g. Figure 3B In 301-a), the input device 302, the virtual / augmented reality logic component 303, the display generation component 304, the camera 305; and the gesture sensor 306 are all integrated into the computer system (e.g., Figure 1A-1B 3 or device 300, such as a smart phone or tablet computer).

[0166] In some computer systems (e.g., 301-b), in addition to the integrated input device 302, virtual / augmented reality logic component 303, display generation component 304, camera 305; and gesture sensor 306, the computer system also communicates with additional devices independent of the computer system, such as an independent input device 307, such as a touch-sensitive surface, stylus, remote control, etc. and / or an independent display generation component 308, such as a virtual reality headset or augmented reality glasses that overlay virtual objects on the physical environment.

[0167] On some computer systems (e.g. Figure 3C In 301-c), input device 307, display generation component 309, camera 311, and / or gesture sensor 312 are separate from and in communication with the computer system. In some embodiments, other combinations of components in computer system 301 and in communication with the computer system are used. For example, in some embodiments, display generation component 309, camera 311, and gesture sensor 312 are incorporated into a headset that is integrated with or in communication with the computer system.

[0168] In some embodiments, the following reference Figures 5A1-5A40 and Figure 5B1-Figure 5B41 All operations described are performed on a single computing device with virtual / augmented reality logic 303 (e.g., Figure 3B However, it should be understood that multiple different computing devices are often linked together to execute the computer system 301-a) described below. Figures 5A1-5A40 and Figure 5B1-Figure 5B41 In any of these embodiments, the following reference is made to the operations described (e.g., a computing device having virtual / augmented reality logic 303 communicating with a separate computing device having display 450 and / or a separate computing device having touch-sensitive surface 451). Figures 5A1-5A40 and Figure 5B1-Figure 5B41 The computing devices depicted are one or more computing devices that include virtual / augmented reality logic 303. Additionally, it should be understood that in various embodiments, virtual / augmented reality logic 303 may be divided among multiple different modules or computing devices; however, for the purposes of this description, virtual / augmented reality logic 303 will primarily be referred to as residing in a single computing device to avoid unnecessarily obscuring other aspects of the embodiments.

[0169] In some embodiments, the virtual / augmented reality logic 303 includes one or more modules that receive and interpret input (e.g., one or more event handlers 190, including those described above with reference to FIG). Figure 1BIn some embodiments, the graphical user interface may be updated based on the interpreted inputs (e.g., one or more object updaters 177 and one or more GUI updaters 178, described in more detail herein), and in response to the interpreted inputs, generates instructions for updating the graphical user interface based on the interpreted inputs, which are then used to update the graphical user interface on the display. Figure 1A and contact motion module 130 in FIG. 3 ), identifying (e.g., by Figure 1B event identifier 180 in ) and / or assigned (e.g., via Figure 1B The interpreted input of the input received by the event classifier 170 in the touch-sensitive surface 451 is used to update the graphical user interface on the display. In some embodiments, the interpreted input is generated by a module on the computing device (e.g., the computing device receives raw contact input data to identify gestures from the raw contact input data). In some embodiments, some or all of the interpreted input is received as interpreted input by the computing device (e.g., the computing device including the touch-sensitive surface 451 processes the raw contact input data to identify gestures from the raw contact input data and sends information indicative of the gesture to the computing device including the virtual / augmented reality logic component 303).

[0170] In some embodiments, both the display and the touch-sensitive surface are connected to a computer system (e.g., Figure 3B For example, the computer system may be a desktop computer or laptop computer with an integrated display (e.g., 340 in FIG. 3 ) and a touchpad (e.g., 355 in FIG. 3 ). For another example, the computing device may be a computer with a touch screen (e.g., Figure 2 112) of a portable multifunction device 100 (e.g., a smart phone, a PDA, a tablet computer, etc.).

[0171] In some embodiments, the touch-sensitive surface is integrated with the computer system, while the display is not integrated with the computer system including the virtual / augmented reality logic 303. For example, the computer system can be a device 300 (e.g., a desktop or laptop computer, etc.) with an integrated touchpad (e.g., 355 in FIG. 3 ), where the integrated touchpad is connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, a television, etc.). For another example, the computer system can be a device 300 (e.g., a desktop or laptop computer, etc.) with an integrated touchpad (e.g., 355 in FIG. 3 ). Figure 2 112) of a portable multifunction device 100 (e.g., a smart phone, PDA, tablet computer, etc.), wherein the touch screen is connected (via a wired or wireless connection) to a separate display (e.g., a computer monitor, a television, etc.).

[0172] In some embodiments, the display is integrated with the computer system, while the touch-sensitive surface is not integrated with the computer system including the virtual / augmented reality logic 303. For example, the computer system can be a device 300 (e.g., a desktop computer, a laptop computer, a television with an integrated set-top box) with an integrated display (e.g., 340 in FIG. 3 ), where the integrated display is connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote trackpad, a portable multifunction device, etc.). For another example, the computer system can be a device 300 (e.g., a desktop computer, a laptop computer, a television with an integrated set-top box) with an integrated display (e.g., 340 in FIG. 3 ). Figure 2 A portable multifunction device 100 (e.g., a smart phone, a PDA, a tablet computer, etc.) having a touch screen 112 therein, wherein the touch screen is connected (via a wired or wireless connection) to a separate touch-sensitive surface (e.g., a remote touchpad, a portable multifunction device in which another touch screen is used as a remote touchpad, etc.).

[0173] In some embodiments, neither the display nor the touch-sensitive surface is connected to a computer system (e.g., Figure 3C For example, the computer system can be a standalone computing device 300 (e.g., a set-top box, a game console, etc.) connected (via a wired or wireless connection) to a standalone touch-sensitive surface (e.g., a remote trackpad, a portable multifunction device, etc.) and a standalone display (e.g., a computer monitor, a television, etc.).

[0174] In some embodiments, the computer system has an integrated audio system (e.g., audio circuitry 110 and speakers 111 in portable multifunction device 100). In some embodiments, the computing device communicates with an audio system that is independent of the computing device. In some embodiments, the audio system (e.g., an audio system integrated into a television unit) is integrated with a separate display. In some embodiments, the audio system (e.g., a stereo system) is a separate system from the computer system and display.

[0175] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0176] Figure 4A An exemplary user interface for an application menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0177] One or more signal strength indicators of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0178] ·time;

[0179] Bluetooth indicator;

[0180] Battery status indicator;

[0181] A tray 408 with icons for commonly used applications, such as:

[0182] o An icon 416 labeled "Phone" of the phone module 138, the icon 416 optionally including an indicator 414 of the number of missed calls or voice messages;

[0183] o An icon 418 of the email client module 140 labeled “Mail,” which optionally includes an indicator 410 of the number of unread emails;

[0184] o An icon 420 labeled "Browser" for the browser module 147; and

[0185] o An icon 422 labeled “Music” of the video and music player module 152;

[0186] as well as

[0187] ○ Icons for other apps, such as:

[0188] ○ Icon 424 labeled "Message" of IM module 141

[0189] o An icon 426 labeled “Calendar” of the calendar module 148;

[0190] o Icon 428 labeled "Photos" of the image management module 144;

[0191] o An icon 430 labeled “Camera” of the camera module 143;

[0192] ○ An icon 432 labeled “Online Video” of the online video module 155;

[0193] ○ Icon 434 labeled “Stock Market” of the Stock Market widget 149-2;

[0194] ○ Icon 436 labeled “Map” of the map module 154

[0195] ○ Icon 438 labeled “Weather” of the weather widget 149-1;

[0196] ○ Icon 440 labeled “Clock” of the alarm clock widget 149-4;

[0197] o An icon 442 labeled “Fitness Support” of the fitness support module 142;

[0198] o An icon 444 labeled "Notepad" of the Notepad module 153; and

[0199] o An icon 446 of a settings application or module labeled “Settings” that provides access to settings for the device 100 and its various applications 136 .

[0200] It should be noted that Figure 4A The icon labels shown in the are merely exemplary. For example, other labels are optionally used for various application icons. In some embodiments, the label of a respective application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0201] Figure 4B A touch-sensitive surface 451 (e.g., Figure 3A a device (e.g., a tablet or touchpad 355) Figure 3A Although many of the examples that follow will be given with reference to input on the touch screen display 112 (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects contact with the touch-sensitive surface 451 at a location that corresponds to a corresponding location on the display (e.g., Figure 4B 460 and 462 in (e.g., in Figure 4B , 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B When the user interface 450 in FIG. 1 is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0202] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting the contact, followed by ceasing to detect the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0203] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3A Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that the touch-sensitive surface 451 in the display includes a touch-screen display (e.g., a touch-sensitive surface 451 in the display), the particular user interface element is adjusted according to the detected input. Figure 1A touch-sensitive display system 112 or Figure 4AIn some implementations (such as touch screens in a touchscreen display), a contact detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by a contact) is detected on the touchscreen display at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted based on the detected input. In some implementations, focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touchscreen display (e.g., by using a tab key or arrow keys to move focus from one button to another); in these implementations, the focus selector moves based on the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user interface element (or contact on the touchscreen display) that is controlled by the user to convey the user's desired interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a trackpad or touchscreen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display). In some embodiments, a focus indicator (e.g., a cursor or selection indicator) is displayed via a display device to indicate the current portion of the user interface that will be affected by input received from one or more input devices.

[0204] In some embodiments, the device's response to an input detected by the device depends on criteria based on the intensity of the contact during the input. For example, for some "light press" inputs, the intensity of the contact exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the device's response to an input detected by the device depends on criteria that include both the intensity of the contact during the input and a time-based criterion. For example, for some "deep press" inputs, the intensity of the contact exceeding a second intensity threshold, which is greater than the first intensity threshold for a light press, during the input triggers a second response, as long as a delay time elapses between the first intensity threshold being met and the second intensity threshold being met. The duration of this delay time is typically less than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, where the delay time increases as the second intensity threshold increases). This delay time helps avoid accidentally identifying a deep press input. As another example, for some "deep press" inputs, a period of reduced sensitivity occurs after the first intensity threshold is reached. During this period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps avoid accidental deep press inputs.For other deep press inputs, the response to detecting the deep press input does not depend on time-based criteria.

[0205] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors, such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Example factors are described in U.S. patent applications 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entireties.

[0206] For example, Figure 4C A dynamic intensity threshold 480 is shown that varies over time based in part on the intensity of the touch input 476 over time. The dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predefined delay time p1 from when the touch input 476 is initially detected, and a second component 478 that tracks the intensity of the touch input 476 over time. The initial high intensity threshold of the first component 474 reduces accidental triggering of a "deep press" response while still allowing an immediate "deep press" response if the touch input 476 provides sufficient intensity. The second component 478 reduces the unintentional triggering of a "deep press" response through gradual intensity fluctuations of the touch input. In some embodiments, when the touch input 476 satisfies the dynamic intensity threshold 480 (e.g., at Figure 4C ), triggering a “deep press” response.

[0207] Figure 4D Another dynamic intensity threshold 486 (eg, intensity threshold IT D ). Figure 4D Two other intensity thresholds are also shown: a first intensity threshold IT H and the second intensity threshold IT L .exist Figure 4D Although the touch input 484 meets the first intensity threshold IT before time p2 H and the second intensity threshold IT L , but no response is provided until a delay time p2 has elapsed at time 482. Additionally, Figure 4D In the example, the dynamic intensity threshold 486 decays over time, wherein the decay is from time 482 (when the trigger is connected to the intensity threshold IT L This type of dynamic intensity threshold reduction begins immediately after the trigger is triggered with a lower threshold intensity (such as the first intensity threshold IT H Or the second intensity threshold IT L ) is triggered after or simultaneously with the unexpected triggering of the dynamic intensity threshold IT D The associated response.

[0208] Figure 4E Yet another dynamic intensity threshold 492 (eg, intensity threshold IT D ).exist Figure 4E In the example, after a delay time p2 has passed since the touch input 490 was initially detected, the trigger is triggered with the intensity threshold IT L At the same time, the dynamic intensity threshold 492 decays after a predefined delay time p1 has passed since the touch input 490 was initially detected. L Reducing the intensity of the touch input 490 after the associated response and then increasing the intensity of the touch input 490 without releasing the touch input 490 may trigger the touch input 490 to be in the same order as the intensity threshold IT. D associated response (eg, at time 494), even when the intensity of touch input 490 is below another intensity threshold (eg, intensity threshold IT L ) is also the case.

[0209] User interface and associated processes

[0210] Attention now turns to embodiments of a user interface (“UI”) and related processes that can be implemented on a computer system (e.g., portable multifunction device 100 or device 300) that includes a display generating component (e.g., a display, a projector, a heads-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a real-time preview of at least a portion of the content within the camera's field of view and, optionally, generates a video output comprising one or more streams of image frames capturing the content within the camera's field of view), and one or more input devices (e.g., a touch-sensitive surface, such as a touch-sensitive remote control, or a touch screen display that also serves as a display generating component, a mouse, a joystick, a stylus controller, and / or a camera that tracks one or more features of a user, such as the position of the user's hands), optionally one or more gesture sensors, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, and optionally one or more tactile output generators (and / or communicates with these components).

[0211] Figures 5A1-5A40 An example user interface is shown for displaying an augmented reality environment and adjusting the appearance of the augmented reality environment and / or the appearance of objects in the augmented reality environment in response to various inputs, as well as switching between viewing a virtual model in the augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figures 6A-6D 、 Figures 7A-7C ,and Figures 8A-8CFor ease of explanation, some embodiments of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, in response to detecting contact on the touch-sensitive surface of the input device 5010 while displaying the user interface on the display of the headset 5008 and the focus indicator shown in the figure, optionally on a computer system having the headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 1 ), a focus indicator is displayed. Figure 5A2 ) perform similar operations.

[0212] Figures 5A1-5A27 An exemplary user interface is shown for displaying an augmented reality environment and adjusting the appearance of the augmented reality environment and / or the appearance of objects in the augmented reality environment in response to different inputs according to some embodiments.

[0213] Figure 5A1-Figure 5A2 Shown is the use of Figure 5A3-Figure 5A40 The context of the user interface being described.

[0214] Figure 5A1 50. The physical space in which the user 5002, the table 5004, and the physical building model 5006 are located is shown. The user 5002 holds the device 100 to view the physical building model 5006 via the display of the device 100 (e.g., on the touch-sensitive display system 112, sometimes referred to as the "touch screen display 112," "touch screen 112," "display 112," or "touch-sensitive display 112" of the device 100, as shown in FIG. Figure 1A 、 Figure 4A 、 Figure 5A4 ). One or more cameras of device 100 (sometimes referred to as the “camera” of device 100) continuously provide a real-time preview of content within the camera’s field of view, including one or more physical objects in the physical space (e.g., wallpaper 5007, table 5004, etc. in a room in the physical space). Device 100 displays an augmented reality environment that includes a representation of at least a portion of the camera’s field of view, the camera’s field of view including a physical object (e.g., a physical building model 5006) and one or more virtual objects (e.g., a virtual model of a building overlaying the physical building model 5006, a virtual tree, etc.), and user 5002 interacts with the augmented reality environment using the touch screen display of device 100.

[0215] Figure 5A2An alternative approach is shown in which a user 5002 views a physical building model 5006 using a computer system that includes a headset 5008 and a separate input device 5010 having a touch-sensitive surface. In this embodiment, the headset 5008 displays an augmented reality environment, and the user 5002 interacts with the augmented reality environment using the separate input device 5010. In some embodiments, the device 100 is used as the separate input device 5010. In some embodiments, the separate input device 5010 is a touch-sensitive remote control, a mouse, a joystick, a stylus controller, etc. In some embodiments, the separate input device 5010 includes one or more cameras that track the position of one or more features of the user 5002, such as the user's hands and movements.

[0216] Figure 5A3-5A4 A view of an augmented reality environment displayed on the touch screen 112 of the device 100 is shown. Figure 5A3 The position of the device 100 relative to the table 5004 and the physical building model 5006 is shown from the perspective of the user 5002. Figure 5A4 Shown from Figure 5A3 . A close-up view of device 100 is provided. Device 100 displays an augmented reality environment that includes a real-time view of a physical space captured by a camera of device 100 and a virtual user interface object (virtual building model 5012). Here, virtual building model 5012 is a 3D virtual model of physical building model 5006 that appears to be attached to or overlaying physical building model 5006 in the camera's field of view (e.g., replacing physical building model 5006 in the augmented reality environment). The displayed augmented reality environment also includes virtual objects that do not correspond to physical objects in the camera's field of view (e.g., virtual trees, virtual bushes, virtual people, and virtual cars) and physical objects that are located in the camera's field of view (e.g., wallpaper 5007). In some embodiments, device 100 displays one or more buttons (e.g., button 5014, button 5016, and button 5018, sometimes referred to as virtual buttons or displayed buttons) for interacting with the augmented reality environment (e.g., as described below with respect to Figure 5A25-Figure 5A27 described above).

[0217] Figure 5A5-Figure 5A6 Shown after user 5002 has moved from in front of table 5004 (e.g., Figure 5A3 ) to the side of table 5004 (e.g., as shown in Figure 5A5 ), different views of the augmented reality environment are displayed on the touch screen 112 of the device 100. Figure 5A5 The position of the device 100 relative to the table 5004 and the physical building model 5006 is shown from the perspective of the user 5002. Figure 5A6 Shown from Figure 5A5A close-up view of the device 100 is shown. Figure 5A5-Figure 5A6 As shown in , the virtual building model 5012 remains anchored to the physical building model 5006, and the view of the virtual building model 5012 changes as the position, shape, and / or orientation of the physical building model 5006 changes in the camera's field of view.

[0218] Figures 5A7-5A14 The appearance of the virtual building model 5012 in the augmented reality environment is shown to be adjusted based on a combination of movement of contacts on the touch screen 112 and movement of the device 100. Reference box 5019 shows the position of the device 100 relative to the table 5004 and the physical building model 5006 from the perspective of the user 5002.

[0219] exist Figure 5A7 , when the device 100 is in a first position relative to the table 5004 and the physical building model 5006 (e.g., as shown by reference box 5019), the device 100 displays the augmented reality environment. Figure 5A8 , the device 100 detects an input on the virtual building model 5012 (eg, by detecting a touch input of a contact 5020 - a on the roof of the virtual building model 5012 ). Figure 5A9-Figure 5A11 , while continuing to detect the input (e.g., while the contact 5020 remains on the touch screen 112), the device 100 detects movement of the input relative to the physical building model 5006 (e.g., a drag gesture by the contact 5020), and adjusts the appearance of the virtual building model 5012 (e.g., raising the virtual roof 5012-a upward from the virtual building model) according to the magnitude of the movement of the input relative to the physical building model 5006. Figure 5A9 In FIG, when contact 5020-b moves a relatively small amount, virtual roof 5012-a is lifted a corresponding small amount. Figure 5A10 In some embodiments, as contact 5020-c moves a larger amount, virtual roof 5012-a is lifted a correspondingly larger amount. Figure 5A11 As shown in FIG, as the virtual roof 5012-a continues to rise, the floors of the virtual building model 5012 rise and expand (for example, showing a virtual first floor 5012-d, a virtual second floor 5012-c, and a virtual third floor 5012-b). Figure 5A9-Figure 5A11 As shown in , as the contact 5020 moves upward, the device 100 updates the display of the virtual building model 5012 so as to maintain the display of the initial contact point on the virtual roof 5012 - a at the location of the contact 5020 .

[0220] exist Figure 5A12-Figure 5A13In FIG. 5 , while contact 5020-d is maintained and remains stationary on touch screen 112, device 100 detects movement of device 100 in physical space (e.g., movement 5022, from Figure 5A12 The first position shown in the reference frame 5019 in FIG. 5 is lower relative to the physical building model 5006 to the first position shown in FIG. Figure 5A13 In response to the input movement (from the movement of the device 100 in the physical space), the device 100 adjusts the appearance of the virtual building model 5012 by raising the virtual roof 5012-a further upward according to the magnitude of the movement. In some embodiments, as Figure 5A13 As shown in , when the appearance of the virtual model 5012 is adjusted according to the magnitude of the movement, the virtual roof 5012 - a is displayed at a position that is beyond the maximum limit of the static state of the virtual roof 5012 - a.

[0221] exist Figure 5A14 In some embodiments, the device 100 stops detecting the input (e.g., contact 5020 is lifted off) and displays the virtual roof 5012-a at a position corresponding to the maximum limit of the static state. Figure 5A13 ) to the virtual roof 5012-a corresponding to the stationary state (e.g., Figure 5A14 ) to the maximum position of the animation transition (for example, from Figures 5A13 to 5A14 ).

[0222] Figure 5A15-Figure 5A16 5024) in physical space when no input is detected on the touch screen 112 (e.g., no touch input is detected on the touch screen 112). Since no input is detected, the movement of the device 100 is from a first position that is lower relative to the physical building model 5006 (e.g., as shown in FIG. Figure 5A15 5019 in the reference frame 5019) to a second position that is higher relative to the physical building model 5006 (e.g., as shown in FIG. Figure 5A16 5019 in the reference box 5019) changes the field of view of the device 100 camera without adjusting the appearance of the virtual building model 5012.

[0223] and Figure 5A15-Figure 5A16 compared to, Figure 5A17-Figure 5A185026-a is detected on the touch screen 112). Figure 5A17 The first position shown in the reference frame 5019 in FIG. 5 is lower relative to the physical building model 5006 to the first position shown in FIG. Figure 5A18 In response to the input movement (from movement of the device 100 in the physical space), the device 100 adjusts the appearance of the virtual building model 5012 by raising the virtual roof 5012-a upward according to the magnitude of the movement.

[0224] exist Figure 5A19-Figure 5A20 , while continuing to detect the input (e.g., while the contact 5026 remains on the touch screen 112), the device 100 detects movement of the input relative to the physical building model 5006 (e.g., a drag gesture performed by the contact 5026), and adjusts the appearance of the virtual building model 5012 according to the amount of movement of the input relative to the physical building model 5006 (e.g., raising the virtual roof 5012-a further upward from the virtual building model 5012). In some embodiments, such as Figure 5A20 As shown in FIG, as the virtual roof 5012-a continues to rise, the floors of the virtual building model 5012 rise and expand (eg, showing the first floor 5012-d, the second floor 5012-c, and the third floor 5012-b).

[0225] like Figures 5A17-5A20 As shown, as the input moves upward (whether the movement of the input is due to movement of the device 100 while the contact (e.g., contact 5026-a) is maintained and remains stationary on the touch screen 112, or the movement of the input is due to movement of the contact across the touch screen 112 while the device 100 remains substantially stationary in physical space), the device 100 updates the display of the virtual building model 5012 so that the initial contact point remains displayed on the virtual roof 5012-a at the location of contact 5026.

[0226] Figures 5A21-5A24 Changing virtual environment settings for an augmented reality environment (e.g., time of day) in response to input of navigating through time in the augmented reality environment is shown. Figures 5A21-5A24, device 100 detects input that changes a setting of the virtual environment (e.g., a left-to-right swipe gesture by contact 5030), and in response, device 100 changes the time of day in the augmented reality environment (e.g., by adjusting the appearance of virtual building model 5012 and applying a filter to portions of the camera's field of view that are not obscured by virtual building model 5012). Figure 5A21 , the time of day in the augmented reality environment is morning, and the shadow of the virtual building model 5012 and the shadows of the virtual objects (e.g., virtual trees, virtual bushes, virtual people, and virtual cars) are located to the right of the objects. When the contact 5030 moves from left to right, the time of day in the augmented reality environment changes from morning to evening (e.g., based on the speed and / or distance of the input movement) (e.g., from Figure 5A21 The morning in Figure 5A22 Noon to Figure 5A23 In the afternoon, Figure 5A24 In some embodiments, in addition to adjusting the appearance of the virtual scene, device 100 also applies a filter to the portion of the live view that is not obscured by the virtual scene (e.g., wallpaper 5007). Figure 5A24 In the embodiment of the present invention, in addition to adjusting the appearance of the virtual scene for night mode (e.g., as shown by the second shading pattern), a different filter is applied to the wallpaper 5007 (e.g., as shown by the first shading pattern) in addition to adjusting the appearance of the virtual scene for night mode (e.g., as shown by the second shading pattern).

[0227] Figure 5A25-Figure 5A27 Changing virtual environment settings for an augmented reality environment in response to input (e.g., a tap input on a displayed button) to switch between different virtual environments of a virtual user interface object (e.g., a virtual building model 5012), where the different virtual environments are associated with different interactions for exploring the virtual user interface object (e.g., predefined virtual environments such as landscape view, interior view, day / night view). Figure 5A25 , the landscape button 5014 is selected, and a landscape view of the virtual building model 5012 is displayed (e.g., with virtual trees, virtual bushes, virtual people, and virtual cars). Figure 5A26-Figure 5A27, the device 100 detects input on the interior button 5016, such as a tap gesture of the contact 5032, and in response, displays an interior view of the virtual building model 5012 (e.g., without virtual trees, without virtual bushes, without virtual people, without virtual cars, but instead displays an expanded view of the virtual building model 5012, with a virtual first floor 5012-d, a virtual second floor 5012-c, a virtual third floor 5012-b, and a virtual roof 5012-a). In some embodiments, when the virtual environment setting is changed (e.g., to an interior view), the surrounding physical environment is blurred (e.g., using a filter). For example, although Figure 5A27 Not shown, but in some embodiments, when the virtual environment setting is changed to an interior view, the wallpaper 5007 is blurred.

[0228] Figures 5A28-5A40 An example user interface is shown for switching between viewing a virtual model in an augmented reality environment and viewing a simulated view of the virtual model from the perspective of an object in the virtual model, according to some embodiments.

[0229] picture Figure 5A4 That way, Figure 5A28 A view of an augmented reality environment displayed on the touch screen 112 of the device 100 is shown, including a real-time view of the physical space captured by the camera of the device 100, a virtual building model 5012, a virtual vehicle 5050, and a virtual person 5060. In addition, Figure 5A28 The reference frame 5019 in FIG is viewed from the perspective of the user 5002 (e.g., as Figure 5A1 and Figure 5A2 ) shows the position of the device 100 relative to the table 5004 and the physical building model 5006.

[0230] Figures 5A29-5A31 Shown from Figure 5A28 Specifically, Figures 5A29-5A31 A view from an augmented reality environment (e.g., Figure 5A28 ) to a transition of a simulated view of the virtual model from the perspective of the virtual vehicle 5050 in the virtual model.

[0231] Figure 5A29 Input 5052 is shown detected at a location corresponding to vehicle 5050 (e.g., a tap gesture on touch screen 112 of device 100, or a selection using a separate input device along with a focus indicator).

[0232] Figure 5A30-Figure 5A31 A transition from a view of the augmented reality environment to a simulated view of the virtual model from the perspective of the vehicle 5050 is shown in response to detecting input 5052. Specifically, Figure 5A30 Shown in Figure 5A29the view displayed on device 100 during an animated transition from the illustrated view to a simulated perspective view of vehicle 5050 (e.g., from the perspective of a person within vehicle 5050, such as a driver or passenger), and Figure 5A31 A simulated perspective view from the perspective of vehicle 5050 is shown.

[0233] In some embodiments, the transition from the view of the augmented reality environment to the simulated perspective view includes an animated transition. Optionally, the transition includes an animation of flying from the location of viewing the augmented reality environment to the location of the vehicle 5050 (e.g., the location of the person inside the vehicle 5050). For example, Figure 5A30 Shown from Figure 5A29 A view of the virtual model of the position between the user's position and the position of the vehicle 5050 (e.g., partially through an animated transition) even if the user has not yet moved the device 100 (e.g., Figure 5A30 The position of the device 100 shown in the reference frame 5019 relative to the physical building model 5006 is the same as Figure 5A29 (same as in ).

[0234] In some embodiments, a portion of the field of view of the device 100 (e.g., the camera of the device 100) continues to be displayed during the animated transition to the perspective view from the vehicle 5050. For example, Figure 5A30 As shown, the edges of the wallpaper 5007 and the table 5004 are displayed during the animated transition to the simulated perspective view (e.g., as if moving from Figure 5A29 Between the user position in and the position of the vehicle 5050, corresponding to Figure 5A30 In some embodiments, the camera's field of view is stopped from being displayed during the animated transition to the perspective view from the vehicle 5050 (e.g., the edges of the wallpaper 5007 and the table 5004 are not displayed during the animated transition, and optionally, corresponding portions of the virtual model are displayed instead).

[0235] exist Figure 5A31 , the simulated perspective view from the vehicle 5050 also shows controls 5054, including directional arrows (up, down, left, and right) for controlling movement (e.g., direction of movement) of the vehicle 5050 (e.g., a virtual object displaying the simulated perspective view). Figure 5A31 In the illustrated embodiment, the upward arrow 5056 controls forward movement of the vehicle 5050. Thus, in some embodiments, the user can control the movement of a corresponding virtual object (e.g., the vehicle 5050) while displaying a simulated view from the perspective of the virtual object. In some embodiments, the user cannot control the movement of a corresponding virtual object (e.g., the virtual vehicle 5050 and / or the virtual person 5060) in the virtual model while displaying a view of the augmented reality environment. For example, in some embodiments, the user cannot Figure 5A28In some embodiments, the vehicle 5050 moves autonomously in the virtual model while displaying the augmented reality environment (e.g., Figure 5A28 ) view.

[0236] Figure 5A32-Figure 5A33 Shown from Figure 5A31 Specifically, Figure 5A32-Figure 5A33 User-controlled movement of vehicle 5050 in the virtual model is shown. Figure 5A32 An upward arrow 5056 (shown in FIG. Figure 5A31 In response to input 5058 on up arrow 5056, vehicle 5050 moves forward in the virtual model. Figure 5A33 , a simulated perspective view of an update of the virtual model corresponding to the forward movement of the vehicle 5050 in the virtual model is shown. Figure 5A33 In the updated simulation perspective in FIG, fewer virtual building models 5012 are visible, and the person 5060 appears smaller than Figure 5A32 Closer.

[0237] Figure 5A34-Figure 5A35 Shown from Figure 5A33 Specifically, Figure 5A34-Figure 5A35 A transition from a simulated view of the virtual model from the perspective of vehicle 5050 to a simulated view of the virtual model from the perspective of virtual person 5060 is shown. Figure 5A34 Input 5062 is shown detected at a location corresponding to person 5060. Figure 5A35 A simulated view of the virtual model from the perspective of person 5060 is shown, displayed in response to detecting input 5062. In some embodiments, device 100 displays an animated transition between the simulated perspective from vehicle 5050 and the simulated perspective from person 5060 (e.g., as if the user were moving from the position of vehicle 5050 (e.g., within vehicle 5050) to the position of person 5060).

[0238] Figure 5A36-Figure 5A37 Shown from Figure 5A35 Specifically, Figure 5A36-Figure 5A37 The view of the virtual model is shown changing from the perspective of person 5060 (eg, the selected virtual object) in response to movement of device 100 (eg, in physical space).

[0239] Figure 5A36 Arrow 5064 indicating movement of device 100 to the left is shown, as well as rotation of device 100 about the z-axis (eg, such that the right edge of device 100 is closer to the user and the left edge of device 100 moves further away from the user). Figure 5A37 An updated simulated perspective view of the virtual model from the perspective of person 5060 is shown displayed in response to detecting movement of device 100. Figure 5A37 The updated simulated perspective in corresponds to the view of the virtual model as if person 5060 moved to the left and was Figure 5A36 The same position as above with his head slightly turned toward the right. Figure 5A37 Reference box 5019 in shows the new position of device 100 relative to physical building model 5006 after device 100 has moved as shown by arrow 5064.

[0240] In some embodiments, when displaying a simulated view from the perspective of person 5060, display controls 5054 (e.g., Figure 5A31 shown in, but Figure 5A35 、 Figure 5A36 ), so that when displaying a simulated view from the perspective of person 5060 (e.g., Figure 5A35 ) when the user uses the arrows on the control 5054 to control the movement of the person 5060 in the virtual model.

[0241] Figures 5A38-5A40 Shown from Figure 5A37 Specifically, Figures 5A38-5A40 Shown from Figure 5A37 Shown is a transition from a simulated perspective view back to a view of an augmented reality environment.

[0242] Figure 5A38 Input 5066 is shown. Figure 5A38 In the embodiment shown, input 5066 is a pinch gesture (e.g., the minimum zoom level of the simulated perspective view from the virtual model). In some embodiments, input 5066 is a gesture (e.g., a tap) on an "empty" location in the virtual model (e.g., a location where the simulated perspective view is not available, such as a patch of grass). In some embodiments, input 5066 is a gesture (e.g., a tap) on an affordance (e.g., an icon such as an "X" for exiting the simulated perspective view) for displaying or redisplaying the augmented reality environment.

[0243] Figure 5A39-Figure 5A40 A transition of a simulated view of a virtual model from the perspective of person 5060 to a view of an augmented reality environment is shown in response to detecting input 5066. Specifically, Figure 5A39 Shown in Figure 5A38 the view displayed on device 100 during the animated transition from the illustrated view to the view of the augmented reality environment, and Figure 5A40A view of the augmented reality environment is shown. In some embodiments, the transition from the simulated perspective to the view of the augmented reality environment includes an animated transition that optionally includes an animation of flying from the position of the virtual object (from which the simulated perspective is displayed) to the position from which the augmented reality environment is viewed.

[0244] Because the device 100 is Figures 5A38-5A40 The position relative to the physical building model 5006 is Figures 5A28-5A30 Different, so Figure 5A40 The augmented reality view shown corresponds to the new position of the device 100 and is different from Figure 5A28 Similarly, Figure 5A39 Shown from Figure 5A38 The location of Zhongren 5060 and Figure 5A40 5019 relative to the physical building model 5006, even if the user has not yet moved the device 100 (e.g., the position of the device 100 shown in reference box 5019 relative to the physical building model 5006). Figures 5A38-5A40 is the same in every image).

[0245] Similar to the reference above Figures 5A29-5A31 In some embodiments, a portion of the field of view of device 100 (e.g., a camera of device 100) is visible during the animated transition from the simulated perspective to the view of the augmented reality environment. Figure 5A39 As shown, wallpaper 5007 is displayed during the transition from the simulated perspective animation (e.g., as from Figure 5A40 Between the position of person 5060 and the user's position, corresponding to Figure 5A39 In some embodiments, the camera's field of view is stopped from being displayed during the animated transition to the view of the augmented reality environment (e.g., wallpaper 5007 is not displayed during the animated transition, and optionally, a corresponding portion of the virtual model is displayed instead).

[0246] Figure 5B1-Figure 5B41 Examples of systems and user interfaces for three-dimensional manipulation of virtual user interface objects are shown in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figures 6A-6D 、 Figures 7A-7C ,and Figures 8A-8CFor ease of explanation, some embodiments of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, in response to detecting contact on the touch-sensitive surface of the input device 5010 while displaying the user interface on the display of the headset 5008 and the focus indicator shown in the figure, optionally on a computer system having the headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 1 ), a focus indicator is displayed. Figure 5B2 ) perform similar operations.

[0247] Figure 5B1-Figure 5B4 Shown is the use of Figure 5B5-Figure 5B41 The context of the user interface being described.

[0248] Figure 5B1 A physical space 5200 is shown with a user 5202 and a table 5204. The device 100 is held by the user 5202 in the user's hand 5206. A reference pad 5208 is located on the table 5204.

[0249] Figure 5B2 A view of a virtual three-dimensional space displayed on the display 112 of the device 100 is shown. The reference mat 5208 is in the field of view of one or more cameras (e.g., optical sensor 164) of the device 100 (hereinafter referred to as "camera", which indicates one or more cameras of the device 100). The display 112 shows a real-time view of the physical space 5200 captured by the camera, including a displayed version 5208b of the physical reference mat 5208a. A virtual user interface object (virtual frame 5210) is displayed in the virtual three-dimensional space displayed on the display 112. In some embodiments, the virtual frame 5210 is anchored to the reference mat 5208b so that when the displayed view 5208b of the reference mat changes in response to movement of the reference mat 5208a in the physical space 5200, the view of the virtual frame 5210 will change (e.g., as Figure 5B2-5B3 ). Similarly, as the view of the displayed version 5208b changes in response to movement of the device 100 relative to the reference pad 5208a, the view of the virtual frame 5210 will change.

[0250] exist Figure 5B3 , reference pad 5208 has been rotated so that the longer side of reference pad 5208a is adjacent to device 100 (and Figure 5B2 , the shorter side of the reference mat 5208a is adjacent to the device 100). Rotation of the reference mat 5208a in the physical space 5200 causes the displayed version 5208b of the reference mat to rotate from Figure 5B2 to Figure 5B3 Rotate.

[0251] exist Figure 5B3-Figure 5B4, the device 100 has been moved closer to the reference mat 5208a. As a result, the size of the displayed version 5208b of the reference mat and the virtual frame 5210 have increased.

[0252] Figure 5B5-Figure 5B41 A larger view of the device 100 is shown, and in order to provide a complete view of the user interface displayed on the display 112, the user's hand 5206 is not shown.

[0253] Figure 5B5 A user interface for creating and adjusting virtual user interface objects is shown displayed on display 112. The user interface includes an avatar 5212, a switch 5214 (e.g., for switching between a virtual reality display mode and an augmented reality display mode), a new object control 5216 (e.g., for adding a new object 5216 to the virtual three-dimensional space displayed by display 112), a color selection palette 5218 including several controls corresponding to available colors (e.g., for selecting a color for a virtual object), and a delete control 5220 (e.g., for removing a virtual user interface object from the virtual three-dimensional space). Figure 5B5 , switch 5214 indicates that the current display mode is an augmented reality display mode (eg, display 112 is displaying a virtual frame 5210 and a view of the physical space 5200 captured by a camera of device 100). Figure 5B37-Figure 5B39 shows a virtual reality display mode. Figure 5B37-Figure 5B39 , changes the appearance of switch 5214 to indicate that virtual reality display mode is active (and that input at switch 5214 will cause a transition from virtual reality display mode to augmented reality display mode).

[0254] Figure 5B6-Figure 5B17 Input that causes the virtual frame 5210 to move is shown.

[0255] exist Figure 5B6 , input (e.g., selection and movement input) is detected by contact 5222 (e.g., contact with the touch-sensitive display 112) on the first surface 5224 of the virtual frame 5210. When the surface of the virtual frame 5210 is selected, the movement of the virtual frame 5210 is limited to movement in a plane parallel to the selected surface. In response to detecting the contact 5222 selecting the first surface 5224 of the virtual frame 5210, a movement protrusion 5226 extending from the virtual frame 5210 is shown to indicate the movement plane of the virtual frame 5210 (e.g., a movement plane parallel to the selected first surface 5224 of the virtual frame 5210).

[0256] exist Figure 5B6-Figure 5B7, contact 5222 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5228. In response to the movement of contact 5222, virtual frame 5210 has moved in the direction indicated by arrow 5228 within the plane indicated by movement protrusion 5226. Figure 5B7-Figure 5B8 , contact 5222 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5230. In response to the movement of contact 5222, virtual frame 5210 has moved in the direction indicated by arrow 5230 within the plane indicated by movement protrusion 5226. Figure 5B9 , contact 5222 has been lifted off of touch-sensitive display 112 and movement protrusion 5226 is no longer displayed.

[0257] exist Figure 5B10 , input (e.g., selection and movement input) made by contact 5232 is detected on second surface 5234 of virtual frame 5210. In response to detecting contact 5232 selecting second surface 5234 of virtual frame 5210, movement protrusion 5236 extending from virtual frame 5210 is shown to indicate a movement plane of virtual frame 5210 (e.g., a movement plane parallel to the selected second surface 5234 of virtual frame 5210).

[0258] exist Figure 5B10-Figure 5B11 , contact 5232 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5238. In response to the movement of contact 5232, virtual frame 5210 has moved in the direction indicated by arrow 5238 within the plane indicated by movement protrusion 5236. As virtual frame 5210 moves upward so that it hovers over displayed reference pad 5208 b, a shadow 5240 of virtual frame 5210 is displayed to indicate that virtual frame 5210 is hovering.

[0259] exist Figure 5B11-Figure 5B12 , contact 5232 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5242. In response to the movement of contact 5232, virtual frame 5210 has moved in the direction indicated by arrow 5242 within the plane indicated by movement protrusion 5236. Figure 5B13 , contact 5232 has been lifted off of touch-sensitive display 112 and movement protrusion 5236 is no longer displayed.

[0260] exist Figure 5B14, input (e.g., selection and movement input) by contact 5233 is detected on first surface 5224 of virtual frame 5210. In response to detecting contact 5233 selecting first surface 5224 of virtual frame 5210, movement protrusion 5237 extending from virtual frame 5210 is shown to indicate a movement plane of virtual frame 5210 (e.g., a movement plane parallel to the selected first surface 5224 of virtual frame 5210).

[0261] exist Figure 5B14-Figure 5B15 , contact 5233 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5239. In response to the movement of contact 5233, virtual frame 5210 has moved within the plane indicated by movement protrusion 5237 in the direction indicated by arrow 5238. Figure 5B10-Figure 5B11 The movement of contact 5232 shown in Figure 5B14-Figure 5B15 The movement direction of the contact 5233 shown in FIG is the same as that of the contact 5233. Since the movement of the contact 5232 occurs while the second surface 5234 of the virtual frame 5210 is selected, Figure 5B10-Figure 5B11 The moving plane of the virtual frame 5210 in the example is different from Figure 5B14-Figure 5B15 5224 of the virtual frame 5210, wherein the movement of the contact 5233 occurs while the first surface 5224 of the virtual frame 5210 is selected. In this way, the selection and movement inputs with the same input movement direction result in different movements of the virtual frame 5210 depending on the surface of the virtual frame 5210 selected.

[0262] exist Figure 5B15-Figure 5B16 , contact 5233 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5243. In response to the movement of contact 5233, virtual frame 5210 has moved in the direction indicated by arrow 5243 within the plane indicated by movement protrusion 5237. Figure 5B17 , contact 5233 has been lifted off of touch-sensitive display 112 and movement protrusion 5237 is no longer displayed.

[0263] Figure 5B18-Figure 5B21 Input for resizing the virtual frame 5210 is shown.

[0264] exist Figure 5B18 , an input (e.g., a resizing input) made by a contact 5244 is detected on the first surface 5224 of the virtual frame 5210. In some embodiments, when the contact remains at a location corresponding to the surface of the virtual object for a time period that increases to above a resizing time threshold, subsequent movement of the contact (and / or movement of the device 100) causes the virtual object to be resized. Figure 5B19, contact 5244 has remained in contact with the first surface 5224 of the virtual box 5210 for a period of time that has increased beyond a resizing time threshold, and a resizing protrusion 5246 is shown indicating an axis (which is perpendicular to the selected first surface 5224) along which the size of the virtual box 5210 will be adjusted in response to subsequent movement of contact 5244.

[0265] exist Figure 5B19-Figure 5B20 , contact 5244 has moved along the path indicated by arrow 5248. In response to the movement of contact 5244, the size of virtual frame 5210 has increased along the axis indicated by resizing protrusion 5246 in the direction indicated by arrow 5248. Figure 5B21 , contact 5244 has been lifted off of touch-sensitive display 112 and protrusion 5246 is no longer displayed.

[0266] Figure 5B22-Figure 5B27 The use of the insertion cursor to place an object and the placement virtual box are shown.

[0267] exist Figure 5B22 In FIG, an input (e.g., a tap input) by contact 5250 is detected at a location corresponding to displayed version 5208b of physical reference pad 5208a. In response to detecting contact 5250, insertion cursor 5252 is displayed on display 112 at a location corresponding to contact 5250; Figure 5B23 , contact 5250 has been lifted from touch-sensitive display 112, and insertion cursor 5252 is shown. In some embodiments, insertion cursor 5252 ceases to be displayed after a predetermined period of time. Figure 5B24 , the insertion cursor 5252 has stopped being displayed, and an input (e.g., a tap input) by the contact 5254 is detected at a position different from the position where the insertion cursor 5252 was displayed (e.g., Figure 5B23 In response to detecting contact 5254, a new insertion cursor 5256 is displayed on display 112 at a position corresponding to contact 5254. Figure 5B25 , contact 5254 has been lifted off of touch-sensitive display 112 and insertion cursor 5256 is shown.

[0268] exist Figure 5B26 , the insertion cursor 5256 has stopped being displayed, and an input (e.g., a tap input) by contact 5258 is detected at a position corresponding to the position where the insertion cursor 5256 was displayed (e.g., Figure 5B25 In response to detecting contact 5258 at the location where the insertion cursor has been placed, a new virtual user interface object (virtual frame 5260) is displayed on display 112 at a location corresponding to contact 5258. Figure 5B27, contact 5258 has been lifted off the touch-sensitive display 112 .

[0269] Figure 5B28-Figure 5B31 Adjustment of the size of the virtual frame 5260 by movement of the device 100 is shown.

[0270] exist Figure 5B28 , input (e.g., resizing input) made by contact 5262 with the touch-sensitive display 112 is detected on the surface 5264 of the virtual frame 5260. Figure 5B29 , contact 5262 has remained in contact with surface 5264 of virtual frame 5260 for a period of time that has increased beyond a resize time threshold, and a resize highlight 5266 is shown indicating an axis (which is perpendicular to the selected surface 5264) along which the virtual frame 5260 will be resized in response to subsequent movement of device 100. Figure 5B29-Figure 5B30 , device 100 moves along the path indicated by arrow 5268 while contact 5262 remains in contact with touch-sensitive display 112. In response to the movement of device 100, the size of virtual frame 5260 increases along the axis indicated by resizing protrusion 5266, as shown in FIG. Figure 5B30 As shown in . Figure 5B31 , contact 5262 has been lifted from touch-sensitive display 112 and resizing highlight 5266 is no longer displayed.

[0271] Figure 5B32-Figure 5B35 Use of the new object control 5216 to insert a new virtual object is shown.

[0272] exist Figure 5B32 , an input (e.g., a tap input) by contact 5270 is detected at a location on displayed version 5208b of physical reference pad 5208a. In response to detecting contact 5270, insertion cursor 5272 is displayed on display 112 at a location corresponding to contact 5270. Figure 5B33 , contact 5270 has been lifted off of touch-sensitive display 112, and insertion cursor 5272 is shown. Figure 5B34 , the insertion cursor 5272 has ceased to be displayed, and an input (e.g., a tap input) is detected by contact 5274 with the touch-sensitive display 112 at a location corresponding to the new object control 5216. Figure 5B35 In response to input at the new object control 5216 (e.g., after placing the insertion cursor 5272), a new virtual user interface object (virtual frame 5276) is displayed on the display 112 at a location corresponding to the location where the insertion cursor 5272 is displayed.

[0273] Figure 5B36-Figure 5B37 A pinch-to-zoom input is shown causing a transition from an augmented reality display mode to a virtual reality display mode. Figure 5B39-Figure 5B40 An input is shown at switch 5214 for returning from virtual reality display mode to augmented reality display mode.

[0274] exist Figure 5B36 , contacts 5278 and 5280 are detected simultaneously with the touch-sensitive display 112. A pinch gesture is detected in which contacts 5278 and 5280 move simultaneously along the paths indicated by arrows 5282 and 5284, respectively, as shown in FIG. Figure 5B36-Figure 5B37 In response to detecting the pinch gesture, the display of virtual frames 5210, 5260, and 5276 is zoomed (e.g., zoomed out so that the displayed size of virtual frames 5210, 5260, and 5276 becomes smaller). In some embodiments, the gesture for zooming causes a transition from an augmented reality display mode to a virtual reality display mode (e.g., because the zoomed view of the frames is no longer aligned with the field of view of the camera of device 100). In some embodiments, in the virtual reality display mode, physical objects in the field of view of the camera of device 100 (e.g., reference mat 5208) cease to be displayed, or virtual (rendered) versions of one or more physical objects are displayed.

[0275] In some embodiments, in virtual reality display mode, virtual objects displayed by device 100 are locked to the reference frame of device 100. Figure 5B37-Figure 5B38 , the position of the device 100 has changed. Because the device 100 is in the virtual reality display mode, the positions of the virtual frames 5210, 5260, and 5276 do not change in response to the changed position of the device 100.

[0276] exist Figure 5B39 , an input (e.g., a tap input) by contact 5286 is detected at a position corresponding to switch 5214. In response to the input made by contact 5286, a transition occurs from the virtual reality display mode to the augmented reality display mode. Figure 5B40 5286. The user interface displayed on display 112 after transitioning to an augmented reality display mode in response to input made by contact 5286 is shown. The transition includes redisplaying the field of view of the camera of device 100 (e.g., redisplaying displayed view 5208b of the reference pad). In some embodiments, the transition includes scaling (e.g., zooming in) the display of virtual frames 5210, 5260, and 5276 (e.g., to realign the frames with the field of view of the camera of device 100).

[0277] In some embodiments, in augmented reality display mode, virtual objects displayed by device 100 are locked to physical space 5200 and / or physical objects in physical space 5200 (e.g., reference mat 5208). Figure 5B40-Figure 5B41, the position of device 100 has changed. Because device 100 is in augmented reality display mode, virtual frames 5210, 5260, and 5276 are locked to reference pad 5208a, and the positions of the virtual frames on display 112 change in response to the changed position of device 100.

[0278] Figure 5C1-Figure 5C30 Examples of systems and user interfaces for switching between viewing modes of a displayed simulated environment are shown in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figures 10A-10E For ease of explanation, some embodiments of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, in response to detecting contact on the touch-sensitive surface of the input device 5010 while displaying the user interface on the display of the headset 5008 and the focus indicator shown in the figure, optionally on a computer system having the headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 1 ), a focus indicator is displayed. Figure 5A2 ) perform similar operations.

[0279] Figure 5C1-Figure 5C2 Shown is the use of Figure 5C3-Figure 5C30 The context of the user interface being described.

[0280] Figure 5C1 A physical space 5200 in which a user and a table 5204 are located is shown. The device 100 is held by the user in the user's hand 5206. A reference mat 5208 is located on the table 5204. A view of the simulated environment is displayed on the display 112 of the device 100. The reference mat 5208 is in the field of view of one or more cameras (e.g., optical sensor 164) of the device 100 (hereinafter referred to as "camera", which indicates one or more cameras of the device 100). The display 112 shows a real-time view of the physical space 5200 captured by the camera, including a displayed version 5208b of the physical reference mat 5208a. Two virtual user interface objects (a first virtual frame 5302 and a second virtual frame 5304) are displayed in the simulated environment displayed on the display 112. In the first viewing mode (e.g., augmented reality viewing mode), the virtual frames 5302 and 5304 are anchored to the reference mat 5208b such that as the displayed view 5208b of the reference mat changes in response to movement of the reference mat 5208a in the physical space 5200, the views of the virtual frames 5302 and 5304 will change (e.g., maintaining a fixed spatial relationship between the virtual frames 5302 and 5304 and the physical environment, including the reference mat 5208a). Similarly, in the first viewing mode, the views of the virtual frames 5302 and 5304 change in response to movement of the device 100 relative to the reference mat 5208a.

[0281] exist Figure 5C2 , the device 100 has been moved closer to the reference mat 5208a. As a result, the size of the displayed version 5208b of the reference mat and the virtual frames 5302 and 5304 have increased.

[0282] Figure 5C3-Figure 5C30 A larger view of the device 100 is shown, and in order to provide a complete view of the user interface displayed on the display 112, the user's hand 5206 is not shown. Figure 5B5 The features of the user interface are further described.

[0283] Figure 5C4-5C6 Input gestures (including swipe up and swipe down) for moving the virtual frame 5302 when the virtual frame is displayed in augmented reality viewing mode are shown. Figure 5C4-5C6 The input gesture is not a gesture that satisfies the mode change criteria (e.g., for changing the viewing mode from an augmented reality viewing mode to a virtual reality viewing mode), so the views of the virtual frames 5302 and 5304 change in response to subsequent movement of the device 100, as shown. Figure 5C7-Figure 5C8 as shown (e.g., such that a fixed spatial relationship is maintained between virtual frames 5302 and 5304 and the physical environment (including reference mat 5208a)).

[0284] Another example of a gesture that does not meet the mode change criteria is a resize gesture (e.g., as described above with reference to Figure 5B18-Figure 5B21 described above).

[0285] exist Figure 5C4 , input (e.g., selection and movement input) by contact 5306 is detected on surface 5308 of virtual frame 5302. In response to detecting contact 5306 selecting surface 5308 of virtual frame 5302, movement protrusion 5310 extending from virtual frame 5302 is shown to indicate a movement plane of virtual frame 5302 (e.g., a movement plane parallel to the selected surface 5308 of virtual frame 5302).

[0286] exist Figure 5C4-5C5 , contact 5306 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5312. In response to the movement of contact 5306, virtual frame 5302 has moved in the direction indicated by arrow 5312 within the plane indicated by movement protrusion 5310. As virtual frame 5302 moves upward so that it hovers over displayed reference pad 5208 b, a shadow 5314 of virtual frame 5302 is displayed to indicate that virtual frame 5210 is hovering.

[0287] exist Figure 5C5-Figure 5C6, contact 5306 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5316. In response to the movement of contact 5306, virtual frame 5302 has moved in the direction indicated by arrow 5316 within the plane indicated by movement protrusion 5310. Figure 5C7 , contact 5306 has been lifted off of touch-sensitive display 112 and movement protrusion 5310 is no longer displayed.

[0288] Figure 5C7-Figure 5C8 Movement of the device 100 is shown along a path indicated by arrow 5318. As the device 100 moves, the positions of the virtual frames 5302 and 5304 displayed by the device 100 change on the display 112 (e.g., such that a fixed spatial relationship is maintained between the virtual frames 5302 and 5304 and the reference mat 5208a in the physical environment of the device 100).

[0289] Figure 5C9-Figure 5C10 An input gesture (a pinch gesture) is shown that satisfies mode change criteria (eg, causes the viewing mode to change from an augmented reality viewing mode to a virtual reality viewing mode).

[0290] exist Figure 5C9 , contacts 5320 and 5324 are detected at touch-sensitive display 112. Figure 5C9-Figure 5C11 , contact 5320 moves along the path indicated by arrow 5322, and contact 5324 moves along the path indicated by arrow 5324. In response to the simultaneous movement of contacts 5320 and 5324 reducing the distance between contacts 5320 and 5324, the displayed view of the simulated environment (including virtual frames 5302 and 5304) is zoomed out (e.g., such that the size of virtual frames 5302 and 5304 increases on display 112). Upon receiving the zoom input, a transition occurs from the augmented reality viewing mode to the virtual reality viewing mode. The transition animation that occurs during the transition includes gradually fading out the displayed view of the physical environment. For example, the displayed view of table 5204 and the displayed view of reference mat 5208a captured by one or more cameras of device 100 gradually fade out (e.g., as shown in FIG. 2 ). Figure 5C10-Figure 5C11 The transition animation includes a gradual fading in of the virtual grid lines of the virtual reference grid 5328 (e.g., as shown in FIG. Figure 5C11-Figure 5C12 During the transition, the appearance of switch 5214 (e.g., for switching between virtual reality display mode and augmented reality display mode) is changed to indicate the current viewing mode (e.g., as shown in FIG. Figure 5C10-Figure 5C11 After contacts 5320 and 5324 are lifted off, virtual boxes 5302 and 5304 in the simulated environment continue to move and decrease in size (e.g., the change in the simulated environment continues to have "momentum," resulting in movement after the input gesture ends).

[0291] exist Figure 5C12-Figure 5C13 , the device 100 is moved along the path indicated by the arrow 5330. Figure 5C9-Figure 5C11 The pinch-to-zoom input gesture causes a change from an augmented reality viewing mode to a virtual reality viewing mode, so the positions of the virtual frames 5302 and 5304 do not change in response to movement of the device 100 (e.g., in the virtual reality viewing mode, no fixed spatial relationship is maintained between the virtual frames 5302 and 5304 and the physical environment).

[0292] exist Figure 5C13-Figure 5C14 , move the device 100 along the path indicated by arrow 5332.

[0293] Figure 5C15-Figure 5C18 An input for inserting a virtual frame into a simulated environment displayed on device 100 while the simulated environment is displayed in a virtual reality viewing mode is shown.

[0294] exist Figure 5C15 In the embodiment, an input (e.g., a tap input) made by contact 5334 is detected on the touch-sensitive display 112. In response to detecting contact 5334, an insertion cursor 5336 is displayed on the display 112 at a position corresponding to contact 5334, as shown in FIG. Figure 5C16 As shown in . Figure 5C17 , the insertion cursor 5336 has ceased to be displayed, and an input (e.g., a tap input) made by contact 5338 is detected at a location corresponding to the new object control 5216. Figure 5C18 In response to input at the new object control 5216 (e.g., after placing the insertion cursor 5336), a new virtual user interface object (virtual frame 5340) is displayed at a location corresponding to the location where the insertion cursor 5336 is displayed.

[0295] Figure 5C19-Figure 5C20 Inputs for manipulating a virtual user interface in a simulated environment displayed on device 100 while the simulated environment is displayed in a virtual reality viewing mode are shown.

[0296] exist Figure 5C19 , input (e.g., selection and movement input) made by contact 5342 is detected on surface 5344 of virtual frame 5340. In response to detecting contact 5342 selecting surface 5344 of virtual frame 5340, movement protrusion 5348 extending from virtual frame 5340 is shown to indicate the movement plane of virtual frame 5340 (e.g., a movement plane parallel to the selected surface 5344 of virtual frame 5340). Figure 5C19-Figure 5C20, contact 5342 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5346. In response to the movement of contact 5342, virtual frame 5340 has moved in the direction indicated by arrow 5346 within the plane indicated by movement protrusion 5348.

[0297] exist Figure 5C21 , contact 5342 has been lifted off of touch-sensitive display 112 and movement protrusion 5384 is no longer displayed.

[0298] Figure 5C22-Figure 5C23 An input gesture (eg, a rotation gesture) for changing the perspective of a simulated environment is shown.

[0299] exist Figure 5C22 , contact 5350 is detected at touch-sensitive display 112. Figure 5C22-Figure 5C23 , contact 5350 moves along the path indicated by arrow 5352. As contact 5350 moves, the simulated environment rotates. Figure 5C23 , the positions of the virtual reference grid 5328 and virtual frames 5302 , 5304 , and 5340 rotate in response to input made by contact 5350 .

[0300] exist Figure 5C24-Figure 5C25 In FIG. 5 , the device 100 is moved along the path indicated by the arrow 5354. Figure 5C24-Figure 5C25 The simulated environment displayed on the display 112 is displayed in a virtual reality viewing mode, so the positions of the virtual frames 5302 and 5304 on the display 112 do not change in response to movement of the device 100.

[0301] Figure 5C26-Figure 5C27 An input gesture (an expand gesture) is shown that causes the viewing mode to change from a virtual reality viewing mode to an augmented reality viewing mode.

[0302] exist Figure 5C26 , contacts 5356 and 5360 are detected at touch-sensitive display 112. Figure 5C26-Figure 5C27 , contact 5356 moves along the path indicated by arrow 5358, and contact 5360 moves along the path indicated by arrow 5362. In response to the simultaneous movement of contacts 5356 and 5360 increasing the distance between contacts 5356 and 5360, the displayed view of the simulated environment (including virtual frames 5302, 5304, and 5340) is zoomed in (e.g., such that the size of virtual frames 5302, 5304, and 5340 increases on display 112). Upon receiving the zoom input, a transition occurs from the virtual reality viewing mode to the augmented reality viewing mode. The transition animation that occurs during the transition includes gradually fading out the virtual reference grid 5328 (e.g., as shown in FIG. 2 ). Figure 5C26-Figure 5C27). The transition animation includes a gradual fading out of the view of the physical environment. For example, the table 5204 and the reference mat 5208a captured by one or more cameras of the device 100 gradually become visible on the display 112 (e.g., as shown). Figure 5C28-Figure 5C30 During the transition, the appearance of the switch 5214 is changed to indicate the current viewing mode (e.g., as shown in FIG. Figure 5C27-Figure 5C28 After contacts 5356 and 5360 are lifted off, virtual boxes 5302, 5304, and 5340 in the simulation environment continue to increase in size, move, and rotate (e.g., until the original space between virtual boxes 5302 and 5304 and reference pad 5208a is restored), as shown in FIG. Figure 5C28-Figure 5C30 shown.

[0303] In some embodiments, the virtual frame 5340 added when the virtual reality viewing mode is active is visible in the alternative reality viewing mode, e.g. Figure 5C30 shown.

[0304] In some embodiments, a change in viewing mode occurs from a virtual reality viewing mode to an augmented reality viewing mode in response to a contact input (e.g., a tap input) at a location corresponding to switch 5214. For example, a change in viewing mode occurs from displaying a virtual reality viewing mode (e.g., as shown in FIG. 5B ) to an augmented reality viewing mode (e.g., as shown in FIG. 5B ). Figure 5C26 ) to augmented reality viewing mode (e.g., as Figure 5C30 In some embodiments, during the transition, a transition animation that is the same as or similar to the animation shown at 5C26-5C30 is displayed.

[0305] Figure 5D1-Figure 5D1 4 shows an example of a system and user interface for updating an indication of a viewing perspective of a second computer system in a simulated environment displayed by a first computer system according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figures 11A-11C For ease of explanation, some embodiments of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, in response to detecting contact on the touch-sensitive surface of the input device 5010 while displaying the user interface on the display of the headset 5008 and the focus indicator shown in the figure, optionally on a computer system having the headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 1 ), a focus indicator is displayed. Figure 5A2 ) perform similar operations.

[0306] Figure 5D1-Figure 5D2 The context in which the user interface described with respect to Figures 5D3-5D14 is used is shown.

[0307] Figure 5D1 A physical space 5400 is shown with two users 5402 and 5408 and a table 5414. A first device 5406 (e.g., device 100) is held in the first user's hand 5404 by the first user 5402. A second device 5412 (e.g., device 100) is held in the second user's hand 5410 by the second user 5408. A reference pad 5416a is located on the table 5414.

[0308] Figure 5D2 A view of a virtual three-dimensional space displayed on a display 5148 (e.g., display 112) of a device 5406 is shown. The reference mat 5416 is in the field of view of one or more cameras (e.g., optical sensor 164) of the device 5406 (hereinafter referred to as "cameras," which indicates one or more cameras of the device 5406). The display 5148 shows a real-time view of the physical space 5400 captured by the cameras, including a displayed version 5416b of the physical reference mat 5416a. A virtual user interface object (virtual frame 5420) is displayed in a simulated environment displayed on the display 5418. In some embodiments, the virtual frame 5420 is anchored to the reference mat 5416b such that when the view of the displayed version 5416b of the reference mat changes in response to movement of the device 100 relative to the reference mat 5416b, the view of the virtual frame 5420 will change. Figure 5B5 The features of the user interface are further described.

[0309] Figures 5D3-5D11 include sub-figures "a," "b," and "c," with sub-figure a illustrating the orientation of the first device 5406 and the second device 5412 relative to the table 5414 in the physical space 5400 (e.g., as shown in FIG. Figure 5D3a ), sub-figure b shows the user interface of the first device 5412 (for example, as Figure 5D3b ), sub-figure c shows the user interface of the second device 5412 (eg, as Figure 5D3c 5D3-5D11 do not show the hands holding the device. Additionally, for clarity, the user interfaces in Figures 5D3-5D11 do not show the bodies of users 5402 and 5408. It should be understood that any part of the body of user 5408 that is within the field of view of the camera of device 5406 will generally be visible in the user interface displayed on device 5406 (although the view of the user's body may be obscured by virtual user interface objects or other user interface elements). For example, Figure 5D2 , the body and hands of user 5408 are visible in the user interface displayed by device 5409.

[0310] Figures 5D3-5D4 illustrate the movement of the second device 5412.

[0311] exist Figure 5D3a , the second device 5412 is displayed at a first position relative to the table 5414 (e.g., a position adjacent to the leftmost edge of the table).

[0312] exist Figure 5D3b , the user interface of device 5406 includes an avatar key 5422 that includes a key avatar 5424 corresponding to device 5406 and a key avatar 5426 corresponding to device 5412. Avatar key 5422 includes a name ("Me") corresponding to key avatar 5424 and a name ("Zoe") corresponding to key avatar 5426. The key avatars shown in the avatar key provide guidance to the avatars (e.g., avatar 5428) displayed in the visible environment, e.g., to help the user of device 5406 understand that avatar 5428 in the simulated environment corresponds to device 5412 of user "Zoe" (e.g., because avatar 5428 in the simulated environment is a cat icon that matches key avatar 5426).

[0313] The simulated environment displayed on the user interface of device 5406 includes a virtual frame 5420 and a displayed view 5416b of a physical reference pad 5416a presented from the perspective of device 5406. The viewing perspective of device 5412 is indicated by a viewing perspective indicator 5432. Viewing perspective indicator 5432 is shown emanating from avatar 5428. In the simulated environment, a representation 5430 of device 5412 (e.g., a view of device 5412 captured by a camera of device 5406 and / or a rendered version of device 5412) is shown.

[0314] exist Figure 5D3c , the user interface of device 5412 includes an avatar key 5434 that includes a key avatar 5436 corresponding to device 5412 and a key avatar 5468 corresponding to device 5406. Avatar key 5434 includes a name ("Me") corresponding to key avatar 5436 and a name ("Gabe") corresponding to key avatar 5438. The key avatars shown in the avatar key provide guidance to the avatars (e.g., avatar 5440) displayed in the visible environment, e.g., to help the user of device 5412 understand that avatar 5440 in the simulated environment corresponds to device 5406 of user "Gabe" (e.g., because avatar 5440 in the simulated environment is a smiley face icon that matches key avatar 5438).

[0315] exist Figure 5D4a In the second device 5412, Figure 5D3a The first position shown relative to the table 5414 is moved to a second position relative to the table (e.g., a position adjacent to the near left side of the table). Figure 5D4b In the device 5406, the user interface has been changed from Figure 5D3bDevice 5412 is shown in the changed position (indicated by avatar 5428 and representation 5430 of the device). The change in viewing perspective of device 5412 is indicated by the change in viewing perspective from Figure 5D3b to Figure 5D4b The different angles of viewing angle indicator 5432 are indicated. The movement of device 5412 is also indicated by Figure 5D3c-Figure 5D4c 5416b and a changed view of the reference pad 5416b and virtual frame 5420 displayed in the user interface of the device 5412 in FIG.

[0316] Figures 5D5-5D7 illustrate the selection and movement of the virtual frame 5420 by the device 5412.

[0317] exist Figure 5D5c , input (e.g., selection and movement input) by contact 5446 is detected on the touch screen display of second device 5412 at a location corresponding to the surface of virtual frame 5420. In response to detecting contact 5446 selecting the surface of virtual frame 5420, movement protrusion 5448 extending from virtual frame 5420 is shown to indicate a movement plane of virtual frame 5420 (e.g., a movement plane parallel to the selected surface of virtual frame 5420).

[0318] exist Figure 5D5b , an interaction indicator 5452 is shown to indicate to the user of the first device 5406 that the second device 5412 is interacting with the virtual frame 5420. The interaction indicator 5452 extends from a position corresponding to the avatar 5428 to a position corresponding to the virtual frame 5420. A control handle 5454 is shown where the interaction indicator 5452 encounters the virtual frame 5420.

[0319] exist Figure 5D5c-Figure 5D6c , contact 5446 has moved along the touch-sensitive display of device 5412 in the direction indicated by arrow 5450. In response to the movement of contact 5446, virtual frame 5420 has moved in the direction indicated by arrow 5450 within the plane indicated by movement protrusion 5448.

[0320] exist Figure 5D5b-Figure 5D6b , when the virtual frame 5420 is moved by movement input detected at the second device 5412, the user interface of the first device 5406 shows movement of the interaction indicator 5452 and the control handle 5454 (e.g., to maintain a connection between the interaction indicator 5452 and the virtual frame 5420).

[0321] exist Figure 5D7c , contact 5446 has been lifted off the touch-sensitive display of device 5412, and movement protrusion 5448 is no longer displayed. Figure 5D7b , the interaction indicator 5452 and the control handle 5454 are no longer displayed (because the device 5412 is not interacting with the virtual frame 5420).

[0322] 5D8-5D11 illustrate resizing of virtual frame 5420 by device 5412 .

[0323] exist Figure 5D8c , input (e.g., resizing input) made by contact 5456 is detected on the touch screen display of the second device 5412 at a location corresponding to the surface of the virtual frame 5420.

[0324] exist Figure 5D8b , an interaction indicator 5462 and a control handle 5464 are shown on the user interface of the first device 5406 to indicate that the second device 5412 is interacting with the virtual frame 5420.

[0325] exist Figure 5D9c , after contact 5456 has been maintained at a position corresponding to the surface of virtual frame 5420 for a period of time that increases to exceed a resizing time threshold, a resizing highlight 5458 is shown to indicate the axis along which the size of virtual frame 5420 will be adjusted in response to subsequent movement of contact 5456 (which is perpendicular to the selected surface of virtual frame 5420).

[0326] Figure 5D9a-Figure 5D10a Second device 5412 is shown moving upward (while contact 5456 is in contact with the touch screen display of second device 5412) to adjust the size of virtual frame 5420. In response to the movement of device 5412, the size of virtual frame 5420 has increased along the axis indicated by resizing protrusion 5458 in the direction of movement of second device 5412.

[0327] exist Figure 5D11c , contact 5456 has been lifted off of touch-sensitive display 112 and protrusion 5458 is no longer displayed.

[0328] As shown in Figures 5D12-5D14, users who are not in the same physical space can view and collaboratively manipulate objects in a simulated environment. For example, a user in a first physical space views a displayed version of a virtual user interface object (e.g., virtual frame 5420) anchored to a first physical reference pad (e.g., 5416a), and a different user at a remote location views a displayed version of the same virtual user interface object anchored to a second physical reference pad (e.g., 5476a).

[0329] Figure 5D12a A first physical space 5400 is shown where two users 5402 and 5408 and a table 5414 are located, as shown in FIG. Figure 5D1 As shown in . Figure 5D12bA second physical space 5470, separate from the first physical space 5400, is shown, in which a third user 5472 and a table 5474 are located. A third device 5478 (e.g., device 100) is held by the third user 5472. A reference pad 5476a is located on the table 5474. The device 5478 of the third user 5472 displays the same simulated environment as the device 5412 of the first user 5408 and the device 5404 of the second user 5402.

[0330] Figure 5D13a Shown as Figure 5D12a The first physical space 5400, and Figure 5D13b Shown as Figure 5D12b The second physical space 5470.

[0331] exist Figure 5D13c , the user interface of first device 5406 includes an avatar key 5422 that includes a key avatar 5480 (for "Stan") corresponding to third device 5478. Within the simulated environment displayed by 5406, an avatar 5482 corresponding to third device 5478 is shown (as indicated by key avatar 5480) at a position relative to a displayed version 5416b of physical reference mat 5416a that corresponds to the position of device 5478 relative to physical reference mat 5476a. The viewing perspective of device 5478 is indicated by viewing perspective indicator 5486. A representation 5484 of device 5478 (e.g., a rendered version of the device) is displayed within the simulated environment displayed by device 5406.

[0332] like Figure 5D13d As shown in , the user interface of the second device 5412 also displays an avatar 5482 corresponding to the third device 5478 , a viewing perspective indicator 5486 for indicating the viewing perspective of the device 5478 , and a representation 5484 of the device 5478 .

[0333] Figure 5D14a Shown as Figure 5D12a The first physical space 5400, and Figure 5D14b Shown as Figure 5D12b The second physical space 5470.

[0334] Figure 5D14c The user interface of the third device 5478 is shown. Figure 5D14c, a virtual frame 5420 is shown anchored to a displayed view 5476b of a physical reference mat 5476a. In the simulated environment displayed by the third device 5478, an avatar 5488 corresponding to the first device 5406 is shown at a position relative to the displayed version 5476b of the physical reference mat 5476a that corresponds to the position of the first device 5406 relative to the physical reference mat 5416a. The viewing perspective of the first device 5406 is indicated by a viewing perspective indicator 5490. A representation 5490 of the first device 5406 (e.g., a rendered version of the first device) is displayed in the simulated environment displayed by the third device 5476. In the simulated environment, an avatar 5494 corresponding to the second device 5412 is shown at a position relative to the displayed version 5476b of the physical reference mat 5476a that corresponds to the position of the second device 5412 relative to the physical reference mat 5416a. The viewing perspective of the second device 5412 is indicated by a viewing perspective indicator 5498. A representation 5496 of the second device 5412 (eg, a rendered version of the second device) is displayed within the simulated environment displayed by the third device 5476 .

[0335] Figure 5E1-Figure 5E32 Examples of systems and user interfaces for placing an insertion cursor are shown in accordance with some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figures 12A-12D For ease of explanation, some embodiments of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, in response to detecting contact on the touch-sensitive surface of the input device 5010 while displaying the user interface on the display of the headset 5008 and the focus indicator shown in the figure, optionally on a computer system having the headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 1 ), a focus indicator is displayed. Figure 5A2 ) perform similar operations.

[0336] Figure 5E1-Figure 5E3 Shown is the use of Figure 5E4-Figure 5E32 The context of the user interface being described.

[0337] Figure 5E1 A physical space 5200 is shown with a user 5202 and a table 5204. The device 100 is held by the user 5202 in the user's hand 5206. A reference pad 5208 is located on the table 5204.

[0338] Figure 5E21 shows a view of a virtual three-dimensional space displayed on the display 112 of the device 100. The reference mat 5208 is in the field of view of one or more cameras (e.g., optical sensor 164) of the device 100 (hereinafter referred to as "camera," which refers to one or more cameras of the device 100). The display 112 shows a real-time view of the physical space 5200 captured by the camera, including a displayed version 5208b of the physical reference mat 5208a.

[0339] exist Figure 5E3 , the device 100 has been moved closer to the reference pad 5208a. As a result, the size of the displayed version 5208b of the reference pad has increased.

[0340] Figure 5E4-Figure 5E32 A larger view of the device 100 is shown, and in order to provide a complete view of the user interface displayed on the display 112, the user's hand 5206 is not shown.

[0341] Figure 5E5-Figure 5E6 Input resulting in placement of an insertion cursor at a first position is shown.

[0342] exist Figure 5E5 , input (e.g., a tap input) by contact 5502 is detected at a first location on displayed version 5208b of physical reference pad 5208a. Figure 5E6 , contact 5502 has been lifted off of touch-sensitive display 112 and insertion cursor 5504 is shown at the location where contact 5502 was detected.

[0343] Figure 5E7-Figure 5E8 An input resulting in placement of the insertion cursor at the second position is shown. Figure 5E7 , an input (e.g., a tap input) by contact 5506 is detected at a position different from the position where the insertion cursor 5504 is displayed. Figure 5E8 , contact 5506 has been lifted off of touch-sensitive display 112 and insertion cursor 5508 is shown at the location where contact 5506 was detected.

[0344] Figure 5E9-Figure 5E10 An input resulting in the insertion of a virtual user interface object is shown. Figure 5E9, input (e.g., a tap input) by contact 5510 is detected at a location corresponding to the location of the insertion cursor 5508. In response to detecting contact 5510 at the location where the insertion cursor is placed, a virtual user interface object (first virtual frame 5512) is displayed on display 112 at a location corresponding to contact 5510, and the insertion cursor 5508 is moved from its previous position on the displayed view 5208b of the reference pad 5208a to the surface 5514 of the first virtual frame 5512. In some embodiments, a shadow 5522 is displayed (e.g., a simulated light causes the first virtual frame 5512 to cast a shadow).

[0345] Figure 5E11-Figure 5E12 An input detected on the surface of a virtual user interface object (first virtual frame 5512) that results in the insertion of an additional virtual user interface object is shown. Figure 5E11 , while insertion cursor 5516 is located on surface 5514, input by contact 5516 (e.g., a tap input) is detected at a location on surface 5514 of first virtual frame 5512. In response to detecting the input by contact 5516, a new virtual user interface object (second virtual frame 5518) is displayed on display 112 at a location corresponding to contact 5510, and insertion cursor 5508 is moved from surface 5514 of first virtual frame 5512 to surface 5520 of second virtual frame 5518. The length of shadow 5522 is increased (so that the shadow appears to be cast by both first virtual frame 5512 and the newly added second virtual frame 5518).

[0346] Figure 5E12-Figure 5E13 The rotation of the physical reference mat 5208 is shown. For example, the user 5202 manually changes the position and / or orientation of the reference mat 5208. As the physical reference mat 5208a rotates, the virtual frames 5512 and 5518 and the shadow 5522 rotate (because the virtual frames 5512 and 5518 are anchored to the displayed view 5208b of the physical reference mat 5208a).

[0347] Figure 5E14-Figure 5E16 Movement of the device 100 is shown. For example, the user 5202 holding the device 100 changes the position and / or orientation of the device. Figure 5E14-Figure 5E15 , as the device 100 moves, the virtual frames 5512 and 5518 and the shadow 5522 move (because the virtual frames 5512 and 5518 are anchored to the displayed view 5208b of the physical reference pad 5208a). Figure 5E15-Figure 5E16 , when the device 100 moves, the virtual frames 5512 and 5518 and the shadow 5522 move.

[0348] Figure 5E17-Figure 5E18An input is shown that changes the position of the insertion cursor 5526 on the virtual box 5518. Figure 5E17 , while the insertion cursor 5526 is located on the surface 5520 of the virtual frame 5518, an input (e.g., a tap input) by a contact 5524 is detected on the surface 5528 of the virtual frame 5518. Figure 5E18 , contact 5524 has been lifted off the touch-sensitive display 112 and the insertion cursor 5508 has moved from surface 5520 of the virtual frame 5518 to surface 5528 of the virtual frame 5518 .

[0349] Figure 5E19-Figure 5E20 An input is shown detected on the surface of the second virtual frame 5518 resulting in the insertion of a third virtual frame 5532. Figure 5E19 , while the insertion cursor 5526 is located on the surface 5268, an input (e.g., a tap input) by a contact 5530 is detected at a position on the surface 5528 of the second virtual frame 5518. In response to detecting the input by the contact 5530, a third virtual frame 5532 is displayed on the display 112 at a position corresponding to the contact 5530, and the insertion cursor 5526 is moved from the surface 5528 of the second virtual frame 5518 to the surface 5526 of the third virtual frame 5532. The shape of the shadow 5522 is changed (so that the shadow appears to be cast by the first virtual frame 5512, the second virtual frame 5518, and the newly added third virtual frame 5532).

[0350] Figure 5E21-Figure 5E22 An input is shown that changes the position of the insertion cursor 5538 on the virtual box 5532. Figure 5E21 , while the insertion cursor 5526 is located on the surface 5534 of the virtual frame 5532, an input (e.g., a tap input) by a contact 5536 is detected on the surface 5538 of the virtual frame 5532. Figure 5E22 , contact 5536 has been lifted off the touch-sensitive display 112 and the insertion cursor 5526 has moved from surface 5534 of virtual frame 5518 to surface 5538 of virtual frame 5532 .

[0351] Figure 5E23-Figure 5E24 Use of new object control 5216 to insert a new virtual user interface object is shown.

[0352] exist Figure 5E23 , while the insertion cursor 5526 is at the surface 5538 of the virtual frame 5532, input (e.g., a tap input) via contact 5542 is detected at a location on the display 112 corresponding to the new object control 5216. Figure 5E24In response to input at a location corresponding to the new object control 5216, a fourth virtual frame 5546 is displayed on the display 112 at a location corresponding to the location where the insertion cursor 5526 is displayed, and the insertion cursor 5526 moves from the surface 5538 of the virtual frame 5532 to the surface 5548 of the fourth virtual frame 5546.

[0353] Figure 5E25-Figure 5E27 Input is shown that causes the fourth virtual frame 5546 to move.

[0354] exist Figure 5E25 , input (e.g., selection and movement input) by contact 5550 is detected on surface 5556 of fourth virtual frame 5546. In response to detecting contact 5550 selecting surface 5556 of fourth virtual frame 5546, a movement protrusion 5552 extending from virtual frame 5546 is shown to indicate a movement plane of fourth virtual frame 5546 (e.g., a movement plane parallel to the selected surface 5556 of virtual frame 5546).

[0355] exist Figure 5E25-Figure 5E26 , contact 5550 has moved along the surface of touch-sensitive display 112 in the direction indicated by arrow 5554. In response to the movement of contact 5550, fourth virtual frame 5546 has moved in the direction indicated by arrow 5554 within the plane indicated by movement protrusion 5552. Figure 5E27 , contact 5550 has been lifted off of touch-sensitive display 112 and movement protrusion 5552 is no longer displayed.

[0356] Figure 5E28-Figure 5E32 Input is shown that causes the fourth virtual frame 5546 to be resized.

[0357] exist Figure 5E28 , an input (eg, a resizing input) made by contact 5258 is detected on touch screen display 112 at a location corresponding to surface 5556 of fourth virtual frame 5546 .

[0358] exist Figure 5E29 , after contact 5255 has been maintained at a position corresponding to surface 5556 of fourth virtual frame 5546 for a period of time that increases to exceed a resizing time threshold, a resizing highlight 5560 is shown to indicate the axis along which the size of virtual frame 5546 will be adjusted in response to subsequent movement of contact 5558 (which is perpendicular to the selected surface of virtual frame 5546).

[0359] exist Figure 5E30-Figure 5E31, contact 5558 has moved on touch screen display 112 along a path indicated by arrow 5562. In response to the movement of contact 5558, the size of virtual frame 5548 has increased along an axis indicated by resizing protrusion 5560 in the direction in which contact 5558 moved.

[0360] exist Figure 5E32 , contact 5558 has been lifted off of touch-sensitive display 112 and protrusion 5560 is no longer displayed.

[0361] Figure 5F1-Figure 5F1 7 shows an example of a system and user interface for displaying an augmented reality environment in a stable operating mode according to some embodiments. The user interface in these figures is used to illustrate the process described below, including Figures 13A-13E For ease of explanation, some embodiments of the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. Similarly, in response to detecting contact on the touch-sensitive surface of the input device 5010 while displaying the user interface on the display of the headset 5008 and the focus indicator shown in the figure, optionally on a computer system having the headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in FIG. 1 ), a focus indicator is displayed. Figure 5A2 ) perform similar operations.

[0362] Figure 5F1-Figure 5F2 The context in which the user interface described with respect to Figures 5F3-5F17 is used is shown.

[0363] Figure 5F1 The physical space 5200 is shown with a user 5202 and a table 5204. The device 100 is held by the user 5202 in the user's hand 5206. An object (physical frame 5602) is located on the table 5204.

[0364] Figure 5F2 An augmented reality environment is shown displayed by the display 112 of the device 100. A table 5204 (referred to as 5204a when referring to a table in a physical space) and a physical frame 5602 are within the field of view of one or more cameras (e.g., optical sensor 164) of the device 100 (hereinafter referred to as "camera," which indicates one or more cameras of the device 100). The display 112 shows a real-time view of the physical space 5200 captured by the camera, including a displayed version 5204b of the table 5204a and a rendered virtual frame 5604 displayed in a simulated environment at a location corresponding to the physical frame 5602 detected by the device 100 camera.

[0365] Figures 5F3-5F17 include sub-figures "a" and "b," where sub-figure a illustrates the orientation of the device 100 in the physical space 5200 relative to the table 5204a and the physical frame 5602 (e.g., as shown in FIG. Figure 5F3a ), sub-figure b shows the user interface of the device 100 (eg, as Figure 5F3b 5F3-5F18 illustrate larger views of device 100 for clarity, and user's hand 5206 is not shown in order to provide a complete view of the user interface displayed on display 112.

[0366] Figure 5F3a-Figure 5F4a The movement of the device 100 relative to the table 5204a and the physical frame 5602 is shown, and the movement is performed on the device 100 (such as Figure 5F3b-Figure 5F4b When the device 100 is in a first position relative to the table 5204a, as shown in FIG. Figure 5F3a As shown, a rendered version 5604 of the physical object 5602 is Figure 5F3b is fully visible in the user interface shown in . Figure 5F4a , the device 100 has moved to a second position relative to the table 5204a, and the rendered version 5604 of the physical object 5602 is at Figure 5F4b 5204b is only partially visible in the user interface shown in FIG. In the unstabilized operating mode, as the device 100 moves, the view of the virtual frame 5604 changes to maintain a fixed spatial relationship between the virtual frame 5604 and the physical frame 5602, and the displayed representation of the camera field of view of the device 100 (e.g., including the displayed table 5204b) is updated based on the movement of the device.

[0367] Figures 5F5-5F8 illustrate inputs (eg, expand-and-contract inputs) that cause the device to display an augmented reality environment in a stable operating mode.

[0368] In Figure 5F5, device 100 is in a first position relative to table 5204. In Figure 5F6, contacts 5606 and 5608 are detected at touch-sensitive display 112 (e.g., Figure 5F6b As shown). Figure 5F6b-Figure 5F7b As shown, contact 5606 moves along the path indicated by arrow 5610, and contact 5608 moves along the path indicated by arrow 5612. In response to the simultaneous movement of contacts 5606 and 5608 increasing the distance between contacts 5606 and 5608, the displayed augmented reality environment (including virtual frame 5604) is zoomed in (e.g., such that the size of virtual frame 5604 increases on display 112). Virtual frame 5604 is re-rendered in response to the zoom input (e.g., Figure 5F8b The larger virtual frame 5604 has Figure 5F5bIn some embodiments, the camera field of view of the device 100 displayed on the display 112 (e.g., the displayed view 5204b of the table 5204a) is not changed in response to the zoom input (e.g., the displayed view 5204b of the table 5204a). Figure 5F5b-Figure 5F8b 5F8 , contacts 5606 and 5608 have been lifted off touch screen 112 .

[0369] In some embodiments, when the device is displaying an augmented reality environment in a stable operating mode, a portion of a virtual user interface object ceases to be displayed due to movement of the device causing the virtual user interface object to extend beyond the field of view of a device camera. Figures 5F8-5F9 illustrate movement of device 100 while device 100 is in a stable operating mode, causing a portion of virtual user interface object 5304 to cease to be displayed. Figure 5F8a-Figure 5F9a The movement of the device 100 relative to the table 5204a and the physical frame 5602 is shown, and the movement is performed on the device 100 (such as Figure 5F8b-Figure 5F9b When the device 100 is in a first position relative to the table 5204a, as shown in FIG. Figure 5F8a As shown, a scaled rendered version 5604 of the physical object 5602 is shown in FIG. Figure 5F8b is fully visible in the user interface shown in . Figure 5F9a , device 100 has moved to a second position relative to table 5204a such that updating the view of virtual frame 5604 to maintain a fixed spatial relationship between virtual frame 5604 and physical frame 5602 causes virtual frame 5604 to extend beyond the field of view of the camera of device 100. As a result, the portion of virtual frame 5604 that extends beyond the field of view of the camera of device 100 is not displayed.

[0370] In some embodiments, when the device is displaying an augmented reality environment in a stable operating mode, and movement of the device causes a virtual user interface object to extend beyond the field of view of the device camera, the augmented reality environment is zoomed out so as to fully display the virtual user interface object. Figure 5F9b to Figure 5F10b , the displayed augmented reality environment (including the virtual frame 5604) has been reduced so that the virtual frame 5604 is fully displayed.

[0371] In some embodiments, in a stable mode of operation, when updating the view of the virtual frame 5604 to maintain a fixed spatial relationship between the virtual frame 5604 and the physical frame 5602 causes the virtual frame 5604 to extend beyond the field of view of the device 100 camera, the virtual frame 5604 is displayed along with a placeholder image at a location corresponding to the portion of the virtual frame 5604 that extends beyond the field of view of the device camera. Figure 5F10b, a rendered version 5604 of the physical object 5602 is displayed along with a placeholder image 5614 (empty space) at a location corresponding to a portion of the virtual frame 5064 that extends beyond the field of view of the device camera. For example, the placeholder image 5614 is displayed at a location in the augmented reality environment that is beyond the field of view of the camera, so no camera data is available for display in the space occupied by the placeholder image 5614.

[0372] Figures 5F10-5F11 illustrate movement of device 100 (returning to the position of device 100 shown in Figures 5F8 and 5F3).

[0373] Figure 5F11a-Figure 5F12a The device 100 is shown moving (e.g., back away from the table 5204a and physical object 5602, making the device 100 appear larger). Figure 5F12b In, as Figure 5F11a-Figure 5F12a As a result of the movement shown in , the size of the virtual object 5604 has changed from Figure 5F11b For illustrative purposes (making the virtual object 5604 in stable mode Figure 5F12b The size of the virtual object 5604 in the unstable mode is different from that in the Figure 5F3b The virtual object 5604 in the same size as Figure 5F11a-Figure 5F12a Movement is shown in to provide a direct comparison of updates to the augmented reality environment in stable and non-stabilized modes of operation.

[0374] Figure 5F12a-Figure 5F13a 5204a and the physical frame 5602, the movement of the device 100 occurs while the device 100 is displaying the augmented reality environment in a stable operating mode. Figure 5F12a As shown, a rendered version 5604 of the physical object 5602 is Figure 5F12b is fully visible in the user interface shown in . Figure 5F13a , the device 100 has moved to a second position relative to the table 5204a, and the rendered version 5604 of the physical object 5602 is at Figure 5F13b In some embodiments, in the stabilized operating mode, as the device 100 moves, the view of the virtual frame 5604 changes so as to maintain a fixed spatial relationship between the virtual frame 5604 and the physical frame 5602, and the displayed representation of the field of view of the camera of the device 100 (e.g., including the displayed table 5204b) changes by an amount that is less than the amount of change that occurs in the unstabilized mode (e.g., the displayed table 5204b changes from 0 to 1 when the device 100 is in the stabilized operating mode). Figure 5F12b-Figure 5F13b The amount of movement is less than that of the table 5204b displayed when the device 100 is in the unstable operating mode. Figure 5F4b to Figure 5F5b amount of movement).

[0375] Figures 5F14-5F16 illustrate the inputs at the stable switch 5616 to transition from the unstable mode of operation to the stable mode of operation. Figure 5F15b , an input (e.g., a tap input) is detected by contact 5618 at a location on the touch screen display 112 corresponding to the stable switch 5616. In response to the input made by contact 5618, the appearance of the stable switch 5616 is changed (e.g., the switch changes from an unshaded state to a shaded state) to indicate that a transition from the unstable operating mode to the stable operating mode has occurred, as shown in FIG. Figure 5F16b shown.

[0376] Figure 5F16a-Figure 5F17a The movement of the device 100 relative to the table 5204a and the physical frame 5602 is shown, and the movement is performed on the device 100 (such as Figure 5F16a-Figure 5F17a When the device 100 is in a first position relative to the table 5204a, as shown in FIG. Figure 5F16a As shown, a rendered version 5604 of the physical object 5602 is Figure 5F16b is fully visible in the user interface shown in . Figure 5F17a , the device 100 has moved to a second position relative to the table 5204a, and the rendered version 5604 of the physical object 5602 is at Figure 5F17b In the stabilized mode of operation, as the device 100 moves, the view of the virtual frame 5604 changes so as to maintain a fixed spatial relationship between the virtual frame 5604 and the physical frame 5602, and the displayed representation of the field of view of the camera of the device 100 (e.g., including the displayed table 5204b) changes by an amount that is less than the amount of change that occurs in the unstabilized mode (e.g., the displayed table 5204b changes from 0 to 1 when the device 100 is in the stabilized mode of operation). Figure 5F15b-Figure 5F16b The amount of movement is less than that of the table 5204b displayed when the device 100 is in the unstable operating mode. Figure 5F4b to Figure 5F5b amount of movement).

[0377] Figures 6A-6Dis a flow chart illustrating a method 600 for adjusting the appearance of a virtual user interface object in an augmented reality environment according to some embodiments. On a computer system (e.g., portable multifunction device 100, 100b) having a display generating component (e.g., a display, a projector, a heads-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of content within the camera's field of view and optionally generates a video output comprising one or more streams of image frames capturing the content within the camera's field of view), and an input device (e.g., a touch-sensitive surface, such as a touch-sensitive remote control, or a touch screen display that also serves as the display generating component, a mouse, a joystick, a stylus controller, and / or a camera that tracks one or more features of a user, such as the position of a user's hand), Figure 1A , device 300, Figure 3A , or a multi-component computer system including a headset 5008 and an input device 5010, Figure 5A2 ) method 600 is performed at. In some embodiments, the input device (e.g., having a touch-sensitive surface) and the display generation component are integrated into the touch-sensitive display. Figure 3B-3C As described, in some embodiments, method 600 is performed at a computer system 301 (e.g., computer system 301-a, 301-b, or 301-c), where corresponding components, such as a display generation component, one or more cameras, one or more input devices, and optionally one or more gesture sensors are each included in or in communication with the computer system 301.

[0378] In some embodiments, the display generating component is a touch screen display, and the input device (e.g., having a touch sensitive surface) is on or integrated with the display generating component. In some embodiments, the display generating component is separate from the input device (e.g., Figure 4B and Figure 5A2 Some operations in method 600 are optionally combined, and / or the order of some operations is optionally changed.

[0379] For ease of explanation, some embodiments will be discussed with reference to operations performed on a computer system having a touch-sensitive display system 112 (e.g., on a device 100 having a touch screen 112) and one or more integrated cameras. However, in response to detecting contact on the touch-sensitive surface of an input device 5010 while displaying a user interface on a display of a headset 5008, optionally on a computer system having a headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as in FIG. 1 ), a user interface is detected on a display of the headset 5008. Figure 5A2Similarly, similar operations may optionally be performed on a computer system having one or more cameras implemented (e.g., in a headset) independently of one or more other components of the computer system (e.g., input devices); and in some such embodiments, "movement of the computer system" corresponds to movement of the one or more cameras of the computer system, or movement of one or more cameras in communication with the computer system.

[0380] As described below, method 600 involves adjusting the appearance of a virtual user interface object (e.g., on a display of a computer system) in an augmented reality environment (e.g., in which reality is augmented with supplemental information that provides the user with additional information not available in the physical world) based on a combination of movement of a computer system (e.g., movement of one or more cameras of the computer system) and movement of contacts on an input device (e.g., a touch screen display) of the computer system. In some embodiments, adjusting the appearance of the virtual user interface object allows the user to access the supplemental information in the augmented reality environment. Adjusting the appearance of a virtual user interface object based on a combination of movement of the computer system and movement of contact on an input device of the computer system provides a user with an intuitive way to adjust the appearance of a virtual user interface object (e.g., by allowing the user to adjust the appearance of a virtual user interface object using only movement of the computer system, only movement of contact on an input device, or a combination of movement of the computer system and movement of contact), and allows the user to expand the range of adjustments available to the user (e.g., by allowing the user to continue adjusting the appearance of a virtual user interface object even if the contact or one or more cameras of the computer system cannot be moved further in the desired direction), thereby enhancing the operability of the device and making the user-device interface more efficient (e.g., by reducing the number of steps required to achieve a desired result when operating the device and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0381] Computer systems (e.g. Figure 5A7 device 100) via a display generating component (e.g., a touch screen 112, Figure 5A7 ) to display (602) an augmented reality environment (e.g., Figure 5A7 Displaying the augmented reality environment includes (604) simultaneously displaying: corresponding physical objects (e.g., a 3D model of a building, a piece of paper with a printed pattern, a poster on a wall or other physical object, a sculpture sitting on a surface, etc.) (e.g., a physical building model 5006, Figure 5A1), wherein the representation is updated when the content of the field of view of the one or more cameras changes (e.g., the representation is a real-time preview of at least a portion of the field of view of the one or more cameras, and the corresponding physical object is included and visible in the camera field of view); and corresponding virtual user interface objects (e.g., a virtual roof of a 3D model of a building, a virtual car parked on a surface represented by a piece of paper with a printed pattern, an interactive logo superimposed on a poster, a virtual 3D mask covering the outline of a sculpture, etc.) at corresponding locations in the representation of the field of view of the one or more cameras (e.g., virtual building model 5012, Figure 5A7 ), wherein the corresponding virtual user interface object (e.g., virtual building model 5012, Figure 5A7 ) has a position determined based on a corresponding physical object (e.g., physical building model 5006) in the field of view of the one or more cameras. For example, in some embodiments, the corresponding virtual user interface object is a graphical object or a 2D or 3D virtual object that appears to be attached to or appears to overlay a corresponding physical object in the field of view of the one or more cameras (e.g., virtual building model 5012 is a 3D virtual object that appears to overlay physical building model 5006, Figure 5A7 ). The position and / or orientation of the corresponding virtual user interface object is determined based on the position, shape and / or orientation of the physical object in the field of view of one or more cameras (e.g., as Figure 5A3 and Figure 5A5 While displaying the augmented reality environment (606), the computer system detects an input (e.g., an input on an input device such as detecting a touch input made by detecting a contact on a touch screen display or touch-sensitive remote control) at a location corresponding to a corresponding virtual user interface object (a location on a touch screen display or touch-sensitive remote control, or movement of a stylus or user's hand when a cursor is at the location of the corresponding virtual user interface object) (e.g., device 100 detects contact 5020-a on a virtual roof of virtual building model 5012, Figure 5A8 ).

[0382] While input continues to be detected (608) (e.g., while the contact remains on an input device, such as when the contact is maintained on a touch screen display or a touch-sensitive remote control) (e.g., while contact 5020 remains on touch screen 112), Figures 5A9-5A13 ), the computer system detects movement of the input relative to a corresponding physical object in the field of view of one or more cameras (e.g., Figures 5A9-5A13 In some embodiments, movement of the input optionally includes movement of the contact across the touch screen display or across the touch-sensitive surface of the touch-sensitive remote control while the computer system (e.g., device 100) remains substantially stationary in physical space (e.g., as shown in FIG. Figures 5A8-5A11In some embodiments, movement of the input optionally includes movement of the device (including the camera) in physical space while maintaining contact and remaining stationary on the touch screen display or touch-sensitive remote control (e.g., as shown in FIG. Figure 5A17-Figure 5A18 In some embodiments, movement of the input optionally includes movement of a contact across a touch screen display or touch sensitive remote control, along with movement of a device including a camera in physical space. In some embodiments, movement of the computer system includes movement of components of a multi-component computer system, such as movement of a virtual reality head mounted display (e.g., as shown in FIG. Figure 5A2 In addition, while continuing to detect input, and in response to detecting movement of the input relative to the corresponding physical object in the field of view of the one or more cameras, the device adjusts the appearance of the corresponding virtual user interface object (e.g., by expanding, contracting, stretching, squeezing together, spreading out, and / or pushing together all or a portion of the virtual user interface object) based on the magnitude of the movement of the input relative to the corresponding physical object. For example, when a contact is detected on a virtual roof of a building model and then the contact moves across the touch screen display, the virtual roof is lifted off the building model in the real-time preview of the camera's field of view (e.g., as Figures 5A8-5A11 and, while the contact is maintained on the touch screen display and the entire device moves in physical space relative to the building model, determining movement of the virtual roof based on both the location of the contact on the touch screen display and the location and orientation of the device relative to the corresponding physical object in physical space (e.g., based on the location of the corresponding physical object shown in the live preview of the camera's field of view) (e.g., as shown in FIG. Figure 5A11-Figure 5A13 ).

[0383] As another example, in a block building application (e.g., as compared to Figure 5B1-Figure 5B41 ), wherein the virtual model is constructed on a corresponding physical object (e.g., a tabletop or a piece of paper with a printed pattern), contact is detected on the block (e.g., in response to a long press input on the block) (e.g., contact 5262, Figure 5B28 ), and the computer system displays a guide of how the block will be scaled (e.g., as shown using scaled projection 5266, Figure 5B29 ), when the contact is maintained on the block and the device as a whole is moved relative to the block (e.g., Figure 5B28-Figure 5B30 While the touch screen is being used (as shown), the scaling of the block (e.g., stretching the block in the guided direction) is determined based on both the location of the contact on the touch screen display (e.g., on a particular side or face of the cube) and the position and orientation of the device relative to the corresponding physical object in physical space (e.g., determined based on the location of the corresponding physical object shown in a live preview of the camera's field of view).

[0384] In some embodiments, the corresponding virtual user interface object (e.g., virtual building model 5012, Figures 5A8-5A13 ) includes (610): adjusting the appearance of the corresponding virtual user interface object by a first adjustment (e.g., a larger amount of relative movement results in a larger adjustment) based on determining that the movement magnitude of the input relative to the corresponding physical object is a first magnitude (e.g., a relatively large movement magnitude) (e.g., Figure 5A9 In comparison, Figure 5A10 and adjusting the appearance of the corresponding virtual user interface object by a second adjustment different from the first adjustment (e.g., a smaller amount of relative movement results in a smaller adjustment) based on determining that the movement magnitude of the input relative to the corresponding physical object is a second magnitude different from the first magnitude (e.g., a smaller movement magnitude). Figure 5A10 In comparison, Figure 5A9 Adjusting the corresponding virtual user interface object by a first adjustment when the input movement magnitude is a first magnitude (e.g., a larger relative movement results in a larger adjustment), and adjusting the corresponding virtual user interface object by a second adjustment when the input movement magnitude is a second magnitude (e.g., a smaller relative movement results in a smaller adjustment), improves visual feedback provided to the user (e.g., by making the computer system appear more responsive to the user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve desired results with desired inputs and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently.

[0385] In some embodiments, corresponding virtual user interface objects (e.g., virtual building models 5012, Figure 5A3-5A6 ) is anchored (612) to a corresponding physical object (e.g., physical building model 5006) in the field of view of the one or more cameras before and after adjustment. For example, in some embodiments, the corresponding virtual user interface object appears to overlay the corresponding physical object in the field of view of the one or more cameras, and when the position and / or orientation of the physical object in the field of view of the one or more cameras changes, the position and / or orientation of the corresponding virtual user interface object changes accordingly (e.g., as Figure 5A3-5A6 In some embodiments, during some or all of the conditioning periods (e.g., Figures 5A3 to 5A5During the transition period, the corresponding virtual user interface object is anchored to the corresponding physical object in the field of view of the one or more cameras. Anchoring the corresponding virtual user interface object to the corresponding physical object improves the visual feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve the desired result with the required input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0386] In some embodiments, the appearance of the corresponding virtual user interface object is adjusted (614) in response to detecting movement of the input relative to the corresponding physical object in the field of view of one or more cameras, regardless of whether the movement of the input is due to: movement of the input on the input device (e.g., movement of contact across the touch screen display or across the touch-sensitive surface of the input device when the input device remains substantially stationary in physical space) (e.g., Figure 5A9-Figure 5A11 ), movement of one or more cameras relative to corresponding physical objects (e.g., movement of a computer system including cameras in physical space while contact is maintained and remains stationary on a touch-sensitive surface of a touch screen display or input device) (e.g., as shown in Figure 5A11-Figure 5A13 Adjusting the appearance of virtual user interface objects regardless of how the input is moved (e.g., by allowing the user to adjust the appearance of virtual user interface objects using only movement of the input on the input device, only movement of the camera relative to the physical object, or a combination of movement of the input and camera) provides the user with an intuitive way to adjust the appearance of virtual user interface objects, improves the visual feedback provided to the user (e.g., by making the computer system appear more responsive to the user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve desired results with desired input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently.

[0387] In some embodiments, the movement of the input relative to the corresponding physical object (616) is based on the movement of the field of view of one or more cameras relative to the corresponding physical object (e.g., due to movement of the computer system including the cameras in physical space) (e.g., as Figure 5A11-Figure 5A13) and movement of input on the input device (e.g., movement of contact across a touch screen display or across a touch-sensitive surface of the input device) (e.g., as shown in Figures 5A8-5A11 Allowing the user to move input relative to a corresponding physical object through movement of the computer system and movement of the contacts provides an intuitive way for the user to move input, improves feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve desired results with desired input and reducing user errors in operating / interacting with the device), which additionally reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently.

[0388] In some embodiments, movement of the input relative to the physical object (618) is based on movement of the input on the input device (e.g., movement of contact across a touch screen display or across a touch-sensitive surface of the input device while the input device remains substantially stationary in physical space) (e.g., as Figures 5A8-5A11 ), and adjusting the appearance of the corresponding virtual user interface object according to the movement magnitude of the input relative to the corresponding physical object (e.g., as Figure 5A11 ), the computer system: detects movement of the field of view of one or more cameras relative to the corresponding physical object (e.g., movement 5022, Figure 5A12 ); and in response to detecting movement of the field of view of the one or more cameras relative to the corresponding physical object, continue to adjust the appearance of the corresponding virtual user interface object (e.g., in the same manner) according to the amount of movement of the field of view of the one or more cameras relative to the corresponding physical object (e.g., as Figure 5A13 In some embodiments, adjusting the appearance of the corresponding virtual user interface object includes moving a portion of the corresponding virtual user interface object, where the movement is initiated by a contact drag on the virtual user interface object and continued by moving the device as a whole (e.g., as shown in FIG. Figures 5A8-5A13 For example, when a contact is detected on a virtual roof of a 3D model of a building and the contact moves across the touch screen display (e.g., in an upward direction), the virtual roof is lifted from the building model in the displayed augmented reality environment (e.g., as shown in FIG. Figures 5A8-5A11 After the virtual roof is raised, as the device as a whole moves in the physical space (e.g., in an upward direction) relative to the building model, the virtual roof continues to rise (e.g., as Figure 5A11-Figure 5A13As shown) (and optionally, the floors of the building model are raised and expanded). In some embodiments, the appearance of the corresponding virtual user interface object is adjusted according to the input movement amount, and then the appearance of the corresponding virtual user interface object continues to be adjusted according to the movement amount of one or more camera fields of view, allowing the user to continue adjusting the appearance of the corresponding virtual user interface object even if the contact cannot move much in the desired direction on the touch screen display (for example, because the touch is at or near the edge of the touch screen display, and further movement of the touch will cause the touch to leave the edge of the touch screen display). For example, using a virtual roof, when the contact is close to the top edge of the touch screen display but the user still wants to continue to raise the roof, even if the contact cannot move higher on the touch screen display, the user can continue to adjust by moving the device or camera to do so (for example, as shown). Figures 5A8-5A13 ). Adjusting the appearance of the corresponding virtual user interface object based on the input movement magnitude, and then continuing to adjust the appearance of the corresponding virtual user interface object based on the movement magnitude of the one or more camera fields of view, allows the user to expand the range of adjustment available to the user (e.g., allowing the user to continue adjusting the appearance of the corresponding virtual user interface object as the computer system moves, even if the contact cannot move further in the desired direction on the touch screen display), thereby enhancing the operability of the device and making the user-device interface more efficient (e.g., by reducing the number of steps required to achieve a desired result when operating the device and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0389] In some embodiments, movement of the input relative to the corresponding physical object is based on (620) movement of the field of view of one or more cameras relative to the corresponding physical object (e.g., due to movement of one or more cameras of the computer system or one or more cameras in communication with the computer system while contact is maintained and remains stationary on the touch-sensitive surface of the touch screen display or input device) (e.g., as Figure 5A17-Figure 5A18 ), after adjusting the appearance of the corresponding virtual user interface object according to the movement magnitude of the input relative to the corresponding physical object (e.g., as Figure 5A18 ), a computer system: detecting movement of an input on an input device (e.g., movement of a (previously stationary) contact across a touch screen display or across a touch-sensitive surface of an input device while the input device remains substantially stationary in physical space) (e.g., as Figure 5A19-Figure 5A20 and in response to detecting movement of the input on the input device, continuing to adjust the appearance of the corresponding virtual user interface object (e.g., as shown in the same manner) according to the amount of movement of the input on the input device; Figure 5A20In some embodiments, adjusting the appearance of the corresponding virtual user interface object includes moving a portion of the corresponding virtual user interface object (e.g., moving virtual roof 5012-a, Figures 5A17-5A20 ), wherein the device is touched and moved as a whole by a (stationary) contact on a virtual user interface object (e.g. Figure 5A17-Figure 5A18 ) to start the movement and continue the movement by contact dragging on the virtual user interface object (e.g., as shown in Figure 5A19-Figure 5A20 For example, when contact is detected on a virtual roof of a 3D model of a building and the device as a whole is moved relative to the building model in physical space (e.g., in an upward direction), the virtual roof is lifted from the building model in a real-time preview of the camera's field of view (e.g., as shown in FIG. Figure 5A17-Figure 5A18 ). After the virtual roof is raised, the virtual roof continues to rise (and optionally, the floors of the building model rise and expand) as the (previously stationary) contact moves across the touch screen display (e.g., in an upward direction) (e.g., as Figure 5A19-Figure 5A20 Adjusting the appearance of a corresponding virtual user interface object based on the amount of movement of one or more camera fields of view, and then continuing to adjust the appearance of the corresponding virtual user interface object based on the amount of movement of the input on the input device, allows the user to expand the range of adjustment available to the user (e.g., allowing the user to continue adjusting the appearance of the corresponding virtual user interface object with the input device even if the computer system (or one or more cameras of the computer system or in communication with the computer system) cannot move further in the desired direction), thereby enhancing the operability of the device and making the user-device interface more efficient (e.g., by reducing the number of steps required to achieve a desired result when operating the device and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0390] In some embodiments, detecting input at a location corresponding to a corresponding virtual user interface object includes (622) detecting input at a first contact point on the corresponding virtual user interface object; and the computer system (e.g., device 100) updating the display of the corresponding virtual user interface object so as to maintain the display of the first contact point on the corresponding virtual user interface object at a location corresponding to the input location (e.g., when displaying the virtual user interface object on a touch screen device, the device updates the corresponding virtual user interface so as to maintain the virtual user interface object under the user's finger, regardless of whether the movement of the input is due to: movement of the input on the input device (e.g., movement of the contact across the touch screen display or across the touch-sensitive surface of the input device while the input device remains substantially stationary in physical space) (e.g., as shown in FIG. 1 ). Figures 5A8-5A11), movement of one or more cameras relative to corresponding physical objects (e.g., movement of a computer system including cameras in physical space while contact is maintained and remains stationary on a touch-sensitive surface of a touch screen display or input device) (e.g., as shown in Figure 5A11-Figure 5A13 ), or a combination of movement of the input on the input device and movement of one or more cameras relative to corresponding physical objects (e.g., movement of a contact across the touch screen display or the touch-sensitive surface of the input device, combined with movement of the computer system including the cameras in physical space). For example, when a contact is detected (e.g., by a user's finger) on a virtual roof of a 3D model of a building (e.g., detected at a location where the virtual roof of the 3D model of the building is displayed on touch screen 112), the movement on the touch screen display and the movement of the computer system are synchronized to maintain the contact at the same point on the virtual roof (e.g., as the contact moves in an upward direction across the touch screen display, the virtual roof lifts and remains under the user's finger, as the device as a whole moves in an upward direction relative to the building model in physical space, the virtual roof lifts and remains under the user's finger, and the virtual roof remains under the user's finger (e.g., moves up or down) based on the combination of movement of the contact and movement of the device as a whole (e.g., as Figures 5A8-5A13 Maintaining the display of the contact point on the corresponding virtual user interface object at a location corresponding to the input location improves the visual feedback provided to the user (e.g., by making the computer system appear more responsive to the user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve desired results with required input and reducing user errors in operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0391] In some embodiments, the input relative to the movement of the corresponding physical object includes (624) movement of the computer system (e.g., movement of the computer system, including one or more cameras, in physical space) (e.g., as Figure 5A17and the movement of the computer system is derived from image analysis, which indicates that one or more reference points within the field of view of the one or more cameras have changed (e.g., changed position or location within the field of view of the one or more cameras) between consecutive images captured by the one or more cameras (e.g., comparison of consecutive image frames and tracking of objects identified in the images). In some embodiments, the image analysis is performed by the computer system. In some embodiments, the image analysis includes tracking three or more reference points within the field of view of the camera. In some embodiments, the new reference point is identified as an old reference point that has moved out of the field of view of the one or more cameras. In some embodiments, the movement of the computer system is derived from the image analysis rather than from using an inertial measurement unit (IMU) of the computer system. In some embodiments, the movement of the computer system is derived from the image analysis in addition to using one or more elements of the IMU of the computer system (e.g., accelerometers, gyroscopes, and / or magnetometers). Detecting movement of a computer system from image analysis improves the feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve desired results with required input and reducing user errors in operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0392] In some embodiments, adjusting the appearance of the corresponding virtual user interface object includes (626) moving at least a portion of the corresponding virtual user interface object, wherein the movement of the corresponding virtual user interface object is based on the physical shape of the corresponding physical object (e.g., based on the shape of the physical model). For example, in some embodiments, the corresponding physical object is a 3D highway model, and the corresponding virtual user interface object is a virtual car. In this embodiment, adjusting the appearance of the virtual car includes moving the virtual car on the 3D highway model, and the movement of the virtual car is based on the physical shape of the 3D highway model (e.g., the virtual car moves on a slope of the 3D highway model). For another example, in some embodiments, the corresponding physical object is a physical building model (e.g., physical building model 5006, Figure 5A1 ), and the corresponding virtual user interface object is a virtual building model (eg, virtual building model 5012, Figure 5A8 ), and adjusting the appearance of the corresponding virtual user interface object includes moving at least a portion of the corresponding virtual user interface object (e.g., virtual roof 5012-a, Figure 5A9Moving corresponding virtual user interface objects based on the physical shapes of corresponding physical objects improves the feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve desired results with required input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0393] In some embodiments, adjusting the appearance of the corresponding virtual user interface object includes (628) moving at least a portion of the corresponding virtual user interface object, wherein the movement of the corresponding virtual user interface object is based on movement of one or more touch inputs (e.g., a swipe input on the input device) and simultaneous movement of the computer system. For example, in some embodiments, adjusting the appearance of a virtual roof of a 3D model of a building includes moving at least a portion of the virtual roof, wherein the movement of the virtual roof is based on movement of a contact moving across the touch screen display (e.g., in an upward direction) and simultaneous movement of the device as a whole in physical space relative to the building model (e.g., in an upward direction) (e.g., if the touch screen display is in an upward direction). Figure 5A17-Figure 5A18 Devices in Mobile 5028 and Figure 5A19-Figure 5A20 5026 occurs simultaneously with the movement of contact 5026 in the 3D highway model). For another example, in some embodiments, the movement of the virtual car is based on the movement of the virtual car being dragged on the slope of the 3D highway model and the movement of the model itself on the display as the device is moving. Allowing the user to move the corresponding virtual user interface object through the movement of the touch input and the simultaneous movement of the computer system provides an intuitive way for the user to move the corresponding virtual user interface object, improves the feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve the desired result with the required input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and more efficiently.

[0394] In some embodiments, adjusting the appearance of the corresponding virtual user interface object includes (630) moving at least a portion of the corresponding virtual user interface object beyond a maximum limit of the stationary state of the corresponding virtual user interface object (e.g., moving the virtual roof 5012-a beyond a maximum limit of its stationary state, such as Figure 5A13) (e.g., based on movement of the contact across the touch screen display or across the touch-sensitive surface of the input device, movement of one or more cameras relative to the corresponding physical object, or a combination of movement of the contact on the input device and movement of the one or more cameras relative to the corresponding physical object), and the computer system: while continuing to detect the input, generates an input signal based on a magnitude of movement of the input relative to the corresponding physical object (e.g., as Figure 5A13 ), displaying the corresponding virtual user interface object at a position exceeding the maximum limit of the static state of the corresponding virtual user interface object; stopping detecting the input (for example, Figure 5A13 and in response to ceasing to detect the input, displaying the corresponding virtual user interface object at a position corresponding to the maximum limit of the stationary state of the corresponding virtual user interface object (e.g., as Figure 5A14 In some embodiments, this includes displaying an animated transition from a corresponding virtual user interface object located at a position exceeding the maximum limit of the rest state to a position corresponding to the maximum limit of the rest state (e.g., in FIG. Figure 5A13 The position in the diagram shows the distance from the virtual roof 5012-a to Figure 5A14 ). In some embodiments, if the corresponding virtual user interface object moves beyond the maximum extent of its maximum resting state based on the movement of the input, the corresponding virtual user interface object quickly jumps back to its maximum resting state when the input is lifted off (e.g., in an animated transition). For example, if the virtual roof of a 3D building model can be displayed as sitting directly on the 3D building model and hovering up to twelve inches above the building model (e.g., the resting state of the virtual roof is between zero inches and twelve inches from the building model), then if the user raises the virtual roof fifteen inches above the building model, the virtual roof will quickly jump back twelve inches above the building model when the user input is lifted off. Moving the corresponding virtual user interface object based on the amount of input movement (even if it exceeds the maximum limit of the corresponding virtual user interface object's resting state) and then displaying the corresponding virtual user interface object quickly jumping back to its maximum resting state when the input is lifted improves the feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by helping the user achieve the desired results with the required input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and more efficiently.

[0395] In some embodiments, the displayed augmented reality environment includes (632): one or more virtual objects that do not correspond to physical objects in the field of view of the one or more cameras (e.g., a virtual car driving in front of a virtual building that is a substitute for a physical model of the building) (e.g., Figure 5A4 ); one or more physical objects (e.g., a table on which a physical model of a building rests) located in the field of view of one or more cameras (e.g., table 5004 and wallpaper 5007, Figure 5A4 and one or more 3D virtual models of one or more physical objects located in the field of view of the one or more cameras, the one or more 3D virtual models replacing the corresponding one or more physical objects (e.g., virtual building model 5012, Figure 5A4 ) (e.g., a replacement of the physical model of the building) (e.g., the corresponding 3D virtual model is projected onto the corresponding physical marker). In some embodiments, the corresponding 3D virtual model of the corresponding physical object in the field of view of the one or more cameras replaces a portion (but not all) of the corresponding physical object (e.g., the 3D virtual model of the statue's head replaces a portion of the physical statue's head in the field of view of the one or more cameras, showing, for example, an internal cross-section of one quarter of the head). In some embodiments, the corresponding 3D virtual model of the corresponding physical object in the field of view of the one or more cameras replaces all of the corresponding physical objects (e.g., the 3D virtual model of the building replaces the entire physical model of the building in the field of view of the one or more cameras) (e.g., the virtual building model 5012 replaces the entire physical building model 5006 in the augmented reality environment, Figure 5A4 In some embodiments, the displayed augmented reality environment includes all three of the above in different layers (e.g., purely virtual objects, physical objects, and 3D virtual models of physical objects). For example, in some embodiments, the displayed augmented reality environment includes a statue in a museum (e.g., a physical object in the field of view of one or more cameras), with a 3D virtual model of the statue's head (e.g., a 3D virtual model of the statue's head showing an interior cross-section of the statue) in a virtual environment with Egyptian pyramids (e.g., a purely virtual object showing the surroundings of the statue where it is initially displayed).

[0396] In some embodiments, the displayed augmented reality environment includes a subset of the above (e.g., including one or more physical objects located in the field of view of one or more cameras and one or more 3D virtual models of the one or more physical objects, but not including one or more purely virtual objects). For example, in some embodiments, using the above example of a statue in a museum, the displayed augmented reality environment includes the statue in the museum, the statue having a 3D virtual model of the statue's head, but not including any purely virtual objects. As another example, in some embodiments, the displayed augmented reality environment includes a physical 3D model of a building on a table or platform (e.g., a physical object in the field of view of one or more cameras), wherein a 3D virtual model of at least a portion of the building (e.g., a 3D virtual model of a portion of the building showing an interior view of the building) is in a virtual outdoor environment (e.g., with virtual objects such as virtual trees around the building, a virtual car driving in front of the building, or a virtual person walking near the building). As the physical 3D model of the building moves in the field of view (e.g., as the user moves the computer by walking around to different sides of the physical building model, due to movement in the physical world of the building model and / or due to movement of the computer system) (e.g., as the user moves the computer by walking around to different sides of the physical building model), the displayed augmented reality environment includes the statue in the museum with the 3D virtual model of the statue's head, but not including any purely virtual objects. Figure 5A3-5A6 As shown, one or more 3D virtual models of the physical 3D model of the building move accordingly. For example, as the user moves around the building to different sides, the 3D virtual model of the portion of the building showing a view of the interior of the building changes to an updated view corresponding to the physical objects in the field of view of the one or more cameras. Displaying an augmented reality environment with virtual objects, physical objects, and 3D virtual models of the physical objects provides a real view (with the one or more physical objects in the field of view of the one or more cameras) together with supplemental information (with the one or more virtual objects and the one or more 3D virtual models) that provides information to the user, thereby enhancing the operability of the device (e.g., by allowing the user to easily access the supplemental information about the one or more physical objects in the field of view of the one or more cameras) and making the user-device interface more efficient (e.g., by helping the user achieve the desired result with the required input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0397] In some embodiments, the corresponding physical object is (634) a 3D marker that is recognizable from different angles, and the corresponding virtual user interface object is a 3D virtual model that is overlaid on the corresponding physical object (e.g., in the displayed augmented reality environment) based on the camera angle of the one or more cameras. In some embodiments, the camera angle of the one or more cameras corresponds to the orientation of the one or more cameras relative to the corresponding physical object. For example, using the above embodiment, where the displayed augmented reality environment includes a statue in a museum with a 3D virtual model of the statue's head, when the camera angle of the one or more cameras is positioned to include the front of the statue in the field of view of the one or more cameras, the displayed augmented reality environment includes the front of the statue and the 3D virtual model in front of the statue's head. When the camera angle changes (e.g., when a user of the device walks around the statue in the museum while viewing the statue in the field of view of the one or more cameras) and the camera angle of the one or more cameras is positioned to include the back of the statue in the field of view of the one or more cameras, the displayed augmented reality environment includes the back of the statue and the 3D virtual model of the back of the statue's head. As the corresponding physical object moves in the field of view (e.g., due to movement of the corresponding physical object in the physical world and / or due to movement of the computer system causing the corresponding physical object to move in the field of view of one or more cameras), the 3D virtual model superimposed on the corresponding physical object moves accordingly (e.g., changes to follow the corresponding physical object). For example, using the embodiment above, where the displayed augmented reality environment includes a physical 3D building model on a table (e.g., physical building model 5006 on table 5004, Figure 5A1 ), which has a 3D virtual model of a portion of a building showing the interior of the building, when the physical 3D building model is moved (e.g., in the field of view of one or more cameras) (e.g., when the user 5002 moves from Figure 5A3 Go to the location shown in Figure 5A5), while viewing the building in the field of view of the one or more cameras, the 3D virtual model of the building superimposed on the physical 3D building model moves accordingly (for example, as the user walks around the physical 3D building model (for example, from the front of the physical 3D building model to the side of the physical 3D building model)), the 3D virtual model changes from the user's new position to display an interior portion of the building in the field of view of the one or more cameras (for example, from displaying the interior in front of the physical 3D building model to displaying the interior in the side of the physical 3D building model). Superimposing the 3D virtual model on the corresponding physical object based on the camera angle of the one or more cameras improves the feedback provided to the user (for example, by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (for example, by helping the user achieve the desired result with the required input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0398] It should be understood that Figures 6A-6D The specific order in which the operations are described in the description is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 700, 800, 900, 1000, 1100, 1200, and 1300) also apply in a similar manner to the details of the processes described above with respect to Figures 6A-6D For example, the contacts, gestures, user interface objects, intensity thresholds, focus indicators, and / or animations described above with reference to method 600 may optionally have one or more of the features of the contacts, gestures, user interface objects, intensity thresholds, focus indicators, and / or animations described herein with reference to other methods described herein (e.g., methods 700, 800, 900, 1000, 1100, 1200, and 1300). For the sake of brevity, these details are not repeated here.

[0399] Figures 7A-7C7 is a flowchart illustrating a method 700 for applying a filter to a real-time image captured by one or more cameras of a computer system in an augmented reality environment according to some embodiments. In a computer system (e.g., portable multifunction device 100, 100b) having a display generating component (e.g., a display, a projector, a heads-up display, etc.), one or more cameras (e.g., a video camera that continuously provides a live preview of at least a portion of content within the camera's field of view and optionally generates a video output comprising one or more streams of image frames capturing the content within the camera's field of view), and an input device (e.g., a touch-sensitive surface such as a touch-sensitive remote control, or a touch screen display that also serves as the display generating component, a mouse, a joystick, a stylus controller, and / or a camera that tracks one or more features of a user, such as the position of a user's hand), a filter is applied to the real-time image captured by one or more cameras of a computer system in an augmented reality environment according to some embodiments. Figure 1A , device 300, Figure 3A , or a multi-component computer system including a headset 5008 and an input device 5010, Figure 5A2 ) method 700 is performed at. In some embodiments, the input device (e.g., having a touch-sensitive surface) and the display generation component are integrated into the touch-sensitive display. Figure 3B-3C As described, in some embodiments, method 700 is performed at computer system 301, where corresponding components, such as display generation components, one or more cameras, one or more input devices, and optionally one or more gesture sensors are each included in computer system 301 or in communication with the computer system.

[0400] In some embodiments, the display generating component is a touch screen display, and the input device (e.g., having a touch sensitive surface) is on or integrated with the display generating component. In some embodiments, the display generating component is separate from the input device (e.g., Figure 4B and Figure 5A2 Some operations in method 700 are optionally combined, and / or the order of some operations is optionally changed.

[0401] For ease of explanation, some embodiments will be discussed with reference to operations performed on a computer system having a touch-sensitive display system 112 (e.g., on a device 100 having a touch screen 112). However, in response to detecting contact on the touch-sensitive surface of an input device 5010 while displaying the user interface shown in the figure on the display of the headset 5008, optionally on a computer system having the headset 5008 and a separate input device 5010 having a touch-sensitive surface (e.g., as shown in the figure), Figure 5A2 ) perform similar operations.

[0402] As described below, method 700 involves applying filters to a representation of a field of view of one or more cameras of a computer system (e.g., a live preview of the field of view of the one or more cameras) in an augmented reality environment (e.g., where reality is augmented by providing supplemental information to a user that is not available in the physical world), wherein the filters are selected based on virtual environment settings for the augmented reality environment. Applying the filters in real time to live images captured by the one or more cameras provides a user with an intuitive way to interact with the augmented reality environment (e.g., by allowing the user to easily change virtual environment settings for the augmented reality environment (e.g., time of day, scene / environment, etc.)) and allows the user to see changes made to the virtual environment in real time, thereby enhancing the operability of the device and making the user-device interface more efficient (e.g., by reducing the number of steps required to achieve a desired result when operating the device and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently. Additionally, changing the view appearance of the physical environment makes the virtual model more visible (eg, compared to a dark model on a very bright background) while still providing the user with information about the physical environment in which the virtual model has been placed.

[0403] Computer system (eg, device 100, Figure 5A21 ) via a display generating component (eg, touch screen 112, Figure 5A21 ) to display (702) an augmented reality environment (e.g., Figure 5A21 Displaying the augmented reality environment includes (704) simultaneously displaying: corresponding physical objects (e.g., a 3D model of a building, a piece of paper with a printed pattern, a poster on a wall or other physical object, a sculpture sitting on a surface, etc.) (e.g., a physical building model 5006, Figure 5A1 ), wherein the representation is updated when the content of the field of view of the one or more cameras changes (e.g., the representation is a real-time preview of at least a portion of the field of view of the one or more cameras, and the corresponding physical object is included and visible in the camera field of view); and corresponding virtual user interface objects (e.g., a virtual roof of a 3D model of a building, a virtual car parked on a surface represented by a piece of paper with a printed pattern, an interactive logo superimposed on a poster, a virtual 3D mask covering the outline of a sculpture, etc.) at corresponding locations in the representation of the field of view of the one or more cameras (e.g., virtual building model 5012, Figure 5A21), where the corresponding virtual user interface object has a position determined based on the corresponding physical object in the field of view of the one or more cameras. For example, in some embodiments, the corresponding virtual user interface object is a graphical object or a 2D or 3D virtual object that appears to be attached to or appears to overlay the corresponding physical object in the field of view of the one or more cameras (e.g., the virtual building model 5012 is a 3D virtual object that appears to overlay the physical building model 5006, Figure 5A21 ). The position and / or orientation of the corresponding virtual user interface object is determined based on the position, shape and / or orientation of the physical object in the field of view of one or more cameras (e.g., as Figure 5A3 and Figure 5A5 While displaying the augmented reality environment, the computer system detects (706) changes to the augmented reality environment (e.g., swipe input to navigate through time in the augmented reality environment, such as Figures 5A21-5A24 (e.g., selecting different display settings among a plurality of display settings corresponding to corresponding physical objects in the field of view of one or more cameras, such as Figure 5A25-Figure 5A27 In response to detecting an input that changes a virtual environment setting, the computer system (708): adjusts the appearance of the corresponding virtual user interface object in accordance with the change made to the virtual environment setting of the augmented reality environment; and applies a filter to at least a portion of the representation of the field of view of the one or more cameras (e.g., a portion of the representation of the field of view of the one or more cameras that is not obscured by the corresponding virtual user interface object), wherein the filter is selected based on the change made to the virtual environment setting (e.g., applying an overall color filter to darken the scene (e.g., as Figure 5A24 ), adding shadows to both physical and virtual objects based on the direction of the virtual sun (e.g., as shown in Figures 5A21-5A23 ), adding additional virtual objects to the scene (and / or removing virtual objects from the scene) based on a selected story (e.g., a historical view, a building, a day in the life of a view, etc.) (e.g., as shown in Figure 5A25-Figure 5A27 as shown), and changing the color temperature, brightness, contrast, sharpness, transparency, etc. of the camera image output before displaying it in a live preview of the camera's field of view).

[0404] In some embodiments, applying the filter to at least a portion of the representation of the field of view of the one or more cameras results in (710) adjusting the appearance of the augmented reality environment in addition to adjusting the appearance of the corresponding virtual user interface object. In some embodiments, applying the filter to a portion of the representation of the field of view of the one or more cameras that is not obscured by the corresponding virtual user interface object (e.g., Figure 5A24, filters are applied to wallpaper that is not obscured by the virtual scene 5007). For example, when a building view is selected, the virtual roof can be removed (e.g., adjusting the appearance of the corresponding virtual user interface object) to display the interior of the physical building model (e.g., as Figure 5A27 ), and a virtual scene showing the interior of a building under construction is overlaid on a live preview of the physical building model, while blurring out the surrounding physical environment (e.g., using a filter). When displaying a time lapse animation showing construction over a period of several days, a light filter is applied to the portions of the live preview that are not obscured by the virtual scene, so that lighting changes throughout the day are applied to the physical objects surrounding the building model that are also included in the live preview (e.g., wallpaper 5007 is also dimmed in night mode, Figure 5A24 ). In some embodiments, a filter is applied to the augmented reality environment, including the corresponding virtual user interface object. For example, in some embodiments, an overall color filter is applied to the entire representation of the field of view of one or more cameras, including the portion occupied by the corresponding virtual user interface object. In addition to adjusting the appearance of the corresponding virtual user interface object, the appearance of the augmented reality environment is also adjusted, improving the visual feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhancing the operability of the device, and making the user-device interface more efficient (e.g., by helping the user achieve the desired results with the required input and reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and more efficiently.

[0405] In some embodiments, the virtual environment setting is changed (712) to a night mode; and applying a filter to at least a portion of the representation of the field of view of the one or more cameras includes: reducing the brightness of an image (or image sequence) captured by the one or more cameras; and applying a color filter to the image (or image sequence) captured by the one or more cameras (e.g., Figure 5A24In some embodiments, filters are applied to images captured by the one or more cameras before the image output is displayed in a real-time preview of the field of view of the one or more cameras (e.g., before the images captured by the one or more cameras are displayed in the augmented reality environment), as described below with respect to operation (726). Applying filters for night mode (e.g., by reducing brightness and applying color filters) improves visual feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by reducing the number of steps required to achieve a desired result and reducing user errors in operating / interacting with the device), which additionally reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and more efficiently.

[0406] In some embodiments, the input for changing the virtual environment setting is (714) a swipe input that navigates through time in the augmented reality environment (e.g., from left to right or from right to left). For example, in some embodiments, when the user swipes on the input device from left to right, the time of day in the augmented reality environment changes from day to night (e.g., based on the speed and / or distance the swipe input moves) (e.g., as Figures 5A21-5A24 Allowing a user to navigate through an augmented reality environment using swipe input provides an intuitive way for the user to change settings of the virtual environment over time, improves feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by reducing the number of steps required to achieve a desired result and reducing user errors in operating / interacting with the device), which additionally reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently.

[0407] In some embodiments, detecting an input that changes a virtual environment setting includes (716) detecting movement of the input that changes the virtual environment setting; adjusting an appearance of a corresponding virtual user interface object based on the change to the virtual environment setting of the augmented reality environment includes gradually adjusting the appearance of the corresponding virtual user interface object based on the movement of the input that changes the virtual environment setting; and applying a filter to at least a portion of a representation of the field of view of one or more cameras includes gradually applying the filter (e.g., as described in detail in the preceding text) based on the movement of the input that changes the virtual environment setting. Figures 5A21-5A24For example, in some embodiments, the filter is gradually applied based on the movement of the input, and the appearance of the corresponding virtual user interface object (e.g., movement of contact 5030 on touch screen 112, Figures 5A21-5A24 Gradually adjusting the appearance of virtual user interface objects and gradually applying filters based on movement of input that changes settings of the virtual environment improves visual feedback provided to the user (e.g., by making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by reducing the number of steps required to achieve a desired result and reducing user errors in operating / interacting with the device), which additionally reduces power usage and improves battery life of the device by enabling the user to use the device more quickly and efficiently.

[0408] In some embodiments, the corresponding virtual user interface object casts (718) a shadow on the corresponding physical object in the augmented reality environment. For example, in some embodiments, the virtual roof of the 3D model of the building casts a shadow on the 3D model of the building. As the time of day or lighting angle changes (e.g., by changing the virtual environment settings), the shadow cast by the corresponding physical object on the corresponding virtual user interface object changes accordingly. For example, Figures 5A21-5A23 As shown, as the time of day changes, the shadow cast by the virtual building model 5012 changes accordingly. Displaying virtual user interface objects with shadows (e.g., cast on physical objects) in an augmented reality environment improves the visual feedback provided to the user (e.g., by making the augmented reality environment more realistic when the user changes virtual environment settings and making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and more efficiently.

[0409] In some embodiments, the corresponding physical object casts (720) a shadow on the corresponding virtual user interface object in the augmented reality environment. For example, in some embodiments, the 3D model of the building casts a shadow on a virtual car parked next to the building. As the time of day or lighting angle changes (e.g., by changing the virtual environment settings or due to the movement of the physical object), the shadow cast by the corresponding physical object on the corresponding virtual user interface object changes accordingly. For example, in some embodiments, as the time of day in the augmented reality environment changes from midday (e.g., when the corresponding shadow of the object in the augmented reality environment is relatively small) to morning or afternoon (e.g., when the corresponding shadow of the object in the augmented reality environment is longer), the shadow cast by the corresponding physical object on the corresponding virtual user interface object changes accordingly (e.g., as the time of day changes from morning to midday, the shadow becomes smaller / shorter, and as the time of day changes from midday to afternoon, the shadow becomes larger / longer). In some embodiments, the 3D virtual model of the corresponding physical object is used to determine where the shadow of the corresponding physical object should be in the augmented reality environment. Although in Figures 5A21-5A24 , the virtual building model 5012 completely covers the physical building model 5006, but if a portion of the physical building model 5006 is exposed, that portion of the physical building model 5006 will cast a similar shadow as the time of day changes in the augmented reality environment. Displaying physical objects with shadows (e.g., cast on virtual user interface objects) in the augmented reality environment improves the visual feedback provided to the user (e.g., by making the augmented reality environment more realistic when the user changes virtual environment settings and making the computer system appear more responsive to user input), enhances the operability of the device, and makes the user-device interface more efficient (e.g., by reducing user errors when operating / interacting with the device), which additionally reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0410] In some embodiments, movement of the corresponding physical object (e.g., due to movement of the corresponding physical object in the physical world and / or due to movement of the computer system that causes the corresponding physical object to move in the field of view of one or more cameras) (e.g., user 5002 moves from a first position (e.g., as shown in FIG. Figure 5A3 ) to a second position (e.g., as shown in Figure 5A5 ) causes (722) one or more changes in the appearance of corresponding virtual user interface objects in the augmented reality environment (e.g., changing the view of the virtual building model 5012 from a front view to a side view, Figure 5A3-5A6In some embodiments, movement of the corresponding physical object causes the corresponding physical object to cast a shadow on the corresponding virtual user interface object in different ways because the ambient light source is at different angles relative to the corresponding physical object as the correspondin...

Claims

1. A method comprising: At a computer system having a display generating component, one or more cameras, and an input device: displaying, via the display generating component, an augmented reality environment, wherein displaying the augmented reality environment comprises simultaneously displaying: a representation of at least a portion of a field of view of the one or more cameras including a corresponding physical object, wherein the representation is updated as content of the field of view of the one or more cameras changes; as well as a first virtual user interface object displayed at a corresponding position in the representation of the field of view of the one or more cameras, wherein the first virtual user interface object has a position determined based on the corresponding physical object in the field of view of the one or more cameras; while displaying the augmented reality environment, detecting a first input corresponding to a selection of the first virtual user interface object; as well as In response to detecting the first input corresponding to selection of the first virtual user interface object: ceasing to display at least a portion of the representation of the field of view of the one or more cameras; as well as A view of a virtual model is displayed, wherein the first virtual user interface object has a corresponding position in the virtual model.

2. The method according to claim 1, comprising: In response to detecting the first input corresponding to selection of the first virtual user interface object, ceasing to display the representation of the field of view of the one or more cameras.

3. The method according to claim 1, comprising: While displaying the augmented reality environment, in response to detecting movement of at least a portion of the computer system that changes the field of view of the one or more cameras, updating the representation of the field of view of the one or more cameras.

4. The method according to claim 1, comprising: While displaying the augmented reality environment, in response to detecting movement of at least a portion of the computer system that changes the perspective of the contents of the field of view of the one or more cameras, updating the representation of the field of view of the one or more cameras and the first virtual user interface object based on the change in the perspective of the contents of the field of view.

5. The method according to claim 1, comprising: In response to detecting the first input corresponding to selection of the first virtual user interface object, displaying an animated transition from the augmented reality environment to the view of the virtual model.

6. The method according to claim 1, comprising: while displaying the view of the virtual model, detecting a second input corresponding to a request to display the augmented reality environment; as well as In response to the second input corresponding to the request to display the augmented reality environment: displaying an animated transition from the view of the virtual model to the augmented reality environment; as well as The augmented reality environment is displayed.

7. A method according to claim 6, wherein displaying the augmented reality environment in response to the second input includes displaying the augmented reality environment according to the field of view of the one or more cameras after detecting the second input, wherein the field of view of the one or more cameras after detecting the second input is different from the field of view of the one or more cameras when the first input is detected.

8. The method of any one of claims 1 to 7, wherein while displaying the view of the virtual model, the first virtual user interface object moves within the virtual model in response to one or more inputs from a user of the computer system.

9. The method according to any one of claims 1 to 7, comprising: While displaying the view of the virtual model, detecting movement of at least a portion of the computer system, and in response to detecting the movement of the computer system, changing the view of the virtual model in accordance with the movement of the computer system.

10. The method according to any one of claims 1 to 7, comprising: While displaying the view of the virtual model, detecting a third input corresponding to a request to exit the virtual model; and In response to detecting the third input: Redisplaying the augmented reality environment; as well as Displaying the view of the virtual model is stopped.

11. A computer system comprising: Display generated components; one or more cameras; Input devices; one or more processors; as well as A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: displaying, via the display generating component, an augmented reality environment, wherein displaying the augmented reality environment comprises simultaneously displaying: a representation of at least a portion of a field of view of the one or more cameras including a corresponding physical object, wherein the representation is updated as content of the field of view of the one or more cameras changes; as well as a first virtual user interface object displayed at a corresponding position in the representation of the field of view of the one or more cameras, wherein the first virtual user interface object has a position determined based on the corresponding physical object in the field of view of the one or more cameras; while displaying the augmented reality environment, detecting a first input corresponding to a selection of the first virtual user interface object; as well as In response to detecting the first input corresponding to selection of the first virtual user interface object: ceasing to display at least a portion of the representation of the field of view of the one or more cameras; as well as A view of a virtual model is displayed, wherein the first virtual user interface object has a corresponding position in the virtual model.

12. The computer system of claim 11, wherein the one or more programs include instructions for executing the method of any one of claims 2 to 10.

13. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system having a display generating component, one or more cameras, and an input device, cause the computer system to: displaying, via the display generating component, an augmented reality environment, wherein displaying the augmented reality environment comprises simultaneously displaying: a representation of at least a portion of a field of view of the one or more cameras including a corresponding physical object, wherein the representation is updated as content of the field of view of the one or more cameras changes; as well as a first virtual user interface object displayed at a corresponding position in the representation of the field of view of the one or more cameras, wherein the first virtual user interface object has a position determined based on the corresponding physical object in the field of view of the one or more cameras; while displaying the augmented reality environment, detecting a first input corresponding to a selection of the first virtual user interface object; as well as In response to detecting the first input corresponding to selection of the first virtual user interface object: ceasing to display at least a portion of the representation of the field of view of the one or more cameras; as well as A view of a virtual model is displayed, wherein the first virtual user interface object has a corresponding position in the virtual model.

14. The computer-readable storage medium of claim 13, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to perform the method of any one of claims 2 to 10.

15. A method comprising: At a computer system having a display generating component, one or more gesture sensors, and an input device: displaying, via the display generating component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, wherein displaying the simulated environment in the first viewing mode comprises: displaying a first virtual user interface object in a virtual model displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system; while displaying the simulated environment, detecting, via the one or more gesture sensors, a first change in a posture of at least a portion of the computer system relative to the physical environment; in response to detecting the first change in the pose of the portion of the computer system, changing the appearance of the first virtual user interface object in the virtual model so as to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; detecting, via the input device, a first gesture corresponding to an interaction with the simulated environment after changing the appearance of the first virtual user interface object based on the first change in the posture of the portion of the computer system; in response to detecting the first gesture corresponding to the interaction with the simulated environment, performing an operation corresponding to the first gesture in the simulated environment; After performing the operation corresponding to the first gesture, detecting, via the one or more gesture sensors, a second change in the posture of the portion of the computer system relative to the physical environment; and In response to detecting the second change in the posture of the portion of the computer system: transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode based on determining that the first gesture satisfies a mode change criterion, wherein the mode change criterion includes requiring that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, wherein displaying the virtual model in the simulated environment in the second viewing mode includes forgoing changing the appearance of the first virtual user interface object to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; and Based on determining that the first gesture does not satisfy the mode change criteria, continue to display the virtual model in the simulated environment in the first viewing mode, wherein displaying the virtual model in the first viewing mode includes changing the appearance of the first virtual user interface object in the virtual model in response to the second change in the posture of the portion of the computer system relative to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment.

16. A computer system comprising: Display generated components; one or more attitude sensors; Input devices; one or more processors; as well as A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: displaying, via the display generating component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, wherein displaying the simulated environment in the first viewing mode comprises: displaying a first virtual user interface object in a virtual model displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system; while displaying the simulated environment, detecting, via the one or more gesture sensors, a first change in a posture of at least a portion of the computer system relative to the physical environment; in response to detecting the first change in the pose of the portion of the computer system, changing the appearance of the first virtual user interface object in the virtual model so as to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; detecting, via the input device, a first gesture corresponding to an interaction with the simulated environment after changing the appearance of the first virtual user interface object based on the first change in the posture of the portion of the computer system; in response to detecting the first gesture corresponding to the interaction with the simulated environment, performing an operation corresponding to the first gesture in the simulated environment; After performing the operation corresponding to the first gesture, detecting, via the one or more gesture sensors, a second change in the posture of the portion of the computer system relative to the physical environment; and In response to detecting the second change in the posture of the portion of the computer system: transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode based on determining that the first gesture satisfies a mode change criterion, wherein the mode change criterion includes requiring that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, wherein displaying the virtual model in the simulated environment in the second viewing mode includes forgoing changing the appearance of the first virtual user interface object to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; and Based on determining that the first gesture does not satisfy the mode change criteria, continue to display the virtual model in the simulated environment in the first viewing mode, wherein displaying the virtual model in the first viewing mode includes changing the appearance of the first virtual user interface object in the virtual model in response to the second change in the posture of the portion of the computer system relative to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment.

17. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system having a display generation component, one or more gesture sensors, and an input device, cause the computer system to: Displaying, via the display generating component, a simulated environment oriented relative to a physical environment of the computer system in a first viewing mode, wherein displaying the simulated environment in the first viewing mode comprises: displaying a first virtual user interface object in a virtual model displayed at a first corresponding location in the simulated environment associated with the physical environment of the computer system; while displaying the simulated environment, detecting, via the one or more gesture sensors, a first change in a posture of at least a portion of the computer system relative to the physical environment; in response to detecting the first change in the pose of the portion of the computer system, changing the appearance of the first virtual user interface object in the virtual model so as to maintain a fixed spatial relationship between the first virtual user interface object and the physical environment; detecting, via the input device, a first gesture corresponding to an interaction with the simulated environment after changing the appearance of the first virtual user interface object based on the first change in the posture of the portion of the computer system; in response to detecting the first gesture corresponding to the interaction with the simulated environment, performing an operation corresponding to the first gesture in the simulated environment; After performing the operation corresponding to the first gesture, detecting, via the one or more gesture sensors, a second change in the posture of the portion of the computer system relative to the physical environment; as well as In response to detecting the second change in the posture of the portion of the computer system: transitioning from displaying the simulated environment including the virtual model in the first viewing mode to displaying the simulated environment including the virtual model in a second viewing mode based on determining that the first gesture satisfies a mode change criterion, wherein the mode change criterion includes requiring that the first gesture corresponds to an input that changes a spatial parameter of the simulated environment relative to the physical environment, wherein displaying the virtual model in the simulated environment in the second viewing mode includes forgoing changing the appearance of the first virtual user interface object to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment; as well as Based on determining that the first gesture does not satisfy the mode change criteria, continue to display the virtual model in the simulated environment in the first viewing mode, wherein displaying the virtual model in the first viewing mode includes changing the appearance of the first virtual user interface object in the virtual model in response to the second change in the posture of the portion of the computer system relative to the physical environment so as to maintain the fixed spatial relationship between the first virtual user interface object and the physical environment.

18. A method comprising: At a first computer system having a first display generating component, one or more first gesture sensors, and a first input device: displaying, via the first display generating component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, wherein displaying the simulated environment comprises simultaneously displaying: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; and a visual indication of a viewing perspective of a second computer system of the simulated environment, wherein the second computer system is a computer system having a second display generating component, one or more second gesture sensors, and a second input device, the second computer system displaying, via the second display generating component of the second computer system, a view of the simulated environment oriented relative to a second physical environment of the second computer system; detecting a change in the viewing perspective of the second computer system of the simulated environment based on a change in a posture of a portion of the second computer system relative to the second physical environment of the second computer system while the simulated environment is displayed via the first display generation component of the first computer system; as well as In response to detecting the change in the viewing perspective of the second computer system of the simulated environment based on the change in the posture of the portion of the second computer system relative to the second physical environment of the second computer system, updating the visual indication of the viewing perspective of the second computer system of the simulated environment displayed by the first display generating component of the first computer system in accordance with the change in the viewing perspective of the second computer system of the simulated environment.

19. A first computer system comprising: a first display generating component; one or more first posture sensors; first input device; one or more processors; as well as A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: displaying, via the first display generating component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, Wherein displaying the simulated environment comprises simultaneously displaying: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; and a visual indication of a viewing perspective of a second computer system of the simulated environment, wherein the second computer system is a computer system having a second display generating component, one or more second gesture sensors, and a second input device, the second computer system displaying, via the second display generating component of the second computer system, a view of the simulated environment oriented relative to a second physical environment of the second computer system; detecting a change in the viewing perspective of the second computer system of the simulated environment based on a change in a posture of a portion of the second computer system relative to the second physical environment of the second computer system while the simulated environment is displayed via the first display generation component of the first computer system; as well as In response to detecting the change in the viewing perspective of the second computer system of the simulated environment based on the change in the posture of the portion of the second computer system relative to the second physical environment of the second computer system, updating the visual indication of the viewing perspective of the second computer system of the simulated environment displayed by the first display generating component of the first computer system in accordance with the change in the viewing perspective of the second computer system of the simulated environment.

20. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a first computer system having a first display generating component, one or more first gesture sensors, and a first input device, cause the first computer system to: displaying, via the first display generating component of the first computer system, a simulated environment oriented relative to a first physical environment of the first computer system, wherein displaying the simulated environment comprises simultaneously displaying: a first virtual user interface object in a virtual model displayed at a corresponding location in the simulated environment associated with the first physical environment of the first computer system; and a visual indication of a viewing perspective of a second computer system of the simulated environment, wherein the second computer system is a computer system having a second display generating component, one or more second gesture sensors, and a second input device, the second computer system displaying, via the second display generating component of the second computer system, a view of the simulated environment oriented relative to a second physical environment of the second computer system; detecting a change in the viewing perspective of the second computer system of the simulated environment based on a change in a posture of a portion of the second computer system relative to the second physical environment of the second computer system while the simulated environment is displayed via the first display generation component of the first computer system; as well as In response to detecting the change in the viewing perspective of the second computer system of the simulated environment based on the change in the posture of the portion of the second computer system relative to the second physical environment of the second computer system, updating the visual indication of the viewing perspective of the second computer system of the simulated environment displayed by the first display generating component of the first computer system in accordance with the change in the viewing perspective of the second computer system of the simulated environment.

Images