DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR SYSTEM-WIDE BEHAVIOR OF 3D MODELS - Patent application

Abstract

To provide a device, method and a graphical user interface for displaying a virtual object in various situations.SOLUTION: A method includes: displaying a representation of a virtual object in a first user interface region on a display; detecting a first input by a contact at a location on a touch-sensitive surface that corresponds to the representation of the virtual object on the display; in response to detecting the first input by the contact, in accordance with a determination that the first input by the contact meets first criteria, displaying a second user interface region on the display, including replacing display of at least a portion of the first user interface region with the representation of a field of view of one or more cameras; and continuously displaying the representation of the virtual object while switching from displaying the first user interface region to displaying the second user interface region.SELECTED DRAWING: Figure 16A

Description

[Technical Field]

[0001] The present invention relates generally to electronic devices that display virtual objects, including but not limited to electronic devices that display virtual objects in various contexts. [Background technology]

[0002] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Interaction with the virtual / augmented reality environment involves the use of input devices, such as touch-sensitive surfaces of computer systems and other electronic computing devices. Exemplary touch-sensitive surfaces include touchpads, touch-sensitive remote controls, and touchscreen displays. These surfaces are used to manipulate user interfaces and objects therein on the display. Exemplary user interface objects include digital images, video, text, icons, and control elements such as buttons and other graphics.

[0003] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, using a series of inputs to orient and position virtual objects in an augmented reality environment is time-consuming and places a significant cognitive burden on the user, detracting from the experience of the virtual / augmented reality environment. Furthermore, these methods take longer than necessary, thereby wasting energy. The latter issue is particularly important in battery-operated devices. Summary of the Invention

[0004] Thus, there is a need for computer systems with improved methods and interfaces for interacting with virtual objects. Such methods and interfaces, optionally complementing or replacing conventional methods for interacting with virtual objects, reduce the number, range, and / or type of inputs from a user, creating a more efficient human-machine interface. In battery-operated devices, such methods and interfaces conserve power and extend the time between battery charges.

[0005] The above-mentioned deficiencies and other problems associated with interfaces for interacting with virtual objects (e.g., user interfaces for augmented reality (AR) and related non-AR interfaces) are reduced or eliminated by the disclosed computer system. In some embodiments, the computer system comprises a desktop computer. In some embodiments, the computer system is portable (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system comprises a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the computer system has (and / or is in communication with) a touchpad. In some embodiments, the computer system has (and / or is in communication with) a touch-sensitive display (also referred to as a "touchscreen" or "touchscreen display"). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing a plurality of functions. In some embodiments, a user interacts with the GUI, in part, through contacts and gestures with a stylus and / or fingers on a touch-sensitive surface. In some embodiments, the functionality optionally includes game playing, image editing, drawing, presenting, word processing, spreadsheet creation, telephony, video conferencing, email, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note taking, and / or digital video playback. Executable instructions for performing these functionality are optionally contained in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0006] According to some embodiments, a method is performed on a computer system having a display, a touch-sensitive surface, and one or more cameras. The method includes displaying a representation of a virtual object in a first user interface area on the display. The method also includes, while displaying the first representation of the virtual object in the first user interface area on the display, detecting a first contact input on the touch-sensitive surface at a location corresponding to the representation of the virtual object on the display. The method also includes, in response to detecting the first contact input, displaying a second user interface area on the display, including replacing a display of at least a portion of the first user interface area with a representation of a field of view of the one or more cameras in accordance with a determination that the first contact input satisfies a first criterion, and continuously displaying a representation of the virtual object while switching from displaying the first user interface area to displaying the second user interface area.

[0007] According to some embodiments, a method is performed on a computer system having a display, a touch-sensitive surface, and one or more cameras. The method includes displaying a first representation of a virtual object in a first user interface area on the display. The method also includes, while displaying the first representation of the virtual object in the first user interface area on the display, detecting a first input via a first contact on the touch-sensitive surface at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display. The method also includes, in response to detecting the first input via the first contact, displaying a second representation of the virtual object in a second user interface area different from the first user interface area in accordance with a determination that the first input via the first contact satisfies a first criterion. The method also includes detecting a second input while displaying a second representation of the virtual object in the second user interface region; and, in response to detecting the second input, altering a display characteristic of the second representation of the virtual object in the second user interface region based on the second input in accordance with a determination that the second input corresponds to a request to manipulate the virtual object in the second user interface region; and, in accordance with a determination that the second input corresponds to a request to display the virtual object in an augmented reality environment, displaying a third representation of the virtual object along with a representation of the field of view of the one or more cameras.

[0008] According to some embodiments, a method is performed on a computer system having a display and a touch-sensitive surface. The method includes, in response to a request to display a first user interface, displaying the first user interface with a representation of a first item. The method also includes, in accordance with a determination that the first item corresponds to a respective virtual three-dimensional object, displaying the representation of the first item with a visual indication that the first item corresponds to the first respective virtual three-dimensional object. The method also includes, in accordance with a determination that the first item does not correspond to a respective virtual three-dimensional object, displaying the representation of the first item without the visual indication. The method also includes, after displaying the representation of the first item, receiving a request to display a second user interface including the second item. The method also includes, in response to the request to display the second user interface, displaying the second user interface with the representation of the second item. The method also includes, in accordance with determining that the second item corresponds to the respective virtual three-dimensional object, displaying a representation of the second item with a visual indication that the second item corresponds to the second respective virtual three-dimensional object. The method also includes, in accordance with determining that the second item does not correspond to the respective virtual three-dimensional object, displaying a representation of the second item without the visual indication.

[0009] According to some embodiments, a method is performed on a computer system having a display generation component, one or more input devices, and one or more cameras. The method includes receiving a request to display a virtual object in a first user interface region that includes at least a portion of a field of view of the one or more cameras. The method also includes, in response to the request to display the virtual object in the first user interface region, displaying, via the display generation component, a representation of the virtual object over at least a portion of a field of view of the one or more cameras included in the first user interface region, where the field of view of the one or more cameras is a view of a physical environment in which the one or more cameras are located. Displaying the representation of the virtual object includes, pursuant to a determination that the object placement criterion is not satisfied, displaying the representation of the virtual object with a first set of visual characteristics and in a first orientation independent of which portions of the physical environment are displayed within the field of view of the one or more cameras, where the object placement criterion, to be satisfied, requires that a placement location of the virtual object be identified within the field of view of the one or more cameras; and, pursuant to a determination that the object placement criterion is satisfied, displaying the representation of the virtual object with a second set of visual characteristics distinct from the first set of visual characteristics and in a second orientation corresponding to a plane in the physical environment detected within the field of view of the one or more cameras.

[0010] According to some embodiments, a method is executed on a computer system having a display generation component, one or more input devices, one or more cameras, and one or more orientation sensors that detect changes in orientation of a device including the one or more cameras. The method includes receiving a request to display an augmented reality view of a physical environment in a first user interface area that includes a representation of a field of view of the one or more cameras. The method also includes, in response to receiving the request to display the augmented reality view of the physical environment, displaying a representation of the field of view of the one or more cameras, and, in accordance with determining that a calibration criterion is not satisfied for the augmented reality view of the physical environment, displaying a calibration user interface object that is dynamically animated in accordance with movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes detecting, via the one or more orientation sensors, a change in orientation of the one or more cameras in the physical environment, and, in response to detecting the change in orientation of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object in accordance with the detected change in orientation of the one or more cameras in the physical environment. The method also includes detecting that a calibration criterion has been met while displaying a calibration user interface object that moves on the display according to the detected change in pose of the one or more cameras in the physical environment. The method also includes ceasing displaying the calibration user interface object in response to detecting that the calibration criterion has been met.

[0011] According to some embodiments, a method is performed on a computer system having a display generation component and one or more input devices including a touch-sensitive surface. The method includes displaying, by the display generation component, a representation of a first perspective of a virtual three-dimensional object in a first user interface area. The method also includes, while displaying the representation of the first perspective of the virtual three-dimensional object in the first user interface area on the display, detecting a first input corresponding to a request to rotate the virtual three-dimensional object relative to the display to display a portion of the virtual three-dimensional object that is not viewable from the first perspective of the virtual three-dimensional object. The method also includes, in response to detecting the first input, in accordance with a determination that the first input corresponds to a request to rotate the three-dimensional object about a first axis, rotating the virtual three-dimensional object about the first axis by an amount determined based on a magnitude of the first input and constrained by a limit imposed on movement that constrains rotation of the virtual three-dimensional object an amount that exceeds a threshold rotation amount about the first axis; and in accordance with a determination that the first input corresponds to a request to rotate the three-dimensional object about a second axis, different from the first axis, rotating the virtual three-dimensional object about the second axis by an amount determined based on the magnitude of the first input, wherein for inputs having a magnitude that exceeds each threshold, the device rotates the virtual three-dimensional object about the second axis by an amount that exceeds the threshold rotation amount.

[0012] According to some embodiments, a method is performed on a computer system having a display generation component and a touch-sensitive surface. The method includes displaying, via the display generation component, a first user interface area including a user interface object associated with a plurality of object manipulation behaviors, including a first object manipulation behavior performed in response to input that satisfies first gesture recognition criteria and a second object manipulation behavior performed in response to input that satisfies second gesture recognition criteria. The method also includes detecting a first portion of an input directed to the user interface object while displaying the first user interface area, including detecting movement of one or more contacts across the touch-sensitive surface, and evaluating the movement of the one or more contacts with respect to both the first gesture recognition criteria and the second gesture recognition criteria while the one or more contacts are detected on the touch-sensitive surface. The method also includes, in response to detecting the first portion of the input, updating an appearance of the user interface object based on the first portion of the input, including: in accordance with a determination that the first portion of the input satisfies the first gesture recognition criterion before the second gesture recognition criterion, changing the appearance of the user interface object according to a first object manipulation behavior based on the first portion of the input and updating the second gesture recognition criterion by increasing a threshold value for the second gesture recognition criterion; and in accordance with a determination that the input satisfies the second gesture recognition criterion before the first gesture recognition criterion, changing the appearance of the user interface object according to the second object manipulation behavior based on the first portion of the input and updating the first gesture recognition criterion by increasing a threshold value for the first gesture recognition criterion.

[0013] According to some embodiments, a method is implemented on a computer system having a display generation component, one or more input devices, one or more audio output generators, and one or more cameras. The method includes displaying, via the display generation component, a representation of a virtual object in a first user interface area including a representation of a field of view of the one or more cameras, where the displaying includes maintaining a first spatial relationship between the representation of the virtual object and a plane detected in a physical environment captured within the field of view of the one or more cameras. The method also includes detecting movement of the device adjusting the field of view of the one or more cameras. In response to detecting movement of the device adjusting the field of view of the one or more cameras, the method also includes adjusting a display of the representation of the virtual object in the first user interface area according to the first spatial relationship between the virtual object and the plane detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted, and generating, via the one or more audio output generators, a first audio alert in response to determining that the movement of the device causes the virtual object to move greater than a threshold amount outside of a displayed portion of the field of view of the one or more cameras.

[0014] According to some embodiments, an electronic device includes a display generation component, optionally one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, optional one or more sensors for detecting intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, optional one or more posture sensors for detecting changes in posture, one or more processors, and a memory having stored thereon one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing or causing the performance of any of the operations of the methods described herein. According to some embodiments, a computer-readable storage medium has stored thereon instructions that, when executed by an electronic device comprising a display generation component, optionally one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, optional one or more sensors that detect intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, and optional one or more posture sensors, cause the device to perform or cause the performance of any of the operations of the methods described herein. According to some embodiments, a graphical user interface of an electronic device comprising a display generation component, optionally one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, one or more sensors that detect intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, optional one or more posture sensors, memory, and one or more processors that execute one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein that are updated in response to input as described in any of the methods described herein.According to some embodiments, an electronic device includes a display generation component, optionally one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, optional one or more sensors that detect intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, optional one or more posture sensors that detect changes in posture, and means for performing or causing the performance of any of the operations of the methods described herein. According to some embodiments, an information processing apparatus for use in an electronic device including a display generation component, optional one or more input devices, optional one or more touch-sensitive surface, optional one or more cameras, optional one or more sensors that detect intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, and optional one or more posture sensors that detect changes in posture includes means for performing or causing the performance of any of the operations of the methods described herein.

[0015] Thus, electronic devices comprising a display generation component, optionally one or more input devices, optional one or more touch-sensitive surfaces, optional one or more cameras, optional one or more sensors for detecting the intensity of contact with the touch-sensitive surface, optional one or more audio output generators, optional one or more device orientation sensors, optional one or more tactile output generators, and optional one or more attitude sensors are provided with improved methods and interfaces for displaying virtual objects in a variety of contexts, thereby increasing the effectiveness, efficiency, and user satisfaction of these devices. These methods and interfaces can complement or replace conventional methods for displaying virtual objects in a variety of contexts. [Brief explanation of the drawings]

[0016] For a better understanding of the various described embodiments, please refer to the following detailed description in connection with the following drawings, in which like reference numerals designate corresponding parts throughout:

[0017] [Figure 1A] FIG. 1 is a block diagram illustrating a portable multifunction device with a touch-sensitive display in accordance with some embodiments.

[0018] [Figure 1B] FIG. 1 is a block diagram illustrating example components for event handling, according to some embodiments.

[0019] [Figure 1C] FIG. 1 is a block diagram illustrating a tactile output module, according to some embodiments.

[0020] [Figure 2] FIG. 1 illustrates a portable multifunction device with a touch screen according to some embodiments.

[0021] [Figure 3] FIG. 1 is a block diagram illustrating an exemplary multifunction device with a display and a touch-sensitive surface in accordance with some embodiments.

[0022] [Figure 4A] 1 illustrates an exemplary user interface for a menu of applications on a portable multifunction device according to some embodiments.

[0023] [Figure 4B] 1A-1C illustrate exemplary user interfaces for a multifunction device with a touch-sensitive surface separate from a display in accordance with some embodiments.

[0024] [Figure 4C] FIG. 10 illustrates an example of a dynamic intensity threshold, according to some embodiments. [Figure 4D] FIG. 10 illustrates an example of a dynamic intensity threshold, according to some embodiments. [Figure 4E] FIG. 10 illustrates an example of a dynamic intensity threshold, according to some embodiments.

[0025] [Figure 4F] FIG. 10 illustrates a set of sample tactile output patterns, according to some embodiments. [Figure 4G] FIG. 10 illustrates a set of sample tactile output patterns, according to some embodiments. [Figure 4H] FIG. 10 illustrates a set of sample tactile output patterns, according to some embodiments. [Figure 4I] FIG. 10 illustrates a set of sample tactile output patterns, according to some embodiments. [Figure 4J] FIG. 10 illustrates a set of sample tactile output patterns, according to some embodiments. [Figure 4K] FIG. 10 illustrates a set of sample tactile output patterns, according to some embodiments.

[0026] [Figure 5A] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5B] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5C] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5D] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5E] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5F] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5G] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5H] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5I] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5J] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5K] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5L] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5M] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5N] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5O] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5P] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5Q] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5R] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5S] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5T] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5U] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5V] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5W] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5X] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5Y] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5Z] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AA] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AB] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AC] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AD] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AE] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AF] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AG] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AH] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AI] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AJ] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AK] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AL] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AM] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AN] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AO] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AP] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AQ] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AR] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AS] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 5AT] FIG. 10 illustrates an exemplary user interface displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments.

[0027] [Figure 6A] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6B] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6C] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6D] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6E]FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6F] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6G] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6H] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6I] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6J]FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6K] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6L] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6M] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6N] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6O]FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6P] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6Q] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6R] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6S] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6T]FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6U] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6V] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6W] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6X] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6Y]FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6Z] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AA] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AB] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AC] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AD]FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AE] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AF] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AG] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AH] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AI]FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras. [Figure 6AJ] FIG. 10 illustrates an exemplary user interface according to some embodiments, displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras.

[0028] [Figure 7A] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7B] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7C] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7D] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7E] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7F1]10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7F2] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7G1] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7G2] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7H] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7I] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7J] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7K] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7L] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7M]10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7N] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7O] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments. [Figure 7P] 10A-10C illustrate exemplary user interfaces displaying items along with visual indications that the items correspond to virtual three-dimensional objects, according to some embodiments.

[0029] [Figure 8A] 10 is a flow diagram illustrating a process for displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 8B] 10 is a flow diagram illustrating a process for displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 8C] 10 is a flow diagram illustrating a process for displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 8D] 10 is a flow diagram illustrating a process for displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments. [Figure 8E] 10 is a flow diagram illustrating a process for displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region, according to some embodiments.

[0030] [Figure 9A] 1 is a flow diagram illustrating a process for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras, according to some embodiments. [Figure 9B] 1 is a flow diagram illustrating a process for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras, according to some embodiments. [Figure 9C] 1 is a flow diagram illustrating a process for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras, according to some embodiments. [Figure 9D] 1 is a flow diagram illustrating a process for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object along with representations of the fields of view of one or more cameras, according to some embodiments.

[0031] [Figure 10A] 1 is a flow diagram illustrating a process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, according to some embodiments. [Figure 10B]1 is a flow diagram illustrating a process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, according to some embodiments. [Figure 10C] 1 is a flow diagram illustrating a process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, according to some embodiments. [Figure 10D] 1 is a flow diagram illustrating a process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, according to some embodiments.

[0032] [Figure 11A] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11B] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11C] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11D] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11E] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11F] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11G]10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11H] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11I] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11J] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11K] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11L] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11M] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11N] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11O] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11P]10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11Q] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11R] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11S] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11T] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11U] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 11V] 10A-10C illustrate example user interfaces that display virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments.

[0033] [Figure 12A] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12B]FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12C] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12D] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12E-1] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12E-2] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12F-1] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12F-2] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12G-1] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12G-2]FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12H-1] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12H-2] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12I-1] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12I-2] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12J] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12K-1] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12K-2] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12L-1]FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 12L-2] FIG. 10 illustrates an exemplary user interface displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments.

[0034] [Figure 13A] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13B] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13C] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13D] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13E] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13F] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13G] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13H] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13I]10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13J] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13K] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13L] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments. [Figure 13M] 10A-10C illustrate exemplary user interfaces for constraining the rotation of a virtual object around an axis, according to some embodiments.

[0035] [Figure 14A] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14B] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14C] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14D] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14E] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14F] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14G] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14H] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14I] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14J] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14K] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14L] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14M] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14N] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14O] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14P] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14Q] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14R] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14S]FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14T] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14U] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14V] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14W] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14X] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14Y] FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14Z]FIG. 10 illustrates an exemplary user interface for increasing a second threshold movement magnitude required for a second object manipulation behavior pursuant to a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments.

[0036] [Figure 14AA] 10 is a flow diagram illustrating operations for increasing a second threshold movement magnitude required for a second object manipulation behavior in accordance with a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14AB] 10 is a flow diagram illustrating operations for increasing a second threshold movement magnitude required for a second object manipulation behavior in accordance with a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14AC] 10 is a flow diagram illustrating operations for increasing a second threshold movement magnitude required for a second object manipulation behavior in accordance with a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 14AD] 10 is a flow diagram illustrating operations for increasing a second threshold movement magnitude required for a second object manipulation behavior in accordance with a determination that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments.

[0037] [Figure 15A] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15B] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15C]FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15D] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15E] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15F] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15G] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15H] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15I] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15J] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15K]FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15L] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15M] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15N] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15O] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15P] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15Q] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15R] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15S]FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15T] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15U] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15V] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15W] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15X] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15Y] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15Z] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AA]FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AB] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AC] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AD] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AE] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AF] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AG] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AH] FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 15AI]FIG. 10 illustrates an example user interface that generates an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments.

[0038] [Figure 16A] 10 is a flow diagram illustrating a process for displaying virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 16B] 10 is a flow diagram illustrating a process for displaying virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 16C] 10 is a flow diagram illustrating a process for displaying virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 16D] 10 is a flow diagram illustrating a process for displaying virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 16E] 10 is a flow diagram illustrating a process for displaying virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 16F] 10 is a flow diagram illustrating a process for displaying virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments. [Figure 16G] 10 is a flow diagram illustrating a process for displaying virtual objects with different visual characteristics depending on whether object placement criteria are met, according to some embodiments.

[0039] [Figure 17A] 10 is a flow diagram illustrating a process for displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 17B]10 is a flow diagram illustrating a process for displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 17C] 10 is a flow diagram illustrating a process for displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments. [Figure 17D] 10 is a flow diagram illustrating a process for displaying calibration user interface objects that are dynamically animated according to the movement of one or more cameras of a device, according to some embodiments.

[0040] [Figure 18A] 1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments. [Figure 18B] 1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments. [Figure 18C] 1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments. [Figure 18D] 1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments. [Figure 18E] 1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments. [Figure 18F] 1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments. [Figure 18G] 1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments. [Figure 18H] 1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments. [Figure 18I]1 is a flow diagram illustrating a process for constraining the rotation of a virtual object about an axis, according to some embodiments.

[0041] [Figure 19A] 1 is a flow diagram illustrating a process for increasing a second threshold movement magnitude required for a second object manipulation behavior upon determining that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 19B] 1 is a flow diagram illustrating a process for increasing a second threshold movement magnitude required for a second object manipulation behavior upon determining that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 19C] 1 is a flow diagram illustrating a process for increasing a second threshold movement magnitude required for a second object manipulation behavior upon determining that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 19D] 1 is a flow diagram illustrating a process for increasing a second threshold movement magnitude required for a second object manipulation behavior upon determining that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 19E] 1 is a flow diagram illustrating a process for increasing a second threshold movement magnitude required for a second object manipulation behavior upon determining that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 19F] 1 is a flow diagram illustrating a process for increasing a second threshold movement magnitude required for a second object manipulation behavior upon determining that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 19G] 1 is a flow diagram illustrating a process for increasing a second threshold movement magnitude required for a second object manipulation behavior upon determining that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments. [Figure 19H] 1 is a flow diagram illustrating a process for increasing a second threshold movement magnitude required for a second object manipulation behavior upon determining that a first threshold movement magnitude has been satisfied for a first object manipulation behavior, according to some embodiments.

[0042] [Figure 20A] 10 is a flow diagram illustrating a process for generating an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 20B] 10 is a flow diagram illustrating a process for generating an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 20C] 10 is a flow diagram illustrating a process for generating an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 20D] 10 is a flow diagram illustrating a process for generating an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 20E] 10 is a flow diagram illustrating a process for generating an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. [Figure 20F] 10 is a flow diagram illustrating a process for generating an audio alert pursuant to a determination that device movement causes a virtual object to move outside the display field of view of one or more device cameras, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0043] A virtual object is a graphical representation of a three-dimensional object in a virtual environment. Conventional methods of interacting with a virtual object to transition it from being displayed in the context of an application user interface (e.g., a two-dimensional application user interface that does not display an augmented reality environment) to being displayed in the context of an augmented reality environment (e.g., an environment that augments the view of the physical world with supplemental information that provides the user with additional information not available in the physical world) often require multiple separate inputs (e.g., a sequence of gestures and button presses) to produce a desired result (e.g., adjusting the size, location, and / or orientation of the virtual object to appear realistic or desired in the augmented reality environment). Furthermore, conventional input methods often involve delays between receiving a request to display an augmented reality environment and displaying the augmented reality environment due to the time required to activate one or more device cameras to capture a view of the physical world and / or to analyze and characterize the view of the physical world in relation to virtual objects that may be placed in the augmented reality environment (e.g., detecting planes and / or surfaces within the captured view of the physical world).Embodiments herein provide users with intuitive ways to display virtual objects in various contexts and / or interact with these virtual objects (e.g., by allowing a user to provide input that switches from displaying a virtual object in the context of an application user interface to displaying the virtual object in an augmented reality environment, allowing a user to change display characteristics of a virtual object before displaying the virtual object in an augmented reality environment (e.g., in a three-dimensional staging environment), providing instructions that allow a user to easily identify a virtual object across multiple applications throughout the system, modifying visual characteristics of an object while determining placement information for the object, providing animated calibration user interface objects that indicate device movements required for calibration, constraining rotation about an axis of a displayed virtual object, increasing a threshold movement magnitude for a manipulation behavior of a second object when a threshold movement magnitude for a manipulation behavior of a first object is met, and providing an audio alert indicating that a virtual object has moved out of the displayed field of view, etc.).

[0044] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways. For example, the systems, methods, and GUIs described herein facilitate displaying virtual objects in an augmented reality environment and adjusting the appearance of the virtual objects displayed in the augmented reality environment in response to various inputs.

[0045] 1A-1B, 2, and 3 provide a description of an exemplary device. FIGS. 4A-4B, 5A-5AT, 6A-6AJ, 7A-7P, 11A-11V, 12A-12L, 13A-13M, 14A-14Z, and 15A-15AI illustrate exemplary user interfaces displaying virtual objects in various contexts. FIGS. 8A-8E illustrate a process of displaying a representation of a virtual object while switching from displaying a first user interface region to displaying a second user interface region. FIGS. 9A-9D illustrate a process of displaying a first representation of a virtual object in a first user interface region, a second representation of the virtual object in a second user interface region, and a third representation of the virtual object through a representation of the field of view of one or more cameras. FIGS. 10A-10D illustrate a process of displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object. FIGS. 16A-16G illustrate a process for displaying a virtual object with different visual characteristics depending on whether object placement criteria are met. FIGS. 17A-17D illustrate a process for displaying a calibrated user object that is dynamically animated according to movement of one or more cameras of a device. FIGS. 18A-18I illustrate a process for constraining rotation about an axis of a virtual object. FIGS. 14AA-14AD and 19A-19H illustrate a process for increasing a second threshold movement magnitude required for a manipulation behavior of a second object according to a determination that a first threshold movement magnitude is met for a manipulation behavior of a first object. FIGS. 20A-20F illustrate a process for generating an audio alert according to a determination that movement of a device causes a virtual object to move outside the displayed field of view of one or more device cameras.The user interfaces of Figures 5A-5AT, 6A-6AJ, 7A-7P, 11A-11V, 12A-12L, 13A-13M, 14A-14Z, and 15A-15AI are used to explain the processes of Figures 8A-8E, 9A-9D, 10A-10D, 14AA-14AD, 16A-16G, 17A-17D, 18A-18I, 19A-19H, and 20A-20F. Exemplary Devices

[0046] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail as not to unnecessarily obscure aspects of the embodiments.

[0047] In this specification, although terms such as "first," "second," etc. are used in some instances to describe various elements, it will be understood that these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, a first contact can be referred to as a second contact, and similarly, a second contact can be referred to as a first contact, without departing from the scope of the various embodiments described. Although a first contact and a second contact are both contacts, they are not the same contact unless the context clearly dictates otherwise.

[0048] The terminology used in the description of the various embodiments set forth herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments set forth herein and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. The term "and / or," as used herein, should also be understood to refer to and include any and all possible combinations of one or more of the associated listed items. It should further be understood that the terms "comprise," "include," "comprises," and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] The term "if" as used herein is, optionally, interpreted to mean "when," or "when," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrase "when it is determined" or "when (a described condition or event) is detected" is, optionally, interpreted to mean "upon determining," or "in response to determining," or "when (a described condition or event) is detected," or "in response to detecting (a described condition or event)," depending on the context.

[0050] Embodiments of electronic devices, their user interfaces, and associated processes for using the devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, devices such as the iPhone®, iPod Touch®, and iPad® manufactured by Apple Inc. of Cupertino, California, USA. Other portable electronic devices, such as laptop or tablet computers having a touch-sensitive surface (e.g., a touchscreen display and / or touchpad), are also optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer having a touch-sensitive surface (e.g., a touchscreen display and / or touchpad).

[0051] In the following description, we describe an electronic device that includes a display and a touch-sensitive surface, but it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0052] The device typically supports a variety of applications, such as one or more of note-taking applications, drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0053] Various applications running on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more features of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or changed for each application and / or within each application. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports various applications with user interfaces that are intuitive and transparent to the user.

[0054] Attention now turns to embodiments of portable devices with touch-sensitive displays. FIG. 1A is a block diagram illustrating portable multifunction device 100 with touch-sensitive display system 112, according to some embodiments. Touch-sensitive display system 112 may conveniently be referred to as a "touch screen" or simply a touch-sensitive display. Device 100 includes memory 102 (optionally including one or more computer-readable storage media), memory controller 122, one or more processing units (CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, input / output (I / O) subsystem 106, other input or control devices 116, and external port 124. Device 100 optionally includes one or more light sensors 164. Device 100 optionally includes one or more intensity sensors 165 that detect the intensity of a contact on device 100 (e.g., a touch-sensitive surface such as touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 that generate tactile output on device 100 (e.g., generate tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0055] It should be understood that device 100 is but one example of a portable multifunction device, and that device 100 may optionally have more or fewer components than shown, may optionally combine two or more components, or may optionally have a different configuration or arrangement of components. The various components shown in FIG. 1A may be implemented in hardware, software, firmware, or a combination thereof, such as one or more signal processing circuits and / or application specific integrated circuits.

[0056] Memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as CPU(s) 120 and peripherals interface 118, is optionally controlled by memory controller 122.

[0057] A peripheral interface 118 can be used to couple input and output peripherals of the device to the CPU(s) 120 and memory 102. The one or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions and process data for the device 100.

[0058] In some embodiments, peripherals interface 118, CPU(s) 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0059] RF (radio frequency) circuitry 108 receives and transmits RF signals, also called electromagnetic signals. RF circuitry 108 converts electrical signals to electromagnetic signals or electromagnetic signals to electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. RF circuitry 108 optionally includes well-known circuits for performing these functions, such as, but not limited to, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module (SIM) card, and memory. RF circuitry 108 optionally communicates via wireless communication with networks, such as the Internet, also called the World Wide Web (WWW), an intranet, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs) and / or metropolitan area networks (MANs), and with other devices. The wireless communication is optionally selected from, but is not limited to, Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), long term evolution (LTE), near field communication (NFC), wideband code division multiple access (W-CDMA), code division multiple access (CDMA), time division multiple access (T ...Multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), voice over Internet Protocol (VoIP), Wi-MAX, protocols for email (e.g., Internet message access protocol (IMAP) and / or post office protocol (POP)), instant messaging (e.g., extensible messaging and presence protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMSP)). The present invention may use any of a number of communication standards, protocols, and technologies, such as SMS (Speech Message Service), or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document.

[0060] Audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and device 100. Audio circuit 110 receives audio data from peripherals interface 118, converts the audio data into electrical signals, and transmits the electrical signals to speaker 111. Speaker 111 converts the electrical signals into sound waves audible to humans. Audio circuit 110 also receives electrical signals converted from sound waves by microphone 113. Audio circuit 110 converts the electrical signals into audio data and transmits the audio data to peripherals interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to memory 102 and / or RF circuit 108 by peripherals interface 118. In some embodiments, audio circuit 110 also includes a headset jack (e.g., 212 in FIG. 2 ). The headset jack provides an interface between audio circuitry 110 and a detachable audio input / output peripheral, such as an output-only headphone or a headset with both an output (e.g., mono or binaural headphones) and an input (e.g., a microphone).

[0061] I / O subsystem 106 couples input / output peripherals on device 100, such as touch-sensitive display system 112 and other input or control devices 116, to peripheral interface 118. I / O subsystem 106 optionally includes display controller 156, light sensor controller 158, intensity sensor controller 159, haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from and send electrical signals to other input or control devices 116. Other input or control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, input controller(s) 160 are optionally coupled to any (or none) of a keyboard, infrared port, USB port, stylus, and / or pointer device such as a mouse. The one or more buttons (e.g., 208 in FIG. 2) optionally include up / down buttons for volume control of speaker 111 and / or microphone 113. The one or more buttons optionally include push buttons (e.g., 206 in FIG. 2).

[0062] Touch-sensitive display system 112 provides an input and output interface between the device and a user. Display controller 156 receives electrical signals from and / or sends electrical signals to touch-sensitive display system 112. Touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term "affordance" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed at that graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, a button, a slider, an icon, a selectable menu item, a switch, a hyperlink, or other user interface control.

[0063] Touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic and / or tactile contact. Touch-sensitive display system 112 and display controller 156 (including any associated modules and / or sets of instructions in memory 102) detect contacts (and any movement or disruption of that contact) on touch-sensitive display system 112 and translate the detected contacts into interactions with user interface objects (e.g., one or more soft keys, icons, web pages, or images) displayed on touch-sensitive display system 112. In some embodiments, the point of contact between touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0064] Touch-sensitive display system 112 optionally uses LCD (liquid crystal display), LPD (light emitting polymer display), or LED (light emitting diode) technology, although other embodiments use other display technologies. Touch-sensitive display system 112 and display controller 156 optionally use any of a number of now-known or later-developed touch-sensing technologies to detect contact and any movement or disruption of contact, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements that determine one or more points of contact with touch-sensitive display system 112. In some embodiments, projected mutual capacitance sensing technology is used, such as that found in the iPhone®, iPod Touch®, and iPad® manufactured by Apple Inc. of Cupertino, California, USA.

[0065] Touch-sensitive display system 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or higher). A user optionally contacts touch-sensitive display system 112 using any suitable object or accessory, such as a stylus, finger, or the like. In some embodiments, the user interface is designed to work with finger contacts and gestures, which may be less precise than stylus input due to the larger area of ​​contact a finger has with the touchscreen. In some embodiments, the device translates coarse finger input into precise pointer / cursor position or commands that perform the user's desired action.

[0066] In some embodiments, in addition to the touchscreen, device 100 optionally includes a touchpad (not shown) for activating or deactivating certain functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display visual output. The touchpad is optionally a touch-sensitive surface separate from touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touchscreen.

[0067] Device 100 also includes a power system 162 that provides power to the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., battery, alternating current (AC)), a recharging system, power fail detection circuitry, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0068] Device 100 also optionally includes one or more light sensors 164. FIG. 1A shows a light sensor coupled to light sensor controller 158 in I / O subsystem 106. Light sensor(s) 164 optionally include charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) phototransistors. Light sensor(s) 164 receive light from the environment projected through one or more lenses and convert this light into data representing an image. In cooperation with imaging module 143 (also called a camera module), light sensor(s) 164 optionally capture still images and / or video. In some embodiments, a light sensor is located on the back of device 100 opposite touch-sensitive display system 112 on the front of the device, allowing the touchscreen to be used as a viewfinder for acquiring still and / or video image capture. In some embodiments, a separate light sensor is positioned on the front of the device to capture an image of the user (e.g., for a selfie, or to allow the user to view other video conference participants on the touchscreen while participating in a video conference, etc.).

[0069] Device 100 also optionally includes one or more contact intensity sensors 165. FIG. 1A shows a contact intensity sensor coupled to intensity sensor controller 159 in I / O subsystem 106. Contact intensity sensor(s) 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of a contact on a touch-sensitive surface). Contact intensity sensor(s) 165 receive contact intensity information (e.g., pressure information or a substitute for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed with or proximate to the touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the back of device 100 opposite touchscreen display system 112, which is located on the front of device 100.

[0070] Device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows proximity sensor 166 coupled to peripherals interface 118. Alternatively, proximity sensor 166 is coupled to input controller 160 in I / O subsystem 106. In some embodiments, the proximity sensor turns off and disables touch-sensitive display system 112 when the multifunction device is placed near a user's ear (e.g., when the user is making a phone call).

[0071] Device 100 also optionally includes one or more tactile output generators 167. FIG. 1A shows tactile output generators coupled to haptic feedback controller 161 in I / O subsystem 106. In some embodiments, tactile output generator(s) 167 include one or more electroacoustic devices, such as speakers or other audio components, and / or electromechanical devices that convert energy into linear movement, such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components that convert electrical signals into tactile output on the device). Tactile output generator(s) 167 receive tactile feedback generation instructions from haptic feedback module 133 and generate tactile outputs on device 100 that can be sensed by a user of device 100. In some embodiments, at least one tactile output generator is juxtaposed with or proximate to a touch-sensitive surface (e.g., touch-sensitive display system 112) and, optionally, generates a tactile output by moving the touch-sensitive surface vertically (e.g., in / out of the surface of device 100) or laterally (e.g., back and forth in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is located on the back of device 100 opposite touch-sensitive display system 112, which is located on the front of device 100.

[0072] Device 100 also optionally includes one or more accelerometers 168. FIG. 1A shows accelerometer 168 coupled to peripherals interface 118. Alternatively, accelerometer 168 is optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed on the touchscreen display in landscape or portrait mode based on an analysis of data received from the one or more accelerometers. Device 100 optionally includes, in addition to accelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) that obtains information regarding the location and orientation (e.g., portrait or landscape) of device 100.

[0073] In some embodiments, software components stored in memory 102 include operating system 126, communications module (or instruction set) 128, touch / motion module (or instruction set) 130, graphics module (or instruction set) 132, haptic feedback module (or instruction set) 133, text input module (or instruction set) 134, Global Positioning System (GPS) module (or instruction set) 135, and applications (or instruction set) 136. Additionally, in some embodiments, as shown in FIGS. 1A and 3 , memory 102 stores device / global internal state 157. Device / global internal state 157 includes one or more of: active application state, which indicates which applications, if any, are currently active; display state, which indicates what applications, views, or other information occupy various regions of touch-sensitive display system 112; sensor state, which includes information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information regarding the device's position and / or orientation.

[0074] Operating system 126 (e.g., an embedded operating system such as iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or VxWorks) includes various software components and / or drivers that control and manage overall system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0075] Communications module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components that process data received by RF circuitry 108 and / or external port 124. External port 124 (e.g., Universal Serial Bus (USB), FIREWIRE®, etc.) is adapted to couple to other devices directly or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used in some devices, such as the iPhone®, iPod Touch®, and iPad®, manufactured by Apple Inc. of Cupertino, California, USA. In some embodiments, the external port is a Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in some devices, such as the iPhone®, iPod Touch®, and iPad®, manufactured by Apple Inc. of Cupertino, California, USA.

[0076] Contact / motion module 130 optionally detects contact with touch-sensitive display system 112 (in cooperation with display controller 156) and with other touch-sensitive devices (e.g., a touchpad or physical click wheel). Contact / motion module 130 includes various software components that perform various operations related to detecting contact (e.g., by a finger or stylus), such as determining if contact has occurred (e.g., detecting a finger down event), determining the intensity of the contact (e.g., contact force or pressure, or a surrogate for contact force or pressure), determining if there is contact movement and tracking that movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining if the contact has ended (e.g., detecting a finger up event or an interruption of contact). Contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point represented by the set of contact data optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions are optionally applied to individual contacts (e.g., single finger contacts or single stylus contacts) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, contact / motion module 130 and display controller 156 detect contacts on the touchpad.

[0077] Contact / motion module 130 optionally detects gesture input by a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different movements, timing, and / or intensities of detected contacts). Thus, gestures are optionally detected by detecting particular contact patterns. For example, detecting a finger tap gesture includes detecting a finger down event (e.g., at an icon) followed by a finger lift (lift-off) event at the same location (or substantially the same location) as the finger down event. In another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger down event followed by one or more finger drag events followed by a finger lift (lift-off) event. Similarly, for a stylus, taps, swipes, drags, and other gestures are optionally detected by detecting particular contact patterns for the stylus.

[0078] In some embodiments, detection of a finger tap gesture depends on the length of time between detecting a finger down event and detecting a finger up event, but not on the strength of the finger contact between detecting the finger down event and detecting the finger up event. In some embodiments, a tap gesture is detected according to determining that the length of time between a finger down event and a finger up event is less than a predetermined value (e.g., less than 0.1 seconds, 0.2 seconds, 0.3 seconds, 0.4 seconds, or 0.5 seconds), regardless of whether the strength of the finger contact during the tap meets a given intensity threshold (greater than a nominal contact-detection intensity threshold), such as a light or deep pressure intensity threshold. Thus, a finger tap gesture need only meet certain input criteria, without requiring that the characteristic intensity of the contact meet a given intensity threshold for that particular input criterion to be met. For clarity, in order for a finger down event to be detected, the finger contact in the tap gesture must typically meet a nominal contact-detection intensity threshold below which the contact is not detected. A similar analysis applies to detecting tap gestures with stylus contact or other contact: if the device can detect contact of a finger or stylus hovering over the touch-sensitive surface, the nominal contact-detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0079] The same concepts apply to other types of gestures as well. For example, swipe gestures, pinch gestures, de-pinch gestures, and / or press and hold gestures are detected based on the satisfaction of criteria that are, optionally, independent of the intensity of the contacts included in the gesture or that do not require the contact(s) making up the gesture to reach an intensity threshold to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts, a pinch gesture is detected based on the movement of two or more contacts toward each other, a de-pinch gesture is detected based on the movement of two or more contacts away from each other, and a press and hold gesture is detected based on the duration of contacts on the touch-sensitive surface whose amount of movement is less than a threshold amount of movement. Thus, a statement that a particular gesture recognition criterion does not require the intensity of the contact(s) to meet a respective intensity threshold for the particular gesture recognition criterion to be met means that the particular gesture recognition criterion can be met even if the contact(s) in the gesture do not reach their respective intensity threshold, and can also be met in situations where one or more of the contacts in the gesture do not reach or exceed their respective intensity threshold. In some embodiments, a tap gesture is detected based on determining that a finger-down event and a finger-up event are detected within a predetermined time period, regardless of whether the contacts are above or below their respective intensity thresholds during the predetermined time period, and a swipe gesture is detected based on determining that a movement of the contact is greater than a predetermined magnitude, even if the contact exceeds its respective intensity threshold at the end of the movement of the contact. Even in implementations in which gesture detection is affected by the intensity of the contact making the gesture (e.g., the device detects a long press earlier when the intensity of the contact exceeds an intensity threshold, or delays detecting a tap input when the intensity of the contact is high), detection of these gestures does not require the contact to reach a particular intensity threshold, as long as the criteria for recognizing the gesture can be met in situations in which the contact does not reach the particular intensity threshold (e.g., even if the amount of time it takes to recognize the gesture varies).

[0080] In some situations, the contact intensity threshold, duration threshold, and movement threshold may be combined in various combinations to create heuristics for distinguishing between two or more different gestures directed at the same input element or region, enabling multiple different interactions with the same input element to provide a richer set of user interactions and responses. A statement that a particular set of gesture recognition criteria does not require the intensity of the contact(s) to meet the respective intensity threshold for that particular gesture recognition criterion to be met does not preclude simultaneously evaluating other intensity-dependent gesture recognition criteria to identify other gestures whose criteria are met when the gesture includes a contact with an intensity exceeding the respective intensity threshold. For example, in some situations, a first gesture recognition criterion of a first gesture that does not require the intensity of the contact(s) to meet the respective intensity threshold for that gesture recognition criterion to be met conflicts with a second gesture recognition criterion of a second gesture that depends on the contact(s) reaching the respective intensity threshold. During such a conflict, a gesture is optionally not recognized as satisfying the first gesture recognition criteria of the first gesture if the second gesture recognition criteria of the second gesture are satisfied first. For example, if the contact reaches the respective intensity threshold before moving a predetermined amount of movement, a deep press gesture is detected rather than a swipe gesture. Conversely, if the contact reaches the respective intensity threshold after moving a predetermined amount of movement, a swipe gesture is detected rather than a deep press gesture. Even in such a situation, the first gesture recognition criteria of the first gesture still do not require that the intensity of the contact(s) meet the respective intensity threshold for the first gesture recognition criteria to be satisfied. This is because if the contact remained below the respective intensity threshold until the end of the gesture (e.g., in the case of a swipe gesture with a contact that does not increase in intensity above the respective intensity threshold), the gesture would have been recognized as a swipe gesture by the first gesture recognition criteria.Thus, a particular gesture recognition criterion that does not require the intensity of the contact(s) to meet a respective intensity threshold for the particular gesture recognition criterion to be satisfied may still be dependent on the intensity of the contact with respect to the intensity threshold, in the sense that (A) in some circumstances, the intensity of the contact may be ignored with respect to the intensity threshold (e.g., for a tap gesture), and / or (B) in some circumstances, the particular gesture recognition criterion (e.g., the recognition criteria for a long press gesture) may not be satisfied if a set of competing intensity-dependent gesture recognition criteria (e.g., the recognition criteria for a long press gesture) recognizes an input as corresponding to an intensity-dependent gesture after the particular gesture recognition criterion recognizes the gesture corresponding to that input (e.g., a long press gesture that competes for recognition with a deep press gesture).

[0081] Graphics module 132 includes various known software components that render and display graphics on touch-sensitive display system 112 or other displays, such as components that modify the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, such as, but not limited to, text, web pages, icons (e.g., user interface objects including soft keys), digital images, video, animation, etc.

[0082] In some embodiments, graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. Graphics module 132 receives one or more codes specifying the graphics to be displayed, including coordinate data and other graphic characteristic data as needed, from an application or the like, and then generates screen image data to output to display controller 156.

[0083] The haptic feedback module 133 includes various software components that generate instructions (e.g., instructions used by the haptic feedback controller 161) that use the tactile output generator(s) 167 to generate tactile outputs at one or more locations on the device 100 in response to user interaction with the device 100.

[0084] Text input module 134 is optionally a component of graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0085] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing, to the camera 143 as photo / video metadata, and to applications that provide location-based services such as weather widgets, local directory widgets, and map / navigation widgets).

[0086] Application 136 optionally includes the following modules (or sets of instructions), or a subset or superset thereof: • a contacts module 137 (sometimes called an address book or contact list); ●Telephone module 138, ●Video conferencing module 139, ● an email client module 140; ● Instant messaging (IM) module 141; ●Training support module 142, a camera module 143 for still and / or video images, ● Image management module 144; ● Browser module 147, ● Calendar module 148, a widget module 149, optionally including one or more of a weather widget 149-1, a stock price widget 149-2, a calculator widget 149-3, an alarm clock widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, as well as user-created widgets 149-6; • A widget creator module 150 for creating user-created widgets 149-6; ● Search module 151, • A video and music player module 152, optionally consisting of a video player module and a music player module; ● Memo module 153, Map module 154, and / or ●Online video module 155.

[0087] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice duplication.

[0088] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in memory 102 or in the application internal state 192 of contacts module 137 in memory 370), such as adding name(s) to an address book, deleting name(s) from an address book, associating phone number(s), email address(es), physical address(es) or other information with names, associating images with names, categorizing and organizing names, and providing phone numbers and / or email addresses to initiate and / or facilitate communication by phone 138, video conference 139, email 140, or IM 141.

[0089] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, telephone module 138 includes executable instructions for entering a series of characters corresponding to a telephone number, accessing one or more telephone numbers in address book 137, modifying an entered telephone number, dialing each telephone number, conducting a conversation, and disconnecting or hanging up when the conversation is completed. As noted above, wireless communication optionally uses any of a number of communication standards, protocols, and technologies.

[0090] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants according to the user's commands.

[0091] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user commands. In conjunction with image management module 144, email client module 140 greatly facilitates creating and sending emails with still or video images captured by camera module 143.

[0092] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, instant messaging module 141 includes executable instructions for entering a series of characters corresponding to an instant message, modifying previously entered characters, transmitting each instant message (e.g., using Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for instant messaging over the telephone, or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for instant messaging over the Internet), receiving instant messages, and viewing received instant messages. In some embodiments, transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported by MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both telephone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0093] In conjunction with the RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, the training support module 142 includes executable instructions to create workouts (e.g., with time, distance, and / or calorie burn goals), communicate with training sensors (in the sports device and smartwatch), receive training sensor data, calibrate sensors used to monitor workouts, select and play music for workouts, and display, store, and transmit workout data.

[0094] In conjunction with touch-sensitive display system 112, display controller 156, light sensor(s) 164, light sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions to capture still images or video (including video streams), store them in memory 102, modify characteristics of the still images or video, and / or delete the still images or video from memory 102.

[0095] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slide show or as an album), and storing still and / or video images.

[0096] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, browser module 147 contains executable instructions for browsing the Internet according to user commands, such as retrieving, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0097] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, email client module 140, and browser module 147, calendar module 148 includes executable instructions to create, display, modify, and store calendars and calendar-related data (e.g., calendar entries, to-do lists, etc.) according to user instructions.

[0098] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget modules 149 are optionally mini-applications downloaded and used by users (e.g., weather widget 149-1, stocks widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or created by users (e.g., user-created widget 149-6). In some embodiments, widgets include Hypertext Markup Language (HTML) files, Cascading Style Sheets (CSS) files, and JavaScript files. In some embodiments, widgets include Extensible Markup Language (XML) files and JavaScript files (e.g., Yahoo! Widgets).

[0099] In association with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, and browser module 147, widget creator module 150 includes executable instructions for creating widgets (e.g., turning user-specified portions of a web page into widgets).

[0100] In association with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, and text input module 134, search module 151 includes executable instructions to search memory 102 for text, music, audio, image, video, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with a user's instructions.

[0101] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that enable a user to download and play recorded music or other audio files stored in one or more file formats, such as MP3 or AAC files, and to display, present, or otherwise play videos (e.g., on touch-sensitive display system 112, or wirelessly or on an external display connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).

[0102] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, notes module 153 includes executable instructions for creating and managing notes, to-do lists, and the like, in accordance with user commands.

[0103] In conjunction with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions to receive, display, modify, and store maps and map-related data (e.g., driving directions, data about businesses and other points of interest at or near a particular location, and other location-based data) in accordance with user instructions.

[0104] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 contains executable instructions that enable a user to access, view, receive (e.g., by streaming and / or downloading), play (e.g., on touchscreen 112, or wirelessly or on an external display connected via external port 124) online videos in one or more file formats such as H.264, send emails with links to particular online videos, and otherwise manage emails. In some embodiments, instant messaging module 141 is used to send links to particular online videos rather than email client module 140.

[0105] Each of the above-identified modules and applications corresponds to a set of executable instructions that perform one or more of the functions described above and methods described herein (e.g., the computer-implemented methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules; thus, various embodiments optionally combine or otherwise reconfigure various subsets of these modules. In some embodiments, memory 102 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 102 optionally stores additional modules and data structures not described above.

[0106] In some embodiments, device 100 is a device in which operation of a predetermined set of functions on the device is performed exclusively through a touchscreen and / or touchpad. Using the touchscreen and / or touchpad as the primary input control device for operation of device 100 optionally reduces the number of physical input control devices (push buttons, dials, etc.) on device 100.

[0107] The set of predetermined functions performed only through the touchscreen and / or touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by a user, navigates device 100 to a main menu, home menu, or root menu from any user interface displayed on device 100. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device rather than a touchpad.

[0108] 1B is a block diagram illustrating exemplary components for event handling according to some embodiments. In some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g., of operating system 126) and a respective application 136-1 (e.g., any of applications 136, 137-155, 380-390 described above).

[0109] Event sorter 170 receives the event information and determines application 136-1 and application view 191 of application 136-1 to deliver the event information to. Event sorter 170 includes event monitor 171 and event dispatcher module 174. In some embodiments, application 136-1 includes application internal state 192 that indicates the current application view(s) that are displayed on touch-sensitive display system 112 when the application is active or running. In some embodiments, device / global internal state 157 is used by event sorter 170 to determine which application(s) are currently active, and application internal state 192 is used by event sorter 170 to determine which application(s) are currently active, and application internal state 192 is used by event sorter 170 to determine which application view(s) to deliver the event information to.

[0110] In some embodiments, application internal state 192 includes additional information such as one or more of resume information used when application 136-1 resumes execution, user interface state information indicating information being displayed or ready to be displayed by application 136-1, a state queue that allows the user to return to a previous state or view of application 136-1, and a redo / undo queue of previous actions taken by the user.

[0111] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user's touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information received from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer(s) 168, and / or microphone 113 (via audio circuitry 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0112] In some embodiments, event monitor 171 sends requests to peripherals interface 118 at predetermined intervals. In response, peripherals interface 118 transmits event information. In other embodiments, peripherals interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or for longer than a predetermined duration).

[0113] In some embodiments, event sorter 170 includes hit view determination module 172 and / or active event recognizer determination module 173 .

[0114] Hit view determination module 172 provides software procedures that determine where a sub-event occurred within one or more views when touch-sensitive display system 112 displays more than one view. A view consists of the controls and other elements that a user can see on the display.

[0115] Another aspect of a user interface associated with an application is the set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch gestures are made. The application view (of the respective application) in which the touch is detected optionally corresponds to a programmatic level within the application's programmatic or view hierarchy. For example, the lowest-level view in which the touch is detected is optionally referred to as the hit view, and the set of events that are recognized as appropriate inputs is optionally determined based at least in part on the hit view of the initial touch that initiates the touch gesture.

[0116] Hit view determination module 172 receives information related to sub-events of a touch gesture. When an application has multiple views organized hierarchically, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should process the sub-events. In most situations, the hit view is the lowest-level view in which the initiating sub-event occurs (i.e., the first sub-event in a sequence of sub-events that form an event or potential event). Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events associated with the same touch or input source for which it was identified as the hit view.

[0117] Active event recognizer determination module 173 determines which view(s) in the view hierarchy should receive the particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive the particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that contain the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive the particular sequence of sub-events. In other embodiments, even if the touch sub-event is completely confined to the area associated with one particular view, views higher in the hierarchy still remain actively participating views.

[0118] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores event information obtained by each event receiver module 182 in an event queue.

[0119] In some embodiments, operating system 126 includes event sorter 170. Alternatively, application 136-1 includes event sorter 170. In still other embodiments, event sorter 170 is a stand-alone module or part of another module stored in memory 102, such as contact / motion module 130.

[0120] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each containing instructions for handling touch events that occur within a respective view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, each application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a User Interface Kit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, each event handler 190 includes one or more of data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event sorter 170. Event handler 190 optionally utilizes or invokes data updater 176, object updater 177, or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views 191 each include one or more event handlers 190. Also, in some embodiments, one or more of the data updater 176, object updater 177, and GUI updater 178 are included in each application view 191.

[0121] Each event recognizer 180 receives event information (e.g., event data 179) from event sorter 170 and identifies an event from the event information. Event recognizer 180 includes event receiver 182 and event comparator 184. In some embodiments, event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (optionally including sub-event delivery instructions).

[0122] Event receiver 182 receives event information from event sorter 170. The event information includes information about a sub-event, such as a touch or a touch movement. Depending on the sub-event, the event information also includes additional information, such as the position of the sub-event. When the sub-event relates to a touch movement, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, the event includes a rotation of the device from one orientation to another (e.g., from portrait to landscape or vice versa), and the event information includes corresponding information about the device's current orientation (also called the device's attitude).

[0123] The event comparator 184 compares the event information to a predetermined event or sub-event definition and determines the event or sub-event, or determines or updates the state of the event or sub-event, based on the comparison. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes a definition of an event (e.g., a predetermined sequence of sub-events), such as Event 1 (187-1) and Event 2 (187-2). In some embodiments, sub-events of Event 187 include, for example, touch start, touch end, touch move, touch cancel, and multiple touches. In one example, the definition of Event 1 (187-1) is a double tap on a displayed object. The double tap includes, for example, a first touch (touch start) on the displayed object for a predetermined period of time, a first lift-off (touch end) for a predetermined period of time, a second touch (touch start) on the displayed object for a predetermined period of time, and a second lift-off (touch end) for a predetermined period of time. In another example, the definition of Event 2 (187-2) is a drag on the displayed object. A drag includes, for example, a touch (or contact) on a displayed object for a predetermined period of time, a movement of the touch across the touch-sensitive display system 112, and a lift-off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0124] In some embodiments, event definitions 187 include a definition of the event for each user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view in which three user interface objects are displayed on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a respective event handler 190, event comparator 184 uses the results of the hit test to determine which event handler 190 to activate. For example, event comparator 184 selects the event handler associated with the sub-event and object that triggers the hit test.

[0125] In some embodiments, each event 187 definition also includes a delay action that delays delivery of the event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognizer.

[0126] If the respective event recognizer 180 determines that the sequence of sub-events does not match any event in the event definition 186, the respective event recognizer 180 enters an event disabled, event failed, or event finished state and thereafter ignores subsequent sub-events of the touch gesture. In this situation, any other event recognizers that remain active for the hit view continue to track and process sub-events of the ongoing touch gesture.

[0127] In some embodiments, each event recognizer 180 includes metadata 183 with configurable properties, flags, and / or lists that indicate to actively involved event recognizers how the event delivery system should perform sub-event delivery. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact with each other or how event recognizers are enabled to interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to various levels in the view or programmatic hierarchy.

[0128] In some embodiments, each event recognizer 180 activates an event handler 190 associated with an event when one or more specific sub-events of the event are recognized. In some embodiments, each event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is distinct from sending (and delaying) the sub-events to the respective hit view. In some embodiments, the event recognizer 180 pops a flag associated with the recognized event, and the event handler 190 associated with the flag catches the flag and performs a predetermined process.

[0129] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the set of sub-events or an actively participating view. The event handler associated with the set of sub-events or an actively participating view receives the event information and performs a predetermined process.

[0130] In some embodiments, data updater 176 creates and updates data used by application 136-1. For example, data updater 176 updates phone numbers used in contacts module 137 or stores video files used by video and music player module 152. In some embodiments, object updater 177 creates and updates objects used by application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends this display information to graphics module 132 for display on the touch-sensitive display.

[0131] In some embodiments, event handler(s) 190 include or have access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the respective application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0132] It should be understood that the above description of event processing of a user's touch on a touch-sensitive display also applies to other forms of user input for operating multifunction device 100 with input devices, not all of which are initiated on a touchscreen. For example, mouse movements and mouse button presses, optionally in conjunction with one or more keyboard presses or holds, contact movements such as tapping, dragging, scrolling on a touchpad, pen stylus input, device movement, verbal commands, detected eye movement, biometric input, and / or any combination thereof, are optionally utilized as inputs corresponding to sub-events that define the recognized event.

[0133] 1C is a block diagram illustrating a tactile output module according to some embodiments. In some embodiments, I / O subsystem 106 (e.g., haptic feedback controller 161 (FIG. 1A) and / or other input controller(s) 160 (FIG. 1A) includes at least some of the example components shown in FIG. 1C. In some embodiments, peripherals interface 118 includes at least some of the example components shown in FIG. 1C.

[0134] In some embodiments, the tactile output module includes a haptic feedback module 133. In some embodiments, the haptic feedback module 133 assembles and combines haptic outputs for user interface feedback from software applications on the electronic device (e.g., feedback responsive to user input corresponding to a displayed user interface, and alerts and other notifications indicating the performance of operations or the occurrence of events in the user interface of the electronic device). The haptic feedback module 133 includes one or more of a waveform module 123 (which provides the waveforms used to generate the tactile output), a mixer 125 (which mixes multiple waveforms, such as waveforms for different channels), a compressor 127 (which reduces or compresses the dynamic range of the waveforms), a low-pass filter 129 (which filters out high-frequency signal components of the waveforms), and a thermal controller 131 (which adjusts the waveform according to thermal conditions). In some embodiments, the haptic feedback module 133 is included in a haptic feedback controller 161 (FIG. 1A). In some embodiments, a separate unit of haptic feedback module 133 (or a separate implementation of haptic feedback module 133) is also included in the audio controller (e.g., audio circuit 110 of FIG. 1A) and used to generate the audio signal. In some embodiments, a single haptic feedback module 133 is used to generate the audio signal and generate the waveform of the tactile output.

[0135] In some embodiments, haptic feedback module 133 also includes trigger module 121 (e.g., a software application, operating system, or other software module that determines the tactile output to generate and initiates the process of generating the corresponding tactile output). In some embodiments, trigger module 121 generates a trigger signal that initiates the generation of a waveform (e.g., by waveform module 123). For example, trigger module 121 generates the trigger signal based on a preset timing criteria. In some embodiments, trigger module 121 receives a trigger signal from outside haptic feedback module 133 (e.g., in some embodiments, haptic feedback module 133 receives a trigger signal from hardware input processing module 146 located outside haptic feedback module 133) and relays this trigger signal to other components within haptic feedback module 133 (e.g., waveform module 123), or to a software application that triggers an action (e.g., by trigger module 121) based on activation of a user interface element (e.g., an application icon or affordance within an application), or to a hardware input device (e.g., an intensity-sensitive input surface such as a home button or an intensity-sensitive touchscreen). In some embodiments, trigger module 121 also receives a tactile feedback generation command (e.g., from haptic feedback module 133 of FIGS. 1A and 3). In some embodiments, trigger module 121 generates a trigger signal in response to haptic feedback module 133 (or trigger module 121 within haptic feedback module 133) receiving a tactile feedback command (e.g., from haptic feedback module 133 of FIGS. 1A and 3).

[0136] Waveform module 123 receives a trigger signal as an input (e.g., from trigger module 121) and, in response to receiving the trigger signal, provides a waveform (e.g., a waveform selected from a predetermined set of waveforms designated for use by waveform module 123, such as the waveforms described in further detail below with reference to Figures 4F-4G) for generating one or more tactile outputs.

[0137] Mixer 125 receives waveforms as inputs (e.g., from waveform module 123) and mixes these waveforms. For example, when mixer 125 receives two or more waveforms (e.g., a first waveform in a first channel and a second waveform in a second channel that at least partially overlaps the first waveform), mixer 125 outputs a composite waveform corresponding to the sum of the two or more waveforms. In some embodiments, mixer 125 also modifies one or more of the two or more waveforms (e.g., by increasing the scale of particular waveform(s) and / or decreasing the scale of the remaining waveforms) to emphasize the particular waveform(s) over the remaining waveforms. In some situations, mixer 125 selects one or more waveforms to remove from the composite waveform (e.g., removes the oldest source waveform when there are waveforms from four or more sources required to be output simultaneously by tactile output generator 167).

[0138] Compressor 127 receives waveforms (e.g., composite waveforms from mixer 125) as inputs and modifies these waveforms. In some embodiments, compressor 127 reduces the waveforms (e.g., according to the physical specifications of tactile output generator 167 (FIG. 1A) or 357 (FIG. 3)) such that the tactile output corresponding to the waveform is reduced. In some embodiments, compressor 127 limits the waveforms, such as by imposing a predetermined maximum amplitude on the waveform. For example, compressor 127 maintains the amplitude of portions of the waveform that do not exceed a predetermined amplitude threshold, while reducing the amplitude of portions of the waveform that exceed a predetermined amplitude threshold. In some embodiments, compressor 127 reduces the dynamic range of the waveform. In some embodiments, compressor 127 dynamically reduces the dynamic range of the waveform so that the composite waveform remains within the performance specifications (e.g., force and / or moving mass displacement limits) of tactile output generator 167.

[0139] Low-pass filter 129 receives waveforms (e.g., compressed waveforms from compressor 127) as inputs and filters (e.g., smooths) these waveforms (e.g., removes or reduces high-frequency signal components in the waveforms). For example, in some cases, compressor 127 includes extraneous signals (e.g., high-frequency signal components) in the compressed waveforms that interfere with the generation of a tactile output and / or exceed the performance specifications of tactile output generator 167 when generating a tactile output in accordance with the compressed waveform. Low-pass filter 129 reduces or removes such extraneous signals in the waveforms.

[0140] Thermal controller 131 receives waveforms (e.g., filtered waveforms from low-pass filter 129) as inputs and adjusts these waveforms according to the thermal conditions of device 100 (e.g., based on an internal temperature detected within device 100, such as the temperature of haptic feedback controller 161, and / or an external temperature detected by device 100). For example, in some cases, the output of haptic feedback controller 161 varies with temperature (e.g., haptic feedback controller 161 generates a first tactile output when haptic feedback controller 161 is at a first temperature and a second tactile output when haptic feedback controller 161 is at a second temperature distinct from the first temperature, in response to receiving the same waveform). For example, the magnitude (or amplitude) of the tactile output may vary with temperature. The waveforms may be modified (e.g., the amplitude of the waveform may be increased or decreased based on temperature) to reduce the effect of temperature changes.

[0141] In some embodiments, haptic feedback module 133 (e.g., trigger module 121) is coupled to hardware input processing module 146. In some embodiments, other input controller(s) 160 of FIG. 1A includes hardware input processing module 146. In some embodiments, hardware input processing module 146 receives input from hardware input device 145 (e.g., other input or control device 116 of FIG. 1A, such as a home button or an intensity-sensitive input surface such as an intensity-sensitive touchscreen). In some embodiments, hardware input device 145 is any input device described herein, such as touch-sensitive display system 112 (FIG. 1A), keyboard / mouse 350 (FIG. 3), touchpad 355 (FIG. 3), one of other input or control devices 116 (FIG. 1A), or an intensity-sensitive home button. In some embodiments, hardware input device 145 comprises an intensity-sensitive home button rather than touch-sensitive display system 112 (FIG. 1A), keyboard / mouse 350 (FIG. 3), or touchpad 355 (FIG. 3). In some embodiments, in response to input from hardware input device 145 (e.g., an intensity-sensitive home button or a touchscreen), hardware input processing module 146 provides one or more trigger signals to haptic feedback module 133 indicating that a user input that meets predetermined input criteria has been detected, such as an input corresponding to a home button "click" (e.g., a "down click" or an "up click") In some embodiments, haptic feedback module 133, in response to an input corresponding to a home button "click," provides a waveform corresponding to a home button "click" to simulate the haptic feedback of pressing a physical home button.

[0142] In some embodiments, the haptic output module includes a haptic feedback controller 161 (e.g., haptic feedback controller 161 of FIG. 1A ) that controls the generation of the tactile output. In some embodiments, the haptic feedback controller 161 is coupled to a plurality of tactile output generators, selects one or more of the plurality of tactile output generators, and sends waveforms to the selected one or more tactile output generators to generate the tactile output. In some embodiments, the haptic feedback controller 161 adjusts the tactile output requests corresponding to the activation of the hardware input device 145 and the tactile output requests corresponding to the software event (e.g., from the haptic feedback module 133), and modifies one or more of the two or more waveforms to emphasize the particular waveform(s) over the rest of the two or more waveforms (e.g., by increasing the scale of the particular waveform(s) and / or decreasing the scale of the remaining waveforms, such as to prioritize the tactile output corresponding to the activation of the hardware input device 145 over the tactile output corresponding to the software event).

[0143] In some embodiments, as shown in FIG. 1C , the output of haptic feedback controller 161 is coupled to an audio circuit of device 100 (e.g., audio circuit 110 of FIG. 1A ) and provides an audio signal to the audio circuit of device 100. In some embodiments, haptic feedback controller 161 provides both the waveform used to generate the tactile output and the audio signal used to provide an audio output in conjunction with generating the tactile output. In some embodiments, haptic feedback controller 161 modifies the audio signal and / or the waveform (used to generate the tactile output) (e.g., by delaying the audio signal and / or waveform) so that the audio output and the tactile output are synchronized. In some embodiments, haptic feedback controller 161 includes a digital-to-analog converter used to convert the digital waveform into an analog signal received by amplifier 163 and / or tactile output generator 167.

[0144] In some embodiments, tactile output module includes amplifier 163. In some embodiments, amplifier 163 receives waveforms (e.g., from haptic feedback controller 161), amplifies the waveforms, and then transmits the amplified waveforms to tactile output generator 167 (e.g., tactile output generator 167 (FIG. 1A) or 357 (FIG. 3)). For example, amplifier 163 amplifies the received waveforms to a signal level according to the physical specifications of tactile output generator 167 (e.g., to a voltage and / or current required by tactile output generator 167 to generate a tactile output, such that the signal transmitted to tactile output generator 167 generates a tactile output corresponding to the waveform received from tactile feedback controller 161), and transmits the amplified waveforms to tactile output generator 167. In response, tactile output generator 167 generates a tactile output (e.g., by oscillating the movable mass in one or more directions relative to a neutral position of the movable mass).

[0145] In some embodiments, tactile output module includes sensor 169 coupled to tactile output generator 167. Sensor 169 detects the state or state change (e.g., mechanical position, physical displacement, and / or movement) of tactile output generator 167 or one or more components of tactile output generator 167 (e.g., one or more moving parts, such as a membrane used to generate the tactile output). In some embodiments, sensor 169 is a magnetic field sensor (e.g., a Hall Effect sensor) or other displacement and / or movement sensor. In some embodiments, sensor 169 provides information (e.g., the position, displacement, and / or movement of one or more parts of tactile output generator 167) to tactile feedback controller 161, which adjusts the waveform output from tactile feedback controller 161 (e.g., the waveform sent to tactile output generator 167, optionally via amplifier 163) according to the information about the state of tactile output generator 167 provided by sensor 169.

[0146] FIG. 2 illustrates portable multifunction device 100 having a touchscreen (e.g., touch-sensitive display system 112 of FIG. 1A ), according to some embodiments. The touchscreen optionally displays one or more graphics in user interface (UI) 200. In these embodiments, as well as other embodiments described below, a user can select one or more of the graphics by making a gesture on the graphic with, for example, one or more fingers 202 (not drawn to scale) or one or more styluses 203 (not drawn to scale). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (left to right, right to left, upward, and / or downward), and / or rolling of a finger in contact with device 100 (right to left, left to right, upward, and / or downward). In some implementations or situations, inadvertent contact with a graphic does not select the graphic, for example, a swipe gesture sweeping over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0147] Device 100 also optionally includes one or more physical buttons, such as a "home" or menu button 204. As mentioned above, menu button 204 is optionally used to navigate to any application 136 within a set of applications optionally running on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key within a GUI displayed on a touchscreen display.

[0148] In some embodiments, device 100 includes a touchscreen display, a menu button 204 (sometimes referred to as a home button 204), push buttons 206 for powering the device on / off and locking the device, volume control button(s) 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and an external docking / charging port 124. Push button 206 is optionally used to power the device on / off by pressing and holding the button down for a predetermined period of time, to lock the device by pressing and releasing the button before the predetermined period of time has elapsed, and / or to unlock the device or begin the unlocking process. Also, in some embodiments, device 100 accepts verbal input through microphone 113 to activate or deactivate certain functions. Device 100 also optionally includes one or more contact intensity sensors 165 that detect the intensity of a contact on touch-sensitive display system 112 and / or one or more tactile output generators 167 that generate a tactile output for a user of device 100.

[0149] FIG. 3 is a block diagram of an exemplary multifunction device having a display and a touch-sensitive surface, according to some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a child's learning toy), a gaming system, or a control device (e.g., a home or commercial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 that interconnect these components. Communication bus 320 optionally includes circuitry (sometimes called a chipset) that interconnects and controls communication between system components. Device 300 includes input / output (I / O) interfaces 330, including a display 340, which is typically a touchscreen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, as well as a touchpad 355, a tactile output generator 357 (e.g., similar to tactile output generator 167 described above with reference to FIG. 1A ) that generates tactile output on device 300, and sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors similar to contact intensity sensor 165 described above with reference to FIG. 1A ). Memory 370 includes high-speed random-access memory such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located remotely from CPU 310.In some embodiments, memory 370 stores programs, modules, and data structures similar to, or a subset of, the programs, modules, and data structures stored in memory 102 of portable multifunction device 100 (FIG. 1A). Additionally, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores drawing module 380, presentation module 382, ​​word processing module 384, website creation module 386, disk authoring module 388, and / or spreadsheet module 390, while memory 102 of portable multifunction device 100 (FIG. 1A) optionally does not store those modules.

[0150] Each of the above-identified elements in FIG. 3 is optionally stored in one or more of the memory devices mentioned above. Each of the above-identified modules corresponds to an instruction set that performs the above-described functions. The above-identified modules or programs (i.e., instruction sets) need not be implemented as separate software programs, procedures, or modules; thus, various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subset of the above-identified modules and data structures. Additionally, memory 370 optionally stores additional modules and data structures not described above.

[0151] Attention is now directed to embodiments of a user interface (“UI”) that is optionally implemented on portable multifunction device 100.

[0152] 4A shows an exemplary user interface 400 for a menu of applications on portable multifunction device 100, according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof: Signal strength indicators for wireless communications such as cellular and Wi-Fi signals; ●Time, ●Bluetooth indicator, ● Battery status indicator, A tray 408 with icons for frequently used applications, such as: an icon 416 for the phone module 138, labeled "Phone," optionally including an indicator 414 of the number of missed calls or voicemail messages; An icon 418 for the email client module 140, labeled "Mail", optionally including an indicator 410 of the number of unread emails; ○ An icon 420 for the browser module 147, labeled "Browser", and ○ An icon 422 for the video and music player module 152 labeled "Music", and Icons for other applications, such as: ○ An icon 424 for the IM module 141 labeled "Messages"; ○ An icon 426 for the calendar module 148, labeled "Calendar"; ○ Icon 428 for the image management module 144, labeled "Photos"; ○ An icon 430 for the camera module 143, labeled "camera"; ○ Icon 432 for the online video module 155, labeled "Online Video"; ○ Icon 434 for stock widget 149-2, labeled "Stocks" ○ Icon 436 for map module 154, labeled "Map"; ○ Icon 438 for weather widget 149-1, labeled "Weather" ○ Icon 440 for alarm clock widget 149-4, labeled "Clock" ○ An icon 442 for the training support module 142, labeled "Training Support"; ○ An icon 444 for the notes module 153, labeled "Notes," and ○ An icon 446 for a settings application or module that provides access to settings related to the device 100 and its various applications 136.

[0153] Note that the icon labels displayed in FIG. 4A are merely examples. For example, other labels are optionally used for various application icons. In some embodiments, the label of each application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0154] FIG. 4B shows an example user interface on a device (e.g., device 300, FIG. 3) that has touch-sensitive surface 451 (e.g., tablet or touchpad 355, FIG. 3) separate from display 450. While many of the following examples are given with reference to input on touchscreen display 112 (where the touch-sensitive surface and display are combined), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as shown in FIG. 4B . In some embodiments, the touch-sensitive surface (e.g., 451 in FIG. 4B ) has a major axis (e.g., 452 in FIG. 4B ) that corresponds to a major axis (e.g., 453 in FIG. 4B ) on the display (e.g., 450). According to these embodiments, the device detects contact with touch-sensitive surface 451 (e.g., 460 and 462 in FIG. 4B ) at locations that correspond to respective locations on the display (e.g., 460 corresponds to 468, and 462 corresponds to 470 in FIG. 4B ). In this manner, when the touch-sensitive surface is separate from the display, user input (e.g., contacts 460 and 462, and their movement) detected by the device on the touch-sensitive surface (e.g., 451 in FIG. 4B ) is used by the device to operate a user interface on the display (e.g., 450 in FIG. 4B ) of the multifunction device. It should be understood that similar methods are optionally used for the other user interfaces described herein.

[0155] Additionally, while the following description is primarily described with reference to finger input (e.g., finger touches, finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of the finger inputs are replaced with input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced with a mouse click (e.g., instead of a touch), followed by cursor movement along the path of the swipe (e.g., instead of a touch movement). As another example, a tap gesture is optionally replaced with a mouse click while the cursor is positioned over the location of the tap gesture (e.g., instead of detecting a touch and then ceasing contact detection). Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice are optionally used simultaneously, or a mouse and finger touches are optionally used simultaneously.

[0156] As used herein, the term "focus selector" refers to an input element that indicates the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other location marker, the cursor functions as a "focus selector" such that when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B) while the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations including a touchscreen display (e.g., touch-sensitive display system 112 of FIG. 1A or the touchscreen of FIG. 6A ) that allows direct interaction with user interface elements on the touchscreen display, contact detected on the touchscreen acts as a “focus selector” such that when input (e.g., a contact press input) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of ​​the user interface to another area of ​​the user interface (e.g., by using the tab key or arrow keys to move focus from one button to another) without a corresponding cursor movement or contact movement on the touchscreen display; in these implementations, the focus selector moves in accordance with the movement of focus between different areas of the user interface.Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on a touchscreen display) that is controlled by the user to communicate the user's intended interaction with the user interface (e.g., by indicating to the device which element of the user interface the user intends to interact with). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of the focus selector (e.g., a cursor, contact, or selection box) over a corresponding button indicates that the user intends to activate the corresponding button (rather than other user interface elements displayed on the device's display).

[0157] As used herein and in the claims, the term “intensity” of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or a proxy for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a value range that includes at least four different values ​​and more typically includes hundreds (e.g., at least 256) different values. The intensity of a contact is optionally determined (or measured) using various techniques and various sensors or combinations of sensors. For example, one or more force sensors under or adjacent to the touch-sensitive surface are optionally used to measure force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted average or sum) to determine an estimated force of the contact. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change in the contact area detected on the touch-sensitive surface, the capacitance and / or change in the capacitance of the touch-sensitive surface proximate the contact, and / or the resistance and / or change in the capacitance of the touch-sensitive surface proximate the contact are optionally used as surrogates for the force or pressure of the contact on the touch-sensitive surface. In some implementations, the surrogate measure for the force or pressure of the contact is used directly to determine whether an intensity threshold is exceeded (e.g., the intensity threshold is stated in units corresponding to the surrogate measure). In some implementations, the surrogate measure for the force or pressure of the contact is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold is exceeded (e.g., the intensity threshold is a pressure threshold measured in units of pressure).Using the intensity of contact as an attribute of user input enables a user to access additional device functionality that may not otherwise be easily accessible by a user in devices of reduced size that have limited area for displaying affordances (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical control such as a knob or button).

[0158] In some embodiments, contact / motion module 130 uses a set of one or more intensity thresholds for determining whether an action has been performed by a user (e.g., for determining whether a user has “clicked” on an icon). In some embodiments, at least a subset of the intensity thresholds are determined according to software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and may be adjusted without modifying the physical hardware of device 100). For example, the mouse “click” threshold of a trackpad or touchscreen display can be set to any of a wide range of predetermined thresholds without modifying the trackpad or touchscreen display hardware. Additionally, in some implementations, a device user is provided with a software setting to adjust one or more of the set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once with a system-level click “intensity” parameter).

[0159] As used herein and in the claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on a plurality of intensity samples. The characteristic intensity is optionally based on a predetermined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) relative to a predetermined event (e.g., after detecting the contact, before detecting liftoff of the contact, before or after detecting the start of contact movement, before detecting the end of the contact, before or after detecting an increase in the intensity of the contact, and / or before or after detecting a decrease in the intensity of the contact). The characteristic intensity of the contact is optionally based on one or more of the maximum intensity of the contact, the mean value of the intensity of the contact, the average value of the intensity of the contact, the top 10% of the intensity of the contact, half the maximum intensity of the contact, 90% of the maximum intensity of the contact, a value generated by low-pass filtering the intensity of the contact starting over a predetermined period or at a predetermined time, etc. In some embodiments, the duration of the contact is used to determine the characteristic intensity (e.g., when the characteristic intensity is an average of the intensity of the contact over time). In some embodiments, the characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an operation is performed by the user. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, a contact with a characteristic intensity that does not exceed the first threshold results in a first operation being performed, a contact with a characteristic intensity that exceeds the first intensity threshold but does not exceed the second intensity threshold results in a second operation being performed, and a contact with a characteristic intensity that exceeds the second intensity threshold results in a third operation being performed. In some embodiments, the comparison between the characteristic intensity and one or more intensity thresholds is used to determine whether to perform one or more operations (e.g., to perform a respective option or to forgo performing a respective operation), rather than to determine whether to perform the first or second operation.

[0160] In some embodiments, portions of a gesture are identified for purposes of determining the characteristic intensity. For example, a touch-sensitive surface may receive a continuous swipe contact (e.g., a drag gesture) that transitions from a start location to an end location, where the intensity of the contact increases. In this example, the characteristic intensity of the contact at the end location may be based on only a portion of the swipe contact (e.g., only the portion of the swipe contact at the end location) rather than the entire continuous swipe contact. In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms filter out small increases or decreases in the intensity of the swipe contact for purposes of determining the characteristic intensity.

[0161] The user interface diagrams described herein optionally include one or more intensity thresholds (e.g., a touch detection intensity threshold IT0, a light pressure intensity threshold IT L , deep pressure intensity threshold IT D (e.g., IT, at least initially L higher than 0.15), and / or one or more other intensity thresholds (e.g., IT L An intensity threshold lower than IT H)). The intensity diagrams are generally not part of the illustrated user interface, but are provided to aid in interpretation of the diagrams. In some embodiments, the light pressure intensity threshold corresponds to an intensity at which the device performs an operation typically associated with a physical mouse button or a trackpad click. In some embodiments, the deep pressure intensity threshold corresponds to an intensity at which the device performs an operation different from an operation typically associated with a physical mouse button or a trackpad click. In some embodiments, when a contact is detected with a characteristic intensity below the light pressure intensity threshold (e.g., above a nominal contact-detection intensity threshold IT0 below which the contact is no longer detected), the device moves a focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an operation associated with the light or deep intensity thresholds. Generally, unless otherwise specified, these intensity thresholds are consistent across different sets of user interface configurations.

[0162] In some embodiments, the device's response to an input detected by the device depends on criteria based on the intensity of contact during the input. For example, for some "light press" inputs, the intensity of contact during the input exceeding a first intensity threshold triggers a first response. In some embodiments, the device's response to an input detected by the device depends on criteria including both the intensity of contact during the input and time-based criteria. For example, for some "deep press" inputs, the intensity of contact during the input exceeding a second intensity threshold greater than the first intensity threshold for light presses triggers a second response only if a delay time has elapsed between meeting the first intensity threshold and meeting the second intensity threshold. This delay time is typically less than 200 ms (milliseconds) in duration (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, with the delay time increasing as the second intensity threshold increases). This delay time helps avoid accidental recognition of deep press inputs. As another example, for some "deep presses," there is a period of reduced sensitivity that occurs after the first intensity threshold is met. During the period of decreased sensitivity, the second intensity threshold is increased. This temporary increase in the second intensity threshold also helps to avoid accidental deep pressure inputs. For other deep pressure inputs, the response to the detection of a deep pressure input does not depend on a time-based criterion.

[0163] In some embodiments, one or more of the input intensity threshold and / or corresponding output vary based on one or more factors such as user settings, contact movement, input timing, running application, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), position of focus selector, etc. Exemplary factors are described in U.S. Patent Application Nos. 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entireties.

[0164] For example, FIG. 4C illustrates a dynamic intensity threshold 480 that varies over time based in part on the intensity of touch input 476 over time. Dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predetermined delay time p1 from when touch input 476 is first detected, and a second component 478 that tracks the intensity of touch input 476 over time. The initial high intensity threshold of first component 474 reduces accidentally triggering a “deep press” response while still allowing an immediate “deep press” response if touch input 476 provides sufficient intensity. Second component 478 reduces unintentionally triggering a “deep press” response due to gradual intensity variations in the touch input. In some embodiments, a “deep press” response is triggered when touch input 476 meets dynamic intensity threshold 480 (e.g., at point 481 in FIG. 4C ).

[0165] FIG. 4D illustrates another dynamic intensity threshold 486 (e.g., intensity threshold I D ) Figure 4D also shows two other intensity thresholds, the first intensity threshold IT H and a second intensity threshold I L In FIG. 4D, the touch input 484 reaches the first intensity threshold IT before time p2. H and a second intensity threshold IT L 4D, the dynamic intensity threshold 486 is set to a value greater than the predetermined delay time p1 after time 482 (second intensity threshold IT L This type of dynamic intensity threshold decays over time, with the decay starting at time 488 (when the response associated with the first intensity threshold I H or the second intensity threshold I L Dynamic intensity threshold I immediately or simultaneously triggers a response associated with a lower intensity threshold, such as D and reducing accidental triggering of the associated response.

[0166] FIG. 4E illustrates yet another dynamic intensity threshold 492 (e.g., intensity threshold I D ) is shown. In FIG. 4E, the intensity threshold ITL The response associated with the intensity threshold I is triggered after a delay time p2 from when the touch input 490 is first detected. At the same time, the dynamic intensity threshold 492 decays after a predetermined delay time p1 from when the touch input 490 is first detected. Thus, the intensity threshold I can be increased or decreased without releasing the touch input 490. L The decrease in intensity of touch input 490 after triggering a response associated with the intensity threshold I, followed by an increase in the intensity of touch input 490, indicates that the intensity of touch input 490 exceeds another intensity threshold, e.g., intensity threshold I. L even when below the intensity threshold IT D A response associated with the event may be triggered (e.g., at time 494).

[0167] Light pressure intensity threshold IT L Light to strong pressure below the threshold IT L and deep pressure intensity threshold IT D Increasing the characteristic intensity of contact to an intensity between 0 and 100 is sometimes referred to as a "light press" input. D From an intensity below the threshold IT D An increase in the characteristic intensity of a contact to an intensity above the contact detection intensity threshold IT0 is sometimes referred to as a "deep press" input. L An increase in the characteristic intensity of the contact to an intensity between ITO and ITO may be referred to as detecting a contact on the touch surface. A decrease in the characteristic intensity of the contact from an intensity above the contact-detection intensity threshold IT0 to an intensity below the contact-detection intensity threshold IT0 may be referred to as detecting lift-off of the contact from the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some figures, shaded circles or ellipses are used to represent the intensity of the contacts on the touch-sensitive surface. In some figures, unshaded circles or ellipses are used to represent each contact on the touch-sensitive surface without specifying the intensity of each contact.

[0168] In some embodiments described herein, one or more operations are performed in response to detecting a gesture including the respective pressure input or in response to detecting the respective pressure input performed on the respective contact (or contacts), where the respective pressure inputs are detected based at least in part on detecting an increase in the intensity of the contact (or contacts) above a pressure input intensity threshold. In some embodiments, the respective operations are performed in response to detecting an increase in the intensity of the respective contact above a pressure input intensity threshold (e.g., the respective operations are performed on the “downstroke” of the respective pressure input). In some embodiments, the pressure input includes an increase in the intensity of the respective contact above a pressure input intensity threshold and a subsequent decrease in the intensity of the contact below the pressure input intensity threshold, where the respective operations are performed in response to detecting a subsequent decrease in the intensity of the respective contact below the pressure input threshold (e.g., the respective operations are performed on the “upstroke” of the respective pressure input).

[0169] In some embodiments, the device employs intensity hysteresis to avoid accidental input, sometimes referred to as “jitter,” and the device defines or selects a hysteresis intensity threshold that has a predetermined relationship to the pressure input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units below the pressure input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the pressure input intensity threshold). Thus, in some embodiments, the pressure input includes an increase in the intensity of each contact above the pressure input intensity threshold and a subsequent decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to the pressure input intensity threshold, and a respective operation is performed in response to detecting a subsequent decrease in the intensity of each contact below the hysteresis intensity threshold (e.g., a respective operation is performed on the “upstroke” of a respective pressure input). Similarly, in some embodiments, a pressure input is detected only when the device detects an increase in the intensity of the contact from an intensity below the hysteresis intensity threshold to an intensity above the pressure input intensity threshold, and optionally a subsequent decrease in the intensity of the contact to an intensity below the hysteresis intensity, and a respective operation is performed in response to detection of that pressure input (e.g., an increase in the intensity of the contact or a decrease in the intensity of the contact, as the case may be).

[0170] For ease of explanation, descriptions of operations performed in response to a pressure input associated with a pressure input intensity threshold or in response to a gesture including a pressure input are optionally triggered in response to detecting an increase in the intensity of the contact above the pressure input intensity threshold, an increase in the intensity of the contact from an intensity below a hysteresis intensity threshold to an intensity above the pressure input intensity threshold, a decrease in the intensity of the contact below the pressure input intensity threshold, or a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to the pressure input intensity threshold. Additionally, in examples described as performing an operation in response to detecting a decrease in the intensity of the contact below a pressure input intensity threshold, the operation is optionally performed in response to detecting a decrease in the intensity of the contact below a hysteresis intensity threshold corresponding to and lower than the pressure input intensity threshold. As described above, in some embodiments, the triggering of such responses also depends on a time-based criterion being met (e.g., a delay time having elapsed between the first intensity threshold being met and the second intensity threshold being met).

[0171] As used herein and in the claims, the term “tactile output” refers to a physical displacement of a device relative to its previous position, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, that is detected by a user with the user's sense of touch. For example, in a situation where a device or a component of the device is in contact with a touch-sensitive user surface (e.g., a user's finger, palm, or other part of the hand), the tactile output produced by the physical displacement is interpreted by the user as a tactile sensation corresponding to a perceived change in a physical property of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a “downclick” or “upclick” of a physical actuator button. In some cases, the user feels a tactile sensation such as a “downclick” or “upclick” even when no movement of a physical actuator button associated with the touch-sensitive surface is physically pressed (e.g., displaced) by the user's movement. As another example, movement of a touch-sensitive surface is optionally interpreted or perceived by a user as "roughness" of the touch-sensitive surface, even when there is no change in the smoothness of the touch-sensitive surface. While such user interpretation of touch depends on the user's personal sensory perception, there are many sensory perceptions of touch that are common to the majority of users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "upclick," "downclick," "roughness"), unless otherwise specified, the generated tactile output corresponds to a physical displacement of the device, or a component of the device, that produces the described sensory perception for a typical (or average) user.Using tactile output to provide haptic feedback to the user enhances the usability of the device and makes the user device interface more efficient (e.g., by assisting the user in providing appropriate inputs when operating / interacting with the device and reducing user errors), as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

[0172] In some embodiments, the tactile output pattern specifies characteristics of the tactile output, such as the amplitude of the tactile output, the shape of the moving waveform of the tactile output, the frequency of the tactile output, and / or the duration of the tactile output.

[0173] When tactile outputs having different tactile output patterns are generated by a device (e.g., via one or more tactile output generators that move a movable mass to generate the tactile output), the tactile outputs can induce different tactile sensations in a user holding or touching the device. While the user's sensations are based on the user's perception of the tactile output, most users are able to distinguish changes in waveform, frequency, and amplitude of the tactile output generated by the device. Thus, the waveform, frequency, and amplitude can be adjusted to inform the user that different operations have been performed. In this manner, tactile output having tactile output patterns designed, selected, and / or engineered to simulate the properties (e.g., size, material, weight, stiffness, smoothness, etc.), behavior (e.g., vibration, displacement, acceleration, rotation, magnification, etc.), and / or interaction (e.g., collision, adhesion, repulsion, attraction, friction, etc.) of objects in a given environment (e.g., a user interface including graphical features and objects, a simulated physical environment with virtual boundaries and virtual objects, a real physical environment with physical boundaries and physical objects, and / or any combination of the above) provides a user with useful feedback that, in some circumstances, reduces input errors and increases efficiency in the user's operation of the device. Additionally, the tactile output is optionally generated to correspond to feedback that is unrelated to the simulated physical properties (e.g., the input starting point or selection of an object). Such tactile output provides a user with useful feedback that, in some circumstances, reduces input errors and increases efficiency in the user's operation of the device.

[0174] In some embodiments, a tactile output having an appropriate tactile output pattern serves as a cue to the occurrence of an event of interest behind the scene within a user interface or device. Examples of events of interest include activation of an affordance (e.g., a real button, a virtual button, or a toggle switch) provided on the device or within the user interface, the success or failure of a requested operation, reaching or crossing a boundary within the user interface, entering a new state, switching input focus between objects, activating a new mode, reaching or crossing an input threshold, detecting or recognizing a type of input or gesture, etc. In some embodiments, a tactile output is provided to serve as a warning or alert to an impending event or outcome that will occur unless a redirection or interrupting input is detected in a timely manner. Tactile output may also be used in other contexts to enrich the user experience, improve device accessibility to users with visual or motor challenges or other accessibility needs, and / or improve the efficiency and functionality of the user interface and / or device. The tactile output is optionally accompanied by an audio output and / or a visible user interface change, which further enhances the user's experience when interacting with the user interface and / or device, further facilitates conveying information about the state of the user interface and / or device, and reduces input errors and increases efficiency in the user's operation of the device.

[0175] 4F-4H provide a set of sample tactile output patterns that can be used individually or in combination, either directly or with one or more transformations (e.g., modulation, amplification, truncation, etc.), to create appropriate tactile feedback for various scenarios and purposes, as described above and with respect to the user interfaces and methods discussed herein. This example of a palette of tactile outputs illustrates how a set of three waveforms and eight frequencies can be used to create an array of tactile output patterns. In addition to the tactile output patterns displayed in these figures, each of these tactile output patterns can optionally be adjusted in amplitude, for example, by changing the gain values ​​for the tactile output patterns for full tap 80 Hz, full tap 200 Hz, mini tap 80 Hz, mini tap 200 Hz, micro tap 80 Hz, and micro tap 200 Hz in FIGS. 4I-4K, as shown by variants with gains of 1.0, 0.75, 0.5, and 0.25, respectively. As shown in Figures 4I-4K, varying the acquisition of the tactile output pattern changes the amplitude of the pattern without changing the frequency of the pattern or the shape of the waveform. In some embodiments, varying the frequency of the tactile output pattern also results in lower amplitude because some tactile output generators are limited by how much force can be applied to the moving mass, and therefore higher frequency movements of the mass force lower amplitudes to ensure that the accelerations required to create the waveform do not require forces outside the operational force range of the tactile output generator (e.g., the peak amplitudes of full taps at 230 Hz, 270 Hz, and 300 Hz are lower than the amplitudes of full taps at 80 Hz, 100 Hz, 125 Hz, and 200 Hz).

[0176] Figures 4F-4K illustrate tactile output patterns having particular waveforms. The waveforms of the tactile output patterns represent a pattern of physical displacement relative to a neutral position (e.g., xzero) over time as the moving mass passes to generate a tactile output in that tactile output pattern. For example, a first set of tactile output patterns (e.g., "full tap" tactile output patterns) shown in Figure 4F each have waveforms that include two complete cycles of oscillation (e.g., oscillations that start and end at a neutral position and cross the neutral position three times). A second set of tactile output patterns (e.g., "mini tap" tactile output patterns) shown in Figure 4G each have waveforms that include one complete cycle of oscillation (e.g., oscillations that start and end at a neutral position and cross the neutral position once). A third set of tactile output patterns (e.g., "micro tap" tactile output patterns) shown in Figure 4H each have waveforms that include one-half of a complete cycle of oscillation (e.g., oscillations that start and end at a neutral position and do not cross the neutral position). The waveform of the tactile output pattern also includes start and end buffers that represent the gradual speed-up and slow-down of the moving mass at the beginning and end of the tactile output. The example waveforms displayed in Figures 4F-4K include xmin and xmax values ​​that represent the maximum and minimum range of movement of the moving mass. For larger electronic devices with larger moving masses, there may be larger or smaller minimum and maximum ranges of mass movement. Although the examples shown in Figures 4F-4K describe movement of a mass in one dimension, however, similar principles apply to movement of a moving mass in two or three dimensions.

[0177] As shown in Figures 4F-4K, each tactile output pattern also has a corresponding characteristic frequency that affects the "pitch" of the tactile sensation felt by the user from a tactile output having that characteristic frequency. For a continuous tactile output, the characteristic frequency represents the number of cycles (e.g., cycles per second) completed by the moving mass of the tactile output generator during a given time period. For a discontinuous tactile output, a discontinuous output signal is generated (e.g., at 0.5, 1, or 2 cycles), and the characteristic frequency value conditions how rapidly the moving mass needs to move to generate a tactile output having that characteristic frequency. As shown in Figures 4F-4H, for each type of tactile output (e.g., defined by its respective waveform (e.g., full tap, mini tap, or micro tap)), higher frequency values ​​correspond to faster movement by the moving mass and therefore generally correspond to shorter times to complete the tactile output (e.g., including the time to complete the required number of cycles for a discontinuous tactile output, plus start and end buffer times). For example, a full tap with a characteristic frequency of 80 Hz takes longer to complete than a full tap with a characteristic frequency of 100 Hz (e.g., 35.4 ms vs. 28.3 ms in Figure 4F). Additionally, for a given frequency, a tactile output with more cycles in its waveform at each frequency will generally take longer to complete than a full tap with a characteristic frequency of 100 Hz (e.g., 35.4 ms vs. 28.3 ms in Figure 4F). Tactile outputs with fewer cycles in their waveforms take longer to complete. For example, a 150 Hz full tap takes longer to complete than a 150 Hz mini-tap (e.g., 19.4 ms vs. 12.8 ms), and a 150 Hz mini-tap takes longer to complete than a 150 Hz micro-tap (e.g., 12.8 ms vs. 9.4 ms). However, for tactile output patterns with different frequencies, this rule does not hold true (e.g., a tactile output with more cycles but a higher frequency may take less time to complete than a tactile output with fewer cycles but a lower frequency, and vice versa). For example, at 300 Hz, a full tap takes about the same time as a mini-tap (e.g., 9.9 ms).

[0178] As shown in FIGS. 4F-4K, a tactile output pattern also has a characteristic amplitude that affects the amount of energy contained in the tactile signal, i.e., the “intensity” of the tactile sensation that can be felt by a user due to a tactile output having that characteristic amplitude. In some embodiments, the characteristic amplitude of a tactile output pattern is related to an absolute or normalized value that represents the maximum displacement of the moving mass from a neutral position when generating a tactile output. In some embodiments, the characteristic amplitude of a tactile output pattern is adjustable, for example, by a fixed or dynamically determined gain factor (e.g., a value between 0 and 1), according to various conditions (e.g., customized based on user interface context and behavior) and / or pre-defined metrics (e.g., input-based metrics and / or user interface-based metrics). In some embodiments, an input-based metric (e.g., an intensity-change metric or an input velocity metric) measures a characteristic of an input (e.g., the rate of change of the characteristic intensity of a contact for a pressure input or the rate of movement of a contact across the touch-sensitive surface) during the input that triggers generation of the tactile output. In some embodiments, a user interface-based metric (e.g., a boundary-wide velocity metric) measures a characteristic of a user interface element (e.g., the velocity of an element's movement across a hidden or visible boundary of the user interface) during a user interface change that triggers the generation of a tactile output. In some embodiments, the characteristic amplitude of the tactile output pattern can be adjusted by an "envelope," where peaks in adjacent cycles can have different amplitudes, where one of the waveforms is further modified by multiplication by a time-varying envelope parameter (e.g., from 0 to 1) to adjust the amplitude of portions of the tactile output in stages over time as the tactile output is being generated.

[0179] Although only characteristic frequencies, amplitudes, and waveforms are shown in the sample tactile output patterns of Figures 4F-4K for illustrative purposes, tactile output patterns having other frequencies, amplitudes, and waveforms can be used for similar purposes. For example, waveforms between 0.5 and 4 cycles can be used. Other frequencies in the range of 60 Hz to 400 Hz can be used as well. User Interface and Related Processes

[0180] Attention is now directed to embodiments of user interfaces (“UI”) and associated processes that may be implemented on an electronic device, such as portable multifunction device 100 or device 300, that includes a display, a touch-sensitive surface, (optionally) one or more tactile output generators for generating tactile output, and (optionally) one or more sensors for detecting the intensity of contact with the touch-sensitive surface.

[0181] 5A-5AT illustrate example user interfaces, according to some embodiments, for displaying representations of virtual objects while switching from displaying a first user interface region to displaying a second user interface region. The user interfaces of these figures are used to illustrate the processes described below, including the processes of FIGS. 8A-8E, 9A-9D, 10A-10D, 16A-16G, 17A-17D, 18A-18I, 19A-19H, and 20A-20F. For ease of explanation, some of the embodiments are described with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representational point corresponding to a finger or stylus contact (e.g., an enantiomeric center of each contact or a point associated with each contact), or an enantiomeric center of two or more contacts detected on touch-sensitive display system 112. However, a similar operation is optionally performed on a device having display 450 and a separate touch-sensitive surface 451 in response to detecting a contact on touch-sensitive surface 451 while displaying a user interface that is shown in the figure on display 450, along with a focus selector.

[0182] FIG. 5A illustrates a real-world context in which the user interface described with respect to FIGS. 5B-5AT is used.

[0183] 5A illustrates a physical space 5002 in which a table 5004 is located. The device 100 is held by a user in the user's hand 5006.

[0184] 5B illustrates a message sending user interface 5008 displayed on display 112. Message sending user interface 5008 includes a message bubble 5010 containing a received text message 5012, a message bubble 5014 containing a sent text message 5016, a message bubble 5018 containing a virtual object received in the message (e.g., a virtual chair 5020), and a virtual object indicator 5022 indicating that virtual chair 5020 is a visible object in augmented reality (e.g., within a representation of the field of view of one or more cameras of device 100). Message sending user interface 5008 also includes a message input area 5024 configured to display a message input.

[0185] 5C-5G illustrate inputs that cause portions of message sending user interface 5008 to be replaced by the field of view of one or more cameras of device 100. In FIG. 5C, contact 5026 with touchscreen 112 of device 100 is detected. The characteristic intensity of the contact, as indicated by intensity level meter 5028, exceeds contact-detection intensity threshold IT0 and is greater than hint pressure intensity threshold IT1. H In FIG. 5D, as shown by the intensity level meter 5028, the hint pressure intensity threshold IT H Increasing the characteristic intensity of contact 5026 above 0 causes the area of ​​message bubble 5018 to increase, the size of virtual chair 5020 to increase, and the send message user interface 5008 to begin to blur behind message bubble 5018 (e.g., to provide the user with visual feedback of the effect of increasing the characteristic intensity of the contact). L Increasing the characteristic intensity of contact 5026 above the deep pressure intensity threshold IT replaces message bubble 5018 with platter 5030, further increases the size of virtual chair 5020, and increases the blur of message send user interface 5008 behind platter 5030. In FIG. 5F, as shown by intensity level meter 5028,D An increase in the characteristic intensity of contact 5026 above causes tactile output generator 167 of device 100 to output a tactile output (as shown at 5032) indicating that the criteria for replacing a portion of message sending user interface 5008 with the field of view of one or more cameras of device 100 have been met.

[0186] In some embodiments, the characteristic intensity of the contact 5026 is determined by a deep pressure intensity threshold IT D 5C-5E is reversible, as illustrated in FIG. 5F. For example, decreasing the characteristic intensity of contact 5026 after the increase illustrated in FIGS. 5D and / or 5E causes an interface state corresponding to the decreased intensity level of contact 5026 to be displayed (e.g., the interface shown in FIG. 5E appears to be a state where the decreased characteristic intensity of contact is below the light pressure intensity threshold IT). L 5D, the interface shown in FIG. 5D indicates that the reduced characteristic strength of the contact exceeds the hint pressure intensity threshold IT H , and the interface shown in FIG. 5C indicates that the reduced characteristic strength of the contact exceeds the hint pressure intensity threshold IT H 5D and / or 5E). In some embodiments, decreasing the characteristic intensity of contact 5026 after the increase illustrated in Figures 5D and / or 5E causes the interface illustrated in Figure 5C to reappear.

[0187] 5F-5J illustrate animated transitions in which portions of the messaging user interface are replaced with the field of view of one or more cameras (hereinafter "cameras") of device 100. From FIG. 5F to FIG. 5G, contact 5026 lifted off touchscreen 112, and virtual chair 5020 rotated toward its final position in FIG. 5I. In FIG. 5G, camera field of view 5034 began to fade in to a view of platter 5030 (as indicated by the dotted line). In FIG. 5H, camera field of view 5034 (e.g., showing the view of physical space 5002 captured by the camera) completed fading in to a view of platter 5030. From FIG. 5H to FIG. 5I, virtual chair 5020 continued its rotation toward its final position in FIG. 5I. 5I , tactile output generator 167 outputs a tactile output (as shown at 5036) to indicate that at least one plane (e.g., floor surface 5038) has been detected in camera's field of view 5034. Virtual chair 5020 is positioned in the detected plane (e.g., pursuant to a determination by device 100 that the virtual object is configured to be placed in an upright orientation on a detected horizontal plane (e.g., floor surface 5038)). The size of virtual chair 5020 is continuously adjusted on display 112 as portions of the message sending user interface change to a representation of camera's field of view 5034 on display 112. For example, the scale of virtual chair 5020 relative to physical space 5002 shown in camera's field of view 5034 is determined based on a predetermined "real-world" size of virtual chair 5020 and / or the detected size of an object (e.g., table 5004) in camera's field of view 5034. 5J, the virtual chair 5020 is shown in its final position with a predetermined orientation relative to the detected floor plane of the camera's field of view 5034. In some embodiments, the initial landing position of the virtual chair 5020 is a predetermined position relative to the detected plane of the camera's field of view, e.g., the center of an open area of ​​the detected plane.In some embodiments, the landing position of contact 5026 is determined according to the lift-off position of contact 5026 (e.g., the initial lift-off position of contact 5026 may differ from the initial touch-down position of contact 5026 due to movement of contact 5026 across touch screen 112 after the criteria for transitioning to the augmented reality environment are met in FIG. 5F).

[0188] 5K-5L illustrate movement of device 100 (e.g., by user's hand 5006) adjusting camera field of view 5034. As device 100 is moved relative to physical space 5002, camera's displayed field of view 5034 changes, and virtual chair 5020 remains attached in the same position and orientation relative to floor surface 5038 in camera's displayed field of view 5034.

[0189] Figures 5M-5Q illustrate inputs that result in movement of virtual chair 5020 across floor surface 5038 of camera's displayed field of view 5034. In Figure 5N, contact 5040 with touchscreen 112 of device 100 is detected at a location corresponding to virtual chair 5020. In Figures 5N-5O, virtual chair 5020 is dragged by contact 5040 as contact 5040 moves along a path indicated by arrow 5042. As virtual chair 5020 is moved by contact 5040, the size of virtual chair 5020 changes, as shown in camera's field of view 5034, to maintain the scale of virtual chair 5020 relative to physical space 5002. For example, in Figures 5N-5P, the size of virtual chair 5020 decreases as virtual chair 5020 moves from the foreground of camera field of view 5034 to a position further from device 100 and closer to table 5004 in camera field of view 5034 (e.g., thereby maintaining the scale of the chair relative to table 5004 in camera field of view 5034). Additionally, as virtual chair 5020 is moved by contact 5040, planes identified in camera field of view 5034 are emphasized. For example, floor plane 5038 is emphasized in Figure 5O. In Figures 5O-5P, virtual chair 5020 continues to be dragged by contact 5040 as contact 5040 moves along the path indicated by arrow 5044. In Figure 5Q, contact 5040 lifts off touchscreen 112. 5N-5Q, the movement path of virtual chair 5020 is constrained by floor surface 5038 of camera field of view 5034, as if virtual chair 5020 were being dragged across floor surface 5038 by contact 5040. In some embodiments, contact 5040 described with respect to FIGS. 5N-5P is a continuation of contact 5026 described with respect to FIGS. 5C-5F (e.g., there is no lift-off of contact 5026, and the same contact that causes a portion of message sending user interface 5008 to be replaced by camera field of view 5034 also drags virtual chair 5020 across camera field of view 5034).

[0190] 5Q-5U illustrate inputs that cause virtual chair 5020 to move from floor surface 5038 to a different plane (e.g., tabletop 5046) detected in camera field of view 5034. In FIG. 5R, contact 5050 with touchscreen 112 of device 100 is detected at a location corresponding to virtual chair 5020. In FIGS. 5R-5S, virtual chair 5020 is dragged by contact 5048 as contact 5048 moves along the path indicated by arrow 5050. As virtual chair 5020 is moved by contact 5048, the size of virtual chair 5020 changes to maintain the scale of virtual chair 5020 relative to physical space 5002, as shown in camera field of view 5034. Additionally, as virtual chair 5020 is moved by contact 5040, tabletop reference plane 5046 is emphasized (e.g., as shown in FIG. 5S). 5S-5T, virtual chair 5020 continues to be dragged by contact 5040 as contact 5048 moves along the path indicated by arrow 5052. In FIG. 5U, contact 5048 has lifted off touchscreen 112, and virtual chair 5020 is placed on table reference surface 5046 in an upright orientation facing the same direction as before.

[0191] 5U-5AD illustrate inputs that drag virtual chair 5020 to the edge of touchscreen display 112, thereby removing camera field of view 5034 from view. In FIG. 5V, contact 5054 with touchscreen 112 of device 100 is detected at a location corresponding to virtual chair 5020. In FIGS. 5V-5W, virtual chair 5020 is dragged by contact 5054 as contact 5054 moves along the path indicated by arrow 5056. In FIGS. 5W-5X, virtual chair 5020 continues to be dragged by contact 5054 to the position displayed in FIG. 5X as contact 5054 moves along the path indicated by arrow 5058.

[0192] 5U-5X causes a transition from displaying camera field of view 5034 on platter 5030 to ceasing to display camera field of view 5034 and returning to fully displaying send message user interface 5008, as shown in FIGS. 5Y-5AD. In FIG. 5Y, camera field of view 5034 begins to fade out on platter 5030. In FIGS. 5Y-5Z, platter 5030 transitions to message bubble 5018. In FIG. 5Z, camera field of view 5034 is no longer displayed. In FIG. 5AA, send message user interface 5008 becomes unblurred and the size of message bubble 5018 returns to the original size of message bubble 5018 (e.g., as shown in FIG. 5B).

[0193] Figures 5AA-5AD illustrate the animated transition of virtual chair 5020 that occurs as virtual chair 5020 moves from the position corresponding to contact 5054 in Figure 5AA to its original position in message sending user interface 5008 (e.g., as shown in Figure 5B). In Figure 5AB, contact 5054 lifts off touch screen 112. In Figures 5AB-5AC, virtual chair 5020 increases in size in incremental steps and rotates toward its final position in Figure 5AD.

[0194] 5B-5AD, the virtual chair 5020 has substantially the same three-dimensional appearance in the message sending user interface 5008 and in the camera's displayed field of view 5034, and the virtual chair 5020 maintains that same three-dimensional appearance during transitions from displaying the message sending user interface 5008 to displaying the camera's field of view 5034, and during reverse transitions. In some embodiments, the representation of the virtual chair 5020 has a different appearance in the application user interface (e.g., the message sending user interface) than in the augmented reality environment (e.g., in the camera's displayed field of view). For example, the virtual chair 5020 optionally has a two-dimensional or more conventional appearance in the application user interface, while having a three-dimensional and more realistic and textured appearance in the augmented reality environment, and the intermediate appearance of the virtual chair 5020 during transitions between displaying the application user interface and displaying the augmented reality environment is a series of interpolated appearances between the two-dimensional appearance and the three-dimensional appearance of the virtual chair 5020.

[0195] 5AE illustrates an internet browser user interface 5060. The internet browser user interface 5060 includes a URL / search input area 5062 configured to display a URL / search input for a web browser and browser controls 5064 (e.g., navigation controls including back and forward buttons, a share control for displaying a share interface, a bookmark control for displaying a bookmark interface, and a tab control for displaying a tab interface). The internet browser user interface 5060 also includes web objects 5066, 5068, 5070, 5072, 5074, and 5076. In some embodiments, each web object includes a link, whereby, in response to a tap input on the respective web object, the linked internet location corresponding to the web object is displayed in the internet browser user interface 5060 (e.g., replacing the display of the respective web object). The web objects 5066, 5068, and 5072 include two-dimensional representations of three-dimensional virtual objects, as indicated by virtual object indicators 5078, 5080, and 5082, respectively. Web objects 5070, 5074, and 5076 comprise two-dimensional images (however, as indicated by the lack of virtual object indicators, the two-dimensional images of web objects 5070, 5074, and 5076 do not correspond to three-dimensional virtual objects). The virtual object corresponding to web object 5068 is lamp object 5084.

[0196] 5AF-5AH illustrate inputs that cause a portion of the internet browser user interface 5060 to be replaced by the camera's field of view 5034. In FIG. 5AF, a contact 5086 is detected with the touchscreen 112 of the device 100. The characteristic intensity of the contact, as indicated by the intensity level meter 5028, exceeds the contact detection intensity threshold IT0 and exceeds the hint pressure intensity threshold IT1. H 5AG, as shown by the intensity level meter 5028, the light pressure intensity threshold ITL Increasing the characteristic intensity of contact 5026 above a deep pressure intensity threshold IT 10 causes the camera's field of view 5034 to be displayed on web object 5068 (e.g., overlaid with virtual lamp 5084). D An increase in the characteristic intensity of contact 5086 above causes camera field of view 5034 to replace a larger portion of internet browser user interface 5060 (e.g., leaving only URL / search input area 5062 and browser controls 5064), and tactile output generator 167 of device 100 outputs a tactile output (as shown at 5088) indicating that the criteria for replacing a portion of internet browser user interface 5060 with camera field of view 5034 have been met. In some embodiments, in response to the inputs described with respect to Figures 5AF-5AH, camera field of view 5034 completely replaces internet browser user interface 506 on touchscreen display 112.

[0197] 5AI-5AM illustrate inputs that result in movement of virtual lamp 5084. In FIGS. 5AI-5AJ, virtual lamp 5084 is dragged by contact 5086 as contact 5086 moves along the path indicated by arrow 5090. As virtual lamp 5084 is moved by contact 5086, the size of virtual lamp 5084 remains constant, and the path of virtual lamp 5084 is, optionally, not constrained by the structure of the physical space captured in the camera's field of view. As virtual lamp 5084 is moved by contact 5086, planes identified in camera's field of view 5034 are emphasized. For example, as virtual lamp 5084 moves over floor plane 5038, floor plane 5038 is emphasized in FIG. 5AJ. In FIGS. 5AJ-5AK, virtual lamp 5084 continues to be dragged by contact 5086 as contact 5086 moves along the path indicated by arrow 5092. 5AK-5AL, as contact 5086 moves along the path indicated by arrow 5094, virtual lamp 5084 continues to be dragged by contact 5086, floor plane 5038 is de-emphasized, and tabletop 5046 is emphasized as virtual lamp 5084 moves over table 5004. In FIG. 5AM, contact 5086 lifts off touchscreen 112. Once contact 5086 has lifted off, the size of virtual lamp 5086 is adjusted to have the correct scale for table 5004 in camera field of view 5034, and virtual lamp 5086 is placed in an upright orientation on tabletop 5046 in camera field of view 5034.

[0198] 5AM-5AQ illustrate inputs dragging virtual lamp 5084 to the edge of touchscreen display 112, thereby removing camera field of view 5034 and restoring internet browser user interface 5060. In FIG. 5AN, contact 5096 with touchscreen 112 of device 100 is detected at a location corresponding to virtual lamp 5084. In FIGS. 5AN-5AO, virtual lamp 5084 is dragged by contact 5096 as contact 5096 moves along the path indicated by arrow 5098. In FIGS. 5AO-5AP, virtual lamp 5084 continues to be dragged by contact 5096 to the position displayed in FIG. 5AP as contact 5054 moves along the path indicated by arrow 5100. In FIG. 5AQ, contact 5096 lifts off touchscreen 112.

[0199] Input by contact 5096, as illustrated in Figures 5AM-5AP, causes a transition from displaying camera field of view 5034 to ceasing to display camera field of view 5034 and returning to fully displaying internet browser user interface 5060, as shown in Figures 5AQ-5AT. In Figure 5AR, camera field of view 5034 begins to fade out (as indicated by the dotted line). In Figures 5AR-5AT, virtual lamp 5084 increases in size and moves toward its original position in internet browser user interface 5060. In Figure 5AS, camera field of view 5034 is no longer displayed, and internet browser user interface 5060 begins to fade in (as indicated by the dotted line). In Figure 5AT, internet browser user interface 5060 is fully displayed, and virtual lamp 5084 has returned to its original size and position within internet browser user interface 5060.

[0200] 6A-6AJ illustrate example user interfaces for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object having representations of the fields of view of one or more cameras, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 8A-8E, 9A-9D, 10A-10D, 16A-16G, 17A-17D, 18A-18I, 19A-19H, and 20A-20F. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representational point corresponding to a finger or stylus contact (e.g., an enantiomeric center of each contact or a point associated with each contact), or an enantiomeric center of two or more contacts detected on touch-sensitive display system 112. However, a similar operation is optionally performed on a device having display 450 and a separate touch-sensitive surface 451 in response to detecting a contact on touch-sensitive surface 451 while displaying a user interface that is displayed as shown on display 450, along with a focus selector.

[0201] 6A illustrates a message sending user interface 5008 that includes a message bubble 5010 containing a received text message 5012, a message bubble 5014 containing a sent text message 5016, a message bubble 5018 containing a virtual object received in the message (e.g., a virtual chair 5020), and a virtual object indicator 5022 that indicates that the virtual chair 5020 is a visible object in augmented reality viewing (e.g., within the field of view of one or more displayed cameras of device 100). The message sending user interface 5008 is described in further detail with respect to FIG. 5B.

[0202] 6B-6C illustrate inputs that cause rotation of virtual chair 5020. In FIG. 6B, contact 6002 is detected with touchscreen 112 of device 100. Contact 6002 moves across touchscreen 112 along the path indicated by arrow 6004. In FIG. 6C, in response to the movement of the contact, message sending user interface 5008 scrolls upward (causing message bubble 5010 to scroll off the display and message bubbles 5014 and 5018 to scroll upward, revealing additional message bubble 6005), and virtual chair 5020 rotates (e.g., tilts upward). The magnitude and direction of the rotation of virtual chair 5020 corresponds to the movement of contact 6002 along the path indicated by arrow 6004. In FIG. 6D, contact 6002 lifts off touchscreen 112. In some embodiments, this rotational behavior of the virtual chair 5020 within the message bubble 5018 is used as an indication that the virtual chair 5020 is a virtual object visible in the augmented reality environment that includes the field of view of the camera of the device 100.

[0203] 6E-6L illustrate an input that causes message sending user interface 5008 to be replaced by staging user interface 6010 and subsequently changes the orientation of virtual chair 5020. In FIG. 6E, contact 6006 with touchscreen 112 of device 100 is detected. As shown by intensity level meter 5028, the characteristic intensity of the contact exceeds contact detection intensity threshold IT0 and exceeds hint pressure intensity threshold IT H 6F, as shown by the intensity level meter 5028, the hint pressure intensity threshold IT H Increasing the characteristic intensity of contact 6006 above 0 causes the area of ​​message bubble 5018 to increase, the size of virtual chair 5020 to increase, and message send user interface 5008 to begin to blur behind message bubble 5018 (e.g., providing the user with visual feedback of the effect of increasing the characteristic intensity of the contact). LIncreasing the characteristic intensity of contact 6006 above 0.001 caused message bubble 5018 to be replaced by platter 6008, further increasing the size of virtual chair 5020, and increasing the blur of message sending user interface 5008 behind platter 6008. In FIG. 6H, as shown by intensity level meter 5028, the deep pressure intensity threshold IT D An increase in the characteristic intensity of contact 6006 above a deep pressure intensity threshold IT 006 causes the send message user interface 5008 to disappear and activates the fade-in of the staging user interface 6010 (indicated by the dotted line). In addition, as shown in FIG. 6H, D An increase in the characteristic intensity of contact 6006 above causes tactile output generator 167 of device 100 to output a tactile output (as shown at 6012) indicating that the criteria for replacing message sending user interface 5008 with staging user interface 6010 have been met.

[0204] In some embodiments, the characteristic intensity of the contact 6006 is determined by a deep pressure intensity threshold IT D 6E-6G are reversible, as illustrated in FIG. 6H. For example, decreasing the characteristic intensity of contact 6006 after the increase illustrated in FIGS. 6F and / or 6G causes a corresponding interface state to be displayed at the decreased intensity level of contact 6006 (e.g., the interface illustrated in FIG. 6G appears to be a state where the decreased characteristic intensity of contact is below the light pressure intensity threshold IT). L 6F, the interface shown in FIG. 6F indicates that the reduced characteristic strength of the contact exceeds the hint pressure intensity threshold IT H , and the interface shown in FIG. 6E indicates that the reduced characteristic strength of the contact exceeds the hint pressure intensity threshold IT H 6F and / or 6G, decreasing the characteristic intensity of contact 6006 causes the interface shown in FIG. 6E to reappear.

[0205] In FIG. 6I, a staging user interface 6010 is displayed. The staging user interface 6010 includes a stage 6014 on which a virtual chair 5020 is displayed. From FIGS. 6H-6I, the virtual chair 5020 is animated to illustrate the transition from the position of the virtual chair 5020 in FIG. 6H to the position of the virtual chair 5020 in FIG. 6I. For example, the virtual chair 5020 is rotated to a predetermined position, orientation, and / or distance relative to the stage 6014 (e.g., such that the virtual chair appears to be supported by the stage 6014). The staging user interface 6010 also includes a back control 6016 that, when activated (e.g., by a tap input at a location corresponding to the back control 6016), causes a previously displayed user interface (e.g., the message sending user interface 5008) to be redisplayed. The staging user interface 6010 also includes a toggle control 6018 that indicates the current display mode (e.g., the current display mode is the staging user interface mode, as indicated by the highlighted "3D" indicator) and that, when activated, transitions to the selected display mode. For example, while the staging user interface 6010 is displayed, a tap input by contact at a location corresponding to the toggle control 6018 (e.g., a location corresponding to the portion of the toggle control 6018 including the text "World") causes the staging user interface 6010 to be replaced with the camera's field of view. The staging user interface 6010 also includes a share control 6020 (e.g., for displaying a share interface).

[0206] 6J-6L illustrate the rotation of virtual chair 5020 relative to stage 6014 caused by movement of contact 6006. In FIGS. 6J-6K, as contact 6006 moves along the path indicated by arrow 6022, virtual chair 5020 rotates (e.g., about a first axis that is perpendicular to the movement of contact 6066). In FIGS. 6K-6L, as contact 6006 moves along the path indicated by arrow 6024 and then along the path indicated by arrow 6025, virtual chair 5020 rotates (e.g., about a second axis that is perpendicular to the movement of contact 6066). In FIG. 6M, contact 6006 lifts off touchscreen 112. In some embodiments, the rotation of virtual chair 5020 is constrained by the surface of stage 6014, as shown in FIGS. 6J-6L. For example, at least one leg of virtual chair 5020 remains in contact with the surface of stage 6014 during rotation of the virtual chair. In some embodiments, the surface of stage 6014 serves as a frame of reference for free rotation and vertical translation of virtual chair 5020 without imposing any particular constraints on the movement of virtual chair 5020.

[0207] 6N-6O illustrate inputs adjusting the displayed size of virtual chair 5020. In FIG. 6N, a first contact 6026 and a second contact 6030 are detected with touchscreen 112. First contact 6026 moves along the path indicated by arrow 6028, and concurrently with the movement of first contact 6026, second contact 6030 moves along the path indicated by arrow 6032. In FIGS. 6N-6O, the displayed size of virtual chair 5020 increases as first contact 6026 and second contact 6030 move along the paths indicated by arrows 6028 and 6032, respectively (e.g., in a depinch gesture). In FIG. 6P, first contact 6030 and second contact 6026 lift off touchscreen 112, and virtual chair 5020 maintains its increased size after liftoff of contacts 6026 and 6030.

[0208] 6Q-6U illustrate inputs that cause the staging user interface 6010 to be replaced by the field of view 6036 of one or more cameras of the device 100. In FIG. 6Q, a contact 6034 is detected with the touchscreen 112 of the device 100. As shown by the intensity level meter 5028, the characteristic intensity of the contact exceeds the contact detection intensity threshold IT0 and exceeds the hint pressure intensity threshold IT1. H In FIG. 6R, as shown by the intensity level meter 5028, the hint pressure intensity threshold IT H As the characteristic intensity of contact 5026 increases above , the staging user interface 6010 begins to blur behind the virtual chair 5020 (as indicated by the dotted line). L An increase in the characteristic intensity of contact 6034 above a deep pressure intensity threshold IT , as shown by intensity level meter 5028 in FIG. 6T, causes staging user interface 6010 to disappear and activates a fade-in (indicated by the dotted line) of camera field of view 6036. D An increase in the characteristic intensity of contact 6034 above a deep pressure intensity threshold IT D 6U , contact 6034 has lifted off touchscreen 112. In some embodiments, the characteristic intensity of contact 6034 exceeds a deep pressure intensity threshold IT D 6Q-6T is reversible, as illustrated in FIG. 6T. For example, decreasing the characteristic intensity of contact 6034 after the increase illustrated in FIG. 6R and / or 6S causes an interface state corresponding to the decreased intensity level of contact 6034 to be displayed.

[0209] 6Q-6U, virtual chair 5020 is positioned on the detected plane (e.g., pursuant to a determination by device 100 that virtual chair 5020 is configured to be placed in an upright orientation on a detected horizontal plane, such as floor surface 5038), and the size of virtual chair 5020 is adjusted (e.g., the scale of virtual chair 5020 relative to physical space 5002 shown in camera field of view 6036 is determined based on a predetermined "real-world" size of virtual chair 5020 and / or the detected size of an object (e.g., table 5004) in camera field of view 6036). The orientation of virtual chair 5020, resulting from a rotation of virtual chair 5020 while staging interface 6010 was displayed (e.g., as described with respect to FIGS. 6J-6K), is maintained as virtual chair 5020 transitions from staging user interface 6010 to camera field of view 6036. For example, the orientation of the virtual chair 5020 relative to the floor surface 5038 is the same as the final orientation of the virtual chair 5020 relative to the surface of the stage 5014. In some embodiments, adjustments to the size of the virtual object 5020 in the staging user interface are taken into account when the size of the virtual chair 5020 is adjusted in the field of view 6036 relative to the size of the physical space 5002.

[0210] 6V-6Y illustrate an input that causes camera field of view 6036 to be replaced by staging user interface 6010. In FIG. 6V, contact 6040 input (e.g., a tap input) is detected at a location corresponding to toggle control 6018 (e.g., a location corresponding to the portion of toggle control 6018 containing the text "3D"). In FIGS. 6W-6Y, in response to contact 6040 input, camera field of view 6036 fades out (as indicated by the dotted lines in FIG. 6W), staging user interface 6010 fades in (as indicated by the dotted lines in FIG. 6X), and staging user interface 6010 is fully displayed (as shown in FIG. 6Y). From FIGS. 6V-6Y, the size of virtual chair 5020 is adjusted and the position of virtual chair 5020 changes (e.g., to restore virtual chair 5020 to a predetermined position and size for the staging user interface).

[0211] 6Z-6AC illustrate an input that causes staging user interface 6010 to be replaced by send message user interface 5008. In FIG. 6Z, input by contact 6042 (e.g., a tap input) is detected at a location corresponding to back control 6016. In FIGS. 6AA-6AC, in response to input by contact 6042, staging user interface 6010 fades out (as indicated by the dotted line in FIG. 6AA), send message user interface 5008 fades in (as indicated by the dotted line in FIG. 6AB), and send message user interface 5008 is fully displayed (as shown in FIG. 6AC). From FIGS. 6Z-6AB, the size, orientation, and position of virtual chair 5020 are continuously adjusted on the display (e.g., to return virtual chair 5020 to the predetermined position, size, and orientation for send message user interface 5008).

[0212] 6AD-6AJ illustrate an input that causes message-sending user interface 5008 to be replaced by camera field of view 6036 (e.g., bypassing the display of staging user interface 6010). In FIG. 6AD, contact 6044 is detected at a location corresponding to virtual chair 5020. The input by contact 6044 includes a long touch gesture (during which contact 6044 is maintained at a location on the touch-sensitive surface corresponding to the representation of virtual object 5020 for at least a predetermined threshold amount of movement) followed by an upward swipe gesture (dragging virtual chair 5020 upward). As shown in FIGS. 6AD-6AE, virtual chair 5020 is dragged upward as contact 6044 moves along the path indicated by arrow 6046. In FIG. 6AE, message-sending user interface 5008 fades out behind virtual chair 5020. As shown in FIGS. 6AE-6AF, virtual chair 5020 continues to be dragged upward as contact 6044 moves along the path indicated by arrow 6048. In FIG. 6AF, camera field of view 5036 fades in behind virtual chair 5020. In FIG. 6AG, in response to input by contact 6044, which includes a long touch gesture followed by an upward swipe gesture, camera field of view 5036 is fully displayed. In FIG. 6AH, contact 6044 lifts off touchscreen 112. In FIGS. 6AH-6AJ, in response to the lift-off of contact 6044, virtual chair 5020 is released (e.g., because virtual chair 5020 is no longer being propelled or dragged by the contact) and falls to a plane (e.g., floor surface 5038, pursuant to a determination that a horizontal (floor) surface corresponds to virtual chair 5020). Additionally, as shown in FIG. 6AJ, the tactile output generator 167 of the device 100 outputs a tactile output (as shown at 6050) to indicate that the virtual chair 5020 has landed on the floor surface 5038.

[0213] 7A-7P illustrate example user interfaces for displaying items with visual indications to indicate that the items correspond to virtual three-dimensional objects, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes in FIGS. 8A-8E, 9A-9D, 10A-10D, 16A-16G, 17A-17D, 18A-18I, 19A-19H, and 20A-20F. For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. In such embodiments, the focus selector is optionally a respective finger or stylus contact, a representational point corresponding to a finger or stylus contact (e.g., a centroid of a respective contact or a point associated with a respective contact), or a centroid of two or more contacts detected on touch-sensitive display system 112. However, a similar operation is optionally performed on a device having display 450 and a separate touch-sensitive surface 451 in response to detecting a contact on touch-sensitive surface 451 while displaying a user interface that is displayed as shown on display 450, along with a focus selector.

[0214] 7A illustrates an input detected while user interface 400 for a menu of applications is displayed. The input corresponds to a request to display a first user interface (e.g., internet browser user interface 5060). In FIG. 7A, input by contact 7000 (e.g., a tap input) is detected at a location corresponding to icon 420 for browser module 147. In response to the input, internet browser user interface 5060 is displayed, as shown in FIG. 7B.

[0215] 7B illustrates internet browser user interface 5060 (e.g., as described in detail with respect to FIG. 5AE). Internet browser user interface 5060 includes web objects 5066, 5068, 5070, 5072, 5074, and 5076. Web objects 5066, 5068, and 5072 include two-dimensional representations of three-dimensional virtual objects, as indicated by virtual object indicators 5078, 5080, and 5082, respectively. Web objects 5070, 5074, and 5076 include two-dimensional images (however, the two-dimensional images of web objects 5070, 5074, and 5076 do not correspond to three-dimensional virtual objects, as indicated by the absence of virtual object indicators).

[0216] 7C-7D illustrate inputs resulting from translational movement (e.g., scrolling) of internet browser user interface 5060. In FIG. 7B, contact 7002 with touch screen 112 is detected. In FIGS. 7C-7D, as contact 7002 moves along the path indicated by arrow 7004, web objects 5066, 5068, 5070, 5072, 5074, and 5076 scroll upward to reveal additional web objects 7003 and 7005. Additionally, as contact 7002 moves along the path indicated by arrow 7004, the virtual objects of web objects 5066, 5068, and 5072, including virtual object indicators 5078, 5080, and 5082, respectively, rotate (e.g., tilt upward) according to the (upward vertical) direction of the input. For example, virtual lamp 5084 tilts upward from a first orientation in FIG. 7C to a second orientation in FIG. 7D. The two-dimensional images of web objects 5070, 5074, and 5076 do not rotate as the contact scrolls through internet browser user interface 5060. In Figure 7E, contact 7002 has lifted off touch screen 112. In some embodiments, the rotational behavior of the objects represented by web objects 5066, 5068, and 5072 is used as a visual indication that these web objects have corresponding three-dimensional virtual objects visible in the augmented reality environment, while the lack of such rotational behavior of the objects represented by web objects 5070, 5074, and 5076 is used as a visual indication that these web objects do not have corresponding three-dimensional virtual objects visible in the augmented reality environment.

[0217] 7F-7G illustrate the parallax effect in which a virtual object rotates on the display in response to changes in the orientation of device 100 relative to the physical world.

[0218] Figure 7F1 illustrates device 100 being held in user hand 5006 by user 7006 so that device 100 is in a substantially vertical orientation. Figure 7F2 illustrates internet browser user interface 5060 displayed by device 100 when device 100 is in the orientation illustrated in Figure 7F1.

[0219] FIG. 7G1 illustrates device 100 being held in user hand 5006 by user 7006 so that device 100 is oriented substantially landscape. FIG. 7G2 illustrates internet browser user interface 5060 displayed by device 100 when device 100 is in the orientation illustrated in FIG. 7G1. From FIG. 7F2 to FIG. 7G2, the virtual object orientations of web objects 5066, 5068, and 5072, including virtual object indicators 5078, 5080, and 5082, respectively, rotate (e.g., tilt upward) in accordance with changes in device orientation. For example, virtual lamp 5084 tilts upward from a first orientation in FIG. 7F2 to a second orientation in FIG. 7G2 in accordance with concurrent changes in device orientation in physical space. The two-dimensional images of web objects 5070, 5074, and 5076 do not rotate as the device orientation changes. In some embodiments, the rotational behavior of the objects represented by web objects 5066, 5068, and 5072 is used as a visual indication that these web objects have corresponding three-dimensional virtual objects visible in the augmented reality environment, while the lack of such rotational behavior of the objects represented by web objects 5070, 5074, and 5076 is used as a visual indication that these web objects do not have corresponding three-dimensional virtual objects visible in the augmented reality environment.

[0220] 7H-7L illustrate inputs corresponding to a request to display a second user interface (e.g., message sending user interface 5008). In FIG. 7H, contact 7008 is detected at a location corresponding to the lower edge of display 112. In FIGS. 7H-7I, contact 7008 moves upward along the path indicated by arrow 7010. In FIGS. 7I-7J, contact 7008 continues to move upward along the path indicated by arrow 7012. In FIGS. 7H-7J, as contact 7008 moves upward from the lower edge of display 112, internet browser user interface 5060 decreases in size, as shown in FIG. 7I, and in FIG. 7J, multitasking user interface 7012 is displayed (e.g., in response to an upward edge swipe gesture by contact 7008). The multitasking user interface 7012 is configured to enable interface selection among various applications having a retained state (e.g., the retained state is the last state of each application when it was the foreground application running on the device) and various control interfaces (e.g., the control center user interface 7014, the internet browser user interface 5060, and the send message user interface 5008, as shown in FIG. 7J). In FIG. 7K, contact 7008 lifts off the touch screen 112. In FIG. 7L, an input (e.g., a tap input) by contact 7016 is detected at a location corresponding to the send message user interface 5008. In response to the input by contact 7016, the send message user interface 5008 is displayed, as shown in FIG. 7M.

[0221] 7M illustrates a message sending user interface 5008 (e.g., as described in further detail with respect to FIG. 5B ) that includes a message bubble 5018 that includes a virtual object received in the message (e.g., a virtual chair 5020) and a virtual object indicator 5022 to indicate that the virtual chair 5020 is a virtual three-dimensional object (e.g., an object visible in an augmented reality view and / or an object visible from different angles). The message sending user interface 5008 also includes a message bubble 6005 that includes a sent text message and a message bubble 7018 that includes a received text message that includes an emoji 7020. The emoji 7020 is a two-dimensional image that does not correspond to a virtual three-dimensional object. For this reason, the emoji 7020 is displayed without a virtual object indicator.

[0222] 7N illustrates a map user interface 7022 including a map 7024, a point of interest information area 7026 for a first point of interest, and a point of interest information area 7032 for a second point of interest. For example, the first point of interest and the second point of interest are search results within or immediately adjacent to the area shown by the map 7024 corresponding to the search input "Apple" in the search input area 7025. In the first point of interest information area 7026, a first point of interest object 7028 is displayed with a virtual object indicator 7030 to indicate that the first point of interest object 7028 is a virtual three-dimensional object. In the second point of interest information area 7032, a second point of interest object 7034 is displayed without a virtual object indicator because the second point of interest object 7034 does not correspond to a virtual three-dimensional object visible in the augmented reality view.

[0223] 7O illustrates a file management user interface 7036 including file management controls 7038, a file management search input area 7040, a file information area 7042 for a first file (e.g., a portable document format (PDF) file), a file information area 7044 for a second file (e.g., a photo file), a file information area 7046 for a third file (e.g., a virtual chair object), and a file information area 7048 for a fourth file (e.g., a PDF file). The third file information area 7046 includes a virtual object indicator 7050 displayed adjacent to a file preview object 7045 in the file information area 7046 to indicate that the third file corresponds to a virtual three-dimensional object. The first file information area 7042, the second file information area 7044, and the fourth file information area 7048 are displayed without a virtual object indicator because the files corresponding to these file information areas do not have corresponding virtual three-dimensional objects visible in the augmented reality environment.

[0224] 7P illustrates an email user interface 7052 including email navigation controls 7054, an email information area 7056, and an email content area 7058 including a representation of a first attachment 7060 and a representation of a second attachment 7062. The representation of the first attachment 7060 includes a virtual object indicator 7064 to indicate that the first attachment is a virtual three-dimensional object visible in the augmented reality environment. The second attachment 7062 is displayed without a virtual object indicator because the second attachment is not a virtual three-dimensional object visible in the augmented reality environment.

[0225] 8A-8E are flowcharts illustrating a method 800 for displaying a representation of a virtual object and switching from displaying a first user interface area to displaying a second user interface area, according to some embodiments. Method 800 is performed on an electronic device (e.g., device 300 of FIG. 3 or portable multifunction device 100 of FIG. 1A ) having a display, a touch-sensitive surface, and one or more cameras (e.g., one or more rear cameras on a side of the device opposite from the display and touch-sensitive surface). In some embodiments, the display is a touchscreen display, and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations of method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0226] Method 800 involves detecting an input via contact on a touch-sensitive surface of a device that displays a representation of a virtual object in a first user interface area. In response to the input, the device replaces a display of at least a portion of the first user interface area with a field of view of one or more cameras of the device and uses criteria for determining whether to continuously display the representation of the virtual object. Replacing a display of at least a portion of the first user interface area with a field of view of one or more cameras and using criteria for determining whether to continuously display the representation of the virtual object enables performance of multiple different types of operations in response to the input. Enabling performance of multiple different types of operations in response to the input (e.g., by replacing a display of at least a portion of the user interface with a field of view of one or more cameras or by maintaining a display of the first user interface area without replacing a display of at least a portion of the first user interface area with a representation of the field of view of one or more cameras) increases the efficiency with which a user can perform these operations, thereby enhancing operability of the device, which in turn improves the device's battery life by reducing power usage and allowing a user to use the device more quickly and efficiently.

[0227] The device displays (802) representations of virtual objects (e.g., graphical representations of three-dimensional objects, such as a virtual chair 5020, a virtual lamp 5084, shoes, furniture, hand tools, decorations, people, emojis, game characters, virtual furniture, etc.) in a first user interface area (e.g., a two-dimensional graphical user interface or portion (e.g., a viewable list of furniture images, an image including one or more selectable objects, etc.)) on the display 112. For example, the first user interface area is the messaging user interface 5008 shown in FIG. 5B or the internet browser user interface 5060 shown in FIG. 5AE. In some embodiments, the first user interface area includes a background other than an image of the physical environment surrounding the device (e.g., the background of the first user interface area is a pre-selected background color / pattern, or a background image that is different from the output imagery captured concurrently by one or more cameras and different from the active content of the field of view of the one or more cameras).

[0228] While displaying a first representation of the virtual object in a first user interface region on the display, the device detects (804) a first input by contact at a location on touch-sensitive surface 112 corresponding to the representation of the virtual object on the display (e.g., the contact is detected on the first representation of the virtual object on a touchscreen display, or the contact is detected on an affordance displayed in parallel in the first user interface region with the first representation of the virtual object and configured to trigger the display of an AR view of the virtual object when activated by the contact). For example, the first input is input by contact 5020 described with reference to Figures 5C-5F or input by contact 5086 described with reference to Figures 5AF-5AL.

[0229] In response to detecting a first input by contact (806), and pursuant to a determination that the first input by contact satisfies first (e.g., AR trigger) criteria (e.g., criteria configured to identify a swipe input, a touch-and-hold input, a press input, a tap input, a hard press having an intensity equal to or greater than a predetermined intensity threshold, or another predetermined type of input gesture associated with triggering camera activation, displaying an augmented reality (AR) view of the physical environment surrounding the device, three-dimensional placement of a virtual object within the augmented reality view of the physical environment, and / or a combination of two or more of the above actions), the device displays a second user interface area on the display that includes replacing the display of at least a portion of the first user interface area with a representation of the field of view of one or more cameras, and the device switches from displaying the first user interface area to displaying the second user interface area while continuously displaying the representation of the virtual object. For example, the second user interface area on the display is the field of view 5034 of the camera of the platter 5030 described with reference to FIG. 5H or the field of view 5034 of the camera described with reference to FIG. 5AH. 5C to 5I, the input by the contact 5026 is deep, and the pressure intensity threshold IT D 5AH, in response to a determination that the input by contact 5086 has an increasing characteristic intensity above a deep pressure intensity threshold IT, virtual chair object 5020 continues to be displayed and switches from displaying a first user interface region (send message user interface 5008) to displaying a second user interface region that replaces the display of a portion of send message user interface 5008 with a camera field of view 5034 of platter 5030. DIn accordance with a determination that the virtual lamp object 5084 has a characteristic intensity that increases above , the virtual lamp object 5084 continues to be displayed and switches from displaying the first user interface region (internet browser user interface 5060) to displaying a second user interface region that replaces the display of a portion of the internet browser user interface 5060 with the camera's field of view 5034.

[0230] In some embodiments, continuously displaying representations of the virtual object includes maintaining the display of the representation of the virtual object or displaying an animated transition of a first representation of the virtual object changing into a second representation of the virtual object (e.g., a view of the virtual object at a different size, from a different viewing angle, in a different drawing style, or at a different location on the display). In some embodiments, the field of view 5034 of one or more cameras displays a live image of the physical environment 5002 surrounding the device, which is updated in real time as the device's position and orientation changes relative to the physical environment (e.g., as illustrated in FIGS. 5K-5L). In some embodiments, the second user interface region completely replaces the first user interface region on the display.

[0231] In some embodiments, the second user interface region overlaps a portion of the first user interface region (e.g., a portion of the first user interface region is shown along an edge of the display or around a border of the display). In some embodiments, the second user interface region pops up next to the first user interface region. In some embodiments, the background in the first user interface region is replaced with the contents of the camera's field of view 5034. In some embodiments, the device displays an animated transition showing the virtual object moving and rotating (e.g., as illustrated in FIGS. 5E-5I ) from a first orientation shown in the first user interface region to a second orientation (e.g., a predetermined orientation relative to the current orientation of the portion of the physical environment captured in the field of view of one or more cameras). For example, the animation includes a transition from displaying a first user interface region along with a two-dimensional representation of the virtual object to displaying a second user interface region along with a three-dimensional representation of the virtual object. In some embodiments, the three-dimensional representation of the virtual object has a predefined anchor plane based on the shape and orientation of the virtual object as shown in the two-dimensional graphical user interface (e.g., the first user interface region). When transitioning to an augmented reality view (e.g., a second user interface area), the three-dimensional representation of the virtual object is moved, resized, and reoriented from the virtual object's original position on the display to a new position on the display (e.g., to the center of the augmented reality view or another predetermined position in the augmented reality view), and during or after the movement is completed, the three-dimensional representation of the virtual object is reoriented so that the three-dimensional representation of the virtual object is at a predetermined position and / or orientation relative to a predetermined plane identified in the field of view of one or more cameras (e.g., a physical surface such as an upright wall or a horizontal floor that can serve as a support plane for the three-dimensional representation of the virtual object).

[0232] In some embodiments, the first criterion includes a criterion that is met when (e.g., according to a determination that) the contact is maintained at a location on the touch-sensitive surface that corresponds to a representation of a virtual object with less than a threshold amount of movement for at least a predetermined time (e.g., a time threshold for a long press) (808). In some embodiments, in accordance with a determination that the contact meets the criterion for recognizing another type of gesture (e.g., a tap), the device performs another predetermined function other than triggering the AR user interface and maintains the display of the virtual object. Continuously displaying the representation of the virtual object and determining whether to replace the display of at least a portion of the first user interface area with the camera's field of view enables the performance of multiple different types of operations in response to the input, depending on whether the contact is maintained at a location on the touch-sensitive surface that corresponds to a representation of a virtual object with less than a threshold amount of movement for at least a predetermined time. Enabling the performance of multiple different types of operations in response to the input increases the efficiency with which a user can perform these operations, thereby enhancing the usability of the device, which in turn improves the device's battery life by reducing power usage and allowing a user to use the device more quickly and efficiently.

[0233] In some embodiments, the first criterion is whether the characteristic intensity of the contact exceeds a first intensity threshold (e.g., a light pressure intensity threshold IT L or deep pressure intensity threshold IT D 5C-5F, the criteria may include a criterion that is met when (e.g., pursuant to a determination that) the characteristic intensity of contact 5026, as indicated by intensity level meter 5028, increases above a deep pressure intensity threshold IT DThe first criterion is met when the characteristic intensity of the contact increases above a first intensity threshold. In some embodiments, pursuant to determining that the contact satisfies the criteria for recognizing another type of gesture (e.g., a tap), the device performs another predetermined function other than triggering the AR user interface and maintains the display of the virtual object. In some embodiments, the first criterion requires that the first input is not a tap input (e.g., the input has an elapsed time between touchdown of the contact and liftoff of the tap contact that is greater than a temporal threshold). Continuously displaying a representation of the virtual object and determining whether to replace the display of at least a portion of the first user interface area with the camera's field of view enables the performance of multiple different types of operations in response to the input, depending on whether the characteristic intensity of the contact increases above a first intensity threshold. Enabling the performance of multiple different types of operations in response to the input increases the efficiency with which a user can perform these operations, thereby enhancing the usability of the device, which in turn improves the device's battery life by reducing power usage and allowing a user to use the device more quickly and efficiently.

[0234] In some embodiments, the first criterion includes a criterion that is met (812) when (e.g., according to a determination that is) the movement of the contact meets a predetermined movement criterion (e.g., the contact moves across the touch-sensitive surface beyond a predetermined threshold location (e.g., a location corresponding to a boundary of the first user interface area, a location that is a threshold distance spaced from the original location of the contact, etc.), the contact moves at a velocity greater than a predetermined threshold velocity, the movement of the contact is terminated by a press input, etc.). In some embodiments, a representation of the virtual object is dragged by the contact during an initial portion of the contact's movement, and the virtual object stops moving with the contact when the movement of the contact attempts to meet the predetermined movement criterion to indicate that the first criterion is being met, and if the movement of the contact continues and the predetermined movement criterion is met by the continued movement of the contact, a transition begins that displays the second user interface area and displays the virtual object within the augmented reality view. In some embodiments, when the virtual object is dragged during an initial portion of the first input, the object size and viewing viewpoint do not change, and once the augmented reality view is displayed and the virtual object is dropped into position in the augmented reality view, the virtual object is displayed with a size and viewing viewpoint that depends on the physical location represented by the drop-off location of the virtual object in the augmented reality view. Continuously displaying a representation of the virtual object and determining whether to replace the display of at least a portion of the first user interface area with the camera's field of view enables the performance of multiple different types of operations in response to the input depending on whether movement of the contact meets predetermined movement criteria. Enabling the performance of multiple different types of operations in response to the input increases the efficiency with which a user can perform these operations, thereby enhancing the usability of the device, and it also improves the device's battery life by reducing power usage and allowing a user to use the device more quickly and efficiently.

[0235] In some embodiments, in response to detecting the first input by touch and in accordance with a determination that the first input by touch satisfies a first criterion, the device outputs a tactile output (814) at one or more tactile output generators 167 (e.g., tactile output 5032 described with reference to FIG. 5F or tactile output 5088 described with reference to FIG. 5AH) to indicate that the first criterion has been met by the first input. In some embodiments, the tactile sensation occurs before the field of view of one or more cameras appears on the screen. For example, the tactile sensation indicates that a first criterion has been met that triggers activation of one or more cameras and next plane detection of the field of view of the one or more cameras. Because it takes time for cameras to be activated and for fields of view to be available for display, the tactile sensation serves as a non-visual signal to the user that the device has detected the required input and presents the augmented reality user interface as soon as the device is ready.

[0236] Outputting a tactile output (e.g., to replace the display of at least a portion of the user interface with the camera's field of view) to indicate that the criteria have been met provides the user with feedback to indicate that the input being provided meets the criteria. Providing improved tactile feedback increases the usability of the device (e.g., by assisting the user in providing appropriate input when operating / interacting with the device and reducing user errors), and in addition, it reduces power usage and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

[0237] In some embodiments, in response to detecting at least an initial portion of the first input (e.g., including detecting a contact, or detecting an input with a contact that meets respective predetermined criteria without meeting the first criteria, or detecting an input that meets the first criteria), the device analyzes the field of view of the one or more cameras to detect one or more planes (e.g., floor surface 5038, tabletop 5046, wall, etc.) in the field of view of the one or more cameras (816). In some embodiments, the one or more cameras are activated in response to detecting at least a portion of the first input, and plane detection is initiated simultaneously with the camera activation. In some embodiments, display of the field of view of the one or more cameras is delayed after activation of the one or more cameras (e.g., from when the one or more cameras are activated until at least one plane is detected in the field of view of the camera). In some embodiments, display of the field of view of the one or more cameras is initiated when the one or more cameras are activated, and plane detection is completed after the field of view is already visible on the display (e.g., in the second user interface area). In some embodiments, after detecting each plane in the field of view of one or more cameras, the device determines the size and / or position of a representation of a virtual object based on the relative position of each plane with respect to the field of view of the one or more cameras. In some embodiments, as the electronic device is moved, the size and / or position of the representation of the virtual object is updated as the position of the field of view of one or more cameras changes with respect to the respective plane (e.g., as described with respect to FIGS. 5K-5L). Determining the size and / or position of the representation of a virtual object based on the position of each plane detected in the field of view of the cameras (e.g., without requiring further user input of the size and / or position of the virtual object with respect to the field of view of the cameras) enhances the usability of the device and, in addition, it reduces power usage and improves the battery life of the device by allowing the user to use the device more quickly and efficiently.

[0238] In some embodiments, analyzing the field of view of the one or more cameras to detect one or more planes of the field of view of the one or more cameras is initiated (818) in response to detecting a contact at a location on the touch-sensitive surface that corresponds to a representation of a virtual object on the display (e.g., in response to detecting contact 5026 at a location on touchscreen 112 that corresponds to virtual chair 5020). For example, activating the cameras and detecting the planes of the camera's field of view may be initiated if a first criterion is met by the first input (e.g., if the characteristic intensity of contact 5026 is greater than or equal to a deep pressure intensity threshold IT, as described with respect to FIG. 5F ). D (before the plane detection increases above ), and begins before the second user interface area is displayed. By beginning plane detection upon detection of any interaction with the virtual object, plane detection can be completed before the AR trigger criteria are met, and thus there is no visible delay for the user in seeing the virtual object transition in the augmented reality view when the AR trigger criteria are met by the first input. Triggering analysis to detect one or more planes in the camera's field of view in response to detecting a contact at the location of the representation of the virtual object (e.g., without requiring further user input to trigger analysis of the camera's field of view) enhances the effectiveness of the device, as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

[0239] In some embodiments, analyzing the field of view of the one or more cameras to detect one or more planes in the field of view of the one or more cameras is in response to detecting that a first criterion is satisfied by a first input by the contact (e.g., a characteristic intensity of the contact 5026 is greater than or equal to a deep pressure intensity threshold IT as described with respect to FIG. 5F ). D, or in response to detecting an increase above . For example, camera activation and detection of a plane in the camera's field of view may begin when a first criterion is met by a first input, and the camera's field of view may be displayed before plane detection is complete. By beginning camera activation and plane detection when the AR trigger criteria are met, camera and plane detection are not activated or continued unnecessarily, which conserves battery power and extends battery and camera life.

[0240] In some embodiments, analyzing the field of view of one or more cameras to detect one or more planes in the field of view of one or more cameras is initiated in response to detecting that an initial portion of the first input satisfies plane detection trigger criteria without satisfying the first criteria (822). For example, camera activation and detection of planes in the camera field of view are initiated when certain criteria (e.g., criteria less stringent than the AR trigger criteria) are satisfied by the initial portion of the first input, and the camera field of view is optionally displayed before plane detection is completed. By initiating camera activation and plane detection after certain criteria are satisfied rather than upon detection of contact, camera and plane detection are not unnecessarily activated or continued, which conserves battery power and extends battery life and camera life. By initiating camera activation and plane detection before the AR trigger criteria are satisfied, the delay (due to camera activation and plane detection) for displaying the virtual object transition to the augmented reality view is reduced when the AR trigger criteria are satisfied by the first input.

[0241] In some embodiments, the device displays (824) a representation of the virtual object in the second user interface area in a respective manner such that the virtual object (e.g., the virtual chair 5020) is oriented at a predetermined angle relative to each plane detected in the field of view 5034 of the one or more cameras (e.g., such that there is no (or only a small) distance separating the underside of the four legs of the virtual chair 5020 from the floor surface 5038). For example, the orientation and / or position of the virtual object relative to each plane is predefined based on the shape and orientation of the virtual object as shown in the two-dimensional graphical user interface (e.g., each plane corresponds to a horizontal physical surface that can serve as a support surface for a three-dimensional representation of the virtual object in the augmented reality view (e.g., a horizontal table surface that supports a vase), or each plane is a vertical physical surface that can serve as a support surface for a three-dimensional representation of the virtual object in the augmented reality view (e.g., a horizontal wall that suspends a virtual picture frame)). In some embodiments, the orientation and / or position of the virtual object is defined by each surface or boundary of the virtual object (e.g., a bottom surface, a bottom boundary point, a side surface, and / or a side boundary point). In some embodiments, the anchor plane corresponding to each plane is one characteristic among a set of characteristics of the virtual object and is specified according to the nature of the physical object that the virtual object is supposed to represent. In some embodiments, the virtual object is positioned at a predetermined orientation and / or position relative to multiple planes detected in the field of view of one or more cameras (e.g., multiple respective sides of the virtual object are associated with respective planes detected in the field of view of the cameras). In some embodiments, if a predefined orientation and / or position for the virtual object is determined relative to a horizontal bottom plane of the virtual object, the bottom plane of the virtual object is displayed on the floor plane detected in the field of view of the cameras (e.g., the horizontal bottom plane of the virtual object is zero distance from the floor plane and parallel to the floor plane).In some embodiments, when the predefined orientation and / or position for the virtual object is determined relative to the virtual object's vertical backplane, the back surface of the virtual object is positioned relative to a wall plane detected in the field of view of one or more cameras (e.g., the virtual object's vertical backplane is zero distance from and parallel to the wall plane). In some embodiments, the virtual object is positioned at a fixed distance relative to each plane and / or at an angle other than zero or perpendicular to each plane. Displaying a representation of the virtual object relative to a plane detected in the field of view of the cameras (e.g., without requiring additional user input to display the virtual object relative to the plane of the camera's field of view) enhances the usability of the device and, in addition, it reduces power usage and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

[0242] In some embodiments, in response to detecting each plane in the field of view of the one or more cameras, the device outputs a tactile output at one or more tactile output generators 167 to indicate detection of each plane in the field of view of the one or more cameras (826). In some embodiments, a respective tactile output is generated for each plane detected in the field of view of the camera (e.g., floor surface 5038 and / or table surface 5046). In some embodiments, the tactile output is generated when plane detection is complete. In some embodiments, the tactile output is accompanied by a visual indication of the field of view plane of the field of view shown in the second user interface portion (e.g., momentary highlighting of the detected field of view plane). Outputting a tactile output to indicate detection of a plane in the field of view of the camera provides feedback to the user to indicate that the plane has been detected. Providing improved tactile feedback enhances device usability (e.g., by helping the user provide appropriate inputs and reducing the additional inputs required to place virtual objects), as well as reducing power usage and improving the device's battery life by allowing the user to use the device more quickly and efficiently.

[0243] In some embodiments, upon switching from displaying the first user interface regi...

Claims

1. 1. A method implemented by a computer system having a display generation component, one or more input devices, and one or more cameras, comprising: displaying, via the display generation component, a representation of a virtual object in a first user interface region that includes a representation of a field of view of one or more cameras, including maintaining a first spatial relationship between the representation of the virtual object and a plane detected in a physical environment captured within the field of view of the one or more cameras; detecting movement of the computer system to adjust the field of view of the one or more cameras; in response to detecting a movement of the computer system to adjust the field of view of the one or more cameras. adjusting a display of the representation of the virtual object in the first user interface region according to the first spatial relationship between the virtual object and the plane detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted; generating a first alert in accordance with a determination that the movement of the computer system causes a greater than threshold amount of the virtual object to move outside of a displayed portion of the field of view of the one or more cameras.

2. The method of claim 1, wherein the computer system includes one or more audio output generators, and generating the first alert includes generating a first audio alert via the one or more audio output generators.

3. The method of claim 1, comprising generating audio associated with the virtual object after the movement of the computer system causes a greater than threshold amount of the virtual object to move outside the display portion of the field of view of the one or more cameras.

4. 4. The method of claim 1, wherein outputting the first alert comprises generating an audio output indicating an amount of the virtual object that remains visible in the displayed portion of the field of view of the one or more cameras.

5. 5. The method of claim 1, wherein outputting the first alert comprises generating an audio output indicating an amount of the displayed portion of the field of view that is obscured by the virtual object.

6. The method according to claim 1, wherein the one or more input devices include a touch-sensitive surface, detecting a contact input at a location on the touch-sensitive surface corresponding to the representation of the field of view of the one or more cameras; 6. The method of claim 1, further comprising: in response to detecting the input, generating a second audio alert in accordance with a determination that the input is detected at a first location on the touch-sensitive surface that corresponds to a first portion of the field of view of the one or more cameras that is not obscured by the virtual object.

7. 7. The method of claim 1, wherein outputting the first alert comprises generating an audio output indicative of an action performed with respect to the virtual object and a state of the virtual object resulting after performance of the action.

8. 8. The method of claim 7, wherein the state of the virtual object resulting after execution of the action is described in the audio output during the first alert relative to a frame of reference corresponding to the physical environment captured within the field of view of the one or more cameras.

9. detecting additional movement of the computer system after generation of the first alert, which further adjusts the field of view of the one or more cameras; in response to detecting the additional movement of the computer system further adjusting the field of view of the one or more cameras. adjusting a display of the representation of the virtual object in the first user interface region according to the first spatial relationship between the virtual object and the plane detected within the field of view of the one or more cameras as the field of view of the one or more cameras is further adjusted; and generating a second alert pursuant to a determination that the additional movement of the computer system causes a greater than a second threshold amount of the virtual object to move into a displayed portion of the field of view of the one or more cameras.

10. The method of claim 9, wherein the computer system includes one or more audio output generators, and generating the second alert includes generating a third audio alert via the one or more audio output generators.

11. displaying the representation of the virtual object in the first user interface region, and detecting a request to switch to another object manipulation type applicable to the virtual object while a first object manipulation type of a plurality of object manipulation types applicable to the virtual object is currently selected for the virtual object; 11. The method of claim 1, further comprising: in response to detecting the request to switch to another object manipulation type applicable to the virtual object, generating an audio output indicating a second object manipulation type of a plurality of object manipulation types applicable to the virtual object, the second object manipulation type being different from the first object manipulation type.

12. detecting a request to perform an object manipulation behavior corresponding to a currently selected object manipulation type after generating the audio output indicating the second object manipulation type of the plurality of object manipulation types applicable to the virtual object; 12. The method of claim 11, further comprising: in response to detecting the request to perform the object manipulation behavior corresponding to the currently selected object manipulation type, performing an object manipulation behavior corresponding to the second object manipulation type.

13. The method according to claim 1, wherein the one or more input devices include a touch-sensitive surface, in response to detecting the request to switch to another object manipulation type applicable to the virtual object; generating an audio alert in conjunction with the audio output indicating the second object manipulation type in response to determining that the second object manipulation type is a continuously adjustable manipulation type to indicate that the second object manipulation type is a continuously adjustable manipulation type; detecting a request to perform an object manipulation behavior corresponding to the second object manipulation type, the request including detecting a swipe input at a location on the touch-sensitive surface corresponding to a portion of the first user interface area that displays the representation of the field of view of the one or more cameras; 13. The method of claim 11, further comprising: in response to detecting the request to perform the object manipulation behavior corresponding to the second object manipulation type, performing the object manipulation behavior corresponding to the second object manipulation type by an amount corresponding to a magnitude of the swipe input.

14. displaying the representation of the virtual object in a second user interface area prior to displaying the representation of the virtual object in the first user interface area, the second user interface area not including a representation of the field of view of one or more cameras; and displaying the representation of the virtual object in the second user interface region, and detecting a request to switch to another action applicable to the virtual object while a first action of a plurality of actions applicable to the virtual object is currently selected for the virtual object; 14. The method of claim 1, further comprising: in response to detecting the request to switch to another action applicable to the virtual object in the second user interface area, generating an audio output indicating a second action of the plurality of actions applicable to the virtual object, the second action being different from the first action.

15. prior to displaying the representation of the virtual object within the first user interface region; detecting a request to display a representation of the virtual object in the first user interface area that includes a representation of the field of view of the one or more cameras while displaying the representation of the virtual object in a second user interface area that does not include a representation of the field of view of the one or more cameras; in response to detecting the request to display a representation of the virtual object within the first user interface region including a representation of the field of view of the one or more cameras; displaying the representation of the virtual object within the first user interface region according to the first spatial relationship between the representation of the virtual object and the plane detected in the physical environment captured within the field of view of the one or more cameras; and generating a fourth audio alert indicating that the virtual object is located within the first user interface area in relation to the physical environment captured within the field of view of the one or more cameras.

16. The method of claim 15 , wherein the fourth audio alert indicates information about an appearance of the virtual object with respect to a displayed portion of the field of view of the one or more cameras.

17. 17. The method of claim 15, comprising generating a tactile output consistent with a placement of the virtual object within the first user interface area relative to the physical environment captured within the field of view of the one or more cameras.

18. The method according to claim 1, wherein the one or more input devices include a touch-sensitive surface, displaying a first control element at a first location within the first user interface area concurrently with a representation of the field of view of the one or more cameras; pursuant to determining that a control fading criterion is satisfied, ceasing to display the first control element within the first user interface region while maintaining display of the representation of the field of view of the one or more cameras within the first user interface region; detecting a touch input at a respective location on the touch-sensitive surface corresponding to the first location in the first user interface area while displaying the first user interface area without displaying the first control element in the first user interface area; and generating a fifth audio alert in response to detecting the touch input, the fifth audio alert including an audio output indicative of an operation corresponding to the first control element.

19. A display generating component; one or more input devices; One or more cameras; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: instructions for displaying, via the display generation component, a representation of a virtual object in a first user interface region that includes a representation of a field of view of one or more cameras, wherein the displaying includes maintaining a first spatial relationship between the representation of the virtual object and a plane detected in a physical environment captured within the field of view of the one or more cameras; detecting movement of the computer system to adjust the field of view of the one or more cameras; in response to detecting a movement of the computer system to adjust the field of view of the one or more cameras. adjusting a display of the representation of the virtual object in the first user interface region according to the first spatial relationship between the virtual object and the plane detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted; 11. A computer system comprising: instructions for generating a first alert in response to a determination that the movement of the computer system causes a greater than threshold amount of the virtual object to move outside a displayed portion of the field of view of the one or more cameras.

20. The computer system of claim 19, wherein the one or more programs include instructions for performing a method according to any one of claims 2 to 18.

21. A computer program comprising instructions, which when executed by a computer system having a display generation component, one or more input devices, and one or more cameras, cause the computer system to: and causing, via the display generation component, a representation of a virtual object in a first user interface region that includes a representation of a field of view of one or more cameras, wherein the displaying includes maintaining a first spatial relationship between the representation of the virtual object and a plane detected in a physical environment captured within the field of view of the one or more cameras, the instructions further including: detecting movement of the computer system to adjust the field of view of the one or more cameras; in response to detecting a movement of the computer system to adjust the field of view of the one or more cameras. adjusting a display of the representation of the virtual object in the first user interface region according to the first spatial relationship between the virtual object and the plane detected in the field of view of the one or more cameras as the field of view of the one or more cameras is adjusted; generating a first alert in response to determining that the movement of the computer system causes a greater than threshold amount of the virtual object to move outside a displayed portion of the field of view of the one or more cameras.

22. The computer program of claim 21, further comprising instructions which, when executed by the computer system, cause the computer system to perform a method according to any one of claims 2 to 18.

Images