Devices, methods, and graphical user interfaces for system-level behavior of 3D models

Abstract

The disclosure relates to devices, methods, and graphical user interfaces for system-level behavior of 3D models. A computer system having a display generation component, one or more input devices, and one or more cameras receives a request to display a virtual object in a first user interface area that includes a field of view of the one or more cameras. In response to the request, the representation of the virtual object is displayed in a first set of visual attributes and a first orientation independent of which portion of the physical environment is displayed in the field of view of the one or more cameras in accordance with a determination that the object placement criteria are not satisfied. In accordance with a determination that the object placement criteria are satisfied, the representation of the virtual object is displayed in a second set of visual attributes and a second orientation, the second set of visual attributes being different from the first set of visual attributes, the second orientation corresponding to a plane.

Description

[0001] This application is a divisional application of the invention patent application with the application date of January 23, 2019, the application number of 201980010176.6, and the title of "Devices, Methods, and Graphical User Interfaces for System-Level Behavior of 3D Models". Technical Field

[0002] The present invention generally relates to electronic devices that display virtual objects, including but not limited to electronic devices that display virtual objects in various scenarios. Background Art

[0003] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices for computer systems and other electronic computing devices such as touch-sensitive surfaces are used to interact with virtual / augmented reality environments. Example touch-sensitive surfaces include touch pads, touch-sensitive remote controls, and touch screen displays. Such surfaces are used to manipulate the user interface and objects therein on the display. Exemplary user interface objects include digital images, videos, text, icons, and control elements (such as buttons) and other graphics.

[0004] However, the methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, using a series of inputs to orient and position virtual objects in an augmented reality environment is tedious, imposes a significant cognitive burden on the user, and detracts from the experience of the virtual / augmented reality environment. Additionally, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the Invention

[0005] Accordingly, there is a need for computer systems having improved methods and interfaces for interacting with virtual objects. Such methods and interfaces optionally supplement or replace conventional methods for interacting with virtual objects. Such methods and interfaces reduce the number, degree, and / or nature of inputs from the user and result in a more effective human-machine interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0006] The computer systems of the present disclosure reduce or eliminate the above deficiencies and other problems associated with interfaces for interacting with virtual objects (e.g., user interfaces for augmented reality (AR) and related non-AR interfaces). In some embodiments, the computer system includes a desktop computer. In some embodiments, the computer system is portable (e.g., a laptop computer, a tablet computer, or a handheld device). In some embodiments, the computer system includes a personal electronic device (e.g., a wearable electronic device such as a watch). In some embodiments, the computer system has a touchpad (and / or communicates with a touchpad). In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display") (and / or communicates with a touch-sensitive display). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, programs, or instruction sets stored in the memory for performing various functions. In some embodiments, the user interacts with the GUI, in part, by contact with a stylus and / or finger and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include playing games, image editing, drawing, presenting, word processing, spreadsheet creation, making phone calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note taking, and / or digital video playback. The executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] According to some embodiments, a method is performed at a computer system having a display, a touch-sensitive surface, and one or more cameras. The method includes displaying a representation of a virtual object in a first user interface region on the display. The method further includes, when the representation of the virtual object is displayed in the first user interface region on the display, detecting a first input by contact at a location on the touch-sensitive surface corresponding to the representation of the virtual object on the display. The method further includes, in response to detecting the first input by contact, and based on determining that the first input by contact meets a first criterion: displaying a second user interface region on the display, which includes replacing at least a portion of the display of the first user interface region, the first user interface region having a representation of the field of view of one or more cameras, and continuously displaying the representation of the virtual object when switching from displaying the first user interface region to displaying the second user interface region.

[0008] According to some embodiments, a method is performed at a computer system having a display, a touch-sensitive surface, and one or more cameras. The method includes displaying a first representation of a virtual object in a first user interface region on the display. The method further includes, when displaying the first representation of the virtual object in the first user interface region on the display, detecting a first input by a first contact at a position on the touch-sensitive surface corresponding to the first representation of the virtual object on the display. The method further includes, in response to detecting the first input by the first contact and based on determining that the input by the first contact meets a first criterion, displaying a representation of the virtual object in a second user interface region, the second user interface region being different from the first user interface region. The method further includes, when displaying a second representation of the virtual object in the second user interface region, detecting a second input, and in response to detecting the second input, based on determining that the second input corresponds to a request to manipulate the virtual object in the second user interface region, changing display attributes of the second representation of the virtual object within the second user interface region; and, based on determining that the second input corresponds to a request to display the virtual object in an augmented reality environment, displaying a third representation of the virtual object having a representation of a field of view of one or more cameras.

[0009] According to some embodiments, a method is performed at a computer system having a display and a touch-sensitive surface. The method includes, in response to a request to display a first user interface, displaying a first user interface having a representation of a first item. The method further includes, based on determining that the first item corresponds to a corresponding virtual three-dimensional object, displaying a representation of the first item having a visual indication indicating that the first item corresponds to the first corresponding virtual three-dimensional object. The method further includes, based on determining that the first item does not correspond to a corresponding virtual three-dimensional object, displaying a representation of the first item without the visual indication. The method further includes, after displaying the representation of the first item, receiving a request to display a second user interface including a second item. The method further includes, in response to the request to display the second user interface, displaying a second user interface having a representation of the second item. The method further includes, based on determining that the second item corresponds to a corresponding virtual three-dimensional object, displaying a representation of the second item having a visual indication indicating that the second item corresponds to the second corresponding virtual three-dimensional object. The method further includes, based on determining that the second item does not correspond to a corresponding virtual three-dimensional object, displaying a representation of the second item without the visual indication.

[0010] According to some embodiments, a method is performed at a computer system having a display generation component, one or more input devices, and one or more cameras. The method includes receiving a request to display a virtual object in a first user interface region, the first user interface region including at least a portion of the field of view of one or more cameras. The method further includes, in response to the request to display a virtual object in the first user interface region, displaying a representation of the virtual object via the display generation component above at least a portion of the field of view of one or more cameras included in the first user interface region, wherein the field of view of the one or more cameras is a view of the physical environment in which the one or more cameras are located. Displaying the representation of the virtual object includes: displaying a representation of the virtual object having a first set of visual attributes and a first orientation based on determining that an object placement criterion is not met, wherein the object placement criterion requires that the placement location of the virtual object be recognizable in the field of view of the one or more cameras in order to meet the object placement criterion, and the first orientation is independent of the portion of the physical environment displayed in the field of view of the one or more cameras; and displaying a representation of the virtual object having a second set of visual attributes and a second orientation based on determining that the object placement criterion is met, the second set of visual attributes being different from the first set of visual attributes, and the second orientation corresponding to a plane in the physical environment detected in the field of view of the one or more cameras.

[0011] According to some embodiments, a method is performed at a computer system having a display generation component, one or more input devices, one or more cameras, and one or more pose sensors for detecting changes in the pose of a device including the one or more cameras. The method includes receiving a request to display an augmented reality view of a physical environment in a first user interface region, the first user interface region including a representation of the field of view of the one or more cameras. The method further includes, in response to receiving the request to display the augmented reality view of the physical environment, displaying a representation of the field of view of the one or more cameras and, based on determining that a calibration criterion for the augmented reality view of the physical environment is not met, displaying a calibration user interface object that animates dynamically based on movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes: detecting, via the one or more pose sensors, changes in the pose of the one or more cameras in the physical environment while the calibration user interface object is being displayed; and, in response to detecting a change in the pose of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object based on the detected change in the pose of the one or more cameras in the physical environment. The method further includes detecting that the calibration criterion is met while the calibration user interface object is being displayed moving on the display based on the detected change in the pose of the one or more cameras in the physical environment. The method further includes, in response to detecting that the calibration criterion is met, stopping the display of the calibration user interface object.

[0012] According to some embodiments, a method is performed at a computer system having a display generation component and one or more input devices including a touch-sensitive surface. The method includes displaying, via the display generation component, a representation of a first perspective of a virtual three-dimensional object in a first user interface region. The method further includes, when the representation of the first perspective of the virtual three-dimensional object is displayed in the first user interface region on the display, detecting a first input corresponding to a request to rotate the virtual three-dimensional object relative to the display about a portion of the virtual three-dimensional object that is not visible from the first perspective of the virtual three-dimensional object. The method further includes, in response to detecting the first input: causing the virtual three-dimensional object to rotate about a first axis by an amount determined based on the magnitude of the first input, the amount being constrained by a movement limit that limits rotation of the virtual three-dimensional object about the first axis beyond a threshold rotation amount, according to determining that the first input corresponds to a request to rotate the three-dimensional object about the first axis; and causing the virtual three-dimensional object to rotate about a second axis different from the first axis by an amount determined based on the magnitude of the first input, wherein, for an input having a magnitude higher than a corresponding threshold, the device causes the virtual three-dimensional object to rotate about the second axis by more than the threshold rotation amount, according to determining that the first input corresponds to a request to rotate the three-dimensional object about a second axis different from the first axis.

[0013] According to some embodiments, a method is performed at a computer system having a display generation component and a touch-sensitive surface. The method includes displaying, via the display generation component, a first user interface region that includes user interface objects associated with a plurality of object manipulation behaviors, the plurality of object manipulation behaviors including a first object manipulation behavior performed in response to an input that satisfies a first gesture recognition criterion and a second object manipulation behavior performed in response to an input that satisfies a second gesture recognition criterion. The method further includes, while the first user interface region is being displayed, detecting a first portion of an input involving the user interface objects, which includes detecting movement of one or more contacts on the touch-sensitive surface, and evaluating the movement of the one or more contacts relative to the first gesture recognition criterion and the second gesture recognition criterion when the one or more contacts are detected on the touch-sensitive surface. The method further includes, in response to detecting the first portion of the input, updating an appearance of the user interface objects based on the first portion of the input, which includes: changing the appearance of the user interface objects according to the first object manipulation behavior and based on the first portion of the input, and updating the second gesture recognition criterion by increasing a threshold of the second gesture recognition criterion, according to determining that the first portion of the input satisfies the first gesture recognition criterion before satisfying the second gesture recognition criterion; and changing the appearance of the user interface objects according to the second object manipulation behavior and based on the first portion of the input, and updating the first gesture recognition criterion by increasing a threshold of the first gesture recognition criterion, according to determining that the input satisfies the second gesture recognition criterion before satisfying the first gesture recognition criterion.

[0014] According to some embodiments, a method is performed at a computer system having a display generation component, one or more input devices, one or more audio output generators, and one or more cameras. The method includes displaying, via the display generation component, a representation of a virtual object in a first user interface region that includes a representation of the field of view of one or more cameras, wherein the display includes maintaining a first spatial relationship between the representation of the virtual object and a plane detected within a physical environment captured in the field of view of one or more cameras. The method further includes detecting movement of a device that adjusts the field of view of one or more cameras. The method further includes, in response to detecting movement of a device that adjusts the field of view of one or more cameras: adjusting the display of the representation of the virtual object in the first user interface region according to the first spatial relationship between the virtual object and a plane detected within the field of view of one or more cameras when adjusting the field of view of one or more cameras, and generating a first audio alert via one or more audio output generators based on determining that the movement of the device causes the virtual object to move outside of a displayed portion of the field of view of one or more cameras and by an amount that exceeds a threshold amount.

[0015] According to some embodiments, an electronic device includes a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more haptic output generators, optionally one or more gesture sensors for detecting gesture changes, one or more processors, and a memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform the operations of any of the methods described herein. According to some embodiments, a computer-readable storage medium has instructions stored therein that, when executed by an electronic device having a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more haptic output generators, and optionally one or more gesture sensors, cause the device to perform the operations of any of the methods described herein or cause the operations of any of the methods described herein to be performed. According to some embodiments, a graphical user interface on an electronic device having a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more haptic output generators, and optionally one or more gesture sensors, a memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements displayed in any of the methods described herein, the one or more elements being updated in response to input as described in any of the methods described herein. According to some embodiments, an electronic device includes: a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more haptic output generators, and optionally one or more gesture sensors for detecting gesture changes; and means for performing or causing to perform the operations of any of the methods described herein.According to some embodiments, an information processing apparatus used in an electronic device having a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more haptic output generators, and optionally one or more gesture sensors includes means for performing the operations of any of the methods described herein or causing the operations of any of the methods described herein to be performed.

[0016] Accordingly, there is provided an improved method and interface for displaying virtual objects in various scenarios for an electronic device having a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting the intensity of contact with a touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more haptic output generators, and optionally one or more gesture sensors, thereby enhancing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for displaying virtual objects in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To better understand the various described embodiments, reference should be made to the following detailed description in conjunction with the accompanying drawings, in which like reference numerals indicate corresponding parts in all the drawings.

[0018] Figure 1A is a block diagram showing a portable multifunctional device having a touch-sensitive display according to some embodiments.

[0019] Figure 1B is a block diagram showing an example component for event handling according to some embodiments.

[0020] Figure 1C is a block diagram showing a haptic output module according to some embodiments.

[0021] Figure 2 shows a portable multifunctional device having a touch screen according to some embodiments.

[0022] Figure 3 is a block diagram of an example multifunctional device having a display and a touch-sensitive surface according to some embodiments.

[0023] Figure 4A shows an example user interface of an application menu on a portable multifunctional device according to some embodiments.

[0024] Figure 4B An example user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0025] Figures 4C to 4E Examples of dynamic intensity thresholds are shown according to some embodiments.

[0026] Figures 4F to 4K A set of sample tactile output patterns according to some embodiments are shown.

[0027] Figures 5A to 5AT An example user interface according to some embodiments is shown for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area.

[0028] Figures 6A to 6AJ An example user interface according to some embodiments is shown, which, according to some embodiments, is used to display a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object with a representation of a field of view of one or more cameras.

[0029] Figures 7A to 7E 、 Figures 7F1 to 7F2 、 Figures 7G1 to 7G2 as well as Figures 7H to 7P An example user interface is shown for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object according to some embodiments.

[0030] Figures 8A to 8E is a flowchart of a process for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area, according to some embodiments.

[0031] Figures 9A to 9D is a flowchart of a process according to some embodiments for displaying a first representation of a virtual object in a first user interface area, displaying a second representation of the virtual object in a second user interface area, and displaying a third representation of the virtual object with a representation of a field of view of one or more cameras.

[0032] Figures 10A to 10D is a flow diagram of a process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, according to some embodiments.

[0033] Figures 11A to 11V An example user interface is shown for displaying virtual objects with different visual attributes depending on whether object placement criteria are met, according to some embodiments.

[0034] Figures 12A to 12D, Figure 12E-1 , Figure 12E-2 , Figure 12F-1 , Figure 12F-2 , Figure 12G-1 , Figure 12G-2 , Figure 12H-1 , Figure 12H-2 , Figure 12I-1 , Figure 12I-2 , Figure 12J , Figure 12K-1 , Figure 12K-2 , Figure 12L-1 and Figure 12L-2 illustrate example user interfaces according to some embodiments for displaying calibration user interface objects that are dynamically animated according to movement of one or more cameras of a device.

[0035] Figures 13A to 13M illustrate an example user interface that constrains rotation of a virtual object about an axis according to some embodiments.

[0036] Figures 14A to 14Z illustrate an example user interface that increases a second threshold movement amount value required for a second object manipulation behavior according to determining that a first object manipulation behavior meets a first threshold movement amount value according to some embodiments.

[0037] Figures 14AA to 14AD illustrate a flowchart according to some embodiments that shows operations for increasing a second threshold movement amount value required for a second object manipulation behavior according to determining that a first object manipulation behavior meets a first threshold movement amount value.

[0038] Figures 15A to 15AI illustrate an example user interface that generates an audio alert according to determining that movement of a device moves a virtual object out of a field of view of one or more device cameras being displayed according to some embodiments.

[0039] Figures 16A to 16G is a flowchart of a process according to some embodiments for displaying a virtual object with different visual attributes according to whether an object placement criterion is met.

[0040] Figures 17A to 17D is a flowchart of a process according to some embodiments for displaying calibration user interface objects that are dynamically animated according to movement of one or more cameras of a device.

[0041] Figures 18A to 18I is a flowchart of a process according to some embodiments for constraining rotation of a virtual object about an axis.

[0042] Figures 19A to 19HFIG. 0 is a flow diagram of a process according to some embodiments for increasing a second threshold amount of movement required for a second object manipulation behavior based on determining that a first object manipulation behavior meets a first threshold amount of movement value.

[0043] Figures 20A to 20F FIG. 4 is a flow diagram of a process according to some embodiments for generating an audio alert based on determining that movement of a device has moved a virtual object out of a displayed field of view of one or more device cameras. DETAILED DESCRIPTION

[0044] A virtual object is a graphical representation of a three-dimensional object in a virtual environment. Conventional methods for interacting with a virtual object to transition the virtual object from a scenario displayed in an application user interface (e.g., a two-dimensional application user interface that does not display an augmented reality environment) to a scenario displayed in an augmented reality environment (e.g., an environment that enhances a view of the physical world with supplementary information that provides additional information not available in the physical world to a user) typically require multiple independent inputs (e.g., a series of gestures and button presses, etc.) to achieve a desired result (e.g., adjusting the size, position, and / or orientation of the virtual object to achieve a realistic or desired appearance in the augmented reality environment). Additionally, conventional input methods typically involve a delay between receiving a request to display the augmented reality environment and displaying the augmented reality environment, which is caused by the time required to activate one or more device cameras to capture a view of the physical world and / or the time required to analyze and characterize the view of the physical world related to virtual objects that can be placed in the augmented reality environment (e.g., detecting planes and / or surfaces in the captured view of the physical world). Embodiments herein provide an intuitive way for a user to display and / or interact with a virtual object in various scenarios (e.g., by allowing a user to provide an input to switch from displaying the virtual object in a scenario of an application user interface to displaying the virtual object in an augmented reality environment; by allowing a user to change display attributes of the virtual object before the virtual object is displayed in the augmented reality environment (e.g., in a three-dimensional staging environment); by providing an indication that allows a user to easily identify a system-level virtual object from multiple applications; by changing visual attributes of an object when determining placement information of the object; by providing an animated calibration user interface object that indicates the device movement required for calibration; by constraining rotation of the displayed virtual object about an axis; by increasing a threshold amount of movement for a second object manipulation behavior when a threshold amount of movement value for a first object manipulation behavior is met; and by providing an audio alert indicating that the virtual object has moved out of the displayed field of view).

[0045] The systems, methods, and GUIs described herein improve user interface interactions with virtual / augmented reality environments in a variety of ways. For example, they make it easier to: display virtual objects in an augmented reality environment and, in response to different inputs, adjust the appearance of virtual objects displayed in the augmented reality environment.

[0046] Below, Figures 1A to 1C 、 Figure 2 and Figure 3 provide a description of an example device. Figures 4A to 4B 、 Figures 5A to 5AT 、 Figures 6A to 6AJ 、 Figures 7A to 7P 、 Figures 11A to 11V 、 Figure 12A through Figure 12L, Figures 13A to 13M 、 Figures 14A to 14Z and Figures 15A to 15AI illustrate example user interfaces for displaying virtual objects in various scenarios. Figures 8A to 8E illustrates a process for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area. Figures 9A to 9D illustrates a process for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object that displays a representation of the field of view of one or more cameras. Figures 10A to 10D illustrates a process for displaying an item having a visual indication that the item corresponds to a virtual three-dimensional object. Figures 16A to 16G illustrates a process for displaying a virtual object having different visual attributes based on whether an object placement criterion is met. Figures 17A to 17D illustrates a process for displaying a calibration user interface object that animates dynamically based on the movement of one or more cameras of a device. Figures 18A to 18I illustrates a process for constraining the rotation of a virtual object about an axis. Figures 14AA to 14AD and Figures 19A to 19H illustrates a process for increasing a second threshold movement value required for a second object manipulation behavior based on determining that a first object manipulation behavior meets a first threshold movement value. Figures 20A to 20F illustrates a process for generating an audio alert based on determining that the movement of a device causes a virtual object to move out of the field of view of one or more displayed device cameras. Figures 5A to 5AT 、 Figures 6A to 6AJ 、 Figures 7A to 7P 、 Figures 11A to 11V 、 Figure 12A through Figure 12L, Figures 13A to 13M 、 Figures 14A to 14Z and Figures 15A to 15AI The user interfaces in Figures 8A to 8E 、 Figures 9A to 9D 、 Figures 10A to 10D 、Figures 14AA to 14AD , Figures 16A to 16G , Figures 17A to 17D , Figures 18A to 18I , Figures 19A to 19H and Figures 20A to 20F the processes in

[0047] Exemplary Device

[0048] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0049] It will also be understood that although in some instances the terms "first", "second", etc. are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0050] The terms used in the description of the various described embodiments herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will also be understood that the term "includes" ("includes", "including", "comprises" and / or "comprising") when used in this specification is specifying the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0051] As used herein, depending on context, the term "if" is optionally construed to mean "when" or "upon" or "in response to determining" or "in response to detecting". Similarly, depending on context, the phrase "if it is determined that..." or "if [stated condition or event] is detected" is optionally construed to mean "when it is determined that..." or "in response to determining that..." or "when [stated condition or event] is detected" or "in response to detecting [stated condition or event]".

[0052] Embodiments of electronic devices, user interfaces for such devices, and related processes of using such devices are described herein. In some embodiments, the device is a portable communication device that also includes other functions such as PDA and / or music player functions, such as a mobile phone. Exemplary embodiments of the portable multifunctional device include, but are not limited to, those from Apple Inc. (Cupertino, California), iPod and devices. Optionally, other portable electronic devices such as a laptop or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad) are used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touchpad).

[0053] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices such as a physical keyboard, a mouse, and / or a joystick.

[0054] The device generally supports a variety of applications, such as one or more of the following: a note-taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0055] The various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or within the respective applications. Thus, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports various applications with a user interface that is intuitive and clear to the user.

[0056] Attention is now turned to an embodiment of a portable device having a touch-sensitive display. Figure 1A FIG. is a block diagram of a portable multifunctional device 100 having a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes called a "touch screen" for convenience and is sometimes abbreviated as a touch-sensitive display. Device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of a contact on device 100 (e.g., a touch-sensitive surface such as the touch-sensitive display system 112 of device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as the touch-sensitive display system 112 of device 100 or the touchpad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0057] It should be understood that device 100 is merely an example of a portable multifunctional device, and device 100 optionally has more or fewer components than those shown, optionally combines two or more components, or optionally has a different configuration or arrangement of these components. Figure 1A The various components shown in are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0058] Memory 102 optionally includes high-speed random access memory and also optionally includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100, such as one or more CPUs 120 and peripheral device interface 118, is optionally controlled by memory controller 122.

[0059] Peripheral device interface 118 can be used to couple input and output peripheral devices of the device to memory 102 and one or more CPUs 120. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0060] In some embodiments, peripheral device interface 118, one or more CPUs 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0061] The RF (Radio Frequency) circuit 108 receives and transmits RF signals, which are also referred to as electromagnetic signals. The RF circuit 108 converts electrical signals into electromagnetic signals / converts electromagnetic signals into electrical signals, and communicates with a communication network and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, a memory, and so on. The RF circuit 108 optionally communicates with the network and other devices via wireless communication, and the network is such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular phone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN)). The wireless communication optionally uses any one of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), Long-Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., IEEE802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g, and / or IEEE802.11n), Voice over Internet Protocol (VoIP), WiMAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including those not yet developed as of the date of submission of this document.

[0062] The audio circuitry 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuitry 110 receives audio data from the peripheral interface 118, converts the audio data into an electrical signal, and transmits the electrical signal to the speaker 111. The speaker 111 converts the electrical signal into sound waves audible to humans. The audio circuitry 110 also receives the electrical signal converted by the microphone 113 based on the sound waves. The audio circuitry 110 converts the electrical signal into audio data and transmits the audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuitry 108 by the peripheral interface 118. In some embodiments, the audio circuitry 110 also includes an earphone jack (e.g., Figure 2 212 in

[0063] ). The earphone jack provides an interface between the audio circuitry 110 and a removable audio input / output peripheral device, such as an output-only headset or an earphone having both an output (e.g., a mono or stereo earphone) and an input (e.g., a microphone). Figure 2 The I / O subsystem 106 couples input / output peripheral devices on the device 100, such as the touch-sensitive display system 112 and other input or control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive electrical signals from / transmit electrical signals to the other input or control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to (or not coupled to) any of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointing device such as a mouse. One or more buttons (e.g., Figure 2 208 in

[0064] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from and / or transmits electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to user interface objects. As used herein, the term "enabling representation" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to the graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0065] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or group of sensors that accepts input from the user based on haptic and / or tactile contact. The touch-sensitive display system 112 and the display controller 156 (along with any associated modules and / or instruction sets in the memory 102) detect contact (and any movement or interruption of the contact) on the touch-sensitive display system 112 and translate the detected contact into an interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0066] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light-emitting polymer display) technology, or LED (light-emitting diode) technology, but in other embodiments uses other display technologies. The touch-sensitive display system 112 and the display controller 156 optionally use any of a variety of touch-sensing technologies now known or later developed, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof, the variety of touch-sensing technologies including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies. In some embodiments, projected mutual capacitance sensing technology is used, such as the technology found in the iPod and found in

[0067] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touchscreen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of the finger on the touchscreen. In some embodiments, the device converts the finger-based rough input into an accurate pointer / cursor position or command for performing the actions desired by the user.

[0068] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of the device, which, unlike the touchscreen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touchscreen.

[0069] The device 100 also includes a power system 162 for powering various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0070] The device 100 optionally also includes one or more optical sensors 164. Figure 1A An optical sensor coupled to the optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In combination with the imaging module 143 (also called the camera module), one or more optical sensors 164 optionally capture still images and / or videos. In some embodiments, the optical sensor is located on the rear of the device 100 opposite the touch-sensitive display system 112 on the front of the device, such that the touchscreen can be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to capture an image of the user (e.g., for selfies, for video conferencing when the user is viewing other video conference participants on the touchscreen, etc.).

[0071] Device 100 optionally further includes one or more contact intensity sensors 165. Figure 1A Shown is a contact intensity sensor coupled to an intensity sensor controller 159 in the I / O subsystem 106. One or more contact intensity sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other intensity sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact intensity sensors 165 receive contact intensity information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact intensity sensor is located on the rear of device 100 opposite the touch-sensitive display system 112 located on the front of device 100.

[0072] Device 100 optionally further includes one or more proximity sensors 166. Figure 1A Shown is a proximity sensor 166 coupled to the peripheral device interface 118. Alternatively, the proximity sensor 166 is coupled to an input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunctional device is placed near the user's ear (e.g., when the user is on a call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0073] Device 100 optionally further includes one or more haptic output generators 167. Figure 1A Shown is a haptic output generator coupled to a haptic feedback controller 161 in the I / O subsystem 106. In some embodiments, one or more haptic output generators 167 include one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other haptic output generating components (e.g., components for converting an electrical signal into a haptic output on the device). The haptic output generator 167 receives haptic feedback generation instructions from the haptic feedback module 133 and generates a haptic output on device 100 that can be sensed by a user of device 100. In some embodiments, at least one haptic output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112), and optionally generates a haptic output by moving the touch-sensitive surface vertically (e.g., into / out of the surface of device 100) or laterally (e.g., backward and forward in the same plane as the surface of device 100). In some embodiments, at least one haptic output generator sensor is located on the rear of device 100 opposite the touch-sensitive display system 112 located on the front of device 100.

[0074] Device 100 optionally further includes one or more accelerometers 168. Figure 1A An accelerometer 168 coupled to the peripheral device interface 118 is shown. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on an analysis of data received from the one or more accelerometers. Device 100 optionally further includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the location and orientation (e.g., portrait or landscape) of device 100.

[0075] In some embodiments, the software components stored in the memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a haptic feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application program (or instruction set) 136. Additionally, in some embodiments, the memory 102 stores a device / global internal state 157, as shown in FIGS. 1A and Figure 3 as shown. The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what application programs, views, or other information occupy the various regions of the touch-sensitive display system 112; a sensor state, including information obtained from the various sensors of the device and other input or control devices 116; and location and / or orientation information about the location and / or attitude of the device.

[0076] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0077] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to be directly coupled to other devices or indirectly coupled via a network (e.g., the Internet, Wireless LAN, etc.). In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used in some iPods and iPod devices of Apple Inc. (Cupertino, California). In some embodiments, the external port is a Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in some iPods and iPod devices of Apple Inc. (Cupertino, California).

[0078] The contact / motion module 130 optionally detects contacts with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or a physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., by a finger or a stylus), such as determining whether a contact has occurred (e.g., detecting a finger press event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift-off event or a contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point optionally includes determining the rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, where the movement of the contact point is represented by a series of contact data. These operations are optionally applied to single-point contacts (e.g., single-finger contacts or stylus contacts) or multi-point simultaneous contacts (e.g., “multi-touch” / multi-finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contacts on the touchpad.

[0079] The contact / motion module 130 optionally detects a user's gesture input. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contact). Thus, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a single-finger tap gesture includes detecting a finger-down event and then detecting a finger-lift (lift-off) event at the same location (or substantially the same location) as the finger-down event (e.g., at an icon location). As another example, detecting a finger-swipe gesture on the touch-sensitive surface includes detecting a finger-down event, then detecting one or more finger-drag events, and then detecting a finger-lift (lift-off) event. Similarly, gestures such as taps, swipes, drags, and others of a stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0080] In some embodiments, detecting a finger tap gesture depends on the length of time between the detected finger-down event and the finger-lift event, but is independent of the finger contact intensity between the finger-down event and the finger-lift event. In some embodiments, a tap gesture is detected based on determining that the length of time between the finger-down event and the finger-lift event is less than a pre-determined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the finger contact intensity during the tap reaches a given intensity threshold (greater than the nominal contact detection intensity threshold), such as a light-press or deep-press intensity threshold. Thus, a finger tap gesture can meet a specific input criterion that does not require the characteristic intensity of the contact to meet a given intensity threshold to meet the specific input criterion. For clarity, finger contact in a tap gesture generally needs to meet the nominal contact detection intensity threshold to detect the finger-down event, below which no contact is detected. A similar analysis applies to detecting tap gestures with a stylus or other contact. In cases where the device can detect a finger or stylus contact hovering above the touch-sensitive surface, the nominal contact detection intensity threshold optionally does not correspond to a physical contact between the finger or stylus and the touch-sensitive surface.

[0081] The same concepts apply in a similar manner to other types of gestures. For example, a swipe gesture, a pinch gesture, a spread gesture, and / or a long press gesture may optionally be detected based on satisfaction of criteria that are independent of the intensity of the contacts included in the gesture or do not require the contacts performing the gesture to reach an intensity threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; a pinch gesture is detected based on the movement of two or more contacts towards each other; a spread gesture is detected based on the movement of two or more contacts away from each other; and a long press gesture is detected based on the duration of a contact having less than a threshold amount of movement on a touch-sensitive surface. Thus, the statement that a particular gesture recognition criterion does not require contact intensity to meet a corresponding intensity threshold to satisfy the particular gesture recognition criterion means that the particular gesture recognition criterion can be satisfied when the contacts in the gesture do not reach the corresponding intensity threshold and can also be satisfied in cases where one or more of the contacts in the gesture reach or exceed the corresponding intensity threshold. In some embodiments, a tap gesture is detected based on determining that a finger down event and a finger up event are detected within a predefined period of time, regardless of whether the contact is above or below the corresponding intensity threshold during the predefined period of time, and a swipe gesture is detected based on determining that the contact movement is greater than a predefined amount, even if the contact is above the corresponding intensity threshold at the end of the contact movement. Even in embodiments where the detection of a gesture is affected by the intensity of the contact performing the gesture (e.g., when the intensity of the contact is above the intensity threshold, the device detects a long press faster, or when the intensity of the contact is higher, the device delays the detection of a tap input), as long as the criteria for recognizing the gesture can be satisfied when the contact does not reach a particular intensity threshold, the detection of these gestures will not require the contact to reach a particular intensity threshold (e.g., even if the amount of time required to recognize the gesture changes).

[0082] In some cases, the contact intensity threshold, the duration threshold, and the movement threshold are combined in various different combinations in order to create heuristics to distinguish two or more different gestures directed at the same input element or region, such that multiple different interactions with the same input element can provide a richer set of user interactions and responses. A statement that a particular gesture recognition criterion does not require the intensity of the contact to meet the corresponding intensity threshold to satisfy the particular gesture recognition criterion does not preclude the simultaneous evaluation of other intensity-related gesture recognition criteria to identify other gestures having criteria that are satisfied when the gesture includes a contact having an intensity above the corresponding intensity threshold. For example, in some cases, a first gesture recognition criterion for a first gesture (which does not require the intensity of the contact to meet the corresponding intensity threshold to satisfy the first gesture recognition criterion) competes with a second gesture recognition criterion for a second gesture (which depends on a contact reaching the corresponding intensity threshold). In such a competition, if the second gesture recognition criterion for the second gesture is satisfied first, the gesture is optionally not recognized as satisfying the first gesture recognition criterion for the first gesture. For example, if the contact reaches the corresponding intensity threshold before moving a predefined amount of movement, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contact moves a predefined amount of movement before reaching the corresponding intensity threshold, a swipe gesture is detected instead of a deep press gesture. Even in such cases, the first gesture recognition criterion for the first gesture still does not require the intensity of the contact to meet the corresponding intensity threshold to satisfy the first gesture recognition criterion, because if the contact remains below the corresponding intensity threshold until the end of the gesture (e.g., a swipe gesture with a contact having an intensity that does not increase above the corresponding intensity threshold), the gesture will be recognized as a swipe gesture by the first gesture recognition criterion. Thus, a particular gesture recognition criterion that does not require the intensity of the contact to meet the corresponding intensity threshold to satisfy the particular gesture recognition criterion will (A) in some cases, ignore the contact intensity relative to the intensity threshold (e.g., for a tap gesture) and / or (B) in some cases, not satisfy the particular gesture recognition criterion (e.g., for a long press gesture), in the sense that if a competing set of intensity-related gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-related gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input, still depends on the contact intensity relative to the intensity threshold (e.g., for a long press gesture competing for recognition with a deep press gesture).

[0083] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch-sensitive display system 112 or other displays, including components for changing the visual impact of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual attributes). As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0084] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives from an application or the like one or more codes specifying the graphics to be displayed, and also receives coordinate data and other graphic attribute data as necessary, and then generates screen image data for output to the display controller 156.

[0085] The haptic feedback module 133 includes various software components for generating instructions (e.g., instructions used by the haptic feedback controller 161) to generate haptic output at one or more locations on the device 100 using the haptic output generator 167 in response to user interaction with the device 100.

[0086] The text input module 134, which is optionally a component of the graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application that requires text input).

[0087] The GPS module 135 determines the location of the device and provides such information for use in various applications (e.g., providing to the phone 138 for location-based dialing; providing to the camera 143 as picture / video metadata; and providing to applications that provide location-based services such as weather widgets, local yellow pages widgets, and map / navigation widgets).

[0088] The application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof:

[0089] · The contacts module 137 (sometimes referred to as an address book or contact list);

[0090] · The phone module 138;

[0091] · The video conferencing module 139;

[0092] · The email client module 140;

[0093] · The instant messaging (IM) module 141;

[0094] · Fitness support module 142;

[0095] · Camera module 143 for still images and / or video images;

[0096] · Image management module 144;

[0097] · Browser module 147;

[0098] · Calendar module 148;

[0099] · Desktop widget module 149, optionally including one or more of the following: weather desktop widget 149-1, stock market desktop widget 149-2, calculator desktop widget 149-3, alarm clock desktop widget 149-4, dictionary desktop widget 149-5, and other desktop widgets obtained by the user, as well as user-created desktop widget 149-6;

[0100] · Desktop widget creator module 150 for forming user-created desktop widget 149-6;

[0101] · Search module 151;

[0102] · Video and music player module 152, optionally composed of a video player module and a music player module;

[0103] · Notepad module 153;

[0104] · Map module 154; and / or

[0105] · Online video module 155.

[0106] Examples of other applications 136 that are optionally stored in the memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, JAVA-supported applications, encryption, digital rights management, speech recognition, and speech reproduction.

[0107] In conjunction with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., the application internal state 192 of the contacts module 137 stored in the memory 102 or the memory 370), including: adding a name to the address book; deleting a name from the address book; associating a phone number, an email address, a physical address, or other information with the name; associating an image with the name; categorizing and classifying the name; providing a phone number and / or an email address to initiate and / or facilitate communication via the phone 138, video conferencing 139, email 140, or instant messaging 141; and so on.

[0108] In conjunction with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the phone module 138 includes executable instructions for: entering a character sequence corresponding to a phone number, accessing one or more phone numbers in the address book 137, modifying an entered phone number, dialing the corresponding phone number, conducting a session, and disconnecting or hanging up when the session is completed. As described above, the wireless communication optionally uses any one of a variety of communication standards, protocols, and technologies.

[0109] In conjunction with the RF circuit 108, the audio circuit 110, the speaker 111, the microphone 113, the touch-sensitive display system 112, the display controller 156, one or more optical sensors 164, the optical sensor controller 158, the contact module 130, the graphics module 132, the text input module 134, the contact list 137, and the phone module 138, the video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between the user and one or more other participants according to user instructions.

[0110] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with the image management module 144, the email client module 140 makes it very easy to create and send emails with static images or video images captured by the camera module 143.

[0111] In combination with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying a previously input character, sending the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messaging), receiving instant messages, and viewing the received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant message" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0112] In combination with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the video and music player module 152, the fitness support module 142 includes executable instructions for creating a fitness (e.g., having time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for the fitness; and displaying, storing, and transmitting fitness data.

[0113] In combination with the touch-sensitive display system 112, the display controller 156, one or more optical sensors 164, the optical sensor controller 158, the contact module 130, the graphics module 132, and the image management module 144, the camera module 143 includes executable instructions for performing the following operations: capturing a still image or video (including a video stream) and storing them in the memory 102, modifying the characteristics of the still image or video, and / or deleting the still image or video from the memory 102.

[0114] In combination with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, and the camera module 143, the image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, tagging, deleting, presenting (e.g., in a digital slide show or album), and storing still images and / or video images.

[0115] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet according to user instructions (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as linking to attachments and other files of web pages).

[0116] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140, and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying, and storing calendars and data associated with the calendars (e.g., calendar entries, to-do items, etc.) according to user instructions.

[0117] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a micro application optionally downloaded and used by the user (e.g., weather desktop widget 149-1, stock market desktop widget 149-2, calculator desktop widget 149-3, alarm clock desktop widget 149-4, and dictionary desktop widget 149-5), or a micro application created by the user (e.g., user-created desktop widget 149-6). In some embodiments, the desktop widget includes HTML (HyperText Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (eXtensible Markup Language) files and JavaScript files (e.g., Yahoo! desktop widget).

[0118] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating desktop widgets (e.g., transferring a user-specified portion of a web page into a desktop widget).

[0119] In combination with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sounds, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) according to user instructions.

[0120] In combination with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, the RF circuit 108, and the browser module 147, the video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats, such as MP3 or AAC files, and executable instructions for displaying, presenting, or otherwise playing back video (e.g., on the touch-sensitive display system 112 or on an external display wirelessly connected via the external port 124). In some embodiments, the device 100 optionally includes the functionality of an MP3 player, such as an iPod (a trademark of Apple Inc.).

[0121] In combination with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the notepad module 153 includes executable instructions for creating and managing notepads, to-do lists, etc. according to user instructions.

[0122] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, and the browser module 147, the map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data of stores and other points of interest at or near a specific location; and other location-based data) according to user instructions.

[0123] In combination with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the audio circuit 110, the speaker 111, the RF circuit 108, the text input module 134, the email client module 140, and the browser module 147, the online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on the touch screen 112 or on an external display wirelessly connected or connected via the external port 124), send an email with a link to a specific online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, the instant messaging module 141 is used instead of the email client module 140 to send a link to a specific online video.

[0124] Each of the above-identified modules and applications corresponds to a set of executable instructions for performing one or more of the above functions and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subgroup of the above modules and data structures. Additionally, memory 102 optionally stores additional modules and data structures not described above.

[0125] In some embodiments, device 100 is a device in which the operation of a predefined set of functions on the device is performed solely via a touchscreen and / or a touchpad. By using the touchscreen and / or the touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0126] The predefined set of functions performed solely via the touchscreen and / or the touchpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates device 100 from any user interface displayed on device 100 to the main menu, home menu, or root menu. In such embodiments, the touchpad is used to implement a "menu button". In some other embodiments, the menu button is a physical push button or other physical input control device rather than a touchpad.

[0127] Figure 1B is a block diagram showing exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A in) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0128] The event classifier 170 receives event information and determines the application 136-1 to which the event information is to be delivered and the application view 191 of the application 136-1. The event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is / are currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0129] In some embodiments, the application internal state 192 includes additional information such as one or more of the following: recovery information to be used when the application 136-1 resumes execution, user interface state information indicating information being displayed by or ready for display by the application 136-1, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0130] The event monitor 171 receives event information from the peripheral device interface 118. The event information includes information about sub-events (e.g., a user touch on the touch-sensitive display system 112 as part of a multi-touch gesture). The peripheral device interface 118 transmits the information it receives from the I / O subsystem 106 or sensors such as a proximity sensor 166, an accelerometer 168, and / or a microphone 113 (via the audio circuit 110). The information received by the peripheral device interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display system 112 or a touch-sensitive surface.

[0131] In some embodiments, the event monitor 171 sends requests to the peripheral device interface 118 at predetermined intervals. In response, the peripheral device interface 118 transmits event information. In other embodiments, the peripheral device interface 118 transmits event information only when there is a significant event (e.g., an input received above a predetermined noise threshold and / or an input received for longer than a predetermined duration).

[0132] In some embodiments, the event classifier 170 further includes a hit view determination module 172 and / or an active event recognizer determination module 173.

[0133] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides a software process for determining where within one or more views a sub-event has occurred. A view consists of the controls and other elements that a user can see on the display.

[0134] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of the corresponding application) in which a touch is detected optionally corresponds to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest-level view in which a touch is detected is optionally referred to as the hit view, and the set of events identified as correct inputs is optionally determined at least in part based on the hit view of the initial touch that starts the touch-based gesture.

[0135] The hit view determination module 172 receives information related to sub-events of a touch-based gesture. When an application has multiple views organized in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest view in the hierarchy that should handle the sub-event. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in the sequence of sub-events that form an event or potential event) occurs. Once the hit view is identified by the hit view determination module, the hit view generally receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0136] The active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognizer determination module 173 determines that all views including the physical location of the sub-event are actively participating views, and thus determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with a particular view, higher views in the hierarchy will still remain as actively participating views.

[0137] The event dispatcher module 174 distributes event information to event recognizers (e.g., event recognizer 180). In embodiments that include the active event recognizer determination module 173, the event dispatcher module 174 delivers the event information to the event recognizer determined by the active event recognizer determination module 173. In some embodiments, the event dispatcher module 174 stores the event information in an event queue, which is retrieved by the corresponding event receiver module 182.

[0138] In some embodiments, the operating system 126 includes an event classifier 170. Alternatively, the application 136-1 includes an event classifier 170. In another embodiment, the event classifier 170 is a stand-alone module or is part of another module (such as the contact / motion module 130) stored in the memory 102.

[0139] In some embodiments, the application 136-1 includes a plurality of event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the application's user interface. Each application view 191 of the application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes a plurality of event recognizers 180. In other embodiments, one or more of the event recognizers 180 are part of a stand-alone module, such as a user interface toolkit (not shown) or a higher-level object from which the application 136-1 inherits methods and other properties. In some embodiments, the corresponding event handlers 190 include one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from the event classifier 170. The event handlers 190 optionally utilize or call the data updater 176, the object updater 177, or the GUI updater 178 to update the application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0140] The corresponding event recognizer 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies an event from the event information. The event recognizer 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event recognizer 180 also includes at least a subset of metadata 183 and event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0141] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as a touch or a touch movement. Depending on the sub-event, the event information also includes additional information such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device pose).

[0142] The event comparator 184 compares the event information with predefined event or sub-event definitions and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 contains definitions of events (e.g., predefined sequences of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, the sub-events in event 187 include, for example, touch start, touch end, touch movement, touch cancellation, and multi-touch. In one example, the definition of event 1 (187-1) is a double-tap on a displayed object. For example, a double-tap includes a first touch (touch start) of a predetermined duration on the displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on the displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, a drag includes a touch (or contact) of a predetermined duration on the displayed object, movement of the touch on the touch-sensitive display system 112, and lift-off of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0143] In some embodiments, the event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, the event comparator 184 performs a hit test to determine which user interface object is associated with the sub-event. For example, in an application view that displays three user interface objects on the touch-sensitive display system 112, when a touch is detected on the touch-sensitive display system 112, the event comparator 184 performs a hit test to determine which one of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, the event comparator 184 selects the event handler associated with the sub-event and the object that triggered the hit test.

[0144] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of event information until it has been determined whether the sub - event sequence does or does not correspond to the event type of the event recognizer.

[0145] When the corresponding event recognizer 180 determines that the sub - event sequence does not match any of the events in the event definition 186, the corresponding event recognizer 180 enters an event - impossible, event - failed, or event - ended state, after which subsequent sub - events of the touch - based gesture are ignored. In such a case, any other event recognizers (if any) for which the hit view remains active continue to track and process the sub - events of the ongoing touch - based gesture.

[0146] In some embodiments, the corresponding event recognizer 180 includes metadata 183 having configurable attributes, flags, and / or lists that indicate how the event delivery system should perform sub - event delivery to the event recognizers actively involved. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, the metadata 183 includes configurable attributes, flags, and / or lists that indicate whether sub - events are delivered to different levels in a view or a programmatic hierarchy.

[0147] In some embodiments, when one or more specific sub - events of an event are recognized, the corresponding event recognizer 180 activates the event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers the event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferring the sending of) sub - events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the recognized event, and the event handler 190 associated with that flag receives the flag and executes a predefined process.

[0148] In some embodiments, the event delivery instruction 188 includes a sub - event delivery instruction that delivers event information about the sub - event without activating the event handler. Instead, the sub - event delivery instruction delivers the event information to the event handler associated with the sub - event sequence or to the actively involved view. The event handler associated with the sub - event sequence or with the actively involved view receives the event information and executes a predetermined process.

[0149] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates telephone numbers used in contact module 137, or stores video files used in video or music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the positions of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on the touch-sensitive display.

[0150] In some embodiments, event handler 190 includes data updater 176, object updater 177, and GUI updater 178 or has access to them. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0151] It should be understood that the above discussion of event handling for user touches on the touch-sensitive display also applies to other forms of user input for operating multifunctional device 100 using an input device, and not all user input is initiated on the touchscreen. For example, mouse movement and mouse button presses optionally in cooperation with single or multiple keyboard presses or holds; contact movement on a touchpad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; voice commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events that define the events to be recognized.

[0152] Figure 1C is a block diagram showing a haptic output module according to some embodiments. In some embodiments, I / O subsystem 106 (e.g., haptic feedback controller 161 ( Figure 1A )) and / or other input controllers 160 ( Figure 1A )) includes Figure 1C at least some of the exemplary components shown. In some embodiments, peripheral device interface 118 includes Figure 1C at least some of the exemplary components shown.

[0153] In some embodiments, the haptic output module includes a haptic feedback module 133. In some embodiments, the haptic feedback module 133 aggregates and combines haptic outputs of user interface feedback from software applications on the electronic device (e.g., feedback in response to user input corresponding to the displayed user interface and cues and other notifications indicating the execution of an operation or the occurrence of an event in the user interface of the electronic device). The haptic feedback module 133 includes one or more of a waveform module 123 (for providing waveforms for generating haptic outputs), a mixer 125 (for mixing waveforms, such as waveforms in different channels), a compressor 127 (for reducing or compressing the dynamic range of the waveforms), a low-pass filter 129 (for filtering high-frequency signal components from the waveforms), and a thermal controller 131 (for adjusting the waveforms according to thermal conditions). In some embodiments, the haptic feedback module 133 is included in a haptic feedback controller 161( Figure 1A ). In some embodiments, individual units of the haptic feedback module 133 (or individual implementations of the haptic feedback module 133) are also included in an audio controller (e.g., audio circuitry 110, Figure 1A ) and are used to generate audio signals. In some embodiments, a single haptic feedback module 133 is used to generate both audio signals and waveforms for generating haptic outputs.

[0154] In some embodiments, the haptic feedback module 133 further includes a trigger module 121 (e.g., a software application, an operating system, or other software module that determines to generate a haptic output and initiates a process for generating the corresponding haptic output). In some embodiments, the trigger module 121 generates a trigger signal for initiating the generation of a waveform (e.g., by the waveform module 123). For example, the trigger module 121 generates the trigger signal based on preset timing criteria. In some embodiments, the trigger module 121 receives a trigger signal from outside the haptic feedback module 133 (e.g., in some embodiments, the haptic feedback module 133 receives a trigger signal from a hardware input processing module 146 located outside the haptic feedback module 133) and relays the trigger signal to other components within the haptic feedback module 133 (e.g., the waveform module 123) or a software application that triggers an operation (as the trigger module 121) based on the activation of a user interface element (e.g., an application icon or an affordance within an application) or a hardware input device (e.g., a home button or an intensity-sensitive input surface, such as an intensity-sensitive touch screen). In some embodiments, the trigger module 121 also receives (e.g., from the haptic feedback module 133, Figure 1A and Figure 3 ) haptic feedback generation instructions. In some embodiments, the trigger module 121 responds to the haptic feedback module 133 (or the trigger module 121 within the haptic feedback module 133) (e.g., from the haptic feedback module 133,Figure 1A and Figure 3 ) receives a haptic feedback instruction and generates a trigger signal.

[0155] The waveform module 123 receives the trigger signal as an input (e.g., from the trigger module 121) and provides a waveform for generating one or more haptic outputs in response to receiving the trigger signal (e.g., a waveform selected from a predefined set of waveforms assigned for use by the waveform module 123, such as the waveforms described in more detail below with reference to Figures 4F-4G the waveforms described in more detail).

[0156] The mixer 125 receives the waveforms as inputs (e.g., from the waveform module 123) and mixes the waveforms together. For example, when the mixer 125 receives two or more waveforms (e.g., a first waveform in a first channel and a second waveform in a second channel that at least partially overlaps the first waveform), the mixer 125 outputs a combined waveform corresponding to the sum of the two or more waveforms. In some embodiments, the mixer 125 also modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the remaining waveforms of the two or more waveforms (e.g., by increasing the scale of the particular waveform and / or decreasing the scale of the other waveforms of the two or more waveforms). In some cases, the mixer 125 selects one or more waveforms to remove from the combined waveform (e.g., when waveforms from more than three sources have been requested to be output simultaneously by the haptic output generator 167, the waveform from the oldest source is discarded).

[0157] The mixer 127 receives the waveforms (e.g., the combined waveform from the mixer 125) as inputs and modifies the waveforms. In some embodiments, the compressor 127 reduces the waveforms (e.g., according to the physical specifications of the haptic output generator 167 ( Figure 1A ) or 357 ( Figure 3 )) such that the haptic outputs corresponding to the waveforms are reduced. In some embodiments, the compressor 127 limits the waveforms, such as by imposing a predefined maximum amplitude on the waveforms. For example, the compressor 127 reduces the amplitude of the waveform portion that exceeds a predefined amplitude threshold while maintaining the amplitude of the waveform portion that does not exceed the predefined amplitude threshold. In some embodiments, the compressor 127 reduces the dynamic range of the waveforms. In some embodiments, the compressor 127 dynamically reduces the dynamic range of the waveforms such that the combined waveform remains within the performance specifications of the haptic output generator 167 (e.g., force and / or movable mass displacement limits).

[0158] The low-pass filter 129 receives a waveform (e.g., the compressed waveform from the compressor 127) as an input and filters the waveform (e.g., smooths it) (e.g., removes or reduces high-frequency signal components in the waveform). For example, in some cases, the compressor 127 includes in the compressed waveform irrelevant signals (e.g., high-frequency signal components) that impede haptic output generation and / or exceed the performance specifications of the haptic output generator 167 when generating a haptic output based on the compressed waveform. The low-pass filter 129 reduces or removes such irrelevant signals in the waveform.

[0159] The thermal controller 131 receives a waveform (e.g., the filtered waveform from the low-pass filter 129) as an input and adjusts the waveform according to the thermal conditions of the device 100 (e.g., based on the internal temperature detected within the device 100, such as the temperature of the haptic feedback controller 161, and / or the external temperature detected by the device 100). For example, in some cases, the output of the haptic feedback controller 161 varies according to temperature (e.g., in response to receiving the same waveform, the haptic feedback controller 161 generates a first haptic output when the haptic feedback controller 161 is at a first temperature and a second haptic output when the haptic feedback controller 161 is at a second temperature different from the first temperature). For example, the magnitude (or amplitude) of the haptic output may vary according to temperature. To reduce the effect of temperature variations, the waveform is modified (e.g., the amplitude of the waveform is increased or decreased based on temperature).

[0160] In some embodiments, the haptic feedback module 133 (e.g., the trigger module 121) is coupled to the hardware input processing module 146. In some embodiments, Figure 1A the other input controllers 160 in include the hardware input processing module 146. In some embodiments, the hardware input processing module 146 receives inputs from a hardware input device 145 (e.g., Figure 1A one of the other input or control devices 116 in, such as a home button or an intensity-sensitive input surface, such as an intensity-sensitive touch screen). In some embodiments, the hardware input device 145 is any of the input devices described herein, such as the touch-sensitive display system 112 ( Figure 1A ), the keyboard / mouse 350 ( Figure 3 ), the touchpad 355 ( Figure 3 ), one of the other input or control devices 116 ( Figure 1A ) or an intensity-sensitive home button. In some embodiments, the hardware input device 145 consists of an intensity-sensitive home button rather than the touch-sensitive display system 112 ( Figure 1A ), the keyboard / mouse 350 ( Figure 3 ) or the touchpad 355 ( Figure 3) Configuration. In some embodiments, in response to an input from a hardware input device 145 (e.g., a force-sensitive home button or a touch screen), the hardware input processing module 146 provides one or more trigger signals to the haptic feedback module 133 to indicate that a user input meeting a predefined input criterion has been detected, such as an input corresponding to a "click" of the main button (e.g., "press click" or "release click"). In some embodiments, the haptic feedback module 133 provides a waveform corresponding to a "click" of the main button in response to an input corresponding to a "click" of the main button, thereby simulating the haptic feedback of pressing a physical main button.

[0161] In some embodiments, the haptic output module includes a haptic feedback controller 161 (e.g., Figure 1A the haptic feedback controller 161 in

[0162] ), which controls the generation of haptic output. In some embodiments, the haptic feedback controller 161 is coupled to a plurality of haptic output generators, and selects one or more of the plurality of haptic output generators and sends waveforms to the selected one or more haptic output generators for generating haptic output. In some embodiments, the haptic feedback controller 161 coordinates haptic output requests corresponding to activating the hardware input device 145 and haptic output requests corresponding to software events (e.g., haptic output requests from the haptic feedback module 133), and modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the remaining waveforms of the two or more waveforms (e.g., by increasing the scale of the particular waveform and / or decreasing the scale of the remaining waveforms of these waveforms to prioritize haptic output corresponding to activating the hardware input device 145 over haptic output corresponding to software events). Figure 1C As shown in Figure 1A ), the output of the haptic feedback controller 161 is coupled to the audio circuit of the device 100 (e.g., the audio circuit 110,

[0163] In some embodiments, the haptic output module includes an amplifier 163. In some embodiments, the amplifier 163 receives a waveform (e.g., from the haptic feedback controller 161) and amplifies the waveform and then sends the amplified waveform to the haptic output generator 167 (e.g., the haptic output generator 167( Figure 1A ) or 357( Figure 3 ) whichever). For example, the amplifier 163 amplifies the received waveform to a signal level that conforms to the physical specifications of the haptic output generator 167 (e.g., amplifies to the voltage and / or current required for the haptic output generator 167 to generate a haptic output such that the signal sent to the haptic output generator 167 generates a haptic output corresponding to the waveform received from the haptic feedback controller 161) and sends the amplified waveform to the haptic output generator 167. In response, the haptic output generator 167 generates a haptic output (e.g., by shifting a movable mass back and forth in one or more dimensions relative to the neutral position of the movable mass).

[0164] In some embodiments, the haptic output module includes a sensor 169, which is coupled to the haptic output generator 167. The sensor 169 detects the state or change in state (e.g., mechanical position, physical displacement, and / or movement) of the haptic output generator 167 or one or more components of the haptic output generator 167 (e.g., one or more moving components for generating a haptic output, such as a membrane). In some embodiments, the sensor 169 is a magnetic field sensor (e.g., a Hall effect sensor) or other displacement and / or motion sensor. In some embodiments, the sensor 169 provides information (e.g., the position, displacement, and / or movement of one or more components in the haptic output generator 167) to the haptic feedback controller 161, and, based on the information provided by the sensor 169 about the state of the haptic output generator 167, the haptic feedback controller 161 adjusts the waveform output from the haptic feedback controller 161 (e.g., the waveform optionally sent to the haptic output generator 167 via the amplifier 163).

[0165] Figure 2 Shows a touch screen according to some embodiments (e.g., Figure 1APortable multifunctional device 100 of the touch-sensitive display system 112). The touch screen optionally displays one or more graphics within the user interface (UI) 200. In these and other embodiments described hereinafter, the user can select one or more of these graphics by making gestures on the graphics, for example, by using one or more fingers 202 (not drawn to scale in the figure) or one or more styli 203 (not drawn to scale in the figure). In some embodiments, when the user interrupts contact with one or more graphics, selection of one or more graphics will occur. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up, and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, from left to right, up, and / or down). In some specific implementations or in some cases, inadvertently contacting a graphic does not select the graphic. For example, when the gesture corresponding to selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0166] Device 100 optionally further includes one or more physical buttons, such as a "home" button, or a menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 among a set of applications optionally executed on the device 100. As an alternative, in some embodiments, the menu button is implemented as a soft key in the GUI displayed on the touch screen display.

[0167] In some embodiments, device 100 includes a touch screen display, a menu button 204 (sometimes referred to as the home button 204), a push button 206 for powering on / off the device and for locking the device, volume adjustment buttons 208, a user identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to power on / off the device by pressing the button and holding the button in the pressed state for a predefined time interval; to lock the device by pressing the button and releasing the button before the predefined time interval has passed; and / or to unlock the device or initiate an unlocking process. In some embodiments, device 100 also accepts voice input for activating or deactivating certain functions through the microphone 113. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112 and / or one or more tactile output generators 167 for generating tactile output for the user of the device 100.

[0168] Figure 3FIG. 0 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface, in accordance with some embodiments. Device 300 need not be portable. In some embodiments, device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home or industrial controller). Device 300 generally includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuitry (sometimes termed a chipset) that interconnects system components and controls the communication between them. Device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350, a touchpad 355, a haptic output generator 357 for generating haptic output on device 300 (e.g., similar to one or more of the haptic output generators 167 described above with reference to Figure 1A ), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch-sensitive sensors, and / or contact intensity sensors similar to one or more of the contact intensity sensors 165 described above with reference to Figure 1A ). Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and optionally includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 370 optionally includes one or more storage devices located remotely from one or more CPUs 310. In some embodiments, memory 370 stores programs, modules, and data structures similar to those stored in memory 102 of portable multifunctional device 100 ( Figure 1A ), or a subgroup thereof. Additionally, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunctional device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while memory 102 of portable multifunctional device 100 ( Figure ) optionally does not store these modules.

[0169] ​Each of the components identified above is optionally stored in one or more of the memory devices mentioned previously. Each of the modules identified above corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., instruction sets) identified above need not be implemented as separate software programs, procedures, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores a subgroup of the modules and data structures described above. Additionally, memory 370 optionally stores additional modules and data structures not described above.

[0170] Attention is now turned to embodiments of a user interface (“UI”) optionally implemented on the portable multifunctional device 100.

[0171] ​ An example user interface 400 of an application menu on the portable multifunctional device 100 is shown according to some embodiments. A similar user interface is optionally implemented on the device 300. In some embodiments, the user interface 400 includes the following elements or subsets or supersets thereof:

[0172] · One or more signal strength indicators for one or more wireless communications (such as cellular signals and Wi-Fi signals);

[0173] · Time;

[0174] · Bluetooth indicator;

[0175] · Battery status indicator;

[0176] · A tray 408 with icons for common applications, such as:

[0177] ○ An icon 416 marked “Phone” for the phone module 138, which optionally includes an indicator 414 of the number of missed calls or voicemails;

[0178] ○ An icon 418 marked “Mail” for the email client module 140, which optionally includes an indicator 410 of the number of unread emails;

[0179] ○ An icon 420 marked “Browser” for the browser module 147; and

[0180] ○ An icon 422 marked “Music” for the video and music player module 152; and

[0181] · Icons for other applications, such as:

[0182] ○ An icon 424 marked “Messages” for the IM module 141;

[0183] ○ The icon 426 marked as "Calendar" of the calendar module 148;

[0184] ○ The icon 428 marked as "Photos" of the image management module 144;

[0185] ○ The icon 430 marked as "Camera" of the camera module 143;

[0186] ○ The icon 432 marked as "Online Video" of the online video module 155;

[0187] ○ The icon 434 marked as "Stock Market" of the stock market desktop applet 149-2;

[0188] ○ The icon 436 marked as "Map" of the map module 154;

[0189] ○ The icon 438 marked as "Weather" of the weather desktop applet 149-1;

[0190] ○ The icon 440 marked as "Clock" of the alarm clock desktop applet 149-4;

[0191] ○ The icon 442 marked as "Fitness Support" of the fitness support module 142;

[0192] ○ The icon 444 marked as "Notepad" of the notepad module 153; and

[0193] ○ The icon 446 for setting the application or module, which provides access to the settings of the device 100 and its various applications 136.

[0194] It should be noted that ​ The icon labels shown in are merely exemplary. For example, other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0195] ​ An exemplary user interface on a device (e.g., the device 300 in ) having a touch-sensitive surface 451 (e.g., a tablet or touchpad 355 in ) separate from the display 450 is shown. Although many of the following examples will be given with reference to input on a touchscreen display 112 (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as shown in ​ In some embodiments, the touch-sensitive surface (e.g., ​ where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface separate from the display, as shown in ​ In some embodiments, the touch-sensitive surface (e.g.,​ 451) in has a main axis corresponding to the main axis (e.g., ​ 453) in (e.g., ​ 452) in. According to these embodiments, the device detects contact with the touch-sensitive surface 451 at a position corresponding to a corresponding position on the display (e.g., ​ 460 and 462) in (e.g., in ​ 460 corresponds to 468 and 462 corresponds to 470). Thus, when the touch-sensitive surface (e.g., ​ 451) in is separate from the display of the multifunctional device (e.g., ​ 450) in, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movements) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0196] In addition, although the following examples are mainly given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by inputs from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of a contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is above the position of the tap gesture (e.g., instead of detecting a contact, followed by stopping the detection of the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and a finger contact are optionally used simultaneously.

[0197] As used herein, the term "focus selector" refers to an input element used to indicate the current part of the user interface with which the user is interacting. In some specific implementations including a cursor or other position marker, the cursor acts as the "focus selector" such that when an input (e.g., a press input) is detected on the touch-sensitive surface (e.g., ​ the touchpad 355 in or ​ the touch-sensitive surface 451 in) above a specific user interface element (e.g., a button, a window, a slider, or other user interface element), the specific user interface element is adjusted according to the detected input. In a touchscreen display including a touchscreen display enabling direct interaction with user interface elements on the touchscreen display (e.g., ​ the touch-sensitive display system 112 in or ​In some specific implementations (such as the touch screen), the contact detected on the touch screen serves as a "focus selector", such that when an input (such as a press input through contact) is detected at the position of a specific user interface element (such as a button, window, slider, or other user interface element) on the touch screen display, the specific user interface element is adjusted according to the detected input. In some specific implementations, the focus moves from one area of the user interface to another area of the user interface without a corresponding movement of the cursor or a movement of the contact on the touch screen display (such as moving the focus from one button to another button by using the tab key or arrow keys); in these specific implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is generally a user interface element (or a contact on the touch screen display) that is controlled by the user to convey the interaction desired by the user with the user interface (such as by indicating to the device the element of the user interface that the user desires to interact with). For example, when a press input is detected on a touch-sensitive surface (such as a touchpad or touch screen), the position of the focus selector (such as a cursor, contact, or selection box) above the corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display).

[0198] As used in this specification and the claims, the "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact on the touch-sensitive surface (e.g., finger contact or stylus contact), or to a surrogate for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values that includes at least four different values and more typically includes hundreds of different values (e.g., at least 256). The intensity of a contact is optionally determined (or measured) using a variety of methods and a variety of sensors or combinations of sensors. For example, one or more force sensors beneath or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some embodiments, force measurements from multiple force sensors are combined (e.g., weighted average or summation) to determine an estimated contact force. Similarly, a pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or change thereof of the contact area detected on the touch-sensitive surface, the capacitance and / or change thereof of the touch-sensitive surface near the contact, and / or the resistance and / or change thereof of the touch-sensitive surface near the contact are optionally used as surrogates for the force or pressure of a contact on the touch-sensitive surface. In some embodiments, the surrogate measurements of the contact force or pressure are directly used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the surrogate measurements). In some embodiments, the surrogate measurement values of the contact force or pressure are converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). Using the intensity of a contact as an attribute of a user input allows a user to access additional device functions that the user would not otherwise be able to easily access on a smaller device with a limited footprint for displaying affordances and / or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or physical / mechanical controls such as knobs or buttons).

[0199] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an operation has been performed by the user (e.g., to determine whether the user has "clicked" on an icon). In some embodiments, at least a subset of the intensity thresholds is determined according to software parameters (e.g., the intensity thresholds are not determined by the activation thresholds of specific physical actuators and can be adjusted without changing the physical hardware of the device 100). For example, the mouse "click" threshold of a touchpad or a touchscreen display can be set to any one of a wide range of predefined thresholds without changing the touchpad or touchscreen display hardware. Additionally, in some embodiments, the user of the device is provided with software settings for adjusting one or more of the intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by using a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0200] As used in the specification and claims, the term "feature intensity" of a contact refers to a feature of the contact based on one or more intensities of the contact. In some embodiments, the feature intensity is based on a plurality of intensity samples. Optionally, the feature intensity is based on a predefined number or set of intensity samples collected during a predefined period of time (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after detection of contact, before detection of contact lift-off, before or after detection of contact start to move, before detection of contact end, before or after detection of increase in intensity of contact, and / or before or after detection of decrease in intensity of contact). Optionally, the feature intensity of the contact is based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, a value generated by low-pass filtering the contact intensity over a predefined period or starting from a predefined time, etc. In some embodiments, the duration of the contact is used in determining the feature intensity (e.g., when the feature intensity is the time average of the intensity of the contact). In some embodiments, the feature intensity is compared to a set of one or more intensity thresholds to determine whether the user has performed an operation. For example, the set of one or more intensity thresholds may include a first intensity threshold and a second intensity threshold. In this example, a contact with a feature intensity not exceeding the first intensity threshold results in a first operation, a contact with a feature intensity exceeding the first intensity threshold but not exceeding the second intensity threshold results in a second operation, and a contact with a feature intensity exceeding the second intensity threshold results in a third operation. In some embodiments, the comparison between the feature intensity and one or more intensity thresholds is used to determine whether to perform one or more operations (e.g., whether to execute the corresponding option or forego execution of the corresponding operation), rather than for determining whether to perform a first operation or a second operation.

[0201] In some embodiments, a portion of a recognized gesture is used to determine a characteristic intensity. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position to an ending position (e.g., a drag gesture), at which ending position the intensity of the contact increases. In this embodiment, the characteristic intensity of the contact at the ending position may be based on only a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the ending position). In some embodiments, a smoothing algorithm may be applied to the intensity of the swipe gesture before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow spikes or dips in the intensity of the swipe contact for the purpose of determining the characteristic intensity.

[0202] The user interface diagrams described herein optionally include various intensity diagrams that show the current intensity of a contact on a touch-sensitive surface relative to one or more intensity thresholds (e.g., a contact detection intensity threshold IT0, a light press intensity threshold IT L , a deep press intensity threshold IT D (e.g., at least initially higher than IT L ) and / or one or more other intensity thresholds (e.g., an intensity threshold IT L lower than IT H ). This intensity diagram is generally not part of the displayed user interface, but is provided to help explain the diagram. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact is detected with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold IT0, a contact below the nominal contact detection intensity threshold is no longer detected), the device will move a focus selector based on the movement of the contact on the touch-sensitive surface without performing an operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different sets of user interface drawings.

[0203] In some embodiments, the response of the device to the detected input depends on criteria based on the intensity of contact during the input. For example, for some "light press" inputs, the intensity of contact exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the response of the device to the input detected by the device depends on criteria including both the intensity of contact during the input and time-based criteria. For example, for some "deep press" inputs, the intensity of contact exceeding a second intensity threshold greater than the first intensity threshold for light presses during the input triggers a second response provided that a delay time elapses between satisfying the first intensity threshold and satisfying the second intensity threshold. The duration of this delay time is typically less than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, where the delay time increases as the second intensity threshold increases). This delay time helps to avoid accidentally recognizing a deep press input. As another example, for some "deep press" inputs, there is a period of reduced sensitivity after the first intensity threshold is reached. During this period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps to avoid accidental deep press inputs. For other deep press inputs, the response to detecting a deep press input does not depend on time-based criteria.

[0204] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors such as user settings, contact motion, input timing, application running, rate at which the intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Exemplary factors are described in U.S. Patent Application Serial Numbers 14 / 399,606 and 14 / 624,296, which are hereby incorporated by reference in their entireties.

[0205] For example, ​ illustrates a dynamic intensity threshold 480 that changes over time based in part on the intensity of a touch input 476 over time. The dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predefined delay time p1 from when the touch input 476 is initially detected, and a second component 478 that tracks the intensity of the touch input 476 over time. The initial high intensity threshold of the first component 474 reduces accidental triggering of a "deep press" response while still allowing an immediate "deep press" response if the touch input 476 provides sufficient intensity. The second component 478 reduces accidental triggering of a "deep press" response by gradual intensity fluctuations of the touch input. In some embodiments, when the touch input 476 meets the dynamic intensity threshold 480 (e.g., at the point 481 in ​ ), a "deep press" response is triggered.

[0206] ​ shows another dynamic intensity threshold 486 (e.g., intensity threshold I D ). ​ Also shown are two other intensity thresholds: a first intensity threshold I H and a second intensity threshold I L . In ​ , although the touch input 484 satisfies the first intensity threshold I H and the second intensity threshold I L before time p2, a response is provided only after a delay time p2 has elapsed at time 482. Similarly in ​ , the dynamic intensity threshold 486 decays over time, where the decay begins at time 488 after a predefined delay time p1 has elapsed from time 482 (when the response associated with the second intensity threshold I L was triggered). This type of dynamic intensity threshold reduction occurs immediately after or simultaneously with the accidental triggering of a response associated with a lower threshold intensity (such as the first intensity threshold I H or the second intensity threshold I L ) and the triggering of a response associated with the dynamic intensity threshold I D .

[0207] ​ shows yet another dynamic intensity threshold 492 (e.g., intensity threshold I D ). In ​ , after a delay time p2 has elapsed from when the touch input 490 was initially detected, the response associated with the intensity threshold I L is triggered. At the same time, the dynamic intensity threshold 492 decays after a predefined delay time p1 has elapsed from when the touch input 490 was initially detected. Thus, reducing the intensity of the touch input 490 after the response associated with the intensity threshold I L is triggered and then increasing the intensity of the touch input 490 without releasing the touch input 490 can trigger the response associated with the intensity threshold I D (e.g., at time 494), even when the intensity of the touch input 490 is below another intensity threshold (e.g., intensity threshold I L ).

[0208] The characteristic intensity of the contact increases from an intensity below the light press intensity threshold IT L to an intensity between the light press intensity threshold IT L and the deep press intensity threshold IT D is sometimes referred to as a "light press" input. The characteristic intensity of the contact increases from an intensity below the deep press intensity threshold IT D to an intensity above the deep press intensity threshold ITD The intensity of which is sometimes referred to as a "deep press" input. The contact feature intensity increases from an intensity below the contact detection intensity threshold IT0 to an intensity between the contact detection intensity threshold IT0 and the light press intensity threshold IT L is sometimes referred to as detecting contact on the touch surface. The contact feature intensity decreasing from an intensity above the contact detection intensity threshold IT0 to an intensity below the contact detection intensity threshold IT0 is sometimes referred to as detecting the contact being lifted from the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some diagrams, a shaded circle or ellipse is used to represent the intensity of contact on the touch-sensitive surface. In some diagrams, a non-shaded circle or ellipse is used to represent the corresponding contact on the touch-sensitive surface without specifying the intensity of the corresponding contact.

[0209] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or contacts), where the corresponding press input is detected at least in part based on the intensity of the detected contact (or contacts) increasing above a press input intensity threshold. In some embodiments, a corresponding operation is performed in response to detecting the intensity of the corresponding contact increasing above the press input intensity threshold (e.g., performing the corresponding operation on the "down stroke" of the corresponding press input). In some embodiments, the press input includes the intensity of the corresponding contact increasing above the press input intensity threshold and the intensity of the contact subsequently decreasing below the press input intensity threshold, and a corresponding operation is performed in response to detecting the intensity of the corresponding contact subsequently decreasing below the press input threshold (e.g., performing the corresponding operation on the "up stroke" of the corresponding press input).

[0210] In some embodiments, the device employs strength hysteresis to avoid unexpected inputs sometimes referred to as "jitter", where the device defines or selects a hysteresis strength threshold having a predefined relationship to a press input strength threshold (e.g., the hysteresis strength threshold is X strength units lower than the press input strength threshold, or the hysteresis strength threshold is 75%, 90%, or some reasonable percentage of the press input strength threshold). Thus, in some embodiments, a press input includes an increase in the strength of a corresponding contact to above the press input strength threshold and a subsequent decrease in the strength of that contact to below the hysteresis strength threshold corresponding to the press input strength threshold, and a corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the hysteresis strength threshold (e.g., performing the corresponding operation on the "upstroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in contact strength from a strength equal to or below the hysteresis strength threshold to a strength equal to or above the press input strength threshold and optionally the contact strength subsequently decreases to a strength equal to or below the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (e.g., depending on the context, an increase in contact strength or a decrease in contact strength).

[0211] For ease of explanation, optionally, a description of an operation performed in response to a press input associated with a press input strength threshold or in response to a gesture including a press input is triggered in response to detecting: an increase in the strength of a contact to above the press input strength threshold, an increase in the strength of a contact from a strength below the hysteresis strength threshold to a strength above the press input strength threshold, a decrease in the strength of a contact to below the press input strength threshold, or a decrease in the strength of a contact to below the hysteresis strength threshold corresponding to the press input strength threshold. Additionally, in examples where an operation is described as being performed in response to detecting a decrease in the strength of a contact to below the press input strength threshold, the operation is optionally performed in response to detecting a decrease in the strength of the contact to below the hysteresis strength threshold corresponding to and less than the press input strength threshold. As described above, in some embodiments, the triggering of these operations also depends on meeting a time-based criterion (e.g., a delay time has elapsed between meeting a first strength threshold and meeting a second strength threshold).

[0212] As used in this specification and the claims, the term "haptic output" refers to a physical displacement of a device relative to a previous position of the device detected by a user using the user's sense of touch, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., the housing), or a displacement of a component relative to the center of mass of the device. For example, in the case of contact between the device or a component of the device and a surface of the user that is sensitive to touch (e.g., a finger, palm, or other part of the user's hand), the haptic output generated by the physical displacement will be interpreted by the user as a sense of touch that corresponds to a perceived change in a physical characteristic of the device or a component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or a touchpad) is optionally interpreted by the user as a "press click" or "release click" of a physical actuation button. In some cases, the user will sense a sense of touch, such as a "press click" or "release click," even when a physical actuation button associated with the touch-sensitive surface that is physically depressed (e.g., displaced) by the user's movement does not move. As another example, movement of the touch-sensitive surface will optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface even when there is no change in the smoothness of the touch-sensitive surface. Although such interpretations of touch by the user will be limited by the user's individual sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a haptic output is described as corresponding to a particular sensory perception of the user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated haptic output corresponds to a physical displacement of the device or a component thereof that will generate the stated sensory perception of a typical (or average) user. Providing haptic feedback to the user using haptic output enhances the operability of the device and makes the user device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating the device / interacting with the device), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0213] In some embodiments, a haptic output pattern specifies characteristics of the haptic output, such as the amplitude of the haptic output, the shape of the motion waveform of the haptic output, the frequency of the haptic output, and / or the duration of the haptic output.

[0214] When a device (e.g., one or more haptic output generators that generate haptic output via a moving movable mass) generates haptic output having different haptic output patterns, the haptic output can produce different tactile sensations in a user holding or touching the device. Although the user's senses are based on the user's perception of the haptic output, most users will be able to discern changes in the waveform, frequency, and amplitude of the haptic output generated by the device. Thus, the waveform, frequency, and amplitude can be adjusted to indicate to the user that different operations have been performed. In this way, haptic output having a haptic output pattern that is designed, selected, and / or arranged to simulate the properties (e.g., size, material, weight, stiffness, smoothness, etc.); behaviors (e.g., oscillation, displacement, acceleration, rotation, stretching, etc.); and / or interactions (e.g., collision, adhesion, repulsion, attraction, friction, etc.) of objects in a given environment (e.g., a user interface including graphical features and objects, a simulated physical environment having virtual boundaries and virtual objects, a real physical environment having physical boundaries and physical objects, and / or any combination of the foregoing) will, in some cases, provide helpful feedback to the user that reduces input errors and increases the efficiency of the user's operation of the device. Additionally, haptic output is optionally generated corresponding to feedback that is unrelated to simulated physical properties such as input thresholds or object selection. Such haptic output will, in some cases, provide helpful feedback to the user that reduces input errors and increases the efficiency of the user's operation of the device.

[0215] In some embodiments, haptic output having a suitable haptic output pattern serves as a cue for an event of interest occurring in a user interface or behind a screen in a device. Examples of events of interest include the activation of an affordance representation (e.g., a real or virtual button, or a toggle switch) provided on the device or in the user interface, the success or failure of a requested operation, reaching or crossing a boundary in the user interface, entering a new state, switching the input focus between objects, activating a new mode, reaching or crossing an input threshold, detecting or recognizing a type of input or gesture, and so on. In some embodiments, haptic output is provided to serve as a warning or cue about an impending event or outcome that will occur unless a change in direction or interruption of the input is detected in a timely manner. Haptic output is also used in other contexts to enrich the user experience, improve the accessibility of the device to users with visual or motor difficulties or other accessibility needs, and / or improve the efficiency and functionality of the user interface and / or the device. Optionally, the haptic output is compared with audio input and / or visual user interface changes, which further enhances the user's experience when interacting with the user interface and / or the device, facilitates better transmission of information about the state of the user interface and / or the device, and which reduces input errors and increases the efficiency of the user's operation of the device.

[0216] ​Provide a set of sample haptic output patterns that can be used, individually or in combination, as is or through one or more transformations (e.g., modulation, amplification, truncation, etc.), for various purposes in various scenarios (such as those described above and those for the user interfaces and methods discussed herein) to generate appropriate haptic feedback. This example of a control panel for haptic output shows how a set of three waveforms and eight frequencies can be used to generate an array of haptic output patterns. In addition to the haptic output patterns shown in these figures, each of these haptic output patterns can optionally be adjusted in amplitude by changing the gain value of the haptic output pattern, as shown, for example, for ​ FullTap 80Hz, FullTap200Hz, MiniTap 80Hz, MiniTap 200Hz, MicroTap 80Hz, and MicroTap 200Hz in ​ , each of which is shown as a variant having gains of 1.0, 0.75, 0.5, and 0.25. As

[0217] ​ shows, changing the gain of the haptic output pattern changes the amplitude of the pattern without changing the frequency of the pattern or the shape of the waveform. In some embodiments, changing the frequency of the haptic output pattern also results in a lower amplitude because some haptic output generators are limited in how much force can be applied to a movable mass, and thus higher frequency movements of the mass are constrained to lower amplitudes to ensure that the acceleration required to generate the waveform does not require a force outside the operating force range of the haptic output generator (e.g., the peak amplitudes of FullTap at 230Hz, 270Hz, and 300Hz are lower than the amplitudes of FullTap at 80Hz, 100Hz, 125Hz, and 200Hz).

[0217] ​ shows a haptic output pattern having a particular waveform. The waveform of a haptic output pattern represents the pattern of physical displacement over time relative to a neutral position (e.g., xzero), through which a movable mass moves to generate a haptic output having that haptic output pattern. For example, ​ each of the first set of haptic output patterns (e.g., the haptic output pattern of "FullTap") shown has a waveform that includes an oscillation having two complete cycles (e.g., an oscillation that starts and ends at the neutral position and crosses the neutral position three times). ​ each of the second set of haptic output patterns (e.g., the haptic output pattern of "MiniTap") shown has a waveform that includes an oscillation having one complete cycle (e.g., an oscillation that starts and ends at the neutral position and crosses the neutral position once). ​Each of the third set of haptic output patterns shown (e.g., the haptic output pattern of “MicroTap”) has a waveform including an oscillation having half a complete cycle (e.g., an oscillation that starts and ends at a neutral position and does not cross the neutral position). The waveform of the haptic output pattern also includes a start buffer and an end buffer representing the gradual acceleration and deceleration of the movable mass at the start and end of the haptic output. ​ The example waveforms shown include xmin and xmax values representing the maximum and minimum degrees of movement of the movable mass. For larger electronic devices with a larger movable mass, the minimum and maximum degrees of movement of the mass can be greater or less. ​ The example shown describes the movement of the mass in one dimension; however, similar principles can also apply to the movement of a movable mass in two or three dimensions.

[0218] As ​ shown, each haptic output pattern also has a corresponding characteristic frequency that affects the “pitch” of the tactile sensation that a user feels from the haptic output having that characteristic frequency. For continuous haptic output, the characteristic frequency represents the number of cycles (e.g., cycles per second) completed by the movable mass of the haptic output generator in a given time period. For discrete haptic output, a discrete output signal is generated (e.g., having 0.5, 1, or 2 cycles), and the characteristic frequency value specifies how fast the movable mass needs to move to generate the haptic output having that characteristic frequency. As ​ shown, for each type of haptic output (e.g., defined by a corresponding waveform, such as FullTap, MiniTap, or MicroTap), a higher frequency value corresponds to a faster movement of the movable mass and thus, generally speaking, corresponds to a shorter haptic output completion time (e.g., the time including the number of cycles required to complete the discrete haptic output plus the start and end buffer times). For example, a FullTap with a characteristic frequency of 80 Hz takes longer to complete than a FullTap with a characteristic frequency of 100 Hz (e.g., at ​Among them, 35.4 ms vs. 28.3 ms). In addition, for a given frequency, a haptic output with more cycles in its waveform takes longer to complete than a haptic output with fewer cycles in its waveform at the same corresponding frequency. For example, FullTap at 150 Hz takes longer to complete than MiniTap at 150 Hz (e.g., 19.4 ms vs. 12.8 ms), and MiniTap at 150 Hz takes longer to complete than MicroTap at 150 Hz (e.g., 12.8 ms vs. 9.4 ms). However, for haptic output patterns with different frequencies, this rule may not apply (e.g., a haptic output with more cycles but a higher frequency may take less time to complete than a haptic output with fewer cycles but a lower frequency, and vice versa). For example, at 300 Hz, FullTap and MiniTap take the same amount of time (e.g., 9.9 ms).

[0219] As ​ shown, the haptic output pattern also has a characteristic amplitude, which affects the amount of energy contained in the haptic signal or the "intensity" of the tactile sensation that the user can feel through the haptic output with that characteristic amplitude. In some embodiments, the characteristic amplitude of the haptic output pattern refers to the absolute or normalized value representing the maximum displacement of the movable mass relative to the neutral position when generating the haptic output. In some embodiments, the characteristic amplitude of the haptic output pattern can be adjusted according to various conditions (e.g., customized based on user interface context and behavior) and / or pre-configured metrics (e.g., input-based metrics, and / or user interface-based metrics), for example, by a gain coefficient (e.g., a value between 0 and 1) determined fixedly or dynamically. In some embodiments, the input-based metric (e.g., intensity change metric or input speed metric) measures the characteristics of the input (e.g., the rate of change of the characteristic intensity of the contact in a press input or the rate of movement of the contact on the touch-sensitive surface) during the input that triggers the generation of the haptic output. In some embodiments, the user interface-based metric (e.g., cross-boundary speed metric) measures the characteristics of the user interface element (e.g., the speed at which the element moves across a hidden or visible boundary in the user interface) during the change of the user interface that triggers the generation of the haptic output. In some embodiments, the characteristic amplitude of the haptic output pattern can be modulated by an "envelope", and the peaks of adjacent cycles can have different amplitudes, where one of the waveforms shown above is further modified by multiplying by an envelope parameter that changes over time (e.g., from 0 to 1) to gradually adjust the amplitude of a part of the haptic output over time when generating the haptic output.

[0220] Although in ​For purposes of illustration, only a specific frequency, amplitude, and waveform are shown in the sample haptic output pattern, but haptic output patterns having other frequencies, amplitudes, and waveforms can be used for similar purposes. For example, a waveform having between 0.5 and 4 cycles can be used. Other frequencies in the range of 60 Hz - 400 Hz can also be used.

[0221] User interface and associated processes

[0222] Attention is now turned to embodiments of a user interface ("UI") and associated processes that can be implemented on an electronic device such as portable multifunctional device 100 or device 300 having a display, a touch-sensitive surface, (optionally) one or more haptic output generators for generating haptic output, and (optionally) one or more sensors for detecting the intensity of contact with the touch-sensitive surface.

[0223] ​ An example user interface according to some embodiments is shown that is used to display a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area. The user interfaces in these figures are used to illustrate the processes described below, including ​ , ​ , ​ , ​ , ​ , ​ , ​ and ​ the processes in. For ease of explanation, some embodiments in the implementation will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a respective finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the respective contact or a point associated with the respective contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, similar operations are optionally performed on a device having a display 450 and a separate touch-sensitive surface 451 in response to detecting a contact on the touch-sensitive surface 451 when the user interface shown in the figures is displayed on the display 450 along with the focus selector.

[0224] ​ Shows a real-world scenario where the user interface described with reference to ​ is used.

[0225] ​ Shows the physical space 5002 in which the table 5004 is located. The device 100 is held by the user in the user's hand 5006.

[0226] ​Shows the instant messaging user interface 5008 displayed on the display 112. The instant messaging user interface 5008 includes: a message bubble 5010 including a received text message 5012, a message bubble 5014 including a sent text message 5016, and a message bubble 5018 including a virtual object received in the message (e.g., virtual chair 5020) and a virtual object indicator 5022, the virtual object indicator indicating that the virtual chair 5020 is an object visible in an augmented reality view (e.g., within a representation of the field of view of one or more cameras of the device 100). The instant messaging user interface 5008 also includes a message input area 5024 configured to display a message input.

[0227] ​ Shows an input that causes a portion of the instant messaging user interface 5008 to be replaced by the field of view of one or more cameras of the device 100. In ​ a contact 5026 with the touch screen 112 of the device 100 is detected. The characteristic intensity of the contact is higher than the contact detection intensity threshold IT0 and lower than the prompt press intensity threshold IT H , as shown by the intensity level meter 5028. In ​ the characteristic intensity of the contact 5026 increases to be higher than the prompt press intensity threshold IT H , which causes the area of the message bubble 5018 to increase, the size of the virtual chair 5020 to increase, and the instant messaging user interface 5008 to start to blur behind the message bubble 5018 (e.g., providing visual feedback to the user of the effect of increasing the characteristic intensity of the contact). In Figure 5E the characteristic intensity of the contact 5026 increases to be higher than the light press intensity threshold IT L , which causes the message bubble 5018 to be replaced by a disk surface 5030, the size of the virtual chair 5020 to further increase, and the instant messaging user interface 5008 to further blur behind the disk surface 5030. In Figure 5F the characteristic intensity of the contact 5026 increases to be higher than the deep press intensity threshold IT D , which causes the haptic output generator 167 of the device 100 to output a haptic output (as shown at 5032), the haptic output indicating that the criterion for replacing a portion of the instant messaging user interface 5008 with the field of view of one or more cameras of the device 100 has been met.

[0228] In some embodiments, before the characteristic intensity of the contact 5026 reaches the deep press intensity threshold IT D (as Figure 5F shown), Figures 5C to 5E the progression shown is reversible. For example, in Figure 5D and / orFigure 5E After the increase shown, decreasing the feature intensity of contact 5026 will cause the interface state corresponding to the decreased intensity level of contact 5026 to be displayed (e.g., based on determining that the decreased feature intensity of the contact is higher than the light press intensity threshold IT L , as shown Figure 5E the interface shown; based on determining that the decreased feature intensity of the contact is higher than the prompt press intensity threshold IT H , as shown Figure 5D the interface shown; and based on determining that the decreased feature intensity of the contact is lower than the prompt press intensity threshold IT H , as shown Figure 5C the interface shown). In some embodiments, after the increase shown in Figure 5D and / or Figure 5E , decreasing the feature intensity of contact 5026 will cause the interface shown in Figure 5C to be redisplayed.

[0229] Figures 5F to 5J shows an animated transition during which a portion of the instant messaging user interface is replaced by the field of view of one or more cameras (hereinafter referred to as "cameras") of device 100. From Figures 5F to 5G , contact 5026 has been lifted off the touch screen 112, and virtual chair 5020 has rotated towards its final position in Figure 5I . In Figure 5G , the field of view 5034 of the camera has begun to fade into view in disc 5030 (as indicated by the dashed line). In Figure 5H , the field of view 5034 of the camera (e.g., showing a view of the physical space 5002 captured by the camera) has completed fading into view in disc 5030. From Figures 5H to 5I , virtual chair 5020 continues to rotate towards its final position in Figure 5I . In Figure 5I , the haptic output generator 167 has output a haptic output indicating that at least one plane (e.g., the floor surface 5038) has been detected in the field of view 5034 of the camera (as shown at 5036). Virtual chair 5020 is placed on the detected plane (e.g., based on the determination of device 100 that the virtual object is configured to be placed in a vertical orientation on the detected horizontal surface, such as the floor surface 5038). When a portion of the instant messaging user interface is converted into a representation of the field of view 5034 of the camera on the display 112, the size of virtual chair 5020 is continuously adjusted on the display 112. For example, the scale of virtual chair 5020 relative to the physical space 5002 as shown in the field of view 5034 of the camera is determined based on the "real world" size of virtual chair 5020 predefined in the field of view 5034 of the camera and / or the size of the detected object (such as table 5004). InFigure 5J In [description], the virtual chair 5020 is shown in its final position, having a predefined orientation relative to the floor surface detected in the field of view 5034 of the camera. In some embodiments, the initial landing position of the virtual chair 5020 is a predefined position relative to the plane detected in the field of view of the camera, such as the center of the unoccupied area of the detected plane. In some embodiments, the initial landing position of the virtual chair 5020 is determined based on the lift-off position of the contact 5026 (e.g., in Figure 5F In [description], the lift-off position of the contact 5026 may be different from the initial downward touch position of the contact 5026, which is caused by the movement of the contact 5026 on the touch screen 112 after the criteria for transitioning to the augmented reality environment are met).

[0230] Figures 5K to 5L Shows the movement of the device 100 that adjusts the field of view 5034 of the camera (e.g., by the user's hand 5006). When the device 100 moves relative to the physical space 5002, the displayed field of view 5034 of the camera changes, and the virtual chair 5020 remains in the same position and orientation relative to the floor surface 5038 in the displayed field of view 5034 of the camera.

[0231] Figures 5M to 5Q Shows an input that causes the virtual chair 5020 to move on the floor surface 5038 in the displayed field of view 5034 of the camera. In Figure 5N In [description], a contact 5040 with the touch screen 112 of the device 100 is detected at the position corresponding to the virtual chair 5020. In Figures 5N to 5O In [description], when the contact 5040 moves along the path indicated by the arrow 5042, the contact 5040 drags the virtual chair 5020. When the virtual chair 5020 is moved by the contact 5040, the size of the virtual chair 5020 changes to maintain the ratio of the virtual chair 5020 relative to the physical space 5002 as shown in the field of view 5034 of the camera. For example, in Figures 5N to 5P In [description], when the virtual chair 5020 moves from the foreground of the field of view 5034 of the camera to a position away from the device 100 and closer to the table 5004 in the field of view 5034 of the camera, the size of the virtual chair 5020 decreases (e.g., so that the ratio of the chair to the table 5004 in the field of view 5034 of the camera is maintained). Additionally, when the virtual chair 5020 is moved by the contact 5040, the plane identified in the field of view 5034 of the camera is highlighted. For example, in Figure 5O In [description], the floor plane 5038 is highlighted. In Figures 5O to 5P In [description], when the contact 5040 moves along the path indicated by the arrow 5044, the contact 5040 continues to drag the virtual chair 5020. In Figure 5QIn [the figure], the contact 5040 has been lifted off the touch screen 112. In some embodiments, such as Figures 5N to 5Q shown, the movement path of the virtual chair 5020 is constrained by the floor surface 5038 in the field of view 5034 of the camera, as if the contact 5040 is dragging the virtual chair 5020 on the floor surface 5038. In some embodiments, such as the contact 5040 described with reference to Figures 5N to 5P is a continuation of the contact 5026 described with reference to Figures 5C to 5F (for example, the contact 5026 is not lifted off, and this contact that causes a part of the instant messaging user interface 5008 to be replaced by the field of view 5034 of the camera also drags the virtual chair 5020 in the field of view 5034 of the camera.

[0232] Figures 5Q to 5U An input is shown that moves the virtual chair 5020 from the floor surface 5038 to a different plane (e.g., the tabletop 5046) detected in the field of view 5034 of the camera. In Figure 5R a contact 5050 with the touch screen 112 of the device 100 is detected at a position corresponding to the virtual chair 5020. In Figures 5R to 5S when the contact 5048 moves along the path indicated by the arrow 5050, the contact 5048 drags the virtual chair 5020. When the virtual chair 5020 is moved by the contact 5048, the size of the virtual chair 5020 changes to maintain the ratio of the virtual chair 5020 relative to the physical space 5002 as shown in the field of view 5034 of the camera. Additionally, when the virtual chair 5020 is moved by the contact 5040, the tabletop plane 5046 is highlighted (e.g., as Figure 5S shown). In Figures 5S to 5T when the contact 5048 moves along the path indicated by the arrow 5052, the contact 5040 continues to drag the virtual chair 5020. In Figure 5U the contact 5048 has been lifted off the touch screen 112, and the virtual chair 5020 is placed on the tabletop plane 5046 in a vertical orientation facing the same direction as before.

[0233] Figures 5U to 5AD An input is shown that drags the virtual chair 5020 to the edge of the touch screen display 112, which causes the field of view 5034 of the camera to stop being displayed. In Figure 5V a contact 5054 with the touch screen 112 of the device 100 is detected at a position corresponding to the virtual chair 5020. In Figures 5V to 5W when the contact 5054 moves along the path indicated by the arrow 5056, the contact 5054 drags the virtual chair 5020. In Figures 5W to 5X when the contact 5054 moves along the path indicated by the arrow 5058, the contact 5054 continues to drag the virtual chair 5020 to Figure 5XThe position shown.

[0234] As Figures 5Y to 5AD shown, an input via Figures 5U to 5X the contact 5054 shown causes a transition from displaying the camera's field of view 5034 in the platter 5030 to ceasing to display the camera's field of view 5034 and returning to fully displaying the instant messaging user interface 5008. In Figure 5Y , the camera's field of view 5034 begins to fade out in the platter 5030. In Figures 5Y to 5Z , the platter 5030 transitions to the message bubble 5018. In Figure 5Z , the camera's field of view 5034 is no longer displayed. In Figure 5AA , the instant messaging user interface 5008 stops blurring, and the size of the message bubble 5018 returns to the original size of the message bubble 5018 (e.g., as Figure 5B shown).

[0235] Figures 5AA to 5AD Illustrated is the animated transition of the virtual chair 5020 as it moves from the position corresponding to the contact 5054 in Figure 5AA to the original position of the virtual chair 5020 in the instant messaging user interface 5008 (e.g., as Figure 5B shown). In Figure 5AB , the contact 5054 has been lifted off the touch screen 112. In Figures 5AB to 5AC , the size of the virtual chair 5020 gradually increases, and the virtual chair rotates towards its final position in Figure 5AD .

[0236] In Figures 5B to 5AD , the virtual chair 5020 has substantially the same three-dimensional appearance within the instant messaging user interface 5008 and within the displayed camera's field of view 5034, and the virtual chair 5020 maintains that same three-dimensional appearance during the transition from displaying the instant messaging user interface 5008 to displaying the camera's field of view 5034 and during the reverse transition. In some embodiments, the representation of the virtual chair 5020 has a different appearance in the application user interface (e.g., the instant messaging user interface) than in the augmented reality environment (e.g., within the displayed camera's field of view). For example, the virtual chair 5020 optionally has a two-dimensional or more dimensional stylized appearance in the application user interface, while having a more realistic three-dimensional textured appearance in the augmented reality environment; and the intermediate appearance of the virtual chair 5020 during the transition between displaying the application user interface and displaying the augmented reality environment is a series of interpolated appearances between the two-dimensional appearance and the three-dimensional appearance of the virtual chair 5020.

[0237] Figure 5AEAn Internet browser user interface 5060 is shown. The Internet browser user interface 5060 includes a URL / search input area 5062 configured to display a URL / search input for a web browser and browser controls 5064 (e.g., navigation controls including a back button and a forward button, a share control for displaying a share interface, a bookmark control for displaying a bookmark interface, and a tab control for displaying a tab interface). The Internet browser user interface 5060 also includes web objects 5066, 5068, 5070, 5072, 5074, and 5076. In some embodiments, the respective web objects include links such that in response to a tap input on the respective web object, the Internet location of the link corresponding to the web object is displayed in the Internet browser user interface 5060 (e.g., replacing the display of the respective web object). The web objects 5066, 5068, and 5072 include two-dimensional representations of three-dimensional virtual objects, as indicated by virtual object indicators 5078, 5080, and 5082, respectively. The web objects 5070, 5074, and 5076 include two-dimensional images (but the two-dimensional images of the web objects 5070, 5074, and 5076 do not correspond to three-dimensional virtual objects, as indicated by the absence of virtual object indicators). The virtual object corresponding to the web object 5068 is a lamp object 5084.

[0238] Figures 5AF to 5AH An input is shown that causes a portion of the Internet browser user interface 5060 to be replaced by the field of view 5034 of the camera. In Figure 5AF , a contact 5086 with the touch screen 112 of the device 100 is detected. The characteristic intensity of the contact is higher than the contact detection intensity threshold IT0 and lower than the prompt press intensity threshold IT H , as shown by the intensity level meter 5028. In Figure 5AG , as shown by the intensity level meter 5028, the characteristic intensity of the contact 5026 increases to be higher than the light press intensity threshold IT L has caused the field of view 5034 of the camera to be displayed in the web object 5068 (e.g., covered by the virtual lamp 5084). In Figure 5AH , as shown by the intensity level meter 5028, the characteristic intensity of the contact 5086 increases to be higher than the deep press intensity threshold IT D causes the field of view 5034 of the camera to replace a larger portion of the Internet browser user interface 5060 (e.g., only leaving the URL / search input area 5062 and the browser controls 5064), and the haptic output generator 167 of the device 100 outputs a haptic output (as shown at 5088), which indicates that the criterion for replacing a portion of the Internet browser user interface 5060 with the field of view 5034 of the camera has been met. In some embodiments, in response to reference to Figures 5AF to 5AHThe described input completely replaces the Internet browser user interface 506 on the touch screen display 112 with the camera's field of view 5034.

[0239] Figures 5AI to 5AM An input that causes the virtual lamp 5084 to move is shown. In Figures 5AI to 5AJ , when the contact 5086 moves along the path indicated by the arrow 5090, the contact 5086 drags the virtual lamp 5084. When the virtual lamp 5084 is moved by the contact 5086, the size of the virtual lamp 5084 remains unchanged, and the path of the virtual lamp 5084 is optionally not constrained by the structure of the physical space captured in the camera's field of view. When the virtual lamp 5084 is moved by the contact 5086, the plane identified in the camera's field of view 5034 is highlighted. For example, in Figure 5AJ , when the virtual lamp 5084 moves above the floor plane 5038, the floor plane 5038 is highlighted. In Figures 5AJ to 5AK , when the contact 5086 moves along the path indicated by the arrow 5092, the contact 5086 continues to drag the virtual lamp 5084. In Figures 5AK to 5AL , when the contact 5086 moves along the path indicated by the arrow 5094, the contact 5086 continues to drag the virtual lamp 5084, stops highlighting the floor plane 5038, and highlights the tabletop 5046 when the virtual lamp 5084 moves above the table 5004. In Figure 5AM , the contact 5086 has lifted off the touch screen 112. When the contact 5086 has lifted off, the size of the virtual lamp 5086 is adjusted to have the correct proportion relative to the table 5004 in the camera's field of view 5034, and the virtual lamp 5086 is placed upright on the tabletop 5046 in the camera's field of view 5034.

[0240] Figures 5AM to 5AQ An input that drags the virtual lamp 5084 to the edge of the touch screen display 112 is shown, which causes the camera's field of view 5034 to stop displaying and the Internet browser user interface 5060 to resume. In Figure 5AN , a contact 5096 with the touch screen 112 of the device 100 is detected at the position corresponding to the virtual lamp 5084. In Figures 5AN to 5AO , when the contact 5096 moves along the path indicated by the arrow 5098, the contact 5096 drags the virtual lamp 5084. In ​ , when the contact 5054 moves along the path indicated by the arrow 5100, the contact 5096 continues to drag the virtual lamp 5084 to the ​ position shown. In ​ , the contact 5096 has lifted off the touch screen 112.

[0241] As ​ shown, by ​The input made via the contact 5096 shown causes a transition from the field of view 5034 of the display camera to ceasing to display the field of view 5034 of the display camera and returning to fully displaying the Internet browser user interface 5060. In ​ , the field of view 5034 of the camera begins to fade out (as indicated by the dashed line). In ​ , the size of the virtual lamp 5084 increases, and the virtual lamp moves towards its original position within the Internet browser user interface 5060. In ​ , the field of view 5034 of the camera is no longer displayed, and the Internet browser user interface 5060 begins to fade in (as indicated by the dashed line). In ​ , the Internet browser user interface 5060 is fully displayed, and the virtual lamp 5084 has returned to its original size and position within the Internet browser user interface 5060.

[0242] ​ An example user interface according to some embodiments is shown, which is used to display a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object that displays a representation of the field of view of one or more cameras. The user interfaces in these figures are used to illustrate the processes described below, including ​ , ​ , ​ , ​ , ​ , ​ , ​ , and ​ the processes in. For ease of explanation, some embodiments in the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, similar operations are optionally performed on a device having a display 450 and a separate touch-sensitive surface 451 in response to detecting a contact on the touch-sensitive surface 451 when the user interface and focus selector shown in the display figures are on the display 450.

[0243] ​Shows an instant messaging user interface 5008, which includes: a message bubble 5010 including a received text message 5012, a message bubble 5014 including a sent text message 5016, and a message bubble 5018 including a virtual object received in the message (e.g., virtual chair 5020) and a virtual object indicator 5022, the virtual object indicator indicating that the virtual chair 5020 is an object visible in an augmented reality view (e.g., within the field of view of one or more cameras of the displayed device 100). Refer to ​ The instant messaging user interface 5008 is described in further detail.

[0244] ​ Shows an input to rotate the virtual chair 5020. In ​ , a contact 6002 with the touch screen 112 of the device 100 is detected. The contact 6002 moves along a path indicated by an arrow 6004 on the touch screen 112. In ​ , in response to the movement of the contact, the instant messaging user interface 5008 scrolls up (such that the message bubble 5010 rolls off the display, such that the message bubbles 5014 and 5018 scroll up, and reveals additional message bubbles 6005), and the virtual chair 5020 rotates (e.g., tilts upward). The amount and direction of rotation of the virtual chair 5020 correspond to the movement of the contact 6002 along the path indicated by the arrow 6004. In ​ , the contact 6002 has been lifted off the touch screen 112. In some embodiments, this rotational behavior of the virtual chair 5020 within the message bubble 5018 is used as an indication that the virtual chair 5020 is a virtual object visible in an augmented reality environment including the field of view of the camera of the device 100.

[0245] ​ Shows an input that causes the instant messaging user interface 5008 to be replaced by a staging user interface 6010 and then changes the orientation of the virtual chair 5020. In ​ , a contact 6006 with the touch screen 112 of the device 100 is detected. The characteristic intensity of the contact is higher than the contact detection intensity threshold IT0 and lower than the prompt press intensity threshold IT H , as shown by the intensity level meter 5028. In ​ , as shown by the intensity level meter 5028, the characteristic intensity of the contact 6006 increases to be higher than the prompt press intensity threshold IT [[ID=2s]] H , which causes the area of the message bubble 5018 to increase, the size of the virtual chair 5020 to increase, and the instant messaging user interface 5008 to start to blur behind the message bubble 5018 (e.g., providing visual feedback to the user of the effect of increasing the characteristic intensity of the contact). In ​In this case, as shown by the intensity level meter 5028, the characteristic intensity of the contact 6006 increases to be higher than the light press intensity threshold IT L , which causes the message bubble 5018 to be replaced by the dial 6008, the size of the virtual chair 5020 to further increase, and the instant messaging user interface 5008 to further blur behind the dial 6008. In ​ , as shown by the intensity level meter 5028, the characteristic intensity of the contact 6006 increases to be higher than the deep press intensity threshold IT D causes the instant messaging user interface 5008 to stop being displayed and initiates the fade-in of the staging user interface 6010 (indicated by the dashed line). Additionally, as ​ shown, the characteristic intensity of the contact 6006 increases to be higher than the deep press intensity threshold IT D causes the haptic output generator 167 of the device 100 to output a haptic output (indicated at 6012), which indicates that the criteria for replacing the instant messaging user interface 5008 with the staging user interface 6010 have been met.

[0246] In some embodiments, before the characteristic intensity of the contact 6006 reaches the deep press intensity threshold IT D (as ​ shown), ​ the progress shown is reversible. For example, after the increase shown in ​ and / or ​ , decreasing the characteristic intensity of the contact 6006 will cause the interface state corresponding to the decreased intensity level of the contact 6006 to be displayed (e.g., depending on determining that the decreased characteristic intensity of the contact is higher than the light press intensity threshold IT L , showing the interface as shown in ​ ; depending on determining that the decreased characteristic intensity of the contact is higher than the prompt press intensity threshold IT H , showing the interface as shown in ​ ; and depending on determining that the decreased characteristic intensity of the contact is lower than the prompt press intensity threshold IT H , showing the interface as shown in ​ ). In some embodiments, after the increase shown in ​ and / or ​ , decreasing the characteristic intensity of the contact 6006 will cause the interface as shown in ​ [[ID=z41]]to be redisplayed.

[0247] In ​ , the staging user interface 6010 is displayed. The staging user interface 6010 includes a pedestal 6014 on which the virtual chair 5020 is displayed. From ​ , the virtual chair 5020 is animated to indicate the movement from the position of the virtual chair 5020 in ​ to ​The transformation of the position of the virtual chair 5020 therein. For example, the virtual chair 5020 is rotated relative to the gantry 6014 to a predefined position, rotated in a predefined orientation, and / or rotated a predefined distance (e.g., such that the virtual chair appears to be supported by the gantry 6014). The staging user interface 6010 also includes a back control 6016, which when activated (e.g., by a tap input at a position corresponding to the back control 6016) causes the previously displayed user interface (e.g., the instant messaging user interface 5008) to be redisplayed. The staging user interface 6010 also includes a switch control 6018, which indicates the current display mode (e.g., the current display mode is the staging user interface mode, as indicated by the highlighted "3D" indicator), and which when activated causes a transition to the selected display mode. For example, when the staging user interface 6010 is displayed, a tap input by contact at a position corresponding to the switch control 6018 (e.g., a position corresponding to the portion of the switch control 6018 that includes the text "World") causes the staging user interface 6010 to be replaced by the field of view of the camera. The staging user interface 6010 also includes a share control 6020 (e.g., a share control for displaying a sharing interface).

[0248] ​ illustrates the rotation of the virtual chair 5020 relative to the gantry 6014 caused by the movement of the contact 6006. In ​ when the contact 6006 moves along the path indicated by the arrow 6022, the virtual chair 5020 rotates (e.g., about a first axis perpendicular to the movement of the contact 6066). In ​ when the contact 6006 moves along the path indicated by the arrow 6024 and then along the path indicated by the arrow 6025, the virtual chair 5020 rotates (e.g., about a second axis perpendicular to the movement of the contact 6066). In ​ the contact 6006 has been lifted off the touch screen 112. In some embodiments, as ​ shown, the rotation of the virtual chair 5020 is constrained by the surface of the gantry 6014. For example, during the rotation of the virtual chair, at least one leg of the virtual chair 5020 remains in contact with the surface of the gantry 6014. In some embodiments, the surface of the gantry 6014 serves as a reference frame for the free rotation and vertical translation of the virtual chair 5020 without imposing specific constraints on the movement of the virtual chair 5020.

[0249] ​ illustrates an input for adjusting the size of the displayed virtual chair 5020. In ​In, a first contact 6026 and a second contact 6030 with the touch screen 112 are detected. The first contact 6026 moves along the path indicated by arrow 6028. While the first contact 6026 is moving, the second contact 6030 moves along the path indicated by arrow 6032. In ​ In, when the first contact 6026 and the second contact 6030 move along the paths indicated by arrows 6028 and 6032 respectively (e.g., in a separation gesture), the size of the displayed virtual chair 5020 increases. In ​ In, the first contact 6030 and the second contact 6026 have lifted off the touch screen 112, and after the contacts 6026 and 6030 have lifted off, the virtual chair 5020 remains at the increased size.

[0250] ​ An input is shown that causes the staging user interface 6010 to be replaced by the field of view 6036 of one or more cameras of the device 100. In ​ In, a contact 6034 with the touch screen 112 of the device 100 is detected. The characteristic intensity of this contact is higher than the contact detection intensity threshold IT0 and lower than the prompt press intensity threshold IT H , as shown by the intensity level meter 5028. In ​ In, as shown by the intensity level meter 5028, the characteristic intensity of the contact 5026 increases to be higher than the prompt press intensity threshold IT H has caused the staging user interface 6010 to start to blur behind the virtual chair 5020 (as indicated by the dashed line). In ​ In, as shown by the intensity level meter 5028, the characteristic intensity of the contact 6034 increases to be higher than the light press intensity threshold IT L causes the staging user interface 6010 to stop being displayed and initiates a fade-in of the field of view 6036 of the camera (as indicated by the dashed line). In ​ In, as shown by the intensity level meter 5028, the characteristic intensity of the contact 6034 increases to be higher than the deep press intensity threshold IT D causes the field of view 6036 of the camera to be displayed. Additionally, as ​ shown, the characteristic intensity of the contact 6034 increases to be higher than the deep press intensity threshold IT D causes the haptic output generator 167 of the device 100 to output a haptic output (as indicated at 6038), which indicates that the criterion for replacing the display of the staging user interface 6010 with the display of the field of view 6036 of the camera has been met. In ​ In, the contact 6034 has lifted off the touch screen 112. In some embodiments, before the characteristic intensity of the contact 6034 reaches the deep press intensity threshold IT D (as ​ shown), ​ the progression shown in​ and / or ​ After the increase shown, reducing the feature intensity of contact 6034 will cause the interface state corresponding to the reduced intensity level of contact 6034 to be displayed.

[0251] From ​ , the virtual chair 5020 is placed on the detected plane (e.g., it is determined according to device 100 that the virtual chair 5020 is configured to be placed in a vertical orientation on the detected horizontal surface, such as the floor surface 5038), and the size of the virtual chair 5020 is adjusted (e.g., based on the "real-world" size of the virtual chair 5020 defined in the camera's field of view 6036 and / or the size of the detected object (such as the table 5004), the ratio of the virtual chair 5020 relative to the physical space 5002 as shown in the camera's field of view 6036 is determined). When the virtual chair 5020 transitions from the staging user interface 6010 to the camera's field of view 6036, the orientation of the virtual chair 5020 caused by the rotation of the virtual chair 5020 when the staging interface 6010 is displayed is maintained (e.g., as described with reference to ​ ). For example, the orientation of the virtual chair 5020 relative to the floor surface 5038 is the same as the final orientation of the virtual chair 5020 relative to the surface of the gantry 5014. In some embodiments, when the size of the virtual chair 5020 is adjusted relative to the size of the physical space 5002 in the field of view 6036, the adjustment of the size of the virtual object 5020 in the staging user interface is taken into account.

[0252] ​ An input is shown that causes the camera's field of view 6036 to be replaced by the staging user interface 6010. In ​ , an input (e.g., a tap input) is detected at a position corresponding to the switching control 6018 (e.g., at a position corresponding to the part of the switching control 6018 that includes the text "3D"). In ​ , in response to the input made by contact 6040, the camera's field of view 6036 fades out (as indicated by the dashed line in ​ ), the staging user interface 6010 fades in (as indicated by the dashed line in ​ ), and the staging user interface 6010 is fully displayed (as shown in ​ ). From ​ , the size of the virtual chair 5020 is adjusted, and the position of the virtual chair 5020 is changed (e.g., returning the virtual chair 5020 to the predefined position and size for the staging user interface).

[0253] ​ An input is shown that causes the staging user interface 6010 to be replaced by the instant messaging user interface 5008. In ​In, an input (e.g., a tap input) is detected at a position corresponding to the backward control 6016 by a contact 6042. In ​ In response to the input by the contact 6042, the staging user interface 6010 fades out (as indicated by the dashed line in ​ , the instant messaging user interface 5008 fades in (as indicated by the dashed line in ​ , and the instant messaging user interface 5008 is fully displayed (as shown in ​ ). From ​ , the size, orientation, and position of the virtual chair 5020 are continuously adjusted on the display (e.g., to return the virtual chair 5020 to a predefined position, size, and orientation for the instant messaging user interface 5008).

[0254] ​ shows an input that causes the instant messaging user interface 5008 to be replaced by the field of view 6036 of the camera (e.g., bypassing the display of the staging user interface 6010). In ​ , a contact 6044 is detected at a position corresponding to the virtual chair 5020. The input by the contact 6044 includes a long touch gesture (during which the contact 6044 remains at a position corresponding to the representation of the virtual object 5020 on the touch-sensitive surface with a movement less than a threshold movement amount for at least a predefined threshold time amount) and a subsequent upward swipe gesture (dragging the virtual chair 5020 upward). As shown in ​ , when the contact 6044 moves along the path indicated by the arrow 6046, the virtual chair 5020 is dragged upward. In ​ , the instant messaging user interface 5008 fades out behind the virtual chair 5020. As shown in ​ , when the contact 6044 moves along the path indicated by the arrow 6048, the virtual chair 5020 continues to be dragged upward. In ​ , the field of view 5036 of the camera fades in behind the virtual chair 5020. In ​ , in response to the input by the contact 6044 including a long touch gesture and a subsequent upward swipe gesture, the field of view 5036 of the camera is fully displayed. In ​ , the contact 6044 is lifted off the touch screen 112. In ​ , in response to the lifting off of the contact 6044, the virtual chair 5020 is released (e.g., because the virtual chair 5020 is no longer constrained or dragged by the contact) and falls onto a plane (e.g., the floor surface 5038, determined according to the horizontal (floor) surface corresponding to the virtual chair 5020). Additionally, as shown in ​ , the haptic output generator 167 of the device 100 outputs a haptic output (as shown at 6050), which indicates that the virtual chair 5020 has landed on the floor surface 5038.

[0255] ​ Illustrates an example user interface according to some embodiments, which is used to display items having visual indications indicating that the items correspond to virtual three-dimensional objects. The user interfaces in these figures are used to illustrate the processes described below, including ​ , ​ , ​ , ​ , ​ , ​ , ​ and ​ the processes in. For ease of explanation, some embodiments in the embodiments will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, in response to detecting a contact on the touch-sensitive surface 451 when the user interface and the focus selector shown in the display figures are on the display 450, similar operations are optionally performed on a device having a display 450 and a separate touch-sensitive surface 451.

[0256] ​ Illustrates the input detected when the user interface 400 displaying the application menu is shown. This input corresponds to a request to display a first user interface (e.g., an Internet browser user interface 5060). In ​ , an input (e.g., a tap input) is detected at a position corresponding to the icon 420 of the browser module 147 by the contact 7000. In response to this input, the Internet browser user interface 5060 is displayed, as shown in ​ .

[0257] ​ Illustrates the Internet browser user interface 5060 (e.g., as described in detail with reference to ​ ). The Internet browser user interface 5060 includes network objects 5066, 5068, 5070, 5072, 5074, and 5076. The network objects 5066, 5068, and 5072 include two-dimensional representations of three-dimensional virtual objects, as indicated by the virtual object indicators 5078, 5080, and 5082, respectively. The network objects 5070, 5074, and 5076 include two-dimensional images (but the two-dimensional images of the network objects 5070, 5074, and 5076 do not correspond to three-dimensional virtual objects, as indicated by the absence of virtual object indicators).

[0258] ​Shows an input that causes the Internet browser user interface 5060 to pan (e.g., scroll). In ​ , a contact 7002 with the touch screen 112 is detected. In ​ , as the contact 7002 moves along the path indicated by arrow 7004, the network objects 5066, 5068, 5070, 5072, 5074, and 5076 scroll upward, thereby revealing additional network objects 7003 and 7005. Additionally, as the contact 7002 moves along the path indicated by arrow 7004, the virtual objects in the network objects 5066, 5068, and 5072, which respectively include virtual object indicators 5078, 5080, and 5082, rotate (e.g., tilt upward) according to the direction of the input (vertically upward). For example, the virtual lamp 5084 tilts upward from the ​ first orientation to the ​ second orientation. When the contact scrolls the Internet browser user interface 5060, the two-dimensional images of the network objects 5070, 5074, and 5076 do not rotate. In ​ , the contact 7002 has lifted off the touch screen 112. In some embodiments, the rotation behavior of the objects depicted in the network objects 5066, 5068, and 5072 is used as a visual indication that these network objects have corresponding three-dimensional virtual objects that are visible in an augmented reality environment, while the absence of such rotation behavior in the objects depicted in the network objects 5070, 5074, and 5076 is used as a visual indication that these network objects do not have corresponding three-dimensional virtual objects that are visible in an augmented reality environment.

[0259] Figures 7F through 7G show a parallax effect where virtual objects rotate on the display in response to a change in the orientation of the device 100 relative to the physical world.

[0260] ​ Shows that the device 100 is held by the user 7006 in the user's hand 5006 such that the device 100 has a substantially vertical orientation. ​ Shows the Internet browser user interface 5060 as displayed by the device 100 when the device 100 is in the ​ orientation shown.

[0261] ​ Shows that the device 100 is held by the user 7006 in the user's hand 5006 such that the device 100 has a substantially horizontal orientation. ​ Shows the Internet browser user interface 5060 as displayed by the device 100 when the device 100 is in the ​ orientation shown. From ​, the orientations of the virtual objects in the network objects 5066, 5068, and 5072, which include virtual object indicators 5078, 5080, and 5082 respectively, rotate (e.g., tilt upward) according to the change in the orientation of the device. For example, according to the simultaneous change in the device orientation in the physical space, the virtual lamp 5084 tilts upward from ​ the first orientation in Figure 7G2 to the second orientation in

[0262] Figures 7H to 7L shows an input corresponding to a request to display a second user interface (e.g., the instant messaging user interface 5008). In Figure 7H , a contact 7008 is detected at a position corresponding to the lower edge of the display 112. In Figures 7H to 7I , the contact 7008 moves upward along the path indicated by the arrow 7010. In Figures 7I to 7J , the contact 7008 continues to move upward along the path indicated by the arrow 7012. In Figures 7H to 7J , when the contact 7008 moves upward from the lower edge of the display 112, the size of the internet browser user interface 5060 decreases, as shown in Figure 7I ; and in Figure 7J , a multitasking user interface 7012 is displayed (e.g., in response to an upward edge swipe gesture made through the contact 7008). The multitasking user interface 7012 is configured to allow selection of interfaces from various applications and various control interfaces (e.g., the control center user interface 7014, the internet browser user interface 5060, and the instant messaging user interface 5008, as shown in Figure 7J ) that have a retained state (e.g., when the corresponding application is the foreground application executing on the device, the retained state is the last state of the corresponding application). In Figure 7K , the contact 7008 has been lifted off the touch screen 112. In Figure 7L , an input (e.g., a tap input) is detected through the contact 7016 at a position corresponding to the instant messaging user interface 5008. In response to the input made through the contact 7016, the instant messaging user interface 5008 is displayed, as shown in Figure 7M .

[0263] Figure 7M shows an instant messaging user interface 5008 including a message bubble 5018 (e.g., as described in further detail with reference to Figure 5B ), the message bubble including a virtual object (e.g., virtual chair 5020) received in the message and a virtual object indicator 5022 that indicates that the virtual chair 5020 is a virtual three-dimensional object (e.g., an object visible in an augmented reality view and / or an object visible from different angles). The instant messaging user interface 5008 also includes a message bubble 6005 including a sent text message and a message bubble 7018 including a received text message including an emoji 7020. The emoji 7020 is a two-dimensional image that does not correspond to a virtual three-dimensional object. Accordingly, the displayed emoji 7020 does not have a virtual object indicator.

[0264] Figure 7N shows a map user interface 7022, which includes a map 7024, a point-of-interest information area 7026 for a first point of interest, and a point-of-interest information area 7032 for a second point of interest. For example, the first point of interest and the second point of interest are search results within or near the area corresponding to the search entry "Apple" in the search input area 7025 shown in the map 7024. In the first point-of-interest information area 7026, the displayed first point-of-interest object 7028 has a virtual object indicator 7030 that indicates that the first point-of-interest object 7028 is a virtual three-dimensional object. In the second point-of-interest information area 7032, the displayed second point-of-interest object 7034 does not have a virtual object indicator because the second point-of-interest object 7034 does not correspond to a virtual three-dimensional object visible in an augmented reality view.

[0265] Figure 7O shows a file management user interface 7036, which includes file management controls 7038, a file management search input area 7040, a file information area 7042 for a first file (e.g., a portable document format (PDF) file), a file information area 7044 for a second file (e.g., a photo file), a file information area 7046 for a third file (e.g., a virtual chair object), and a file information area 7048 for a fourth file (e.g., a PDF file). The third file information area 7046 includes a virtual object indicator 7050 displayed adjacent to a file preview object 7045 in the file information area 7046, the virtual object indicator indicating that the third file corresponds to a virtual three-dimensional object. The displayed first file information area 7042, second file information area 7044, and fourth file information area 7048 do not have virtual object indicators because the files corresponding to these file information areas do not have corresponding virtual three-dimensional objects visible in an augmented reality environment.

[0266] Figure 7P An email user interface 7052 is shown, which includes an email navigation control 7054, an email information area 7056, and an email content area 7058 including representations of a first attachment 7060 and a second attachment 7062. The representation of the first attachment 7060 includes a virtual object indicator 7064 that indicates that the first attachment is a virtual three-dimensional object visible in an augmented reality environment. The displayed second attachment 7062 does not have a virtual object indicator because the second attachment is not a virtual three-dimensional object visible in an augmented reality environment.

[0267] Figures 8A to 8E is a flowchart of a method 800 for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area according to some embodiments. The method 800 is performed at an electronic device (e.g., Figure 3 device 300 in Figure 1A or the portable multifunctional device 100 in

[0268] having a display, a touch-sensitive surface, and one or more cameras (e.g., one or more rear cameras on a side of the device opposite the display and the touch-sensitive surface). In some embodiments, the display is a touchscreen display and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display and the touch-sensitive surface are separate. Some operations in the method 800 are optionally combined and / or the order of some operations is optionally changed.

[0269] The device displays (802) a representation of a virtual object (e.g., a graphical representation of a three-dimensional object such as a virtual chair 5020, a virtual lamp 5084, a shoe, furniture, a hand tool, an ornament, a person, an emoji, a game character, virtual furniture, etc.) in a first user interface area on the display 112 (e.g., a two-dimensional graphical user interface or a part thereof (e.g., a browsable list of furniture images, an image containing one or more selectable objects, etc.)). For example, the first user interface area is an instant messaging user interface 5008 as shown in Figure 5B or an Internet browser user interface 5060 as shown in Figure 5AE . In some embodiments, in addition to an image of the physical environment around the device, the first user interface area further includes a background (e.g., the background of the first user interface area is a preselected background color / pattern or a background image that is different from the output image simultaneously captured by one or more cameras and different from the real-time content in the field of view of one or more cameras).

[0270] When a first representation of a virtual object is displayed in the first user interface area on the display, the device detects (804) a first input by contact at a position on the touch-sensitive surface 112 corresponding to the representation of the virtual object on the display (e.g., detecting contact on the first representation of the virtual object on a touchscreen display or detecting contact on an affordance representation that is simultaneously displayed in the first user interface area with the first representation of the virtual object and is configured to trigger the display of an AR view of the virtual object when invoked by contact). For example, the first input is an input made by contact 5020 as described with reference to Figures 5C to 5F or an input made by contact 5086 as described with reference to Figures 5AF to 5AL .

[0271] In response to detecting the first input by contact (806), based on determining that the first input by contact meets a first (e.g., AR-trigger) criterion (e.g., the AR-trigger criterion is a criterion configured to recognize a swipe input, a touch-and-hold input, a press input, a tap input, a force press with an intensity higher than a predefined intensity threshold, or another type of predefined input gesture, and the criterion is associated with triggering the activation of the camera, the display of an augmented reality (AR) view of the physical environment around the device, the placement of a three-dimensional representation of the virtual object inside the augmented reality view of the physical environment, and / or a combination of two or more of the above actions): The device displays a second user interface area on the display, which includes replacing at least a part of the display of the first user interface area with a representation of the field of view of one or more cameras, and the device continuously displays the representation of the virtual object when switching from displaying the first user interface area to displaying the second user interface area. For example, the second user interface area on the display is as shown in reference to Figure 5HThe field of view 5034 of the camera in the described panel 5030 or as described with reference to Figure the field of view 5034 of the camera. In ​ it, based on determining that the input via contact 5026 has a characteristic intensity that increases to be higher than the deep press intensity threshold IT D when switching from displaying a first user interface area (instant messaging user interface 5008) to displaying a second user interface area, continuously display the virtual chair object 5020, where displaying the second user interface area means replacing a part of the instant messaging user interface 5008 with the field of view 5034 of the camera in the panel 5030. In ​ it, based on determining that the input via contact 5086 has a characteristic intensity that increases to be higher than the deep press intensity threshold IT D when switching from displaying a first user interface area (Internet browser user interface 5060) to displaying a second user interface area, continuously display the virtual lamp object 5084, where displaying the second user interface area means replacing a part of the Internet browser user interface 5060 with the field of view 5034 of the camera.

[0272] In some embodiments, continuously displaying a representation of a virtual object includes maintaining the display of the representation of the virtual object or displaying an animated transition in which a first representation of the virtual object changes to a second representation of the virtual object (e.g., views of the virtual object with different sizes, from different perspectives, with different rendering styles, or at different positions on the display). In some embodiments, the field of view 5034 of one or more cameras displays a real-time image of the physical environment 5002 around the display device, and this real-time image is updated in real time when the position and orientation of the device relative to the physical environment change (e.g., as ​ shown). In some embodiments, the second user interface area completely replaces the first user interface on the display.

[0273] In some embodiments, the second user interface area covers a part of the first user interface area (e.g., a part of the first user interface area is shown along the edge of the display or around the boundary of the display). In some embodiments, the second user interface area pops up next to the first user interface area. In some embodiments, the background within the first user interface area is replaced with the content of the field of view 5034 of the camera. In some embodiments, the device displays a virtual object moving and rotating from a first orientation as shown in the first user interface area (e.g., as ​An animated transition from a first orientation (as shown) to a second orientation (e.g., an orientation predefined relative to the current orientation of a portion of the physical environment captured in the field of view of one or more cameras). For example, the animation includes a transition from displaying a two-dimensional representation of a virtual object when a first user interface region is displayed to displaying a three-dimensional representation of the virtual object when a second user interface region is displayed. In some embodiments, the three-dimensional representation of the virtual object has an anchoring plane predefined based on the shape and orientation of the virtual object as shown in a two-dimensional graphical user interface (e.g., the first user interface region). When transitioning to an augmented reality view (e.g., the second user interface region), the three-dimensional representation of the virtual object is moved, resized, and reoriented so that the virtual object moves from its original position on the display to a new position on the display (e.g., to the center of the augmented reality view or to another predefined position in the augmented reality view), and during or at the end of the movement, the three-dimensional representation of the virtual object is reoriented so that the three-dimensional representation of the virtual object is at a predefined position and / or orientation relative to a predefined plane (e.g., a physical surface that can serve as a support plane for the three-dimensional representation of the virtual object, such as a vertical wall or a horizontal floor surface) identified in the field of view of one or more cameras.

[0274] In some embodiments, the first criterion includes (808) a criterion satisfied when a movement of a contact at a position corresponding to a representation of a virtual object on a touch-sensitive surface is maintained at a movement amount less than a threshold amount for at least a predefined amount of time (e.g., a long-press time threshold) (e.g., determined as follows). In some embodiments, based on determining that the contact satisfies a criterion for identifying another type of gesture (e.g., a tap), while maintaining the display of the virtual object, the device also performs another predefined function in addition to triggering the AR user interface. Whether to continuously display the representation of the virtual object when replacing at least a portion of the display of the first user interface region with the field of view of the camera is determined based on whether the contact at the position corresponding to the representation of the virtual object on the touch-sensitive surface is maintained at a movement amount less than a threshold amount for at least a predefined amount of time, which enables multiple different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to the input improves the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0275] In some embodiments, the first criterion includes (810) a criterion satisfied when (e.g., determined as follows) the characteristic intensity of the contact increases above a first intensity threshold (e.g., a light-press intensity threshold IT L or a deep-press intensity threshold IT D ). For example, as referred to in ​as described, when the feature intensity of contact 5026 increases to be higher than the deep press intensity threshold IT D the criteria are met, as indicated by the intensity level meter 5028. In some embodiments, based on determining that the contact meets the criteria for identifying another type of gesture (e.g., a tap), while maintaining the display of the virtual object, the device also performs another predefined function in addition to triggering the AR user interface. In some embodiments, the first criteria require that the first input is not a tap input (e.g., the input has a duration between the downward touch of the contact and the lift-off of the contact that is greater than a tap time threshold). Based on whether the feature intensity of the contact increases to be higher than the first intensity threshold, it is determined whether to continuously display a representation of the virtual object when replacing at least a portion of the display of the first user interface region with the field of view of the camera, which enables a variety of different types of operations to be performed in response to the input. Enabling a variety of different types of operations to be performed in response to the input improves the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0276] In some embodiments, the first criterion includes (812) the criterion that is satisfied when the movement of the contact (e.g., as determined according to the following) meets a predefined movement criterion (e.g., the contact moves beyond a predefined threshold position on the touch-sensitive surface (e.g., a position corresponding to the boundary of the first user interface region, a position at a threshold distance from the original position of the contact, etc.), the contact moves at a speed greater than a predefined threshold speed, the movement of the contact ends under a press input, etc.). In some embodiments, during an initial portion of the movement of the contact, the representation of the virtual object is dragged by the contact, and when the movement of the contact is about to meet the predefined movement criterion, the virtual object stops moving under the contact to indicate that the first criterion is about to be satisfied; and if the movement of the contact continues and the continued movement of the contact causes the predefined movement criterion to be satisfied, a transition to displaying a second user interface region and displaying the virtual object within the augmented reality view is initiated. In some embodiments, when the virtual object is dragged during an initial portion of the first input, the object size and viewing perspective do not change, and once the augmented reality view is displayed and the virtual object drops to a position within the augmented reality view, a virtual object with a size and viewing perspective depending on the physical position represented by the drop position of the virtual object in the augmented reality view is displayed. Whether to continuously display the representation of the virtual object when replacing at least a portion of the display of the first user interface region with the field of view of the camera is determined based on whether the movement of the contact meets the predefined movement criterion, which enables various different types of operations to be performed in response to the input. Enabling various different types of operations to be performed in response to the input improves the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0277] In some embodiments, in response to detecting a first input via a contact, and based on determining that the first input via the contact has met the first criterion, a device having one or more haptic output generators 167 outputs (814) a haptic output that indicates that the first input meets the first criterion (e.g., the haptic output 5032 as described with reference to ​ or the haptic output 5088 as described with reference to ​ . In some embodiments, the haptic sensation is generated before the field of view of one or more cameras appears on the display. For example, the haptic sensation indicates that the first criterion for triggering the activation of one or more cameras and subsequently triggering plane detection in the field of view of one or more cameras is satisfied. Since it takes time to activate the cameras and make the field of view displayable, the haptic sensation serves as a non-visual signal for the user indicating that the device has detected the necessary input and will present the augmented reality user interface as soon as it is ready.

[0278] Tactile output indicating that an output indication criterion (e.g., for replacing at least a portion of a user interface with a field of view of a camera) is met provides feedback to the user indicating that the provided input meets the criterion. Providing improved tactile feedback enhances the operability of the device (e.g., by assisting the user in providing appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0279] In some embodiments, in response to detecting at least an initial portion of a first input (e.g., including: detecting contact; or detecting input by contact that meets a corresponding predefined criterion but does not meet the first criterion; or detecting an input that meets the first criterion), the device analyzes (816) the fields of view of one or more cameras to detect one or more planes in the fields of view of the one or more cameras (e.g., floor surface 5038, tabletop 5046, wall, etc.). In some embodiments, one or more cameras are activated in response to detecting at least an initial portion of the first input, and plane detection is initiated while the cameras are activated. In some embodiments, the display of the fields of view of the one or more cameras is delayed after the one or more cameras are activated (e.g., from the time the one or more cameras are activated until at least one plane is detected in the fields of view of the cameras). In some embodiments, the display of the fields of view of the one or more cameras is initiated at the time the one or more cameras are activated, and plane detection is completed after the fields of view are visible on the display (e.g., in a second user interface region). In some embodiments, after a corresponding plane is detected in the fields of view of the one or more cameras, the device determines the size and / or position of a representation of a virtual object based on the position of the corresponding plane relative to the fields of view of the one or more cameras. In some embodiments, when the electronic device moves, the size and / or position of the representation of the virtual object is updated as the position of the fields of view of the one or more cameras relative to the corresponding plane changes (e.g., as described ​ ). Determining the size and / or position of a representation of a virtual object based on the position of a corresponding plane detected in the fields of view of the cameras (e.g., without the need for further user input to size and / or position the virtual object relative to the fields of view of the cameras) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0280] In some embodiments, in response to detecting contact at a location on a touch-sensitive surface corresponding to a representation of a virtual object on a display (e.g., in response to detecting contact 5026 at a location on touch screen 112 corresponding to virtual chair 5020), an analysis of the fields of view of one or more cameras is initiated (818) to detect one or more planes in the fields of view of the one or more cameras. For example, before a first input meets a first criterion (e.g., before the characteristic intensity of contact 5026 increases above a deep press intensity threshold IT D as described with reference to ​ ), and before a second user interface region is displayed, activation of the camera and detection of planes in the field of view of the camera are started. By starting to detect planes when any interaction with the virtual object is detected, plane detection can be completed before an AR trigger criterion is met, and thus there is no visual delay for the user when, when the first input meets the AR trigger criterion, the virtual object transitions into an augmented reality view. Initiating an analysis to detect one or more planes in the field of view of a camera (e.g., without the need for further user input to initiate the analysis of the field of view of the camera) in response to detecting contact at the location of the representation of the virtual object improves the efficiency of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0281] In some embodiments, in response to detecting that a first input made by contact meets a first criterion (e.g., in response to detecting that the characteristic intensity of contact 5026 increases above a deep press intensity threshold IT D as described with reference to ​ ), an analysis of the fields of view of one or more cameras is initiated (820) to detect one or more planes in the fields of view of the one or more cameras. For example, when the first input meets the first criterion, activation of the camera and detection of planes in the field of view of the camera are started, and the field of view of the camera is displayed before plane detection is completed. By starting camera activation and plane detection when the AR trigger criterion is met, the camera and plane detection are not activated and kept running unnecessarily, which saves battery power and extends battery life and camera life.

[0282] In some embodiments, in response to detecting that an initial portion of a first input meets a plane detection trigger criterion and does not meet a first criterion, an analysis of the field of view of one or more cameras is initiated (822) to detect one or more planes in the field of view of the one or more cameras. For example, when the initial portion of the first input meets some criteria (e.g., criteria that are less stringent than the AR trigger criterion), activation of the camera and detection of planes in the field of view of the camera are started, and optionally, the field of view of the camera is displayed before plane detection is complete. By starting camera activation and plane detection after certain criteria are met rather than upon detection of contact, the camera and plane detection are not unnecessarily activated and kept running, which saves battery power and extends battery life and camera life. By starting camera activation and plane detection before the AR trigger criterion is met, the latency (caused by camera activation and plane detection) for displaying virtual objects that transition to an augmented reality view when the first input meets the AR trigger criterion is reduced.

[0283] In some embodiments, the device displays (824) a representation of a virtual object in a second user interface region in a corresponding manner such that the virtual object (e.g., virtual chair 5020) is oriented at a predefined angle relative to a corresponding plane detected in the field of view 5034 of one or more cameras (e.g., such that there is no distance (or a minimum distance) between the lower sides of the four legs of the virtual chair 5020 and the floor surface 5038). For example, the orientation and / or position of the virtual object relative to the corresponding plane is predefined based on the shape and orientation of the virtual object as shown in a two-dimensional graphical user interface (e.g., the corresponding plane corresponds to a horizontal physical surface that can act as a support surface for a three-dimensional representation of the virtual object in an augmented reality view (e.g., a horizontal tabletop for supporting a vase), or the corresponding plane is a vertical physical surface that can act as a support surface for a three-dimensional representation of the virtual object in an augmented reality view (e.g., a vertical wall for hanging a virtual picture frame)). In some embodiments, the orientation and / or position of the virtual object is defined by a corresponding surface or boundary of the virtual object (e.g., a bottom surface, bottom boundary points, side surfaces, and / or side boundary points). In some embodiments, the anchoring plane corresponding to the corresponding plane is an attribute in a set of attributes of the virtual object, and the anchoring plane is specified according to the nature of the physical object that the virtual object is supposed to represent. In some embodiments, the virtual object is placed at a predefined orientation and / or position relative to a plurality of planes detected in the field of view of one or more cameras (e.g., multiple corresponding sides of the virtual object are associated with the corresponding planes detected in the field of view of the camera). In some embodiments, if the predefined orientation and / or position of the virtual object is defined relative to the horizontal bottom plane of the virtual object, the bottom plane of the virtual object is displayed on the floor plane detected in the field of view of the camera (e.g., the horizontal bottom plane of the virtual object is parallel to the floor plane and the distance between them is zero). In some embodiments, if the predefined orientation and / or position of the virtual object is defined relative to the vertical rear plane of the virtual object, the rear surface of the virtual object is placed against the wall plane detected in the field of view of one or more cameras (e.g., the vertical rear plane of the virtual object is parallel to the wall plane and the distance between them is zero). In some embodiments, the virtual object is placed at a position at a fixed distance from the corresponding plane or at an angle other than a zero angle or a right angle relative to the corresponding plane. Displaying the representation of the virtual object relative to the plane detected in the field of view of the camera (e.g., without further user input to display the virtual object relative to the plane in the field of view of the camera) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0284] In some embodiments, in response to detecting a respective plane in the field of view of one or more cameras, a device having one or more haptic output generators 167 outputs (826) a haptic output that indicates the detection of the respective plane in the field of view of the one or more cameras. In some embodiments, a respective haptic output is generated for each plane detected in the field of view of the camera (e.g., floor surface 5038 and / or tabletop 5046). In some embodiments, the haptic output is generated upon completion of plane detection. In some embodiments, the haptic output is accompanied by a visual indication (e.g., momentary highlighting of the detected field-of-view plane) of the field-of-view plane in the second user interface portion. Outputting a haptic output that indicates the detection of a plane in the field of view of the camera provides the user with feedback that the plane has been detected. Providing improved haptic feedback enhances the operability of the device (e.g., by assisting the user in providing appropriate input and reducing unnecessary additional input for placing virtual objects), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0285] In some embodiments, when switching from displaying a first user interface area to displaying a second user interface area, the device displays (828) an animation of a representation of a virtual object (e.g., moving, rotating, resizing, and / or re-rendering in a different style, etc.) transitioning to a predefined position relative to a respective plane in the second user interface area (e.g., as ​ shown), and in combination with displaying a representation of a virtual object at a predefined angle relative to the respective plane (e.g., at a predefined orientation and / or position relative to the respective plane, and its size, rotation angle, and appearance in the augmented reality view upon reaching the final state to be shown), a device having one or more haptic output generators 167 outputs a haptic output that indicates that the virtual object is displayed at the predefined angle relative to the respective plane in the second user interface area. For example, as ​ shown, in combination with displaying a virtual chair 5020 at a predefined angle relative to the floor surface 5038, the device outputs a haptic output 5036. In some embodiments, the generated haptic output is configured to have characteristics (e.g., frequency, number of cycles, modulation, amplitude, accompanying audio wave, etc.) that reflect the following properties of the virtual object or the physical object represented by the virtual object: weight (e.g., heavy vs. light), material (e.g., metal, cotton, wood, marble, liquid, rubber, glass), size (e.g., large vs. small), shape (e.g., thin vs. thick, long vs. short, round vs. pointed, etc.), elasticity (e.g., elastic vs. rigid), nature (e.g., playful vs. solemn, mild vs. strong, etc.), and other properties. For example, the haptic output uses ​One or more of the haptic output patterns shown. In some embodiments, a preset distribution including one or more changes over time of one or more features corresponds to a virtual object (e.g., an emoji). For example, a "bouncing" haptic output distribution is provided for the "smiling" emoji virtual object. The haptic output indicating the placement of the representation of the virtual object relative to a corresponding plane provides the user with feedback indicating that the representation of the virtual object has been automatically placed relative to the corresponding plane. Providing improved haptic feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing unnecessary additional input for placing the virtual object), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0286] In some embodiments (830), the haptic output has a haptic output distribution corresponding to a feature of the virtual object (e.g., simulating physical properties such as size, density, mass, and / or material). In some embodiments, the haptic output distribution has features (e.g., frequency, number of cycles, modulation, amplitude, accompanying audio wave, etc.) that vary based on one or more features of the virtual object (e.g., weight, material, size, shape, and / or elasticity). For example, the haptic output uses ​ One or more of the haptic output patterns shown. In some embodiments, as the size, weight, and / or mass of the virtual object increases, the amplitude and / or duration of the haptic output also increase. In some embodiments, the haptic output pattern is selected based on the virtual material constituting the virtual object. The haptic output having a distribution corresponding to the features of the virtual object provides the user with feedback indicating information about the features of the virtual object. Providing improved haptic feedback enhances the operability of the device (e.g., by helping the user provide appropriate input; by reducing unnecessary additional input for placing the virtual object; and by providing a way for the user to perceive the features of the virtual object without cluttering the user interface with displayed information about these features), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0287] In some embodiments, when the representation of the virtual object is displayed in the second user interface region, the device detects (832) an adjustment of the field of view 5034 of one or more cameras (e.g., as ​movement of the device (e.g., lateral movement and / or rotation of the device) as shown, and in response to detecting the movement of the device, when adjusting the field of view of one or more cameras, according to a fixed spatial relationship (e.g., orientation and / or position) between the virtual object and a corresponding plane (e.g., the floor surface 5038) in the field of view of the one or more cameras (e.g., the virtual object is displayed on the display in an orientation and position such that a fixed angle between the representation of the virtual object and the plane is maintained (e.g., the virtual object appears to remain at a fixed position on the plane or rolls along the field of view plane)), the device adjusts the representation of the virtual object (e.g., the virtual chair 5020) in the second user interface region. For example, in ​ , when the device 100 moves, the virtual chair 5020 in the second user interface region including the field of view 5034 of the camera maintains a fixed orientation and position relative to the floor surface 5038. In some embodiments, the virtual object appears stationary and unchanged relative to the surrounding physical environment 5002, that is, when the field of view of one or more cameras changes as the device moves relative to the surrounding physical environment, the size, position, and / or orientation of the representation of the virtual object on the display change as the device position and / or orientation change. Adjusting the representation of the virtual object according to the fixed relationship between the virtual object and the corresponding plane (e.g., no further user input is required to maintain the position of the virtual object relative to the corresponding plane) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0288] In some embodiments, (e.g., at a time corresponding to replacing at least a portion of the display of the first user interface region with a representation of the field of view of one or more cameras), the device displays (834) an animation (e.g., movement, rotation about one or more axes, and / or scaling) of the representation of the virtual object (e.g., the virtual chair 5020) that is continuously displayed when switching from displaying the first user interface region to displaying the second user interface region (e.g., as ​as shown). For example, the animation includes a transition from displaying a two-dimensional representation of a virtual object when displaying a first user interface region to displaying a three-dimensional representation of the virtual object when displaying a second user interface region. In some embodiments, the three-dimensional representation of the virtual object has a predefined orientation relative to the current orientation of a portion of the physical environment captured in the field of view of one or more cameras. In some embodiments, when transitioning to an augmented reality view, the representation of the virtual object is moved, resized, and reoriented so that the virtual object moves from an initial position on the display to a new position on the display (e.g., the center of the augmented reality view or another predefined position in the augmented reality view), and during or at the end of the movement, the virtual object is reoriented so that the virtual object forms a fixed angle relative to a plane detected in the field of view of the camera (e.g., a physical surface that can support the representation of the virtual object, such as a vertical wall or a horizontal floor surface). In some embodiments, when an animated transition occurs, the lighting of the virtual object and / or the shadow cast by the virtual object is adjusted (e.g., to match the ambient lighting detected in the field of view of one or more cameras). Displaying the animation when the representation of the virtual object switches from displaying a first user interface region to displaying a second user interface region provides the user with feedback indicating that the first input meets the first criterion. Providing improved feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0289] In some embodiments, when the second user interface region is displayed on the display, the device detects (836) a second input made by a second contact (e.g., contact 5040), where the second input includes (optionally, a press or touch input for selecting the representation of the virtual object made by the second contact and) a movement of the second contact along a first path on the display (e.g., as ​ shown), and in response to detecting the second input made by the second contact, the device moves the representation of the virtual object (e.g., virtual chair 5020) in the second user interface region along a second path corresponding to (e.g., the same as or constrained by) the first path. In some embodiments, the second contact is different from the first contact and is detected after the first contact is lifted (e.g., as shown by contact 5040 in ​ which is in ​After the lift-off of the contact 5026 in it is detected). In some embodiments, the second contact is the same as the first contact that remains continuously on the touch-sensitive surface (e.g., as shown by the input through the contact 5086, which meets the AR trigger criteria and then moves on the touch screen 112 to move the virtual lamp 5084). In some embodiments, a swipe input on the virtual object rotates the virtual object, and the movement of the virtual object is optionally constrained by a plane in the field of view of the camera (e.g., the swipe input rotates the representation of the chair on the floor plane in the field of view of the camera). Moving the representation of the virtual object in response to detecting the input provides the user with feedback indicating that the position of the displayed virtual object can be moved in response to user input. Providing improved feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power consumption and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0290] In some embodiments, when the representation of the virtual object moves along a second path based on a contact-based movement and a corresponding plane associated with the virtual object, the device adjusts (838) the size of the representation of the virtual object (e.g., based on the virtual distance from the representation of the virtual object to the user to maintain an accurate perspective of the virtual object in the field of view). For example, in ​ when the virtual chair moves deeper into the field of view 5034 of the camera, away from the device 100, and towards the table 5004, the size of the virtual chair 5020 decreases. Adjusting the size of the representation of the virtual object when the representation of the virtual object moves along a second path based on a contact-based movement and a plane associated with the virtual object (e.g., no further user input is required to adjust the size of the representation of the virtual object to maintain the representation of the virtual object in a realistic size relative to the environment in the field of view of the camera) enhances the operability of the device, and in turn reduces power consumption and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0291] In some embodiments, when the representation of the virtual object moves along a second path, the device maintains (840) a first size of the representation of the virtual object (e.g., as ​ shown), the device detects the termination of a second input made through a second contact (e.g., including detecting the lift-off of the second contact, as ​ shown), and in response to detecting the termination of the second input made through the second contact, the device places the representation of the virtual object at a drop position in a second user interface area (e.g., on the desktop 5046) and displays the representation of the virtual object having a second size at the drop position in the second user interface area, the second size being different from the first size (e.g., ​After the input termination through contact 5086, the size of the virtual lamp 5084 is different from ​ the size of the virtual lamp 5084 before the input termination through contact 5086). For example, when being dragged by contact, the size and viewing perspective of the object do not change, and when the object lands at the final position in the augmented reality view, an object is displayed having a size and viewing perspective determined based on a physical position corresponding to the landing position of the virtual object shown in the field of view of the camera in the physical environment, such that according to determining the landing position as a first position in the field of view of the camera, the object has a second size, and according to determining the landing position as a second position in the field of view of the camera, the object has a third size different from the second size, wherein the second size and the third size are selected based on the distance between the landing position and one or more cameras. Displaying a representation of the virtual object with a changed size in response to detecting the termination of a second input for moving the virtual object (e.g., no further user input is required to adjust the size of the virtual object to keep the virtual object at a realistic size relative to the environment in the field of view of the camera) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0292] In some embodiments, according to determining that the movement of the second contact along a first path on the display satisfies a second criterion (e.g., at the end of the first path, the contact is within a threshold distance, or outside the edge of the display (e.g., bottom edge, top edge, and / or side edge) or the edge of a second user interface area), the device (842): stops displaying the second user interface area including a representation of the field of view of one or more cameras, and redisplay the (complete) first user interface area having a representation of the virtual object (e.g., if a part of the previous first user interface area was simultaneously displayed with the second user interface area, after the second user interface area is no longer displayed, the device displays the complete first user interface area). For example, in response to the movement of the contact 5054 dragging the virtual chair 5054 to the edge of the touch screen 112, as ​ shown, stops displaying the field of view 5034 of the camera, and redisplay the complete instant messaging user interface 5008, as ​ shown. In some embodiments, as the contact approaches the edge of the display or the edge of the second user interface area, the second user interface area fades out (e.g., as ​ shown), and / or the first user interface area (the part that was not displayed or blocked) fades in (e.g., as ​as shown). In some embodiments, the gesture for transitioning from a non-AR view (e.g., a first user interface region) to an AR view (e.g., a second user interface region) is the same as the gesture for transitioning from an AR view to a non-AR view. For example, a drag gesture on a virtual object that is beyond a threshold position in the currently displayed user interface (e.g., within a threshold distance of the boundary of the currently displayed user interface region or beyond the boundary of the currently displayed user interface region) causes a transition from the currently displayed user interface region to the corresponding user interface region (e.g., from displaying a first user interface region to displaying a second user interface region, or alternatively, from displaying a second user interface region to displaying a first user interface region). In some embodiments, a visual indication (e.g., fading out the currently displayed user interface region and fading in the corresponding user interface) is shown before the first / second criterion is met, and the visual indication is reversible if the input continues and the first / second criterion is not met before detecting the termination of the input (e.g., lifting off of the contact). Redisplaying the first user interface in response to detecting an input that meets the input criteria provides additional control options without cluttering the second user interface with additional displayed controls (e.g., controls for displaying the first user interface from the second user interface). Providing additional control options without cluttering the second user interface with additional displayed controls enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0293] In some embodiments, at a time corresponding to redisplaying the first user interface region, the device displays (844) an animated transition (e.g., movement, rotation about one or more axes, and / or scaling) of the representation of the virtual object from being displayed in the second user interface region to being displayed in the first user interface region (e.g., as shown by the animation of the virtual chair 5020 in ​ ). Displaying an animated transition of the representation of the virtual object from being displayed in the second user interface to being displayed in the first user interface (e.g., without requiring further user input to reposition the virtual object in the first user interface) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0294] In some embodiments, when a second contact moves along a first path, the device changes (846) the visual appearance of one or more corresponding planes identified in the field of view of one or more cameras (e.g., highlighting, marking, outlining, and / or otherwise visually altering the appearance of the one or more planes), the one or more corresponding planes corresponding to the current position of the contact. For example, when the contact 5042 moves along as ​When dragging the virtual chair 5020 along the paths indicated by the arrows 5042 and 5044 in [description], the floor surface 5038 is highlighted (e.g., compared to ​ before the contact 5042 moves). In some embodiments, in accordance with determining that the contact is at a position corresponding to a first plane detected in the field of view of the camera, the first plane is highlighted. In accordance with determining that the contact has moved to a position corresponding to a second plane detected in the field of view of the camera (e.g., as shown in ​ ), the highlighting of the first plane (e.g., the floor surface 5038) is stopped and the second plane (e.g., the tabletop 5046) is highlighted. In some embodiments, multiple planes are highlighted simultaneously. In some embodiments, the first plane among the multiple visually changed planes is visually changed in a manner different from the way other planes are visually changed to indicate that the contact is at a position corresponding to the first plane. Changing the visual appearance of one or more corresponding planes identified in the field of view of the camera provides the user with feedback indicating that the plane has been identified (e.g., a virtual object can be positioned relative to the plane). Providing improved visual feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0295] In some embodiments, in response to detecting a first input by contact, in accordance with determining that the first input by contact meets a third (e.g., staging user interface display) criterion (e.g., the staging user interface display criterion is a criterion configured to identify a swipe input, a touch-and-hold input, a press input, a tap input, or a force press with an intensity higher than a predefined intensity threshold), the device displays (848) a third user interface region on the display, which includes replacing at least a portion of the first user interface region (e.g., including replacing a 2D image of a virtual object with a 3D model of the virtual object). In some embodiments, when displaying the staging user interface (e.g., as referred to in ​When describing the staging user interface 6010, the device updates the appearance of the representation of the virtual object based on the detected input corresponding to the staging user interface (e.g., as described in more detail below with reference to method 900). In some embodiments, when another input is detected while the virtual object is displayed in the staging user interface and the input meets the criteria for transitioning to display a second user interface area, the device replaces the display of the staging user interface with the second user interface area while continuously displaying the virtual object. More details are described with respect to method 900. Displaying a third user interface based on determining that the first input meets a third criterion provides additional control options without cluttering the first user interface with additional displayed controls (e.g., controls for displaying the third user interface from the first user interface). Providing additional control options without cluttering the second user interface with additional displayed controls enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0296] In some embodiments, based on determining that a first input made by contact (e.g., a swipe input corresponding to scrolling a first user interface area or a tap input corresponding to a request to display a web page or email corresponding to content in the first user interface area) does not meet a first (e.g., AR trigger) criterion, the device maintains (850) the display of the first user interface area without replacing at least a portion of the display of the first user interface area with a representation of the field of view of one or more cameras (e.g., as described with reference to ​ described). Using the first criterion to determine whether to maintain the display of the first user interface area or continuously display the representation of the virtual object when replacing at least a portion of the display of the first user interface area with the field of view of one or more cameras enables a variety of different types of operations to be performed in response to the input. Enabling a variety of different types of operations to be performed in response to the input (e.g., by replacing at least a portion of the display of the user interface with the field of view of one or more cameras, or by maintaining the display of the first user interface area without replacing at least a portion of the display of the first user interface area with a representation of the field of view of one or more cameras) improves the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0297] It should be understood that for ​The specific order of operations described herein is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. Those of ordinary skill in the art will envision various ways to reorder the operations described herein. Additionally, it should be noted that the details of the other processes described herein with respect to other methods (e.g., methods 900 and 1000) apply in a similar manner to the method 800 described above with respect to ​ the method 800. For example, the contact, input, virtual object, user interface region, intensity threshold, haptic output, field of view, movement, and / or animation described above with reference to the method 800 optionally have one or more of the characteristics of the contact, input, virtual object, user interface region, intensity threshold, haptic output, field of view, movement, and / or animation described herein with reference to other methods (e.g., methods 900, 1000, 16000, 17000, 18000, 19000, and 20000). For the sake of brevity, these details are not repeated here.

[0298] ​ is a flowchart showing a method 900 according to some embodiments for displaying a first representation of a virtual object in a first user interface region, a second representation of the virtual object in a second user interface region, and a third representation of the virtual object showing a field of view of one or more cameras. The method 900 is performed at an electronic device (e.g., ​ the device 300 in ​ or the portable multifunctional device 100 in

[0299] As described below, method 900 involves detecting an input made by contact at a touch-sensitive surface of a device, the input being for displaying a representation of a virtual object in a first user interface (e.g., a two-dimensional graphical user interface). In response to the first input, the device uses criteria to determine whether to display a second representation of the virtual object in a second user interface (e.g., a staging user interface in which a three-dimensional representation of a movable virtual object, the three-dimensional representation of the virtual object can be resized and / or reoriented). When the second representation of the virtual object is displayed in the second user interface, in response to a second input, the device changes a display attribute of the second representation of the virtual object based on the second input, or displays a third representation of the virtual object in a third user interface that includes the field of view of one or more cameras of the device. Enabling a variety of different types of operations to be performed in response to an input (e.g., by changing the display attribute of the virtual object or displaying the virtual object in the third user interface) improves the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0300] The device displays (902) a first representation of a virtual object (e.g., a graphical representation of a three-dimensional object such as a virtual chair 5020, a virtual lamp 5084, a shoe, furniture, a hand tool, an ornament, a person, an emoji, a game character, virtual furniture, etc.) in a first user interface area on a display 112 (e.g., a two-dimensional graphical user interface or a portion thereof (e.g., a browsable list of furniture images, an image containing one or more selectable objects, etc.)). For example, the first user interface area is an instant messaging user interface 5008 as ​ shown. In some embodiments, in addition to an image of the physical environment surrounding the device, the first user interface area also includes a background (e.g., the background of the first user interface area is a preselected background color / pattern or a background image that is different from the output image simultaneously captured by one or more cameras and different from the real-time content in the field of view of one or more cameras).

[0301] When a first representation of a virtual object is displayed in a first user interface region on a display, the device detects (904) a first input by a first contact at a location on the touch-sensitive surface corresponding to the first representation of the virtual object on the display (e.g., a first contact is detected on the first representation of the virtual object on a touchscreen display, or a first contact is detected on an affordance representation (e.g., switch control 6018) that is simultaneously displayed in the first user interface region with the first representation of the virtual object, the affordance representation being configured to trigger the display of an AR view (e.g., field of view 6036 of a camera) and / or a staging user interface 6010 including a representation of the virtual object (e.g., virtual chair 5020) when invoked by the first contact). For example, the first input is an input made by contact 6006 as described with reference to ​ The input made by contact 6006.

[0302] In response to detecting the first input by the first contact and based on determining that the first input by the first contact meets a first (e.g., staging trigger) criterion (e.g., the staging trigger criterion is configured to identify a swipe input, a touch-and-hold input, a press input, a tap input, a down touch of a contact, an initial movement of a contact, or another type of predefined input gesture, and the staging trigger criterion is associated with triggering the activation of a camera and / or detecting a field plane in the field of view of the camera), the device displays (906) a second representation of the virtual object in a second user interface region that is different from the first user interface region (e.g., the second user interface region is the staging user interface 6010, which does not include the field of view of the camera and includes an analog three-dimensional space in which a three-dimensional representation of the virtual object can be manipulated (e.g., rotated or moved) in response to user input). For example, in ​ In accordance with determining that the input made by contact 6006 has a characteristic intensity that increases to above a deep press intensity threshold IT D The virtual chair object 5020 is displayed in the staging user interface 6010 (e.g., as ​ Shown), and the staging user interface is different from the instant messaging user interface 5008 (e.g., as ​ Shown).

[0303] In some embodiments, in response to detecting the first input and based on determining that the first input meets the staging trigger criterion, the device displays a first animation transition that shows a movement and reorientation from a first orientation as shown in the first user interface region (e.g., the first orientation of the virtual chair 5020 as shown in the instant messaging user interface 5008 in ​ To a second orientation (e.g., the second orientation of the virtual chair 5020 determined based on the gantry plane 6014, as ​6014).

[0304] While displaying the second representation of the virtual object in the second user interface area, the device detects (908) a second input (e.g., ​ In some embodiments, detecting the second input includes: detecting one or more second contacts at locations on the touch screen corresponding to a second representation of the virtual object; detecting a second contact on an affordance that is configured to trigger display of an augmented reality view of the physical environment surrounding the device when invoked by the second contact; detecting movement of the second contact; and / or detecting lift-off of the second contact. In some embodiments, the second input is a continuation of the first input made by the same contact (e.g., the second input is a continuation of the first input made by the same contact). ​ After the first input by contact 6006 is shown ​ The input is made through contact 6034 as shown (e.g., the contact is not lifted)), or it is a separate input made through an entirely different contact (e.g., the second input is made in the ​ After the first input by contact 6006 is shown ​ The input made by contact 6034 as shown (e.g., contact lift-off)), or the continuation of the input made by another contact (e.g., the second input is made in the ​ After the first input by contact 6006 is shown ​ For example, the second input may be a continuation of a swipe input, a second tap input, a second press input, a press input subsequent to the first input, a second touch-and-hold input, a continuous touch continuing from the first input, etc.

[0305] In response to detecting a second input (910): Based on determining that the second input corresponds to a request to manipulate a virtual object in a second user interface region (e.g., without transitioning to an augmented reality view), the device changes the display attributes of a second representation of the virtual object within the second user interface region based on the second input, and based on determining that the second input corresponds to a request to display the virtual object in an augmented reality environment, the device displays a third representation of the virtual object having a representation of the field of view of one or more cameras (e.g., the device displays a third user interface including the field of view 6036 of one or more cameras and places a three-dimensional representation of the virtual object (e.g., virtual chair 5020) on a virtual plane (e.g., floor surface 5038) detected within the field of view of the camera corresponding to a physical plane (e.g., the floor) in the physical environment 5002 around the device).

[0306] In some embodiments, the second input corresponding to a request to manipulate a virtual object in a second user interface region is a pinch or a swipe by a second contact at a location on the touch-sensitive surface corresponding to the second representation of the virtual object in the second user interface region. For example, the second input is an input made by contact 6006 as ​ shown or an input made by contacts 6026 and 6030 as ​ shown.

[0307] In some embodiments, the second input corresponding to a request to display a virtual object in an augmented reality environment is a tap input, a press input, or a touch-and-hold or press input and subsequent drag input at or from a location on the touch-sensitive surface corresponding to the representation of the virtual object in the second user interface region. For example, the second input is a deep press input made by contact 6034 as ​ shown.

[0308] In some embodiments, changing the display attributes of the second representation of the virtual object within the second user interface region based on the second input includes rotating about one or more axes (e.g., by vertical and / or horizontal swipes), resizing (e.g., pinching to resize), tilting about one or more axes (e.g., by tilting the device), changing the viewing angle (e.g., by horizontally moving the device, which in some embodiments is used to analyze the field of view of one or more cameras to detect one or more field-of-view planes), and / or changing the color of the representation of the virtual object. For example, changing the display attributes of the second representation of the virtual object includes, in response to a horizontal swipe gesture made by contact 6006 as ​ shown, rotating the virtual chair 5020; in response to a diagonal swipe gesture made by contact 6006 as ​ shown, rotating the virtual chair 5020; or in response to as​ The illustrated separation gesture performed by contacts 6026 and 6030 increases the size of the virtual chair 5020. In some embodiments, the amount of change in the displayed property of the second representation of the virtual object is associated with the amount of change in the property of the second input (e.g., the distance or speed of movement of the contact, the strength of the contact, the duration of the contact, etc.).

[0309] In some embodiments, the second input is determined to be consistent with the image in the augmented reality environment (e.g., in the field of view 6036 of one or more cameras, as shown in FIG. ​ In response to a request to display a virtual object as described above, the device displays a second animated transition that shows the virtual object being displayed from a corresponding orientation relative to the virtual plane on the display (e.g., ​ The virtual chair 5020 shown in FIG. 5 is reoriented to a third orientation (eg, ​ The three-dimensional representation of the virtual object is reoriented to a third orientation that is based on a current orientation of a portion of the physical environment captured in the field of view of the one or more cameras. For example, the three-dimensional representation of the virtual object is reoriented so that the three-dimensional representation of the virtual object is at a fixed angle relative to a predefined plane (e.g., floor surface 5038) identified in a real-time image of the physical environment 5002 captured in the field of view of the camera (e.g., a physical surface that can support the three-dimensional representation of the virtual object, such as a vertical wall or a horizontal floor surface). In some embodiments, in at least one aspect, the orientation of the virtual object in the augmented reality view is constrained by the orientation of the virtual object in the staging user interface. For example, when the virtual object is transitioned from the staging user interface to the augmented reality view, the rotation angle of the virtual object about at least one axis of the three-dimensional coordinate system is maintained (e.g., as shown with reference to ​ Described, as referenced ​ In some embodiments, the rotation of the virtual chair 5020 is maintained. In some embodiments, the light source projected on the representation of the virtual object in the second user interface area is a virtual light source. In some embodiments, the third representation of the virtual object in the third user interface area is illuminated by a real-world light source (e.g., as detected in and / or determined by the field of view of the one or more cameras).

[0310] In some embodiments, the first criterion includes (912) a criterion that is satisfied when, for example, as determined hereinafter, the first input includes a tap input made at a position corresponding to the virtual object indicator 5022 on the touch-sensitive surface (e.g., an indicator that overlaps and / or is proximate to a representation of a virtual object on the display, such as an icon). For example, the virtual object indicator 5022 provides an indication (e.g., as described in more detail hereinafter with reference to method 1000) that the virtual object corresponding to the virtual object indicator is visible in the staging view (e.g., the staging user interface 6010) and the augmented reality view (e.g., the field of view 6036 of the camera). Determining whether to display a second representation of the virtual object in the second user interface region based on whether the first input includes a tap input enables a variety of different types of operations to be performed in response to the first input. Enabling a variety of different types of operations to be performed in response to an input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0311] In some embodiments, the first criterion includes (914) a criterion that is satisfied when, for example, as determined hereinafter, the first contact is held at a position corresponding to the first representation of the virtual object on the touch-sensitive surface with a movement less than a threshold movement amount for at least a predefined threshold time amount (e.g., a long press time threshold). For example, the first criterion is satisfied by a touch-and-hold input. In some embodiments, the first criterion includes a criterion that requires moving the first contact after the first contact is held at a position corresponding to the representation of the virtual object on the touch-sensitive surface with a movement less than a threshold movement amount for at least a predefined threshold time amount in order to satisfy the criterion. For example, the first criterion is satisfied by a touch-and-hold input and a subsequent drag input. Determining whether to display a second representation of the virtual object in the second user interface region based on whether the contact is held at a position corresponding to the representation of the virtual object on the touch-sensitive surface with a movement less than a threshold movement amount for at least a predefined time amount enables a variety of different types of operations to be performed in response to the first input. Enabling a variety of different types of operations to be performed in response to an input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0312] In some embodiments, the first criterion includes (916) a criterion that is satisfied when, for example, as determined hereinafter, the characteristic intensity of the first contact increases above a first intensity threshold (e.g., a deep press intensity threshold IT D ) ​ As described, when the characteristic intensity of contact 6034 increases above the deep press intensity threshold IT DWhen the criteria are met, as indicated by the intensity level meter 5028. In some embodiments, based on determining that the contact meets the criteria for identifying another type of gesture (e.g., a tap), while maintaining the display of the virtual object, the device also performs another predefined function in addition to triggering the second (e.g., staging) user interface. In some embodiments, the first criteria requires that the first input not be a tap input (e.g., a forceful tap input where the intensity reaches above a threshold intensity before detecting a lift-off of the contact within a tap time threshold of the initial downward touch of the contact). In some embodiments, the first criteria includes a criteria that requires moving the first contact after the intensity of the first contact exceeds a first intensity threshold in order to meet the criteria. For example, the first criteria is met by a press input followed by a drag input. Whether to display the virtual object in the second user interface region is determined based on whether the characteristic intensity of the contact increases above the first intensity threshold such that a variety of different types of operations can be performed in response to the first input. Enabling a variety of different types of operations to be performed in response to an input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0313] In some embodiments, in response to detecting a first input via a first contact and based on determining that the first input via the first contact meets a second criteria (e.g., an interface scrolling criteria), the device scrolls (918) the first user interface region (and the representation of the virtual object) in a direction corresponding to the direction of movement of the first contact (e.g., the first criteria is not met and the representation of the virtual object in the second user interface region is not displayed), where the second criteria requires that the first input includes a movement of the first contact in a direction across the touch-sensitive surface that is greater than a threshold distance (e.g., the second criteria is met by a swipe gesture such as a vertical swipe or a horizontal gesture). For example, as referenced ​As described, an upward vertical swipe gesture via contact 6002 causes the instant messaging user interface 5008 and the virtual chair 5020 to scroll upward. In some embodiments, the first criterion further requires that the first input include movement of the first contact greater than a threshold distance in order to meet the first criterion, and the device determines whether the first input meets the first criterion (e.g., staging trigger criterion) or the second criterion (e.g., interface scroll criterion) based on whether an initial portion of the first input (e.g., a touch hold or press on a representation of a virtual object) meets an object selection criterion. In some embodiments, a swipe input initiated at a touch location outside the location of the virtual object and the AR icon of the virtual object meets the second criterion. Depending on whether the first input meets the second criterion, it is determined whether to scroll a first user interface region in response to the first input, which enables a variety of different types of operations to be performed in response to the first input. Enabling a variety of different types of operations to be performed in response to an input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0314] In some embodiments, in response to detecting a first input via a first contact and based on determining that the first input via the first contact meets a third (e.g., AR trigger) criterion, the device displays (920) a third representation of a virtual object having a representation of the field of view of one or more cameras. For example, as described with reference to ​ a long touch input via contact 6044 and a subsequent upward drag input of dragging the virtual chair 5020 via contact 6044 cause the field of view 6036 of the camera to display the virtual chair 5020.

[0315] In some embodiments, the third criterion includes criteria that are determined to be met, for example, based on: one or more cameras being active; the device orientation falling within a defined range (e.g., from a defined original orientation, a defined range of rotational angles about one or more axes); the input via contact including a selection input (e.g., a long touch) and a subsequent drag input (movement of the contact that moves a virtual object on the display) (e.g., moving within a range a predetermined distance from an edge of the display); the characteristic intensity of the contact increasing to be above an AR trigger intensity threshold (e.g., a light press threshold IT L or a deep press threshold IT D);The duration of the contact is increased to be greater than an AR trigger duration threshold (e.g., a long press threshold); and / or the distance by which the contact moves is increased to be greater than an AR trigger distance threshold (e.g., a long swipe threshold). In some embodiments, a control (e.g., a switching control 6018) for displaying a representation of a virtual object in a second user interface region (e.g., a staging user interface 6010) is displayed in a user interface (e.g., a third user interface region that replaces at least a portion of the second user interface region) that includes the representation of the virtual object and the field of view 6036 of one or more cameras.

[0316] In some embodiments, when transitioning directly from a first user interface region (e.g., a non-AR, non-staging, touchscreen UI view) to a third user interface region (e.g., an augmented reality view), the device displays an animated transition that shows a three-dimensional representation of a virtual object being reoriented from a corresponding orientation represented in the touchscreen UI (e.g., a non-AR, non-staging view) on the display to a predefined orientation relative to a current orientation of a portion of the physical environment captured in the field of view of one or more cameras. For example, as ​ shown, when transitioning directly from a first user interface region (e.g., an instant messaging user interface 5008, as ​ shown) to a third user interface region (e.g., an augmented reality user interface that includes the field of view 6036 of a camera, as ​ shown), the virtual chair 5020 changes from a first orientation as ​ shown to a predefined orientation relative to the floor surface 5038 in the physical environment 5002 as captured in the field of view 6036 of the camera (e.g., as ​ shown). For example, the three-dimensional representation of the virtual object is reoriented such that the three-dimensional representation of the virtual object is at a fixed angle relative to a predefined plane (e.g., a physical surface that can support the three-dimensional representation of the virtual object, such as a vertical wall or a horizontal floor surface (e.g., the floor surface 5038)) identified in a real-time image of the physical environment 5002. Whether to display a third representation of the virtual object with the field of view of the camera is determined in response to the first input based on whether the first input meets a third criterion, which enables multiple different types of operations to be performed in response to the first input. Enabling multiple different types of operations to be performed in response to an input improves the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0317] In some embodiments, in response to detecting a first input via a first contact, the device determines (922) the current device orientation (e.g., orientation relative to the physical environment surrounding the device) via one or more device orientation sensors, and a third criterion (e.g., an AR trigger criterion) requires the current device orientation to be within a first orientation range in order to satisfy the third criterion (e.g., when the angle between the device and the ground is less than a threshold angle, the second criterion is satisfied, which indicates that the device is sufficiently parallel to the ground (to bypass the gap state)). In some embodiments, a first criterion (e.g., a staging trigger criterion) requires the current device orientation to be within a second orientation range in order to satisfy the first criterion (e.g., when the angle between the device and the ground is within the threshold and up to 90 degrees, the first criterion is satisfied, which indicates that the device is sufficiently vertical relative to the ground to first enter the gap state). Depending on whether the device orientation is within the orientation range, it is determined whether to display a third representation of a virtual object having a field of view of a camera in response to the first input, which enables a variety of different types of operations to be performed in response to the first input. Enabling a variety of different types of operations to be performed in response to an input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0318] In some embodiments, at least one display attribute (e.g., size, shape, respective angles about the yaw, pitch, and roll axes, etc.) of the second representation of the virtual object is applied (924) to the third representation of the virtual object. For example, as described with reference to ​ when the third representation of the virtual chair 5020 is displayed in an augmented reality view including the field of view 6036 of the camera (e.g., as ​ shown), as described with reference to ​The rotation of the second representation of the virtual chair 5020 in the staging user interface 6010 for the described application is maintained. In some embodiments, in at least one aspect, the orientation of the virtual object in the augmented reality view is constrained by the orientation of the virtual object in the staging user interface. For example, when transitioning the virtual object from the staging view to the augmented reality view, the rotation angle of the virtual object about at least one axis of a predefined three-dimensional coordinate system (e.g., yaw, pitch, and roll axes) is maintained. In some embodiments, if the second representation of the virtual object has been manipulated in some way (e.g., changing size, shape, texture, orientation, etc.) by user input, at least one display property of the second representation of the virtual object is applied only to the third representation of the virtual object. In other words, changes made in the staging view are maintained when the object is shown in the augmented reality view or used to constrain the appearance of the object in the augmented reality view in one or more ways. Applying at least one display property of the second representation of the virtual object to the third representation of the virtual object (e.g., without requiring further user input to apply the same display property to the second representation and the third representation of the virtual object) enhances the operability of the device (e.g., by allowing the user to apply rotation to the second virtual object when the large version of the virtual object is displayed in the second user interface and to apply the rotation to the third representation of the virtual object shown in the field of view of one or more cameras), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0319] In some embodiments, in response to detecting at least an initial portion of a first input via a first contact (926) (e.g., including: detecting the first contact; or detecting an input via the first contact that meets a corresponding predefined criterion but does not meet a first criterion; or detecting an input that meets the first criterion): the device activates one or more cameras (e.g., activates the cameras without immediately displaying the field of view of the cameras on the display), and the device analyzes the field of view of the one or more cameras to detect one or more planes in the field of view of the one or more cameras. In some embodiments, the display of the field of view of the one or more cameras is delayed 6036 after activating the one or more cameras (e.g., until a second input corresponding to a request to display a virtual object in an augmented reality environment is detected, until at least one field of view plane is detected, or until a field of view plane corresponding to an anchor plane defined for the virtual object is detected). In some embodiments, the field of view of the one or more cameras is displayed 6036 at a time corresponding to the activation of the one or more cameras (e.g., simultaneously with the activation of the one or more cameras). In some embodiments, the field of view of the one or more cameras is displayed 6036 before a plane is detected in the field of view of the one or more cameras (e.g., in response to detecting a first input via a contact and based on a determination, the field of view of the one or more cameras is displayed). Activating the cameras and detecting one or more field of view planes by analyzing the field of view of the cameras in response to detecting an initial portion of the first input (e.g., before displaying a third representation of a virtual object having a representation of the field of view of the one or more cameras) improves the efficiency of the device (e.g., by reducing the amount of time required to determine the position and / or orientation of the third representation of the virtual object relative to the corresponding plane in the field of view of the cameras), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0320] In some embodiments, in response to detecting a corresponding plane (e.g., a floor surface 5038) in the field of view of the one or more cameras, a device having one or more haptic output generators 167 outputs (928) a haptic output that indicates the detection of the corresponding plane in the field of view of the one or more cameras. In some embodiments, the field of view 6036 may be shown before a field of view plane is recognized. In some embodiments, additional user interface controls and / or icons are overlaid on the real-world image in the field of view after at least one field of view plane is detected or after all field of view planes are recognized. Outputting a haptic output that indicates the detection of a plane in the field of view of the camera provides the user with feedback indicating that the plane has been detected. Providing improved haptic feedback enhances the operability of the device (e.g., by helping the user provide appropriate inputs and reducing unnecessary additional inputs for placing virtual objects), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0321] In some embodiments, the size of the third representation of the virtual object on the display is determined (930) based on the distance between the simulated real-world size of the virtual object and the position (e.g., the plane to which the virtual object is attached, such as the ground surface 5038) in the field of view 6036 of one or more cameras that has a fixed spatial relationship with the third representation of the virtual object (e.g., the virtual chair 5020). In some embodiments, the size of the third representation of the virtual object is constrained such that the ratio of the size of the third representation of the virtual object to the field of view of one or more cameras is maintained. In some embodiments, one or more physical size parameters (e.g., length, width, depth, and / or radius) are defined for the virtual object. In some embodiments, in a second user interface (e.g., the staging user interface), the virtual object is not constrained by its defined physical size parameters (e.g., the size of the virtual object can vary in response to user input). In some embodiments, the third representation of the virtual object is constrained by its defined size parameters. When a user input for changing the position of the virtual object relative to the physical environment represented in the augmented reality view is detected, or when a user input for changing the zoom level of the field of view is detected, or when a user input for moving relative to the physical environment around the device is detected, the appearance (e.g., size, viewing perspective) of the virtual object will change in a manner constrained by the fixed spatial relationship between the virtual object and the physical environment (e.g., as represented by the fixed spatial relationship between the anchoring plane of the virtual object and the plane in the augmented reality environment) and a fixed ratio based on the predefined size parameters of the virtual object and the actual size of the physical environment. Determining the size of the third representation of the virtual object based on the simulated real-world size of the virtual object and the distance between one or more cameras and the position in the field of view of the camera (e.g., no further user input is required to re-size the third representation of the virtual object to simulate the real-world size of t...

Claims

1. A method for displaying a representation of a virtual object, comprising: at a computer system having a display generation component, one or more input devices, and one or more cameras: displaying, via the display generation component, a representation of a virtual object in a first user interface area that includes a representation of the field of view of one or more cameras, wherein the display includes maintaining a first spatial relationship between the representation of the virtual object and a plane detected within a physical environment captured in the field of view of the one or more cameras; detecting movement of the computer system that adjusts the field of view of the one or more cameras; and in response to detecting movement of the computer system that adjusts the field of view of the one or more cameras: adjusting the display of the representation of the virtual object in the first user interface area according to the first spatial relationship between the virtual object and the plane detected within the field of view of the one or more cameras when adjusting the field of view of the one or more cameras, and generating a first alert based on determining that the movement of the computer system causes more than a threshold amount of the virtual object to move outside of a display portion of the field of view of the one or more cameras.

2. The method according to claim 1, wherein the computer system includes one or more audio output generators, and generating the first alert includes generating a first audio alert via the one or more audio output generators.

3. The method according to any one of claims 1 to 2, including generating audio associated with the virtual object after the movement of the computer system causes more than a threshold amount of the virtual object to move outside of a display portion of the field of view of the one or more cameras.

4. The method according to any one of claims 1 to 2, wherein outputting the first alert includes generating an audio output that indicates an amount of the virtual object that remains visible on a display portion of the field of view of the one or more cameras.

5. The method according to any one of claims 1 to 2, wherein outputting the first alert includes generating an audio output that indicates an amount of the display portion of the field of view covered by the virtual object.

6. The method according to any one of claims 1 to 2, wherein the one or more input devices include a touch-sensitive surface, and the method includes: detecting input by contact at a position on the touch-sensitive surface corresponding to the representation of the field of view of the one or more cameras; and in response to detecting the input and based on determining that the input is detected at a first position on the touch-sensitive surface corresponding to a first portion of the field of view of the one or more cameras not occupied by the virtual object, generating a second audio alert.

7. The method according to any one of claims 1 to 2, wherein outputting the first alert includes generating an audio output that indicates an operation performed with respect to the virtual object and a resulting state of the virtual object after performing the operation.

8. The method according to claim 7, wherein in the audio output of the first alert, the resulting state of the virtual object after performing the operation is described relative to a reference frame corresponding to the physical environment captured in the field of view of the one or more cameras.

9. The method according to any one of claims 1 to 2, comprising: detecting further movement of the computer system, the further movement further adjusting the field of view of the one or more cameras after generating the first alert; and in response to detecting the further movement of the computer system that further adjusts the field of view of the one or more cameras: adjusting the display of the representation of the virtual object in the first user interface region according to the first spatial relationship between the virtual object and the plane detected within the field of view of the one or more cameras when further adjusting the field of view of the one or more cameras, and, generating a second alert according to determining that the further movement of the computer system causes the virtual object to move into the display portion of the field of view of the one or more cameras by more than a second threshold amount.

10. The method according to claim 9, wherein the computer system includes one or more audio output generators, and generating the second alert includes generating a third audio alert via the one or more audio output generators.

11. The method according to any one of claims 1 to 2, comprising: when the representation of the virtual object is displayed in the first user interface region and a first object manipulation type among a plurality of object manipulation types applicable to the virtual object is currently selected for the virtual object, detecting a request to switch to another object manipulation type applicable to the virtual object; and in response to detecting the request to switch to another object manipulation type applicable to the virtual object, generating an audio output that speaks the name of a second object manipulation type among the plurality of object manipulation types applicable to the virtual object, wherein the second object manipulation type is different from the first object manipulation type.

12. The method according to claim 11, comprising: after generating the audio output that speaks the name of the second object manipulation type among the plurality of object manipulation types applicable to the virtual object, detecting a request to perform an object manipulation behavior corresponding to the currently selected object manipulation type; and in response to detecting the request to perform the object manipulation behavior corresponding to the currently selected object manipulation type, performing an object manipulation behavior corresponding to the second object manipulation type.

13. The method according to claim 11, wherein the one or more input devices include a touch-sensitive surface, and the method comprises: in response to detecting the request to switch to another object manipulation type applicable to the virtual object: generating an audio alert in combination with the audio output that speaks the name of the second object manipulation type to indicate that the second object manipulation type is a continuously adjustable manipulation type according to determining that the second object manipulation type is a continuously adjustable manipulation type; Detecting a request to perform an object manipulation behavior corresponding to the second object manipulation type, including detecting a swipe input at a position corresponding to a portion of the first user interface region that displays the representation of the field of view of the one or more cameras on the touch-sensitive surface; and In response to detecting the request to perform the object manipulation behavior corresponding to the second object manipulation type, performing the object manipulation behavior corresponding to the second object manipulation type with an amount corresponding to the magnitude of the swipe input.

14. The method according to any one of claims 1 to 2, comprising: Before displaying the representation of the virtual object in the first user interface region, displaying the representation of the virtual object in a second user interface region that does not include a representation of the field of view of the one or more cameras; When the representation of the virtual object is displayed in the second user interface region and a first operation among a plurality of operations applicable to the virtual object is currently selected for the virtual object, detecting a request to switch to another operation applicable to the virtual object; and In response to detecting the request to switch to another operation applicable to the virtual object in the second user interface region, generating an audio output that speaks the name of a second operation among the plurality of operations applicable to the virtual object, wherein the second operation is different from the first operation.

15. The method according to any one of claims 1 to 2, comprising: Before displaying the representation of the virtual object in the first user interface region: When displaying the representation of the virtual object in a second user interface region that does not include a representation of the field of view of the one or more cameras, detecting a request to display the representation of the virtual object in the first user interface region that includes a representation of the field of view of the one or more cameras; and In response to detecting the request to display the representation of the virtual object in the first user interface region that includes a representation of the field of view of the one or more cameras: Displaying the representation of the virtual object in the first user interface region according to a first spatial relationship between the representation of the virtual object and a plane detected in the physical environment captured in the field of view of the one or more cameras; and Generating a fourth audio alert that indicates that the virtual object is placed in the first user interface region relative to the physical environment captured in the field of view of the one or more cameras.

16. The method according to claim 15, wherein the fourth audio alert indicates information about the appearance of the virtual object relative to a displayed portion of the field of view of the one or more cameras.

17. The method according to claim 15, comprising: Generating a tactile output in combination with the placement of the virtual object in the first user interface region relative to the physical environment captured in the field of view of the one or more cameras.

18. The method according to any one of claims 1 to 2, wherein the one or more input devices include a touch-sensitive surface, and the method includes: Displaying a first control at a first position in the first user interface region while displaying a representation of the field of view of the one or more cameras; Stopping displaying the first control in the first user interface region while maintaining the display of the representation of the field of view of the one or more cameras in the first user interface region, based on determining that a fade-out criterion is met; When the first user interface region is being displayed but the first control is not being displayed in the first user interface region, detecting a touch input at a corresponding position on the touch-sensitive surface that corresponds to the first position in the first user interface region; And In response to detecting the touch input, generating a fifth audio alert that includes an audio output specifying an operation corresponding to the first control.

19. A computer system, comprising: A display generation component; One or more input devices; One or more cameras; One or more processors; And A memory that stores one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods according to claims 1 to 18.

20. A computer program comprising instructions that, when executed by a computer system having a display generation component, one or more input devices, and one or more cameras, cause the computer system to perform any of the methods according to claims 1 to 18.

21. A method, comprising: At a device having a display, a touch-sensitive surface, and one or more cameras: Displaying a representation of a virtual object in a first user interface region on the display; When the first representation of the virtual object is being displayed in the first user interface region on the display, detecting a first input by contact at a position on the touch-sensitive surface that corresponds to the representation of the virtual object on the display; In response to detecting the first input by the contact: Based on determining that the first input by the contact meets a first criterion: Displaying a second user interface region on the display, including replacing at least a portion of the display of the first user interface region with a representation of the field of view of the one or more cameras; And Continuously displaying the representation of the virtual object when switching from displaying the first user interface region to displaying the second user interface region.

22. A method, comprising: At a device having a display, a touch-sensitive surface, and one or more cameras: Displaying a first representation of a virtual object in a first user interface region on the display; When the first representation of the virtual object is being displayed in the first user interface region on the display, detecting a first input by a first contact at a position on the touch-sensitive surface that corresponds to the first representation of the virtual object on the display; In response to detecting the first input via the first contact and based on determining that the first input via the first contact meets a first criterion, display a second representation of the virtual object in a second user interface region that is different from the first user interface region; When displaying the second representation of the virtual object in the second user interface region, detect a second input; And In response to detecting the second input: Based on determining that the second input corresponds to a request to manipulate the virtual object in the second user interface region, change a display attribute of the second representation of the virtual object within the second user interface region based on the second input; And Based on determining that the second input corresponds to a request to display the virtual object in an augmented reality environment, display a third representation of the virtual object together with a representation of the field of view of the one or more cameras.

23. A method, comprising: At a device having a display generation component, one or more input devices, one or more cameras, and one or more pose sensors for detecting pose changes of the device including the one or more cameras: Receive a request to display an augmented reality view of a physical environment in a first user interface region that includes a representation of the field of view of the one or more cameras; In response to receiving the request to display the augmented reality view of the physical environment, display the representation of the field of view of the one or more cameras and, based on determining that the augmented reality view of the physical environment does not meet a calibration criterion, display a calibration user interface object that is dynamically animated based on movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes: When displaying the calibration user interface object, detect pose changes of the one or more cameras in the physical environment via the one or more pose sensors; and In response to detecting the pose changes of the one or more cameras in the physical environment, adjust at least one display parameter of the calibration user interface object based on the detected pose changes of the one or more cameras in the physical environment; When displaying the calibration user interface object that moves on the display based on the detected pose changes of the one or more cameras in the physical environment, detect that the calibration criterion is met; and In response to detecting that the calibration criterion is met, stop displaying the calibration user interface object.

24. A method, comprising: At a device having a display generation component and a touch-sensitive surface: Via the display generation component, display a first user interface region that includes user interface objects associated with a plurality of object manipulation behaviors, the plurality of object manipulation behaviors including a first object manipulation behavior performed in response to an input that meets a first gesture recognition criterion and a second object manipulation behavior performed in response to an input that meets a second gesture recognition criterion; When displaying the first user interface area, detect a first portion of an input to the user interface object, including detecting movement of one or more contacts on the touch-sensitive surface, and when the one or more contacts are detected on the touch-sensitive surface, evaluate the movement of the one or more contacts against both the first gesture recognition criterion and the second gesture recognition criterion; In response to detecting the first portion of the input, update an appearance of the user interface object based on the first portion of the input, including: Based on determining that the first portion of the input meets the first gesture recognition criterion before meeting the second gesture recognition criterion: Change the appearance of the user interface object based on the first portion of the input according to the first object manipulation behavior; and Update the second gesture recognition criterion by increasing a threshold of the second gesture recognition criterion; and Based on determining that the input meets the second gesture recognition criterion before meeting the first gesture recognition criterion: Change the appearance of the user interface object based on the first portion of the input according to the second object manipulation behavior; and Update the first gesture recognition criterion by increasing a threshold of the first gesture recognition criterion.

Images