Device, method and graphical user interface for system-level behavior of 3D models

Abstract

A computer system having a display generation component, one or more input devices, and one or more cameras, receives a request to display a virtual object in a first user interface area that includes a field of view of the one or more cameras. In response to the request, based on determining that the object placement criteria are not met, displaying the representation of the virtual object with a first set of visual attributes and a first orientation, the first orientation being independent of which portion of the physical environment is displayed in the field of view of the one or more cameras. Based on determining that the object placement criteria are met, displaying the representation of the virtual object with a second set of visual attributes and a second orientation, the second set of visual attributes being different from the first set of visual attributes, the second orientation corresponding to a plane.

Description

Technical Field

[0001] The present invention generally relates to electronic devices that display virtual objects, including but not limited to electronic devices that display virtual objects in various scenarios. Background Art

[0002] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices such as touch-sensitive surfaces for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Example touch-sensitive surfaces include touchpads, touch-sensitive remote controls, and touchscreen displays. Such surfaces are used to manipulate user interfaces on displays and objects therein. Exemplary user interface objects include digital images, videos, text, icons, and control elements (such as buttons), as well as other graphics.

[0003] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, using a series of inputs to orient and position virtual objects in an augmented reality environment is cumbersome, imposes a significant cognitive burden on the user, and detracts from the experience of the virtual / augmented reality environment. Furthermore, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the Invention

[0004] Therefore, there is a need for computer systems with improved methods and interfaces for interacting with virtual objects. Such methods and interfaces optionally supplement or replace conventional methods for interacting with virtual objects. Such methods and interfaces reduce the amount, extent, and / or nature of input from the user and produce a more efficient human-computer interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0005] The computer system of the present disclosure reduces or eliminates the above defects and other problems associated with interfaces for interacting with virtual objects (e.g., user interfaces for augmented reality (AR) and related non-AR interfaces). In some embodiments, the computer system includes a desktop computer. In some embodiments, the computer system is portable (e.g., a laptop, tablet computer, or handheld device). In some embodiments, the computer system includes a personal electronic device (e.g., a wearable electronic device, such as a watch). In some embodiments, the computer system has a touchpad (and / or communicates with a touchpad). In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display") (and / or communicates with a touch-sensitive display). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs or instruction sets stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI in part through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include playing games, image editing, drawing, displaying, word processing, spreadsheet making, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note taking and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0006] According to some embodiments, a method is performed at a computer system having a display, a touch-sensitive surface, and one or more cameras. The method includes displaying a representation of a virtual object in a first user interface area on the display. The method also includes, while displaying the first representation of the virtual object in the first user interface area on the display, detecting a first input by contact at a location on the touch-sensitive surface corresponding to the representation of the virtual object on the display. The method also includes, in response to detecting the first input by contact, based on determining that the first input by contact meets a first criterion: displaying a second user interface area on the display, which includes replacing the display of at least a portion of the first user interface area, the first user interface area having a representation of the field of view of the one or more cameras, and continuously displaying the representation of the virtual object when switching from displaying the first user interface area to displaying the second user interface area.

[0007] According to some embodiments, a method is performed at a computer system having a display, a touch-sensitive surface, and one or more cameras. The method includes displaying a first representation of a virtual object in a first user interface area on the display. The method also includes, while displaying the first representation of the virtual object in the first user interface area on the display, detecting a first input made by a first contact on the touch-sensitive surface at a location corresponding to the first representation of the virtual object on the display. The method also includes, in response to detecting the first input made by the first contact, and based on determining that the input made by the first contact meets a first criterion, displaying a representation of the virtual object in a second user interface area, the second user interface area being different from the first user interface area. The method also includes, while displaying a second representation of the virtual object in the second user interface area, detecting a second input, and in response to detecting the second input, based on determining that the second input corresponds to a request to manipulate the virtual object in the second user interface area, changing display properties of the second representation of the virtual object in the second user interface area based on the second input; and, based on determining that the second input corresponds to a request to display the virtual object in an augmented reality environment, displaying a third representation of the virtual object, the virtual object having a representation of the field of view of the one or more cameras.

[0008] According to some embodiments, a method is performed at a computer system having a display and a touch-sensitive surface. The method includes, in response to a request to display a first user interface, displaying a first user interface having a representation of a first item. The method also includes, based on a determination that the first item corresponds to a corresponding virtual three-dimensional object, displaying a representation of the first item, the representation of the first item having a visual indication indicating that the first item corresponds to a first corresponding virtual three-dimensional object. The method also includes, based on a determination that the first item does not correspond to a corresponding virtual three-dimensional object, displaying the representation of the first item without the visual indication. The method also includes, after displaying the representation of the first item, receiving a request to display a second user interface including a second item. The method also includes, in response to the request to display the second user interface, displaying the second user interface having a representation of the second item. The method also includes, based on a determination that the second item corresponds to a corresponding virtual three-dimensional object, displaying a representation of the second item, the representation of the second item having a visual indication indicating that the second item corresponds to a second corresponding virtual three-dimensional object. The method also includes, based on a determination that the second item does not correspond to a corresponding virtual three-dimensional object, displaying the representation of the second item without the visual indication.

[0009] According to some embodiments, a method is performed at a computer system having a display generation component, one or more input devices, and one or more cameras. The method includes receiving a request to display a virtual object in a first user interface area, the first user interface area including at least a portion of a field of view of the one or more cameras. The method also includes, in response to the request to display the virtual object in the first user interface area, displaying, via the display generation component, a representation of the virtual object over at least a portion of a field of view of the one or more cameras included in the first user interface area, wherein the field of view of the one or more cameras is a view of a physical environment within which the one or more cameras are located. Displaying the representation of the virtual object includes: based on determining that an object placement criterion is not met, displaying a representation of the virtual object having a first set of visual attributes and a first orientation, wherein the object placement criterion requires that a placement position of the virtual object be recognizable in the field of view of the one or more cameras in order to meet the object placement criterion, the first orientation being independent of a portion of the physical environment displayed in the field of view of the one or more cameras; and based on determining that the object placement criterion is met, displaying a representation of the virtual object having a second set of visual attributes and a second orientation, the second set of visual attributes being different from the first set of visual attributes, the second orientation corresponding to a plane in the physical environment detected in the field of view of the one or more cameras.

[0010] According to some embodiments, a method is performed at a computer system having a display generation component, one or more input devices, one or more cameras, and one or more gesture sensors for detecting a change in the gesture of a device including the one or more cameras. The method includes receiving a request to display an augmented reality view of a physical environment in a first user interface area, the first user interface area including a representation of a field of view of the one or more cameras. The method also includes, in response to receiving the request to display the augmented reality view of the physical environment, displaying the representation of the field of view of the one or more cameras, and, in response to determining that a calibration criterion for the augmented reality view of the physical environment is not met, displaying a calibration user interface object that dynamically animates based on movement of the one or more cameras in the physical environment, wherein displaying the calibration user interface object includes: detecting a change in the gesture of the one or more cameras in the physical environment via the one or more gesture sensors while displaying the calibration user interface object; and, in response to detecting the change in the gesture of the one or more cameras in the physical environment, adjusting at least one display parameter of the calibration user interface object based on the detected change in the gesture of the one or more cameras in the physical environment. The method also includes, while displaying the calibration user interface object that moves on the display based on the detected change in the gesture of the one or more cameras in the physical environment, detecting that the calibration criterion is met. The method also includes, in response to detecting that the calibration criterion is met, ceasing to display the calibration user interface object.

[0011] According to some embodiments, a method is performed at a computer system having a display generation component and one or more input devices including a touch-sensitive surface. The method includes displaying, by the display generation component, a representation of a first perspective of a virtual three-dimensional object in a first user interface area. The method also includes, while displaying the representation of the first perspective of the virtual three-dimensional object in the first user interface area on the display, detecting a first input corresponding to a request to rotate the virtual three-dimensional object relative to the display displaying a portion of the virtual three-dimensional object that is not visible from the first perspective of the virtual three-dimensional object. The method also includes, in response to detecting the first input: based on determining that the first input corresponds to a request to rotate the three-dimensional object about a first axis, rotating the virtual three-dimensional object about the first axis by an amount determined based on a magnitude of the first input, and the amount is constrained by a movement limit that limits the virtual three-dimensional object from rotating about the first axis by more than a threshold amount; and based on determining that the first input corresponds to a request to rotate the three-dimensional object about a second axis different from the first axis, rotating the virtual three-dimensional object about the second axis by an amount determined based on the magnitude of the first input, wherein for inputs having a magnitude above a corresponding threshold, the device rotates the virtual three-dimensional object about the second axis by more than the threshold amount.

[0012] According to some embodiments, a method is performed at a computer system having a display generation component and a touch-sensitive surface. The method includes displaying a first user interface area via the display generation component, the first user interface area including a user interface object associated with a plurality of object manipulation behaviors, the plurality of object manipulation behaviors including a first object manipulation behavior performed in response to an input that satisfies a first gesture recognition criterion and a second object manipulation behavior performed in response to an input that satisfies a second gesture recognition criterion. The method also includes, while displaying the first user interface area, detecting a first portion of the input involving the user interface object, including detecting movement of one or more contacts on the touch-sensitive surface, and when the one or more contacts are detected on the touch-sensitive surface, evaluating the movement of the one or more contacts relative to the first gesture recognition criterion and the second gesture recognition criterion. The method also includes, in response to detecting the first portion of the input, updating the appearance of the user interface object based on the first portion of the input, comprising: in accordance with determining that the first portion of the input satisfies the first gesture recognition criteria before satisfying the second gesture recognition criteria, changing the appearance of the user interface object in accordance with the first object manipulation behavior and based on the first portion of the input, and updating the second gesture recognition criteria by increasing a threshold value for the second gesture recognition criteria; and in accordance with determining that the input satisfies the second gesture recognition criteria before satisfying the first gesture recognition criteria, changing the appearance of the user interface object in accordance with the second object manipulation behavior based on the first portion of the input, and updating the first gesture recognition criteria by increasing the threshold value for the first gesture recognition criteria.

[0013] According to some embodiments, a method is performed at a computer system having a display generation component, one or more input devices, one or more audio output generators, and one or more cameras. The method includes displaying, via the display generation component, a representation of a virtual object in a first user interface area, the first user interface area including a representation of a field of view of the one or more cameras, wherein the display includes maintaining a first spatial relationship between the representation of the virtual object and a plane detected within a physical environment captured in the field of view of the one or more cameras. The method also includes detecting movement of a device that adjusts the field of view of the one or more cameras. The method also includes, in response to detecting movement of the device that adjusts the field of view of the one or more cameras: when adjusting the field of view of the one or more cameras, adjusting the display of the representation of the virtual object in the first user interface area according to the first spatial relationship between the virtual object and the plane detected within the field of view of the one or more cameras, and generating, via the one or more audio output generators, a first audio alert based on determining that the movement of the device causes the virtual object to move outside the displayed portion of the field of view of the one or more cameras and the movement exceeds a threshold amount.

[0014] According to some embodiments, an electronic device includes a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, optionally one or more posture sensors for detecting changes in posture, one or more processors, and a memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. According to some embodiments, a computer-readable storage medium has instructions stored therein that, when executed by an electronic device having a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting the intensity of contact with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors, cause the device to perform or cause the performance of any of the methods described herein. According to some embodiments, a graphical user interface on an electronic device having a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors, a memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements displayed in any of the methods described herein, the one or more elements being updated in response to input, as described in any of the methods described herein. According to some embodiments, the electronic device includes: a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors for detecting changes in posture; and means for performing or causing the performance of operations of any of the methods described herein.According to some embodiments, an information processing device used in an electronic device having a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors for detecting changes in posture includes a device for performing the operations of any method described herein or causing the operations of any method described herein to be performed.

[0015] Thus, improved methods and interfaces for displaying virtual objects in various scenarios are provided for electronic devices having display generation components, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more gesture sensors, thereby increasing the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can supplement or replace conventional methods for displaying virtual objects in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0017] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0018] Figure 1B is a block diagram illustrating example components for event handling according to some embodiments.

[0019] Figure 1C is a block diagram illustrating a tactile output module according to some embodiments.

[0020] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0021] Figure 3 is a block diagram of an example multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0022] Figure 4A An example user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0023] Figure 4B An example user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0024] Figures 4C to 4E Examples of dynamic intensity thresholds are shown according to some embodiments.

[0025] Figures 4F to 4K A set of sample tactile output patterns according to some embodiments are shown.

[0026] Figures 5A to 5AT An example user interface according to some embodiments is shown for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area.

[0027] Figures 6A to 6AJ An example user interface according to some embodiments is shown, which, according to some embodiments, is used to display a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object with a representation of a field of view of one or more cameras.

[0028] 7A to 7E 、 Figures 7F1 to 7F2 、 Figure 7G1 to Figure 7G2 as well as Figures 7H to 7P An example user interface is shown for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object according to some embodiments.

[0029] Figures 8A to 8E is a flowchart of a process for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area, according to some embodiments.

[0030] 9A to 9D is a flowchart of a process according to some embodiments for displaying a first representation of a virtual object in a first user interface area, displaying a second representation of the virtual object in a second user interface area, and displaying a third representation of the virtual object with a representation of a field of view of one or more cameras.

[0031] 10A to 10D is a flow diagram of a process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object, according to some embodiments.

[0032] Figures 11A to 11V An example user interface is shown for displaying virtual objects with different visual attributes depending on whether object placement criteria are met, according to some embodiments.

[0033] 12A to 12D、 Figure 12E-1 、 Figure 12E-2 、 Figure 12F-1 、 Figure 12F-2 、 Figure 12G-1 、 Figure 12G-2 、 Figure 12H-1 、 Figure 12H-2 、 Figure 12I-1 、 Figure 12I-2 、 Figure 12J 、 Figure 12K-1 、 Figure 12K-2 、 Figure 12L-1 and Figure 12L-2 An example user interface according to some embodiments is shown for displaying a calibration user interface object that is dynamically animated based on movement of one or more cameras of a device.

[0034] Figures 13A to 13M An example user interface for constraining the rotation of a virtual object around an axis is shown according to some embodiments.

[0035] Figures 14A to 14Z An example user interface is shown for increasing a second threshold movement magnitude required for a second object manipulation behavior based on determining that a first object manipulation behavior satisfies a first threshold movement magnitude according to some embodiments.

[0036] Figures 14AA to 14AD A flowchart is shown according to some embodiments, illustrating the operations of increasing a second threshold movement magnitude required for a second object manipulation behavior based on determining that a first object manipulation behavior satisfies a first threshold movement magnitude.

[0037] Figures 15A to 15AI An example user interface is shown for generating an audio alert based on determining that movement of the device causes a virtual object to move outside the field of view of one or more displayed device cameras in accordance with some embodiments.

[0038] Figures 16A to 16G is a flowchart of a process for displaying virtual objects with different visual attributes depending on whether object placement criteria are met, according to some embodiments.

[0039] 17A to 17D is a flowchart of a process according to some embodiments for displaying a calibration user interface object that is dynamically animated based on movement of one or more cameras of a device.

[0040] 18A to 18I is a flowchart of a process for constraining the rotation of a virtual object about an axis, according to some embodiments.

[0041] Figures 19A to 19His a flow chart of a process for increasing a second threshold movement magnitude required for a second object manipulation behavior based on a determination that a first object manipulation behavior satisfies a first threshold movement magnitude, according to some embodiments.

[0042] 20A to 20F is a flowchart of a process according to some embodiments for generating an audio alert based on determining that movement of the device causes a virtual object to move outside the displayed field of view of one or more device cameras. DETAILED DESCRIPTION

[0043] A virtual object is a graphical representation of a three-dimensional object in a virtual environment. Conventional methods of interacting with a virtual object to transition the virtual object from being displayed in the context of an application user interface (e.g., a two-dimensional application user interface that does not display an augmented reality environment) to being displayed in the context of an augmented reality environment (e.g., an environment in which the view of the physical world is augmented with supplemental information that provides the user with additional information not available in the physical world) typically require multiple independent inputs (e.g., a series of gestures and button presses, etc.) to achieve a desired result (e.g., adjusting the size, position, and / or orientation of the virtual object to achieve a realistic or desired appearance in the augmented reality environment). In addition, conventional input methods typically involve a delay between receiving a request to display the augmented reality environment and displaying the augmented reality environment due to the time required to activate one or more device cameras to capture a view of the physical world and / or the time required to analyze and characterize the view of the physical world relative to a virtual object that may be placed in the augmented reality environment (e.g., detecting planes and / or surfaces in the captured view of the physical world). Embodiments herein provide users with intuitive ways to display and / or interact with virtual objects in various scenarios (e.g., by allowing the user to provide input to switch from displaying a virtual object in the context of an application user interface to displaying a virtual object in an augmented reality environment; by allowing the user to change display properties of a virtual object before the virtual object is displayed in the augmented reality environment (e.g., in a three-dimensional staging environment); by providing indications that allow the user to easily identify system-level virtual objects from multiple applications; by changing the visual properties of an object when determining placement information for the object; by calibrating user interface objects by providing animations that indicate device movements required for calibration; by constraining rotation of a displayed virtual object about an axis; by increasing a threshold movement magnitude for a second object manipulation behavior when a threshold movement magnitude for a first object manipulation behavior is met; and by providing an audio alert indicating that a virtual object has moved out of a displayed field of view).

[0044] The systems, methods, and GUIs described herein improve user interface interactions with virtual / augmented reality environments in a variety of ways. For example, they make it easier to display virtual objects in an augmented reality environment and to adjust the appearance of virtual objects displayed in an augmented reality environment in response to various inputs.

[0045] under, Figures 1A to 1C 、 Figure 2 and Figure 3 A description of an example device is provided. Figures 4A to 4B 、 Figures 5A to 5AT 、 Figures 6A to 6AJ 、 Figures 7A to 7P 、 Figures 11A to 11V 、 Figure 12A To Figure 12L, Figures 13A to 13M 、 Figures 14A to 14Z as well as Figures 15A to 15AI Example user interfaces for displaying virtual objects in various scenarios are shown. Figures 8A to 8E A process for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area is shown. 9A to 9D A process is shown for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object with a representation of a field of view of one or more cameras. 10A to 10D A process for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object is shown. Figures 16A to 16G A process for displaying virtual objects with different visual attributes depending on whether object placement criteria are met is shown. 17A to 17D A process for displaying a calibration user interface object that dynamically animates based on movement of one or more cameras of a device is shown. 18A to 18I A process for constraining the rotation of a virtual object about an axis is shown. Figures 14AA to 14AD as well as Figures 19A to 19H A process is shown for increasing a second threshold movement magnitude required for a second object manipulation behavior based on a determination that a first object manipulation behavior satisfies a first threshold movement magnitude. 20A to 20F A process for generating an audio alert based on determining that movement of the device causes a virtual object to move outside of a displayed field of view of one or more device cameras is shown. Figures 5A to 5AT 、 Figures 6A to 6AJ 、 Figures 7A to 7P 、 Figures 11A to 11V 、 Figure 12A To Figure 12L, Figures 13A to 13M 、 Figures 14A to 14Z as well as Figures 15A to 15AI The user interface in Figures 8A to 8E 、 9A to 9D 、 10A to 10D 、 Figures 14AA to 14AD 、 Figures 16A to 16G 、 17A to 17D 、 18A to 18I 、 Figures 19A to 19H as well as 20A to 20F in the process.

[0046] Exemplary devices

[0047] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure various aspects of the embodiments.

[0048] It will also be understood that, although in some cases, the terms "first," "second," etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are simply used to distinguish one element from another. For example, a first contact can be named a second contact, and similarly, a second contact can be named a first contact without departing from the scope of the various described embodiments. Both a first contact and a second contact are contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0049] The terms used in the description of the various embodiments described herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a" and "the" are intended to also include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "and / or" used herein refer to and encompass any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "includes," "including," "comprises," and / or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, elements, and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0050] As used herein, the term "if" is optionally interpreted to mean "when" or "upon," or "in response to determining," or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0051] Embodiments of electronic devices, user interfaces for such devices, and processes associated with using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or music player functions. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices, such as laptops or tablets with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads), are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0052] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0053] The device typically supports a variety of applications, such as one or more of the following applications: a note-taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital camcorder application, a web browsing application, a digital music player application, and / or a digital video player application.

[0054] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as the touch-sensitive surface) optionally supports the various applications with a user interface that is intuitive and clear to the user.

[0055] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and sometimes simply as a touch-sensitive display. The device 100 includes memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, RF circuitry 108, audio circuitry 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input or control devices 116, and external ports 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting intensity of contacts on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0056] It should be understood that device 100 is merely one example of a portable multifunction device and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing circuits and / or application specific integrated circuits.

[0057] Memory 102 optionally includes high-speed random access memory and optionally also includes non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to memory 102 by other components of device 100 (such as one or more CPUs 120 and peripheral device interface 118) is optionally controlled by a memory controller 122.

[0058] Peripherals interface 118 may be used to couple the device's input and output peripherals to memory 102 and CPU(s) 120. Processor(s) 120 runs or executes various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0059] In some embodiments, peripherals interface 118, one or more CPUs 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0060] RF (radio frequency) circuitry 108 receives and transmits RF signals, also known as electromagnetic signals. RF circuitry 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via the electromagnetic signals. RF circuitry 108 optionally includes well-known circuitry for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuitry 108 optionally communicates with a network and other devices via wireless communications, such as the Internet (also known as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols not yet developed on the date of filing of this document.

[0061] The audio circuit 110, the speaker 111, and the microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset with both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0062] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes up / down buttons for volume control of the speaker 111 and / or microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206 in the ).

[0063] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some visual outputs or all of the visual outputs correspond to user interface objects. As used herein, the term "indicator" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object that is configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0064] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from the user based on tactile and / or haptic contact. The touch-sensitive display system 112 and the display controller 156 (together with any associated modules and / or instruction sets in the memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0065] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In some embodiments, projected mutual capacitance sensing technology is used, such as the ATmega 2000 from Apple Inc. (Cupertino, California). iPod and Technology found in.

[0066] The touch-sensitive display system 112 optionally has a video resolution exceeding 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi, or greater). The user optionally uses any suitable object or appendage, such as a stylus, finger, or the like, to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as precise as stylus-based input due to the larger contact area of ​​a finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the action desired by the user.

[0067] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112 or an extension of the touch-sensitive surface formed by the touch screen.

[0068] Device 100 also includes a power system 162 for powering the various components. Power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuitry, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0069] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The one or more optical sensors 164 receive light projected from the environment through one or more lenses and convert the light into data representing an image. In conjunction with the imaging module 143 (also called a camera module), the one or more optical sensors 164 optionally capture still images and / or video. In some embodiments, the optical sensor is located on the rear portion of the device 100 opposite the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for still image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for selfies, for video conferencing while the user views other video conference participants on the touch screen, etc.).

[0070] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor is shown coupled to a force sensor controller 159 in the I / O subsystem 106. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed with or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front portion of the device 100.

[0071] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0072] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1A A tactile output generator is shown coupled to the tactile feedback controller 161 in the I / O subsystem 106. In some embodiments, one or more tactile output generators 167 include one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components for converting electrical signals into tactile outputs on the device). The tactile output generators 167 receive tactile feedback generation instructions from the tactile feedback module 133 and generate tactile outputs on the device 100 that can be felt by the user of the device 100. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112) and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward toward the surface of the device 100) or laterally (e.g., back and forth in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on the back of the device 100 opposite the touch-sensitive display system 112 located on the front of the device 100.

[0073] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripherals interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, information is displayed on the touch screen display in a portrait view or a landscape view based on analysis of data received from the one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0074] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in Figures 1A and 1B. Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which application, if any, is currently active; a display state, which indicates what applications, views, or other information occupies various areas of the touch-sensitive display system 112; a sensor state, which includes information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information regarding the device's position and / or posture.

[0075] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OSX, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitates communication between various hardware and software components.

[0076] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device that is compatible with some of Apple Inc. (Cupertino, California). iPod A multi-pin (e.g., 30-pin) connector that is the same as, similar to, and / or compatible with the 30-pin connector used in iPod devices. In some embodiments, the external port is compatible with some of Apple Inc. (Cupertino, California) iPod A Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in iPod devices.

[0077] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there has been movement of the contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact break). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or stylus contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multi-finger contact). In some embodiments, contact / motion module 130 and display controller 156 detect contact on a touchpad.

[0078] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0079] In some embodiments, detecting a finger tap gesture depends on the length of time between detecting a finger press event and a finger lift event, but is independent of the intensity of the finger contact between the finger press event and the finger lift event. In some embodiments, a tap gesture is detected based on determining that the length of time between the finger press event and the finger lift event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the intensity of the finger contact during the tap reaches a given intensity threshold (greater than a nominal contact detection intensity threshold), such as a light press or deep press intensity threshold. Thus, a finger tap gesture can meet a specific input criterion that does not require the characteristic intensity of the contact to meet a given intensity threshold to meet the specific input criterion. For clarity, the finger contact in a tap gesture is generally required to meet a nominal contact detection intensity threshold to detect a finger press event, below which the contact is not detected. A similar analysis applies to detecting a tap gesture by a stylus or other contact. In cases where the device is capable of detecting contact by a finger or stylus hovering over the touch-sensitive surface, the nominal contact detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0080] The same concepts apply in a similar manner to other types of gestures. For example, a swipe gesture, a pinch gesture, an expand gesture, and / or a long press gesture may be optionally detected based on the satisfaction of a criterion that is independent of the intensity of the contacts included in the gesture or that does not require the contacts performing the gesture to reach an intensity threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; a zoom gesture is detected based on the movement of two or more contacts toward each other; a zoom gesture is detected based on the movement of two or more contacts away from each other; and a long press gesture is detected based on the duration of a contact on the touch-sensitive surface having less than a threshold amount of movement. Thus, the statement that a particular gesture recognition criterion does not require the intensity of the contacts to meet the corresponding intensity threshold in order to meet the particular gesture recognition criterion means that the particular gesture recognition criterion can be met when the contacts in the gesture do not meet the corresponding intensity threshold, and can also be met when one or more contacts in the gesture meets or exceeds the corresponding intensity threshold. In some embodiments, a tap gesture is detected based on determining that a finger down event and a finger up event are detected within a predefined time period, regardless of whether the contact is above or below a corresponding intensity threshold during the predefined time period, and a swipe gesture is detected based on determining that the contact moves by more than a predefined amount, even if the contact is above the corresponding intensity threshold at the end of the contact movement. Even in specific implementations where detection of gestures is affected by the intensity of the contact performing the gesture (e.g., the device detects a long press more quickly when the intensity of the contact is above an intensity threshold, or the device delays detection of a tap input when the intensity of the contact is higher), detection of these gestures does not require the contact to reach a particular intensity threshold (e.g., even if the amount of time required to recognize the gesture varies), as long as the criteria for recognizing the gesture can be met without the contact reaching the particular intensity threshold.

[0081] In some cases, contact intensity thresholds, duration thresholds, and movement thresholds are combined in various combinations to create heuristics that distinguish between two or more different gestures directed at the same input element or area, enabling multiple different interactions with the same input element to provide a richer set of user interactions and responses. The statement that a particular set of gesture recognition criteria does not require the intensity of a contact to meet a corresponding intensity threshold to meet the particular gesture recognition criteria does not preclude the simultaneous evaluation of other intensity-related gesture recognition criteria to identify other gestures whose criteria are met when the gesture includes a contact with an intensity above the corresponding intensity threshold. For example, in some cases, a first gesture recognition criterion for a first gesture (which does not require the intensity of the contact to meet the corresponding intensity threshold to meet the first gesture recognition criterion) competes with a second gesture recognition criterion for a second gesture (which depends on the contact meeting the corresponding intensity threshold). In such a competition, if the second gesture recognition criterion for the second gesture is met first, the gesture is optionally not recognized as meeting the first gesture recognition criterion for the first gesture. For example, if the contact reaches the corresponding intensity threshold before the contact moves a predefined amount, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contact moves a predefined amount before the contact reaches the corresponding intensity threshold, a swipe gesture is detected instead of a deep press gesture. Even in this case, the first gesture recognition criterion for the first gesture still does not require the intensity of the contact to meet the corresponding intensity threshold in order to satisfy the first gesture recognition criterion, because if the contact remains below the corresponding intensity threshold until the gesture ends (e.g., a swipe gesture with an intensity of the contact that does not increase above the corresponding intensity threshold), the gesture will be recognized as a swipe gesture by the first gesture recognition criterion. Therefore, a particular gesture recognition criterion that does not require the intensity of the contact to meet the corresponding intensity threshold in order to satisfy the particular gesture recognition criterion may (A) in some cases ignore the intensity of the contact relative to the intensity threshold (e.g., for a tap gesture) and / or (B) in some cases fail to satisfy the particular gesture recognition criterion (e.g., for a long press gesture) if a set of competing intensity-related gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-related gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input, and in this sense still depends on the intensity of the contact relative to the intensity threshold (e.g., for a long press gesture competing with a deep press gesture for recognition).

[0082] The graphics module 132 includes various known software components for rendering and displaying graphics on the touch-sensitive display system 112 or other display, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of the displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, but not limited to, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0083] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes specifying the graphics to be displayed from an application program, etc., along with coordinate data and other graphic attribute data, if necessary, and then generates screen image data for output to the display controller 156.

[0084] Haptic feedback module 133 includes various software components for generating instructions (e.g., instructions used by haptic feedback controller 161) to generate tactile output at one or more locations on device 100 using tactile output generator 167 in response to user interaction with device 100.

[0085] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0086] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as a weather widget, local pages widget, and map / navigation widget).

[0087] Application 136 optionally includes the following modules (or instruction sets), or a subset or superset thereof:

[0088] ● Contacts module 137 (sometimes called address book or contact list);

[0089] ● Telephone module 138;

[0090] ● Video conferencing module 139;

[0091] ● Email client module 140;

[0092] Instant messaging (IM) module 141;

[0093] ●Fitness support module 142;

[0094] A camera module 143 for still and / or video images;

[0095] Image management module 144;

[0096] ●Browser module 147;

[0097] ●Calendar module 148;

[0098] Widget module 149, which optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5 and other widgets obtained by the user, and user-created widgets 149-6;

[0099] A widget creator module 150 for forming a user-created widget 149-6;

[0100] ●Search module 151;

[0101] • Video and music player module 152, optionally consisting of a video player module and a music player module;

[0102] ●Notepad module 153;

[0103] ● Map module 154; and / or

[0104] ●Online video module 155.

[0105] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0106] In combination with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and so on.

[0107] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing the corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communication optionally uses any of a variety of communication standards, protocols, and technologies.

[0108] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and phone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0109] In conjunction with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0110] In conjunction with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for entering a character sequence corresponding to an instant message, modifying previously entered characters, sending the corresponding instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocols for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs), or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or Enhanced Messaging Service (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs, or IMPS).

[0111] In combination with the RF circuitry 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, the text input module 134, the GPS module 135, the map module 154, and the video and music player module 152, the fitness support module 142 includes executable instructions for creating a workout (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring a workout; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0112] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0113] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing still images and / or video images.

[0114] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching for, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0115] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0116] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes HTML (Hypertext Markup Language) files, CSS (Cascading Style Sheets) files, and JavaScript files. In some embodiments, the desktop widget includes XML (Extensible Markup Language) files and JavaScript files (e.g., the Yahoo! desktop widget).

[0117] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page into a desktop widget).

[0118] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0119] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player, such as an iPod (trademark of Apple Inc.).

[0120] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notes module 153 includes executable instructions for creating and managing notes, to-do lists, etc. according to user instructions.

[0121] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving directions; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0122] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats, such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0123] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above-mentioned modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0124] In some embodiments, device 100 is a device where operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touchpad. By using a touch screen and / or a touchpad as the primary input control device for operating device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on device 100 is optionally reduced.

[0125] A predefined set of functions that are exclusively performed through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to a main menu, home menu, or root menu. In such embodiments, the touchpad is used to implement a "menu button." In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0126] Figure 1B is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0127] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, the application 136-1 includes an application internal state 192 that indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0128] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating information being displayed by the application 136-1 or information ready for display by the application, a state queue for enabling the user to return to a previous state or view of the application 136-1, and a redo / undo queue of previous actions taken by the user.

[0129] Event monitor 171 receives event information from peripherals interface 118. The event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, accelerometer 168, and / or microphone 113 (through audio circuit 110). The information that peripherals interface 118 receives from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0130] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0131] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0132] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0133] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to programmatic levels within the application's programmatic or view hierarchy. For example, the lowest-level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined based at least in part on the hit view of the initial touch that started the touch-based gesture.

[0134] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies a hit view as the lowest view in the hierarchy where sub-events should be processed. In most cases, the hit view is the lowest-level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0135] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, views higher in the hierarchy will still remain actively participating views.

[0136] Event dispatcher module 174 dispatches event information to event recognizers (e.g., event recognizer 180). In embodiments that include active event recognizer determination module 173, event dispatcher module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event dispatcher module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0137] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0138] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events that occur within a corresponding view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, a corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more of event recognizers 180 are part of a separate module, such as a user interface toolkit (not shown) or a higher-level object from which application 136-1 inherits methods and other properties. In some embodiments, a corresponding event handler 190 includes one or more of the following: a data updater 176, an object updater 177, a GUI updater 178, and / or event data 179 received from an event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177, or GUI updater 178 to update the application's internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0139] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0140] The event receiver 182 receives event information from the event classifier 170. The event information includes information about sub-events such as touches or touch movements. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of a touch, the event information optionally also includes the rate and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another (e.g., from a portrait orientation to a landscape orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0141] The event comparator 184 compares the event information with a predefined event or sub-event definition and determines the event or sub-event based on the comparison, or determines or updates the state of the event or sub-event. In some embodiments, the event comparator 184 includes an event definition 186. The event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, the sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0142] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and the object that triggered the hit test.

[0143] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0144] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definitions 186, the corresponding event recognizer 180 enters the event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0145] In some embodiments, the corresponding event identifier 180 includes metadata 183 having configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery for actively participating event identifiers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event identifiers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0146] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating an event handler 190 is different from sending (and deferred sending) sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a flag associated with the identified event, and the event handler 190 associated with the flag receives the flag and performs a predefined process.

[0147] In some embodiments, event delivery instructions 188 include sub-event delivery instructions that deliver event information about a sub-event without activating an event handler. Instead, the sub-event delivery instructions deliver the event information to an event handler associated with the sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with the actively participating view receives the event information and performs a predetermined process.

[0148] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the location of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0149] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0150] It should be understood that the above discussion of event handling for user touches on a touch-sensitive display also applies to other forms of user input utilizing input devices to operate the multifunction device 100, and not all user input is initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touchpad, such as taps, drags, scrolls, etc.; stylus input; movement of the device; spoken commands; detected eye movement; biometric input; and / or any combination thereof, are optionally used as input corresponding to sub-events defining the event to be recognized.

[0151] Figure 1C is a block diagram illustrating a tactile output module according to some embodiments. In some embodiments, the I / O subsystem 106 (e.g., the tactile feedback controller 161 ( Figure 1A ) and / or other input controller 160 ( Figure 1A ))include Figure 1C At least some of the exemplary components shown. In some embodiments, the peripheral device interface 118 includes Figure 1C At least some of the example components shown.

[0152] In some embodiments, the tactile output module includes a tactile feedback module 133. In some embodiments, the tactile feedback module 133 aggregates and combines tactile output of user interface feedback from software applications on the electronic device (e.g., feedback in response to user input corresponding to the displayed user interface and prompts and other notifications indicating the execution of operations or the occurrence of events in the user interface of the electronic device). The tactile feedback module 133 includes one or more of a waveform module 123 (for providing waveforms for generating tactile output), a mixer 125 (for mixing waveforms, such as waveforms in different channels), a compressor 127 (for reducing or compressing the dynamic range of the waveform), a low-pass filter 129 (for filtering out high-frequency signal components in the waveform), and a thermal controller 131 (for adjusting the waveform according to thermal conditions). In some embodiments, the tactile feedback module 133 is included in a tactile feedback controller 161 ( Figure 1A In some embodiments, a separate unit of the tactile feedback module 133 (or a separate specific implementation of the tactile feedback module 133) is also included in the audio controller (e.g., audio circuit 110, Figure 1A ) and used to generate an audio signal. In some embodiments, a single tactile feedback module 133 is used to generate an audio signal and to generate a waveform for a tactile output.

[0153] In some embodiments, the haptic feedback module 133 also includes a trigger module 121 (e.g., a software application, operating system, or other software module that determines that a tactile output is to be generated and initiates a process for generating the corresponding tactile output). In some embodiments, the trigger module 121 generates a trigger signal for initiating (e.g., by the waveform module 123) the generation of a waveform. For example, the trigger module 121 generates the trigger signal based on a preset timing standard. In some embodiments, the trigger module 121 receives the trigger signal from outside the haptic feedback module 133 (e.g., in some embodiments, the haptic feedback module 133 receives the trigger signal from a hardware input processing module 146 located outside the haptic feedback module 133) and relays the trigger signal to other components within the haptic feedback module 133 (e.g., the waveform module 123) or a software application that triggers an operation (with the trigger module 121) based on activation of a user interface element (e.g., an application icon or an affordance within an application) or a hardware input device (e.g., a home button or an intensity-sensitive input surface, such as an intensity-sensitive touch screen). In some embodiments, the trigger module 121 also (e.g., from the haptic feedback module 133, Figure 1A and Figure 3 ) receives the tactile feedback generation instruction. In some embodiments, the trigger module 121 responds to the tactile feedback module 133 (or the trigger module 121 in the tactile feedback module 133) (e.g., from the tactile feedback module 133, Figure 1A and Figure 3 ) receives tactile feedback instructions and generates a trigger signal.

[0154] Waveform module 123 receives as input a trigger signal (e.g., from trigger module 121) and, in response to receiving the trigger signal, provides a waveform for generating one or more tactile outputs (e.g., a waveform selected from a predefined set of waveforms designated for use by waveform module 123, such as those described below with reference to Figures 4F-4G waveforms described in more detail).

[0155] The mixer 125 receives waveforms as input (e.g., from the waveform module 123) and mixes the waveforms together. For example, when the mixer 125 receives two or more waveforms (e.g., a first waveform in a first channel and a second waveform in a second channel that at least partially overlaps with the first waveform), the mixer 125 outputs a combined waveform corresponding to the sum of the two or more waveforms. In some embodiments, the mixer 125 also modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the rest of the two or more waveforms (e.g., by increasing the scale of the particular waveform and / or reducing the scale of the other waveforms in the waveforms). In some cases, the mixer 125 selects one or more waveforms to remove from the combined waveform (e.g., when waveforms from more than three sources have been requested to be output simultaneously by the tactile output generator 167, the waveform from the oldest source is discarded).

[0156] Mixer 127 receives waveforms (e.g., combined waveforms from mixer 125) as input and modifies these waveforms. In some embodiments, compressor 127 reduces these waveforms (e.g., based on the tactile output generator 167 ( Figure 1A ) or 357( Figure 3 ) such that the tactile output corresponding to these waveforms is reduced. In some embodiments, compressor 127 limits the waveforms, such as by imposing a predefined maximum amplitude on the waveforms. For example, compressor 127 reduces the amplitude of the waveform portion that exceeds a predefined amplitude threshold, while maintaining the amplitude of the waveform portion that does not exceed the predefined amplitude threshold. In some embodiments, compressor 127 reduces the dynamic range of the waveforms. In some embodiments, compressor 127 dynamically reduces the dynamic range of the waveforms such that the combined waveform remains within the performance specifications of tactile output generator 167 (e.g., force and / or movable mass displacement limits).

[0157] Low-pass filter 129 receives a waveform (e.g., a compressed waveform from compressor 127) as input and filters (e.g., smoothes) the waveform (e.g., removes or reduces high-frequency signal components in the waveform). For example, in some cases, compressor 127 may include extraneous signals (e.g., high-frequency signal components) in the compressed waveform that interfere with generating tactile output and / or exceed the performance specifications of tactile output generator 167 when generating tactile output based on the compressed waveform. Low-pass filter 129 reduces or removes such extraneous signals in the waveform.

[0158] Thermal controller 131 receives a waveform (e.g., a filtered waveform from low-pass filter 129) as input and adjusts the waveform based on the thermal condition of device 100 (e.g., based on an internal temperature detected within device 100, such as the temperature of haptic feedback controller 161, and / or an external temperature detected by device 100). For example, in some cases, the output of haptic feedback controller 161 varies based on temperature (e.g., in response to receiving the same waveform, haptic feedback controller 161 generates a first tactile output when haptic feedback controller 161 is at a first temperature and generates a second tactile output when haptic feedback controller 161 is at a second temperature different from the first temperature). For example, the magnitude (or amplitude) of the tactile output may vary based on temperature. To reduce the effects of temperature changes, the waveform is modified (e.g., the amplitude of the waveform is increased or decreased based on temperature).

[0159] In some embodiments, the tactile feedback module 133 (e.g., trigger module 121) is coupled to the hardware input processing module 146. In some embodiments, Figure 1A Other input controllers 160 in the embodiment include a hardware input processing module 146. In some embodiments, the hardware input processing module 146 receives input from a hardware input device 145 (e.g., Figure 1A In some embodiments, the hardware input device 145 is any input device described herein, such as a touch-sensitive display system 112 ( Figure 1A )、Keyboard / Mouse 350( Figure 3 )、Touchpad 355( Figure 3 ), one of the other input or control devices 116 ( Figure 1A ) or a strength-sensitive home button. In some embodiments, the hardware input device 145 consists of a strength-sensitive home button rather than a touch-sensitive display system 112 ( Figure 1A )、Keyboard / Mouse 350( Figure 3 ) or touchpad 355( Figure 3). In some embodiments, in response to input from a hardware input device 145 (e.g., an intensity-sensitive home button or a touch screen), the hardware input processing module 146 provides one or more trigger signals to the tactile feedback module 133 to indicate that user input that meets predefined input criteria has been detected, such as input corresponding to a home button "click" (e.g., a "press click" or a "release click"). In some embodiments, the tactile feedback module 133 provides a waveform corresponding to a home button "click" in response to the input corresponding to the home button "click", thereby simulating the tactile feedback of pressing a physical home button.

[0160] In some embodiments, the tactile output module includes a tactile feedback controller 161 (e.g. Figure 1A 145 ). In some embodiments, the tactile feedback controller 161 is coupled to a plurality of tactile output generators and selects one or more tactile output generators from the plurality of tactile output generators and sends a waveform to the selected one or more tactile output generators for generating a tactile output. In some embodiments, the tactile feedback controller 161 coordinates a tactile output request corresponding to activation of the hardware input device 145 and a tactile output request corresponding to a software event (e.g., a tactile output request from the tactile feedback module 133), and modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the remaining waveforms of the two or more waveforms (e.g., by increasing the scale of the particular waveform and / or reducing the scale of the remaining waveforms of the waveforms to prioritize the tactile output corresponding to activation of the hardware input device 145 over the tactile output corresponding to the software event).

[0161] In some embodiments, as Figure 1C As shown, the output of the tactile feedback controller 161 is coupled to the audio circuit of the device 100 (eg, the audio circuit 110, Figure 1A ) and provides the audio signal to the audio circuitry of device 100. In some embodiments, haptic feedback controller 161 provides both a waveform for generating a tactile output and an audio signal for providing an audio output along with generating the tactile output. In some embodiments, haptic feedback controller 161 modifies the audio signal and / or the waveform (for generating the tactile output) so that the audio output and the tactile output are synchronized (e.g., by delaying the audio signal and / or the waveform). In some embodiments, haptic feedback controller 161 includes a digital-to-analog converter for converting the digital waveform into an analog signal, which is received by amplifier 163 and / or tactile output generator 167.

[0162] In some embodiments, the tactile output module includes an amplifier 163. In some embodiments, the amplifier 163 receives a waveform (e.g., from the tactile feedback controller 161), amplifies the waveform, and then sends the amplified waveform to the tactile output generator 167 (e.g., the tactile output generator 167 ( Figure 1A ) or 357( Figure 3 )). For example, amplifier 163 amplifies the received waveform to a signal level that meets the physical specifications of tactile output generator 167 (e.g., to a voltage and / or current required by tactile output generator 167 to generate a tactile output such that the signal sent to tactile output generator 167 generates a tactile output corresponding to the waveform received from haptic feedback controller 161) and sends the amplified waveform to tactile output generator 167. In response, tactile output generator 167 generates a tactile output (e.g., by displacing the movable mass back and forth in one or more dimensions relative to the neutral position of the movable mass).

[0163] In some embodiments, the tactile output module includes a sensor 169 coupled to the tactile output generator 167. The sensor 169 detects the state or state change (e.g., mechanical position, physical displacement, and / or movement) of the tactile output generator 167 or one or more components of the tactile output generator 167 (e.g., one or more moving components for generating a tactile output, such as a membrane). In some embodiments, the sensor 169 is a magnetic field sensor (e.g., a Hall effect sensor) or other displacement and / or motion sensor. In some embodiments, the sensor 169 provides information (e.g., the position, displacement, and / or movement of one or more components in the tactile output generator 167) to the tactile feedback controller 161, and, based on the information about the state of the tactile output generator 167 provided by the sensor 169, the tactile feedback controller 161 adjusts the waveform output from the tactile feedback controller 161 (e.g., the waveform optionally sent to the tactile output generator 167 via the amplifier 163).

[0164] Figure 2 A device with a touch screen (e.g., Figure 1AIn some embodiments, the portable multifunction device 100 includes a touch-sensitive display system 112 (e.g., a touch screen). The touch screen optionally displays one or more graphics within a user interface (UI) 200. In these embodiments, and in other embodiments described below, a user can select one or more of the graphics by, for example, making a gesture on the graphics using one or more fingers 202 (not drawn to scale in the figures) or one or more styluses 203 (not drawn to scale in the figures). In some embodiments, selection of the one or more graphics occurs when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down), and / or rolling of a finger that has made contact with the device 100 (from right to left, from left to right, up and / or down). In some implementations or in some cases, inadvertent contact with a graphic does not select the graphic. For example, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application when the gesture corresponding to selection is a tap.

[0165] The device 100 optionally also includes one or more physical buttons, such as a "home" button, or a menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0166] In some embodiments, the device 100 includes a touch screen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for powering the device on / off and for locking the device, a volume adjustment button 208, a subscriber identity module (SIM) card slot 210, a headset jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by pressing the button and releasing the button before the predefined time interval has passed; and / or to unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through the microphone 113. The device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112 and / or one or more tactile output generators 167 for generating tactile output for the user of the device 100.

[0167] Figure 3is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. The device 300 does not have to be portable. In some embodiments, the device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a gaming system, or a control device (e.g., a home controller or an industrial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. The device 300 includes an input / output (I / O) interface 330 having a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above referenced devices). Figure 1A The one or more tactile output generators 167 described above), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or sensors similar to those described above) Figure 1A The memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the one or more CPUs 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ), or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100( Figure 1A )'s memory 102 optionally does not store these modules.

[0168] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device.Each module in the above-mentioned identified module corresponds to a group of instructions for performing the functions described above.The above-mentioned identified module or program (that is, instruction set) need not be implemented as independent software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments.In some embodiments, memory 370 optionally stores the subset of above-mentioned modules and data structure.In addition, memory 370 optionally stores other modules and data structure not described above.

[0169] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0170] Figure 4A An example user interface 400 of an applications menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0171] ● one or more signal strength indicators of one or more wireless communications (such as cellular signals and Wi-Fi signals);

[0172] ● Time;

[0173] Bluetooth indicator;

[0174] Battery status indicator;

[0175] A tray 408 with common application icons, such as:

[0176] o An icon 416 labeled "Phone" for the phone module 138, which optionally includes an indicator 414 of the number of missed calls or voice messages;

[0177] o An icon 418 of the email client module 140 labeled "Mail," which optionally includes an indicator 410 of the number of unread emails;

[0178] o An icon 420 labeled "Browser" for the browser module 147; and

[0179] o An icon 422 labeled "Music" for the video and music player module 152; and

[0180] Icons for other applications, such as:

[0181] o Icon 424 labeled "Messages" of the IM module 141;

[0182] o An icon 426 labeled “Calendar” of the calendar module 148;

[0183] o Icon 428 labeled "Photos" of the image management module 144;

[0184] o An icon 430 labeled “Camera” of the camera module 143;

[0185] ○ An icon 432 labeled “Online Video” of the online video module 155;

[0186] ○ Icon 434 labeled “Stock Market” of the Stock Market widget 149-2;

[0187] o An icon 436 labeled “Map” of the map module 154;

[0188] ○ Icon 438 labeled “Weather” of the weather widget 149-1;

[0189] ○ Icon 440 labeled “Clock” of the alarm clock widget 149-4;

[0190] o An icon 442 labeled “Fitness Support” of the fitness support module 142;

[0191] o An icon 444 of the Notes module 153 labeled "Notes"; and

[0192] o An icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136 .

[0193] It should be noted that Figure 4A The icon labels shown in the are merely exemplary. For example, other labels are optionally used for various application icons. In some embodiments, the label of a respective application icon includes the name of the application corresponding to the respective application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0194] Figure 4B A touch-sensitive surface 451 (e.g., Figure 3 a device (e.g., a tablet or touchpad 355) Figure 3 Although many of the examples that follow will be given with reference to input on the touch screen display 112 (where the touch-sensitive surface and the display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects contact with the touch-sensitive surface 451 at a location that corresponds to a corresponding location on the display (e.g., Figure 4B 460 and 462 in (e.g., in Figure 4B , 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B When the user interface 450 in FIG. 4 is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0195] In addition, although the following examples are primarily given with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). For another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting the contact, followed by ceasing to detect the contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or mice and finger contacts are optionally used simultaneously.

[0196] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that the touch-sensitive surface 451 in the display is included, the touch-sensitive surface 451 can be used to directly interact with the user interface elements on the touch-screen display. Figure 1A touch-sensitive display system 112 or Figure 4AIn some implementations (such as touch screens in a touchscreen display), a contact detected on the touchscreen acts as a "focus selector," such that when input (e.g., a press input by a contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, the particular user interface element is adjusted based on the detected input. In some implementations, focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touchscreen display (e.g., by using a tab key or arrow keys to move focus from one button to another); in these implementations, the focus selector moves based on the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is typically a user interface element (or contact on the touchscreen display) that is controlled by the user to convey the user's desired interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display).

[0197] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or to a surrogate (surrogate) for the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, the force measurements from multiple force sensors are combined (e.g., weighted averaged or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether an intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). Using the intensity of the contact as an attribute of the user input allows the user to access additional device functions that the user cannot easily access on a smaller device with limited real estate for displaying indications (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical controls / mechanical controls such as knobs or buttons).

[0198] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some embodiments, the user of the device is provided with a software setting for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0199] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of the contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact begins to move, before contact ends, before or after contact is detected to increase in intensity, and / or before or after contact is detected to decrease in intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, a value generated by low-pass filtering the contact intensity over a predefined time period or starting from a predefined time, etc. In some embodiments, the duration of the contact is used in determining the feature strength (e.g., when the feature strength is the average of the strength of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first strength threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second strength threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more strength thresholds is used to determine whether to perform one or more operations (e.g., whether to execute the corresponding option or give up performing the corresponding operation), rather than to determine whether to perform the first operation or the second operation.

[0200] In some embodiments, a portion of a gesture is identified for use in determining the characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position and reaches an ending position (e.g., a drag gesture) where the strength of the contact increases. In this embodiment, the characteristic strength of the contact at the ending position may be based on only a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the ending position). In some embodiments, a smoothing algorithm may be applied to the strength of the swipe gesture before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the intensity of the swipe contact to achieve the purpose of determining the characteristic strength.

[0201] The user interface diagrams described herein optionally include various intensity diagrams that illustrate the intensity of contacts on the touch-sensitive surface relative to one or more intensity thresholds (e.g., contact detection intensity threshold IT0, light press intensity threshold IT1, and contact detection intensity threshold IT2). L , deep press intensity threshold IT D (For example, at least initially above IT L ) and / or one or more other intensity thresholds (e.g., L Low intensity threshold IT H )). This intensity graph is not typically part of the displayed user interface, but is provided to help explain the graph. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact is detected having a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold IT0, contacts below the nominal contact detection intensity threshold are no longer detected), the device will move the focus selector according to the movement of the contact on the touch-sensitive surface without performing the operation associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface figures.

[0202] In some embodiments, the device's response to an input detected by the device depends on criteria based on the intensity of the contact during the input. For example, for some "light press" inputs, the intensity of the contact exceeding a first intensity threshold during the input triggers a first response. In some embodiments, the device's response to an input detected by the device depends on criteria that include both the intensity of the contact during the input and a time-based criterion. For example, for some "deep press" inputs, the intensity of the contact exceeding a second intensity threshold, which is greater than the first intensity threshold for a light press, during the input triggers a second response, as long as a delay time elapses between the first intensity threshold being met and the second intensity threshold being met. The duration of this delay time is typically less than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second intensity threshold, where the delay time increases as the second intensity threshold increases). This delay time helps avoid accidentally identifying a deep press input. As another example, for some "deep press" inputs, a period of reduced sensitivity occurs after the first intensity threshold is reached. During this period of reduced sensitivity, the second intensity threshold increases. This temporary increase in the second intensity threshold also helps avoid accidental deep press inputs.For other deep press inputs, the response to detecting the deep press input does not depend on time-based criteria.

[0203] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors, such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Exemplary factors are described in U.S. patent application serial numbers 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0204] For example, Figure 4C A dynamic intensity threshold 480 is shown that varies over time based in part on the intensity of the touch input 476 over time. The dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predefined delay time p1 from when the touch input 476 is initially detected, and a second component 478 that tracks the intensity of the touch input 476 over time. The initial high intensity threshold of the first component 474 reduces accidental triggering of a "deep press" response while still allowing an immediate "deep press" response if the touch input 476 provides sufficient intensity. The second component 478 reduces the unintentional triggering of a "deep press" response through gradual intensity fluctuations of the touch input. In some embodiments, when the touch input 476 satisfies the dynamic intensity threshold 480 (e.g., at Figure 4C ), triggering a “deep press” response.

[0205] Figure 4D Another dynamic intensity threshold 486 (eg, intensity threshold I D ). Figure 4D Two other intensity thresholds are also shown: a first intensity threshold I H and the second intensity threshold I L .exist Figure 4D Although the touch input 484 meets the first intensity threshold I before time p2, H and the second intensity threshold I L , but no response is provided until a delay time p2 has elapsed at time 482. Similarly, Figure 4D In the example, the dynamic intensity threshold 486 decays over time, wherein the decay is from time 482 (triggering the second intensity threshold I L This type of dynamic intensity threshold reduction begins at time 488 after the trigger is triggered with a lower threshold intensity (such as the first intensity threshold I H Or the second intensity threshold I L ) is triggered after or simultaneously with the response associated with the dynamic intensity threshold I D The associated response.

[0206] Figure 4E Yet another dynamic intensity threshold 492 (eg, intensity threshold I D ).exist Figure 4E In the example, after a delay time p2 has passed since the touch input 490 was initially detected, the trigger is triggered with the intensity threshold I L At the same time, the dynamic intensity threshold 492 decays after a predefined delay time p1 has passed since the touch input 490 was initially detected. L The associated response then reduces the intensity of the touch input 490, and then increases the intensity of the touch input 490 without releasing the touch input 490, which may trigger an action that is consistent with the intensity threshold I. D The associated response (e.g., at time 494) is performed even when the intensity of touch input 490 is below another intensity threshold (e.g., intensity threshold I L ) is also the case.

[0207] The contact characteristic strength is lower than the light press strength threshold IT L The intensity increases to between the light press intensity threshold IT L and deep press intensity threshold IT D The intensity between the deep press intensity threshold IT is sometimes referred to as a "light press" input. D The intensity of the deep press increases to above the deep press intensity threshold ITD The intensity of the contact feature increases from an intensity below the contact detection intensity threshold IT0 to an intensity between the contact detection intensity threshold IT0 and the light press intensity threshold IT0. L A decrease in the characteristic intensity of a contact from an intensity above the contact detection intensity threshold IT0 to an intensity below the contact detection intensity threshold IT0 is sometimes referred to as detecting liftoff of the contact from the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some illustrations, shaded circles or ellipses are used to represent the intensity of contacts on the touch-sensitive surface. In some illustrations, unshaded circles or ellipses are used to represent corresponding contacts on the touch-sensitive surface without specifying the intensity of the corresponding contacts.

[0208] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting an increase in the strength of the contact (or multiple contacts) to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting an increase in the strength of the corresponding contact to above the press input strength threshold (e.g., the corresponding operation is performed on a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and a subsequent decrease in the strength of the contact to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the press input threshold (e.g., the corresponding operation is performed on an "up stroke" of the corresponding press input).

[0209] In some embodiments, the device employs intensity hysteresis to avoid unintended input, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases below the hysteresis intensity threshold (e.g., performing the corresponding operation on an "up stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact strength increases from an strength equal to or less than a hysteresis strength threshold to an strength equal to or greater than a press input strength threshold and, optionally, the contact strength subsequently decreases to an strength equal to or less than the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact strength or a decrease in contact strength, depending on the circumstances).

[0210] For ease of explanation, descriptions of operations performed in response to a press input associated with a press input intensity threshold, or in response to a gesture including a press input, are optionally triggered in response to detecting: the intensity of the contact increasing above the press input intensity threshold, the intensity of the contact increasing from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, the intensity of the contact decreasing below the press input intensity threshold, or the intensity of the contact decreasing below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where operations are described as being performed in response to detecting a decrease in the intensity of the contact below the press input intensity threshold, the operations are optionally performed in response to detecting a decrease in the intensity of the contact below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold. As described above, in some embodiments, triggering of these operations is also dependent on satisfying a time-based criterion (e.g., a delay time having elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold).

[0211] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device that will be detected by a user using the user's sense of touch. For example, when a device or a component of the device is in contact with a surface that is touch-sensitive to a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click," even when the physical actuation button associated with the touch-sensitive surface that was physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such interpretation of touch by the user will be limited by the user's individualized sensory perception, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the tactile output generated corresponds to a physical displacement of the device or a component thereof that would generate the sensory perception of a typical (or average) user. Providing tactile feedback to the user using tactile output enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0212] In some embodiments, the tactile output pattern specifies characteristics of the tactile output, such as the amplitude of the tactile output, the shape of the motion waveform of the tactile output, the frequency of the tactile output, and / or the duration of the tactile output.

[0213] When a device (e.g., one or more tactile output generators that generate tactile output via movement of a movable mass) generates tactile output with different tactile output patterns, the tactile output can produce different tactile sensations in a user holding or touching the device. While a user's senses are based on their perception of the tactile output, most users will be able to discern changes in the waveform, frequency, and amplitude of the tactile output generated by the device. Therefore, the waveform, frequency, and amplitude can be adjusted to indicate to the user that different operations have been performed. Thus, tactile output with tactile output patterns designed, selected, and / or arranged to simulate the characteristics (e.g., size, material, weight, stiffness, smoothness, etc.); behavior (e.g., oscillation, displacement, acceleration, rotation, extension, etc.); and / or interaction (e.g., collision, adhesion, repulsion, attraction, friction, etc.) of objects in a given environment (e.g., a user interface including graphical features and objects, a simulated physical environment with virtual boundaries and virtual objects, a real physical environment with physical boundaries and physical objects, and / or any combination of the foregoing) can, in some cases, provide helpful feedback to the user, reducing input errors and improving the user's efficiency in operating the device. In addition, tactile output is optionally generated to correspond to feedback that is unrelated to the simulated physical properties (such as input threshold or object selection). Such tactile output will provide helpful feedback to the user in some cases, which reduces input errors and improves the efficiency of the user's operation of the device.

[0214] In some embodiments, a tactile output with a suitable tactile output pattern serves as a prompt for an event of interest in the user interface or behind the screen in the device. Examples of events of interest include activation of an affordance (e.g., a real or virtual button or toggle switch) provided on the device or in the user interface, the success or failure of a requested operation, reaching or crossing a boundary in the user interface, entering a new state, switching input focus between objects, activating a new mode, reaching or crossing an input threshold, detecting or recognizing a type of input or gesture, and the like. In some embodiments, a tactile output is provided to serve as a warning or prompt for an upcoming event or result that would otherwise occur unless a change in direction or interrupt input is detected in a timely manner. Tactile output is also used in other contexts to enrich the user experience, improve the accessibility of the device for users with visual or motor difficulties or other accessibility needs, and / or improve the efficiency and functionality of the user interface and / or device. Optionally, the tactile output is compared with audio input and / or visual user interface changes, which further enhances the user's experience when the user interacts with the user interface and / or device, and facilitates better transmission of information about the state of the user interface and / or device, and reduces input errors and improves the efficiency of the user's operation on the device.

[0215] Figures 4F to 4HA set of sample tactile output patterns are provided that can be used, individually or in combination, as is or through one or more transformations (e.g., modulation, amplification, truncation, etc.) to generate suitable tactile feedback in various scenarios for various purposes (such as those described above and those described for the user interfaces and methods discussed herein). This example of a control panel for tactile output shows how a set of three waveforms and eight frequencies can be used to generate an array of tactile output patterns. In addition to the tactile output patterns shown in these figures, each of these tactile output patterns is optionally adjusted in amplitude by changing the gain value of the tactile output pattern, as shown, for example, for Figures 4I to 4K 80Hz, FullTap200Hz, MiniTap 80Hz, MiniTap 200Hz, MicroTap 80Hz, and MicroTap 200Hz, are shown as variations with gains of 1.0, 0.75, 0.5, and 0.25, respectively. Figures 4I to 4K As shown, changing the gain of a tactile output pattern changes the amplitude of the pattern without changing the frequency of the pattern or changing the shape of the waveform. In some embodiments, changing the frequency of the tactile output pattern also results in a lower amplitude because some tactile output generators are limited by how much force can be applied to the movable mass, so the higher frequency movement of the mass is constrained to a lower amplitude to ensure that the acceleration required to generate the waveform does not require forces outside the operating force range of the tactile output generator (e.g., the peak amplitudes of FullTap at 230 Hz, 270 Hz, and 300 Hz are lower than the amplitudes of FullTap at 80 Hz, 100 Hz, 125 Hz, and 200 Hz).

[0216] Figures 4F to 4K A tactile output pattern having a particular waveform is shown. The waveform of the tactile output pattern represents a pattern of physical displacement relative to a neutral position (e.g., xzero) versus time through which the movable mass passes to generate a tactile output having the tactile output pattern. For example, Figure 4F The first set of tactile output patterns shown (eg, the "FullTap" tactile output pattern) each have a waveform including oscillations having two complete cycles (eg, oscillations that start and end at a neutral position and pass through the neutral position three times). Figure 4G The second set of tactile output patterns shown (eg, the “MiniTap” tactile output patterns) each have a waveform including an oscillation having one complete cycle (eg, an oscillation that starts and ends at a neutral position and passes through the neutral position once). Figure 4HThe third set of tactile output patterns shown (e.g., the "MicroTap" tactile output patterns) each have a waveform that includes an oscillation with half a complete cycle (e.g., an oscillation that begins and ends at a neutral position and does not pass through the neutral position). The waveforms of the tactile output patterns also include starting and ending buffers that represent the gradual acceleration and deceleration of the movable mass at the beginning and end of the tactile output. Figures 4F to 4K The example waveform shown includes xmin and xmax values ​​representing the maximum and minimum extents of movement of the movable mass. For larger electronic devices with larger movable masses, the minimum and maximum extents of movement of the mass may be larger or smaller. Figures 4F to 4K The examples shown describe the movement of a mass in 1 dimension, however, similar principles can also be applied to the movement of a movable mass in two or three dimensions.

[0217] like Figures 4F to 4K As shown, each tactile output pattern also has a corresponding characteristic frequency, which affects the "pitch" of the tactile sensation felt by the user from the tactile output having that characteristic frequency. For continuous tactile output, the characteristic frequency represents the number of cycles (e.g., cycles per second) that the movable mass of the tactile output generator completes in a given time period. For discrete tactile output, a discrete output signal is generated (e.g., having 0.5, 1, or 2 cycles), and the characteristic frequency value specifies how fast the movable mass needs to move to generate a tactile output having that characteristic frequency. Figures 4F to 4H As shown, for each type of tactile output (e.g., defined by a corresponding waveform, such as FullTap, MiniTap, or MicroTap), higher frequency values ​​correspond to faster movement of the movable mass and, therefore, generally speaking, to a shorter tactile output completion time (e.g., including the time required to complete the number of cycles of the discrete tactile output plus the start and end buffer times). For example, a FullTap with a characteristic frequency of 80 Hz takes longer to complete (e.g., in seconds) than a FullTap with a characteristic frequency of 100 Hz. Figure 4F, 35.4ms vs. 28.3ms). In addition, for a given frequency, a tactile output with more cycles in its waveform at the corresponding frequency takes longer to complete than a tactile output with fewer cycles in its waveform at the same corresponding frequency. For example, a 150Hz FullTap takes longer to complete than a 150Hz MiniTap (e.g., 19.4ms vs. 12.8ms), and a 150Hz MiniTap takes longer to complete than a 150Hz MicroTap (e.g., 12.8ms vs. 9.4ms). However, for tactile output modes with different frequencies, this rule may not apply (e.g., a tactile output with more cycles but a higher frequency may take a shorter amount of time to complete than a tactile output with fewer cycles but a lower frequency, and vice versa). For example, at 300Hz, FullTap takes the same amount of time as MiniTap (e.g., 9.9ms).

[0218] like Figures 4F to 4K As shown, the tactile output pattern also has a characteristic amplitude, which affects the amount of energy contained in the tactile signal, or the "intensity" of the tactile sensation that the user can feel through the tactile output having the characteristic amplitude. In some embodiments, the characteristic amplitude of the tactile output pattern refers to an absolute or normalized value representing the maximum displacement of the movable mass relative to the neutral position when generating the tactile output. In some embodiments, the characteristic amplitude of the tactile output pattern can be adjusted according to various conditions (e.g., customized based on user interface context and behavior) and / or preconfigured metrics (e.g., input-based metrics, and / or user interface-based metrics), such as by a fixed or dynamically determined gain factor (e.g., a value between 0 and 1). In some embodiments, an input-based metric (e.g., an intensity change metric or an input speed metric) measures a characteristic of the input during the period that triggers the generation of the tactile output (e.g., the rate of change of the characteristic intensity of the contact in a press input or the rate of movement of the contact on the touch-sensitive surface). In some embodiments, a user interface-based metric (e.g., a boundary-crossing velocity metric) measures a characteristic of a user interface element (e.g., the speed at which the element moves across a hidden or visible boundary in the user interface) during a user interface change that triggers generation of a tactile output. In some embodiments, a characteristic amplitude of the tactile output pattern can be "envelope" modulated, and the peaks of adjacent cycles can have different amplitudes, where one of the waveforms shown above is further modified by multiplying by an envelope parameter that changes over time (e.g., from 0 to 1) to gradually adjust the amplitude of a portion of the tactile output over time as the tactile output is generated.

[0219] Although Figures 4F to 4KFor illustrative purposes, only specific frequencies, amplitudes, and waveforms are shown in the sample tactile output patterns. However, tactile output patterns with other frequencies, amplitudes, and waveforms can also be used for similar purposes. For example, a waveform with between 0.5 and 4 cycles can be used. Other frequencies in the range of 60 Hz-400 Hz can also be used.

[0220] User interface and associated processes

[0221] Attention now turns to embodiments of a user interface (“UI”) and associated processes that may be implemented on an electronic device such as portable multifunction device 100 or device 300 having a display, a touch-sensitive surface, (optionally) one or more tactile output generators for generating tactile output, and (optionally) one or more sensors for detecting intensity of contact with the touch-sensitive surface.

[0222] Figures 5A to 5AT An example user interface is shown for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figures 8A to 8E 、 9A to 9D 、 10A to 10D 、 Figures 16A to 16G 、 17A to 17D 、 18A to 18I 、 Figures 19A to 19H as well as 20A to 20F For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device with touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, in response to detecting a contact on touch-sensitive surface 451 when the user interface shown in the figure is displayed on display 450 together with the focus selector, similar operations are optionally performed on a device with display 450 and a separate touch-sensitive surface 451.

[0223] Figure 5A shows where reference is used Figures 5B to 5AT Real-world scenarios for the described user interface.

[0224] Figure 5A A physical space 5002 is shown in which a table 5004 is located. The device 100 is held by a user in the user's hand 5006.

[0225] Figure 5BAn instant messaging user interface 5008 is shown displayed on the display 112. The instant messaging user interface 5008 includes a message bubble 5010 including a received text message 5012, a message bubble 5014 including a sent text message 5016, and a message bubble 5018 including a virtual object (e.g., a virtual chair 5020) received in the message and a virtual object indicator 5022 indicating that the virtual chair 5020 is an object visible in the augmented reality view (e.g., within the representation of the field of view of one or more cameras of the device 100). The instant messaging user interface 5008 also includes a message input area 5024 configured to display a message input.

[0226] Figures 5C to 5G An input is shown that causes a portion of the instant messaging user interface 5008 to be replaced by the field of view of one or more cameras of the device 100. Figure 5C In the embodiment, a contact 5026 is detected with the touch screen 112 of the device 100. The characteristic intensity of the contact is higher than the contact detection intensity threshold IT0 and lower than the prompt press intensity threshold IT0. H , as shown by the intensity level meter 5028. Figure 5D In the example, as shown by the intensity level meter 5028, the characteristic intensity of the contact 5026 increases to above the prompt press intensity threshold IT H , which causes the area of ​​the message bubble 5018 to increase, the size of the virtual chair 5020 to increase, and the instant messaging user interface 5008 to begin to blur behind the message bubble 5018 (e.g., providing visual feedback to the user of the effect of increasing the characteristic strength of the contact). Figure 5E , as shown by the intensity level meter 5028, the characteristic intensity of contact 5026 increases to above the light press intensity threshold IT L , which causes the message bubble 5018 to be replaced by the disk 5030, the size of the virtual chair 5020 to be further increased, and the instant messaging user interface 5008 to be further blurred behind the disk 5030. Figure 5F , as shown by the intensity level meter 5028, the characteristic intensity of the contact 5026 increases to above the deep press intensity threshold IT D , which causes the tactile output generator 167 of the device 100 to output a tactile output (as shown at 5032) indicating that the criteria for replacing a portion of the instant messaging user interface 5008 with the field of view of one or more cameras of the device 100 have been met.

[0227] In some embodiments, when the characteristic strength of contact 5026 reaches the deep press strength threshold IT D (like Figure 5F Before Figures 5C to 5E The progression shown is reversible. For example, Figure 5D and / or Figure 5E After the increase shown, decreasing the characteristic intensity of contact 5026 will cause the interface state corresponding to the reduced intensity level of contact 5026 to be displayed (e.g., based on determining that the reduced characteristic intensity of the contact is above the light press intensity threshold IT L , as shown in Figure 5E The interface shown; according to the determination that the reduced characteristic intensity of the contact is higher than the prompt pressing intensity threshold IT H , as shown in Figure 5D The interface shown; and according to determining that the reduced characteristic intensity of the contact is lower than the prompt press intensity threshold IT H , as shown in Figure 5C In some embodiments, in Figure 5D and / or Figure 5E After the increase shown, reducing the characteristic strength of contact 5026 will result in Figure 5C The shown interface is redisplayed.

[0228] Figures 5F to 5J An animated transition is shown during which a portion of the instant messaging user interface is replaced by the field of view of one or more cameras (hereinafter referred to as "cameras") of device 100. Figure 5F to Figure 5G , contact 5026 has been lifted off the touch screen 112, and the virtual chair 5020 has been moved toward Figure 5I The final position of the rotation. Figure 5G , the camera's field of view 5034 has begun to fade into view in the disk 5030 (as indicated by the dotted line). Figure 5H , the camera's field of view 5034 (e.g., showing the view of the physical space 5002 captured by the camera) has been faded into view on disk 5030. Figures 5H to 5I , the virtual chair 5020 continues to move towards its Figure 5I The final position of the rotation. Figure 5I , tactile output generator 167 has output a tactile output (as shown at 5036) indicating that at least one plane (e.g., floor surface 5038) has been detected in the camera's field of view 5034. Virtual chair 5020 is placed on the detected plane (e.g., based on a determination by device 100 that the virtual object is configured to be placed in a vertical orientation on a detected horizontal surface, such as floor surface 5038). As a portion of the instant messaging user interface is converted to a representation of the camera's field of view 5034 on display 112, the size of virtual chair 5020 is continuously adjusted on display 112. For example, the scale of virtual chair 5020 relative to physical space 5002 as shown in the camera's field of view 5034 is determined based on a "real world" size of virtual chair 5020 predefined in the camera's field of view 5034 and / or the size of a detected object (such as table 5004). Figure 5J , the virtual chair 5020 is displayed in its final position with a predefined orientation relative to the floor surface detected in the camera's field of view 5034. In some embodiments, the initial landing position of the virtual chair 5020 is a predefined position relative to the plane detected in the camera's field of view, such as at the center of an unoccupied area of ​​the detected plane. In some embodiments, the initial landing position of the virtual chair 5020 is determined based on the lift-off position of the contact 5026 (e.g., at Figure 5F , the lift-off position of contact 5026 may be different from the initial touch-down position of contact 5026, resulting from movement of contact 5026 on touch screen 112 after the criteria for transitioning to the augmented reality environment are met).

[0229] Figures 5K to 5L Movement of the device 100 is shown (e.g., by a user's hand 5006) adjusting the camera's field of view 5034. As the device 100 moves relative to the physical space 5002, the displayed camera's field of view 5034 changes, and the virtual chair 5020 remains in the same position and orientation relative to the floor surface 5038 in the displayed camera's field of view 5034.

[0230] Figures 5M to 5Q An input is shown that causes the virtual chair 5020 to move across a floor surface 5038 in the displayed camera's field of view 5034. Figure 5N , contact 5040 is detected with the touch screen 112 of the device 100 at a location corresponding to the virtual chair 5020. Figures 5N to 5O , contact 5040 drags virtual chair 5020 as contact 5040 moves along the path indicated by arrow 5042. As virtual chair 5020 moves through contact 5040, the size of virtual chair 5020 changes to maintain the proportion of virtual chair 5020 relative to physical space 5002 as shown in the camera's field of view 5034. For example, in Figures 5N to 5P , as the virtual chair 5020 moves from the foreground of the camera's field of view 5034 to a position farther from the device 100 and closer to the table 5004 in the camera's field of view 5034, the size of the virtual chair 5020 decreases (e.g., so that the proportion of the chair in the camera's field of view 5034 relative to the table 5004 is maintained). Additionally, as the virtual chair 5020 is moved by contact 5040, the plane identified in the camera's field of view 5034 is highlighted. For example, in Figure 5O , the floor plane 5038 is highlighted. Figures 5O to 5P , as contact 5040 moves along the path indicated by arrow 5044, contact 5040 continues to drag virtual chair 5020. Figure 5Q, contact 5040 has been lifted off touch screen 112. In some embodiments, as Figures 5N to 5Q As shown, the movement path of the virtual chair 5020 is constrained by the floor surface 5038 in the camera's field of view 5034, as if the contact 5040 is dragging the virtual chair 5020 on the floor surface 5038. In some embodiments, as shown in FIG. Figures 5N to 5P The contact 5040 is described as referring to Figures 5C to 5F Continuation of the described contact 5026 (e.g., contact 5026 is not lifted, and this contact that causes a portion of the instant messaging user interface 5008 to be replaced by the camera's field of view 5034) also drags the virtual chair 5020 in the camera's field of view 5034.

[0231] Figures 5Q to 5U Input is shown that causes the virtual chair 5020 to move from the floor surface 5038 to a different plane (e.g., the tabletop 5046) detected in the camera's field of view 5034. Figure 5R , contact 5050 is detected with the touch screen 112 of the device 100 at a location corresponding to the virtual chair 5020. Figures 5R to 5S 5048, the contact 5048 drags the virtual chair 5020 as it moves along the path indicated by the arrow 5050. As the virtual chair 5020 is moved by the contact 5048, the size of the virtual chair 5020 changes to maintain the proportion of the virtual chair 5020 relative to the physical space 5002 as shown in the camera's field of view 5034. Additionally, as the virtual chair 5020 is moved by the contact 5040, the desktop plane 5046 is highlighted (e.g., as shown in FIG. Figure 5S shown). Figures 5S to 5T , contact 5040 continues to drag virtual chair 5020 as contact 5048 moves along the path indicated by arrow 5052. Figure 5U , contact 5048 has been lifted off the touch screen 112 and the virtual chair 5020 is placed on the tabletop plane 5046 in a vertical orientation facing the same direction as before.

[0232] Figures 5U to 5AD An input is shown dragging the virtual chair 5020 to the edge of the touch screen display 112, which causes the camera's field of view 5034 to cease display. Figure 5V , contact 5054 is detected with the touch screen 112 of the device 100 at a location corresponding to the virtual chair 5020. Figure 5V to Figure 5W , when contact 5054 moves along the path indicated by arrow 5056, contact 5054 drags virtual chair 5020. Figures 5W to 5X When contact 5054 moves along the path indicated by arrow 5058, contact 5054 continues to drag virtual chair 5020 to Figure 5XPosition shown.

[0233] like Figures 5Y to 5AD As shown, through Figures 5U to 5X The input by contact 5054 is shown to cause a transition from displaying the camera's field of view 5034 in disk 5030 to ceasing to display the camera's field of view 5034 and returning to fully displaying instant messaging user interface 5008. Figure 5Y In , the camera's field of view 5034 begins to fade out in the disk 5030. Figures 5Y to 5Z , the disk 5030 transitions to the message bubble 5018. Figure 5Z The camera's field of view is no longer displayed. Figure 5AA , the instant messaging user interface 5008 stops blurring and the size of the message bubble 5018 returns to the original size of the message bubble 5018 (e.g., Figure 5B shown).

[0234] Figures 5AA to 5AD shows that when the virtual chair 5020 is Figure 5AA The position corresponding to the contact 5054 in the instant messaging user interface 5008 is moved to the original position of the virtual chair 5020 (e.g., Figure 5B ) occurs when the virtual chair 5020 is animated. Figure 5AB , contact 5054 has been lifted off touch screen 112. Figures 5AB to 5AC In the example, the size of the virtual chair 5020 gradually increases, and the virtual chair moves toward its Figure 5AD The final position in the rotation.

[0235] exist Figures 5B to 5AD In some embodiments, the virtual chair 5020 has substantially the same three-dimensional appearance within the instant messaging user interface 5008 and within the displayed camera's field of view 5034, and the virtual chair 5020 maintains this same three-dimensional appearance during the transition from displaying the instant messaging user interface 5008 to displaying the camera's field of view 5034, and during the reverse transition. In some embodiments, the representation of the virtual chair 5020 has a different appearance in the application user interface (e.g., the instant messaging user interface) than in the augmented reality environment (e.g., in the displayed camera's field of view). For example, the virtual chair 5020 optionally has a two-dimensional or more dimensional stylized appearance in the application user interface while having a more realistic three-dimensional textured appearance in the augmented reality environment; and the intermediate appearance of the virtual chair 5020 during the transition between displaying the application user interface and displaying the augmented reality environment is a series of interpolated appearances between the two-dimensional appearance and the three-dimensional appearance of the virtual chair 5020.

[0236] Figure 5AEInternet browser user interface 5060 is shown. Internet browser user interface 5060 includes a URL / search input area 5062, which is configured to display a URL / search input for a web browser and browser controls 5064 (e.g., navigation controls including back and forward buttons, a share control for displaying a share interface, a bookmark control for displaying a bookmark interface, and a tab control for displaying a tab interface). Internet browser user interface 5060 also includes network objects 5066, 5068, 5070, 5072, 5074, and 5076. In some embodiments, the corresponding network objects include links, so that in response to a tap input on the corresponding network object, the Internet location of the link corresponding to the network object is displayed in the internet browser user interface 5060 (e.g., replacing the display of the corresponding network object). Network objects 5066, 5068, and 5072 include two-dimensional representations of three-dimensional virtual objects, as indicated by virtual object indicators 5078, 5080, and 5082, respectively. Network objects 5070 , 5074 , and 5076 include two-dimensional images (but the two-dimensional images of network objects 5070 , 5074 , and 5076 do not correspond to three-dimensional virtual objects, as indicated by the absence of virtual object indicators.) The virtual object corresponding to network object 5068 is light object 5084 .

[0237] Figures 5AF to 5AH An input is shown that causes a portion of the internet browser user interface 5060 to be replaced by the camera's field of view 5034. Figure 5AF In the embodiment, a contact 5086 is detected with the touch screen 112 of the device 100. The characteristic intensity of the contact is higher than the contact detection intensity threshold IT0 and lower than the prompt press intensity threshold IT0. H , as shown by the intensity level meter 5028. Figure 5AG , as shown by the intensity level meter 5028, the characteristic intensity of contact 5026 increases to above the light press intensity threshold IT L The camera's field of view 5034 has been made to appear on the network object 5068 (e.g., covered by the virtual light 5084). Figure 5AH , as shown by intensity level meter 5028, the characteristic intensity of contact 5086 increases above the deep press intensity threshold IT D The camera's field of view 5034 is caused to replace a larger portion of the internet browser user interface 5060 (e.g., leaving only the URL / search input area 5062 and the browser controls 5064), and the tactile output generator 167 of the device 100 outputs a tactile output (as shown at 5088) indicating that the criteria for replacing a portion of the internet browser user interface 5060 with the camera's field of view 5034 has been met. In some embodiments, in response to a reference to Figures 5AF to 5AHWith the input described, the camera's field of view 5034 completely replaces the internet browser user interface 506 on the touch screen display 112.

[0238] Figure 5AI to Figure 5AM An input is shown that causes the virtual light 5084 to move. Figures 5AI to 5AJ , contact 5086 drags virtual light 5084 as contact 5086 moves along the path indicated by arrow 5090. As virtual light 5084 moves through contact 5086, the size of virtual light 5084 does not change, and the path of virtual light 5084 is optionally not constrained by the structure of the physical space captured in the camera's field of view. As virtual light 5084 moves through contact 5086, the plane identified in the camera's field of view 5034 is highlighted. For example, in Figure 5AJ , as the virtual light 5084 moves over the floor plane 5038, the floor plane 5038 is highlighted. Figures 5AJ to 5AK , contact 5086 continues to drag virtual light 5084 as contact 5086 moves along the path indicated by arrow 5092. Figures 5AK to 5AL , as contact 5086 moves along the path indicated by arrow 5094, contact 5086 continues to drag virtual light 5084, stops highlighting floor plane 5038, and highlights tabletop 5046 as virtual light 5084 moves over table 5004. Figure 5AM , contact 5086 has been lifted off the touch screen 112. When contact 5086 has been lifted off, the size of the virtual light 5086 is adjusted to have the correct proportions relative to the table 5004 in the camera's field of view 5034, and the virtual light 5086 is placed in a vertical orientation on the table top 5046 in the camera's field of view 5034.

[0239] Figure 5AM to Figure 5AQ An input of dragging the virtual light 5084 to the edge of the touch screen display 112 is shown, which causes the camera's field of view 5034 to cease displaying and the internet browser user interface 5060 to resume. Figure 5AN , contact 5096 is detected with touch screen 112 of device 100 at a location corresponding to virtual light 5084. Figures 5AN to 5AO , as contact 5096 moves along the path indicated by arrow 5098, contact 5096 drags virtual light 5084. Figures 5AO to 5AP When contact 5054 moves along the path indicated by arrow 5100, contact 5096 continues to drag virtual light 5084 to Figure 5AP The location shown. Figure 5AQ , contact 5096 has been lifted off touch screen 112 .

[0240] like Figures 5AQ to 5AT As shown, through Figure 5AM to Figure 5APInput by contact 5096 is shown causing a transition from displaying the camera's field of view 5034 to ceasing to display the camera's field of view 5034 and returning to fully displaying the internet browser user interface 5060. Figure 5AR , the camera's field of view 5034 begins to fade out (as indicated by the dotted line). Figures 5AR to 5AT , the size of the virtual light 5084 increases, and the virtual light moves toward its original position in the Internet browser user interface 5060. Figure 5AS , the camera's field of view 5034 is no longer displayed, and the internet browser user interface 5060 begins to fade in (as indicated by the dotted line). Figure 5AT , the internet browser user interface 5060 is fully displayed and the virtual light 5084 has returned to its original size and position within the internet browser user interface 5060.

[0241] Figures 6A to 6AJ An example user interface is shown for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object with a representation of the field of view of one or more cameras, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including Figures 8A to 8E 、 9A to 9D 、 10A to 10D 、 Figures 16A to 16G 、 17A to 17D 、 18A to 18I 、 Figures 19A to 19H as well as 20A to 20F For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, in response to detecting a contact on the touch-sensitive surface 451 while displaying the user interface on the display 450 shown in the figure and the focus selector, similar operations are optionally performed on a device having a display 450 and a separate touch-sensitive surface 451.

[0242] Figure 6AInstant messaging user interface 5008 is shown including message bubble 5010 including received text message 5012, message bubble 5014 including sent text message 5016, and message bubble 5018 including a virtual object (e.g., virtual chair 5020) received in the message and a virtual object indicator 5022 indicating that virtual chair 5020 is an object visible in the augmented reality view (e.g., within the field of view of one or more cameras of device 100 as shown). Figure 5B The instant messaging user interface 5008 is described in further detail.

[0243] Figures 6B to 6C An input is shown that causes the virtual chair 5020 to rotate. Figure 6B , contact 6002 is detected with touch screen 112 of device 100. Contact 6002 moves along a path indicated by arrow 6004 on touch screen 112. Figure 6C , in response to movement of the contact, instant messaging user interface 5008 scrolls upward (causing message bubble 5010 to scroll off the display, causing message bubbles 5014 and 5018 to scroll upward, and revealing additional message bubble 6005), and virtual chair 5020 rotates (e.g., tilts upward). The magnitude and direction of the rotation of virtual chair 5020 corresponds to the movement of contact 6002 along the path indicated by arrow 6004. Figure 6D , contact 6002 has been lifted off touch screen 112. In some embodiments, this rotational behavior of virtual chair 5020 within message bubble 5018 is used as an indication that virtual chair 5020 is a virtual object visible in the augmented reality environment that includes the field of view of the camera of device 100.

[0244] Figures 6E to 6L An input is shown that causes the instant messaging user interface 5008 to be replaced by the staging user interface 6010 and subsequently changes the orientation of the virtual chair 5020. Figure 6E In the embodiment, a contact 6006 is detected with the touch screen 112 of the device 100. The characteristic intensity of the contact is higher than the contact detection intensity threshold IT0 and lower than the prompt press intensity threshold IT0. H , as shown by the intensity level meter 5028. Figure 6F In the example, as shown by the intensity level meter 5028, the characteristic intensity of the contact 6006 increases to above the prompt press intensity threshold IT H , which causes the area of ​​the message bubble 5018 to increase, the size of the virtual chair 5020 to increase, and the instant messaging user interface 5008 to begin to blur behind the message bubble 5018 (e.g., providing visual feedback to the user of the effect of increasing the characteristic strength of the contact). Figure 6G, as shown by the intensity level meter 5028, the characteristic intensity of the contact 6006 increases to above the light press intensity threshold IT L , which causes the message bubble 5018 to be replaced by the disk 6008, the size of the virtual chair 5020 to be further increased, and the instant messaging user interface 5008 to be further blurred behind the disk 6008. Figure 6H , as shown by the intensity level meter 5028, the characteristic intensity of the contact 6006 increases to above the deep press intensity threshold IT D The instant messaging user interface 5008 is stopped from being displayed and a fade-in (indicated by a dotted line) of the staging user interface 6010 is initiated. Figure 6H As shown, the characteristic intensity of contact 6006 increases to above the deep press intensity threshold IT D Causes tactile output generator 167 of device 100 to output a tactile output (as indicated at 6012) indicating that the criteria for replacing instant messaging user interface 5008 with staging user interface 6010 have been met.

[0245] In some embodiments, when the characteristic strength of contact 6006 reaches the deep press strength threshold IT D (like Figure 6H Before Figures 6E to 6G The progression shown is reversible. For example, Figure 6F and / or Figure 6G After the increase shown, decreasing the characteristic intensity of contact 6006 will cause the interface state corresponding to the reduced intensity level of contact 6006 to be displayed (e.g., based on determining that the reduced characteristic intensity of the contact is above the light press intensity threshold IT L , as shown in Figure 6G The interface shown; according to the determination that the reduced characteristic intensity of the contact is higher than the prompt pressing intensity threshold IT H , as shown in Figure 6F The interface shown; and according to determining that the reduced characteristic intensity of the contact is lower than the prompt press intensity threshold IT H , as shown in Figure 6E In some embodiments, in Figure 6F and / or Figure 6G After the increase shown, reducing the characteristic strength of contact 6006 will result in Figure 6E The shown interface is redisplayed.

[0246] exist Figure 6I , a staging user interface 6010 is displayed. The staging user interface 6010 includes a stage 6014 on which a virtual chair 5020 is displayed. Figures 6H to 6I , the virtual chair 5020 is animated to indicate Figure 6H The position of the virtual chair 5020 in Figure 6I6014. For example, the virtual chair 5020 is rotated to a predefined position relative to the stand 6014, rotated in a predefined orientation, and / or rotated a predefined distance (e.g., so that the virtual chair appears to be supported by the stand 6014). The staging user interface 6010 also includes a back control 6016, which, when activated (e.g., by a tap input at a location corresponding to the back control 6016), causes the previously displayed user interface (e.g., the instant messaging user interface 5008) to be redisplayed. The staging user interface 6010 also includes a toggle control 6018, which indicates the current display mode (e.g., the current display mode is the staging user interface mode, as indicated by the highlighted “3D” indicator), and which, when activated, causes a transition to a selected display mode. For example, when the staging user interface 6010 is displayed, a tap input by contact at a location corresponding to the toggle control 6018 (e.g., a location corresponding to a portion of the toggle control 6018 including the text "World") causes the staging user interface 6010 to be replaced by the camera's field of view. The staging user interface 6010 also includes a sharing control 6020 (e.g., a sharing control for displaying a sharing interface).

[0247] Figures 6J to 6L The rotation of the virtual chair 5020 relative to the stage 6014 caused by the movement of the contact 6006 is shown. Figures 6J to 6K , as contact 6006 moves along the path indicated by arrow 6022, virtual chair 5020 rotates (e.g., about a first axis perpendicular to the movement of contact 6066). Figures 6K to 6L , as contact 6006 moves along the path indicated by arrow 6024 and then along the path indicated by arrow 6025, virtual chair 5020 rotates (e.g., about a second axis perpendicular to the movement of contact 6066). Figure 6M , contact 6006 has been lifted off touch screen 112. In some embodiments, as Figures 6J to 6L As shown, the rotation of the virtual chair 5020 is constrained by the surface of the stage 6014. For example, during the rotation of the virtual chair, at least one leg of the virtual chair 5020 remains in contact with the surface of the stage 6014. In some embodiments, the surface of the stage 6014 serves as a reference frame for the free rotation and vertical translation of the virtual chair 5020 without placing specific constraints on the movement of the virtual chair 5020.

[0248] Figures 6N to 6O An input for adjusting the size of the displayed virtual chair 5020 is shown. Figure 6NIn FIG, a first contact 6026 and a second contact 6030 with the touch screen 112 are detected. The first contact 6026 moves along the path indicated by the arrow 6028, and while the first contact 6026 moves, the second contact 6030 moves along the path indicated by the arrow 6032. Figures 6N to 6O , when the first contact 6026 and the second contact 6030 move along the paths indicated by arrows 6028 and 6032, respectively (e.g., in a split gesture), the size of the displayed virtual chair 5020 increases. Figure 6P , the first contact 6030 and the second contact 6026 have lifted off the touch screen 112, and the virtual chair 5020 maintains the increased size after the contacts 6026 and 6030 are lifted off.

[0249] Figures 6Q to 6U An input is shown that causes the staging user interface 6010 to be replaced by the field of view 6036 of one or more cameras of the device 100. Figure 6Q In the embodiment, a contact 6034 is detected with the touch screen 112 of the device 100. The characteristic intensity of the contact is higher than the contact detection intensity threshold IT0 and lower than the prompt press intensity threshold IT0. H , as shown by the intensity level meter 5028. Figure 6R In the example, as shown by the intensity level meter 5028, the characteristic intensity of the contact 5026 increases to above the prompt press intensity threshold IT H The staging user interface 6010 has been made to begin to blur behind the virtual chair 5020 (as indicated by the dotted line). Figure 6S , as shown by the intensity level meter 5028, the characteristic intensity of the contact 6034 increases to above the light press intensity threshold IT L The staging user interface 6010 ceases to be displayed and a fade-in of the camera's field of view 6036 is initiated (indicated by the dashed line). Figure 6T , as shown by the intensity level meter 5028, the characteristic intensity of the contact 6034 increases to above the deep press intensity threshold IT D The camera's field of view is displayed as 6036. Figure 6T As shown, the characteristic intensity of contact 6034 increases to above the deep press intensity threshold IT D Causes tactile output generator 167 of device 100 to output a tactile output (as indicated at 6038) indicating that the criteria for replacing the display of staging user interface 6010 with the display of camera's field of view 6036 has been met. Figure 6U In some embodiments, the contact 6034 has been lifted off the touch screen 112. In some embodiments, the characteristic strength of the contact 6034 reaches the deep press strength threshold IT D (like Figure 6T Before Figures 6Q to 6T The progression shown is reversible. For example, Figure 6R and / or Figure 6S After the increase shown, decreasing the characteristic intensity of contact 6034 will cause the interface state corresponding to the decreased intensity level of contact 6034 to be displayed.

[0250] from Figures 6Q to 6U , the virtual chair 5020 is placed on the detected plane (e.g., based on the device 100 determining that the virtual chair 5020 is configured to be placed in a vertical orientation on a detected horizontal surface, such as the floor surface 5038), and the size of the virtual chair 5020 is adjusted (e.g., based on the "real world" size of the virtual chair 5020 defined in the camera's field of view 6036 and / or the size of a detected object (such as the table 5004), the scale of the virtual chair 5020 as shown in the camera's field of view 6036 relative to the physical space 5002). When the virtual chair 5020 transitions from the staging user interface 6010 to the camera's field of view 6036, the orientation of the virtual chair 5020 caused by the rotation of the virtual chair 5020 when the staging interface 6010 is displayed is maintained (e.g., as shown with reference to Figures 6J to 6K For example, the orientation of the virtual chair 5020 relative to the floor surface 5038 is the same as the final orientation of the virtual chair 5020 relative to the surface of the stage 5014. In some embodiments, when the size of the virtual chair 5020 is adjusted relative to the size of the physical space 5002 in the field of view 6036, the adjustment to the size of the virtual object 5020 in the staging user interface is taken into account.

[0251] Figures 6V to 6Y An input is shown that causes the camera's field of view 6036 to be replaced by the staging user interface 6010. Figure 6V , an input (e.g., a tap input) by contact 6040 is detected at a position corresponding to toggle control 6018 (e.g., a position corresponding to a portion including the text “3D” on toggle control 6018). Figures 6W to 6Y In response to input via contact 6040, the camera's field of view 6036 fades out (e.g., Figure 6W ), the staging user interface 6010 fades in (as indicated by the dotted line in Figure 6X ), and the staging user interface 6010 is fully displayed (as indicated by the dashed line in Figure 6Y From Figures 6V to 6Y , the size of the virtual chair 5020 is adjusted, and the position of the virtual chair 5020 is changed (eg, returning the virtual chair 5020 to a predefined position and size for the staging user interface).

[0252] Figures 6Z to 6AC An input is shown that causes the staging user interface 6010 to be replaced by the instant messaging user interface 5008. Figure 6Z, an input (e.g., a tap input) by contact 6042 is detected at a location corresponding to back control 6016. Figures 6AA to 6AC In response to input via contact 6042, the staging user interface 6010 fades out (e.g., Figure 6AA ), the instant messaging user interface 5008 fades in (as indicated by the dotted line in Figure 6AB ), and the instant messaging user interface 5008 is fully displayed (as indicated by the dotted line in Figure 6AC From Figures 6Z to 6AB , continuously adjusting the size, orientation, and position of the virtual chair 5020 on the display (e.g., to return the virtual chair 5020 to a predefined position, size, and orientation for the instant messaging user interface 5008).

[0253] Figures 6AD to 6AJ An input is shown that causes the instant messaging user interface 5008 to be replaced by the camera's field of view 6036 (e.g., bypassing the display of the staging user interface 6010). Figure 6AD , contact 6044 is detected at a location corresponding to virtual chair 5020. Input through contact 6044 includes a long touch gesture (during which contact 6044 is maintained on the touch-sensitive surface at a location corresponding to the representation of virtual object 5020 with a movement less than a threshold movement amount for at least a predefined threshold amount of time) followed by an upward swipe gesture (dragging virtual chair 5020 upward). Figures 6AD to 6AE As shown, when contact 6044 moves along the path indicated by arrow 6046, virtual chair 5020 is dragged upward. Figure 6AE In the example, the instant messaging user interface 5008 fades out behind the virtual chair 5020. Figures 6AE to 6AF As shown, when contact 6044 moves along the path indicated by arrow 6048, virtual chair 5020 continues to be dragged upward. Figure 6AF , the camera's field of view 5036 fades in behind the virtual chair 5020. Figure 6AG In FIG, in response to input through contact 6044 comprising a long touch gesture followed by an upward swipe gesture, the camera's field of view 5036 is fully displayed. Figure 6AH , contact 6044 is lifted off touch screen 112. Figures 6AH to 6AJ In response to the lifting of contact 6044, the virtual chair 5020 is released (e.g., because the virtual chair 5020 is no longer constrained or dragged by the contact) and falls onto a plane (e.g., floor surface 5038, based on determining that the horizontal (floor) surface corresponds to the virtual chair 5020). Figure 6AJ As shown, the tactile output generator 167 of the device 100 outputs a tactile output (as shown at 6050) indicating that the virtual chair 5020 has landed on the floor surface 5038.

[0254] Figures 7A to 7P An example user interface is shown for displaying an item with a visual indication that the item corresponds to a virtual three-dimensional object according to some embodiments. The user interface in these figures is used to illustrate the process described below, including Figures 8A to 8E 、 9A to 9D 、 10A to 10D 、 Figures 16A to 16G 、 17A to 17D 、 18A to 18I 、 Figures 19A to 19H as well as 20A to 20F For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to a finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, in response to detecting a contact on the touch-sensitive surface 451 while displaying the user interface on the display 450 shown in the figure and the focus selector, similar operations are optionally performed on a device having a display 450 and a separate touch-sensitive surface 451.

[0255] Figure 7A An input is shown that is detected while displaying the user interface 400 of the application menu. The input corresponds to a request to display a first user interface (e.g., an Internet browser user interface 5060). Figure 7A In the example, an input (eg, a tap input) by contact 7000 is detected at a position corresponding to the icon 420 of the browser module 147. In response to the input, an Internet browser user interface 5060 is displayed, such as Figure 7B shown.

[0256] Figure 7B An Internet browser user interface 5060 is shown (e.g., as shown in FIG. Figure 5AE Detailed description is provided for the description of FIG. 5060. Internet browser user interface 5060 includes network objects 5066, 5068, 5070, 5072, 5074, and 5076. Network objects 5066, 5068, and 5072 include two-dimensional representations of three-dimensional virtual objects, as indicated by virtual object indicators 5078, 5080, and 5082, respectively. Network objects 5070, 5074, and 5076 include two-dimensional images (although the two-dimensional images of network objects 5070, 5074, and 5076 do not correspond to three-dimensional virtual objects, as indicated by the absence of virtual object indicators).

[0257] Figures 7C to 7DAn input is shown that causes the internet browser user interface 5060 to translate (e.g., scroll). Figure 7B , contact 7002 with the touch screen 112 is detected. Figures 7C to 7D , when contact 7002 moves along the path indicated by arrow 7004, network objects 5066, 5068, 5070, 5072, 5074, and 5076 scroll upward, thereby revealing additional network objects 7003 and 7005. In addition, when contact 7002 moves along the path indicated by arrow 7004, the virtual objects in network objects 5066, 5068, and 5072, including virtual object indicators 5078, 5080, and 5082, respectively, rotate (e.g., tilt upward) according to the direction of the input (upward vertically). For example, virtual light 5084 moves from Figure 7C The first orientation in the Figure 7D When the touch scrolling internet browser user interface 5060 is engaged, the two-dimensional images of the network objects 5070, 5074 and 5076 do not rotate. Figure 7E , contact 7002 has been lifted off the touch screen 112. In some embodiments, the rotational behavior of the objects depicted in network objects 5066, 5068, and 5072 is used as a visual indication that these network objects have corresponding three-dimensional virtual objects visible in the augmented reality environment, while the absence of such rotational behavior of the objects depicted in network objects 5070, 5074, and 5076 is used as a visual indication that these network objects do not have corresponding three-dimensional virtual objects visible in the augmented reality environment.

[0258] 7F-7G illustrate a parallax effect in which a virtual object rotates on the display in response to changes in the orientation of device 100 relative to the physical world.

[0259] Figure 7F1 The device 100 is shown being held by a user 7006 in the user's hand 5006 such that the device 100 has a substantially vertical orientation. Figure 7F2 As shown when the device 100 is in Figure 7F1 The orientation shown shows the device 100 displaying an internet browser user interface 5060 .

[0260] Figure 7G1 The device 100 is shown being held by a user 7006 in the user's hand 5006 such that the device 100 has a substantially horizontal orientation. Figure 7G2 As shown when the device 100 is in Figure 7G1 The device 100 displays an internet browser user interface 5060 in the orientation shown. Figure 7F2 to Figure 7G2, the orientations of the virtual objects in network objects 5066, 5068, and 5072, including virtual object indicators 5078, 5080, and 5082, respectively, rotate (e.g., tilt upward) in accordance with the change in the orientation of the device. For example, virtual light 5084 changes from Figure 7F2 The first orientation in the Figure 7G2 When the orientation of the device changes, the two-dimensional images of network objects 5070, 5074, and 5076 do not rotate. In some embodiments, the rotational behavior of the objects depicted in network objects 5066, 5068, and 5072 is used as a visual indication that these network objects have corresponding three-dimensional virtual objects visible in the augmented reality environment, while the absence of such rotational behavior of the objects depicted in network objects 5070, 5074, and 5076 is used as a visual indication that these network objects do not have corresponding three-dimensional virtual objects visible in the augmented reality environment.

[0261] Figures 7H to 7L An input corresponding to a request to display a second user interface (e.g., instant messaging user interface 5008) is shown. Figure 7H , contact 7008 is detected at a location corresponding to the lower edge of display 112. Figures 7H to 7I , contact 7008 moves upward along the path indicated by arrow 7010. Figures 7I to 7J , contact 7008 continues to move upward along the path indicated by arrow 7012. Figures 7H to 7J , when contact 7008 moves upward from the lower edge of display 112, the size of Internet browser user interface 5060 decreases, as shown in FIG. Figure 7I shown; and Figure 7J , a multitasking user interface 7012 is displayed (e.g., in response to an upward edge swipe gesture performed by contact 7008). The multitasking user interface 7012 is configured to allow access to various applications and various control interfaces (e.g., control center user interface 7014, internet browser user interface 5060, and instant messaging user interface 5008, as shown) that have a preserved state (e.g., the preserved state is the last state of the corresponding application when the corresponding application was the foreground application executing on the device). Figure 7J Select the interface as shown in the figure. Figure 7K , contact 7008 has been lifted off touch screen 112. Figure 7L In the example, an input (e.g., a tap input) is detected by contact 7016 at a position corresponding to instant messaging user interface 5008. In response to the input by contact 7016, instant messaging user interface 5008 is displayed, as shown in FIG. Figure 7M shown.

[0262] Figure 7M An instant messaging user interface 5008 including a message bubble 5018 is shown (e.g., as shown in FIG. Figure 5B ), the message bubble includes a virtual object received in the message (e.g., a virtual chair 5020) and a virtual object indicator 5022, which indicates that the virtual chair 5020 is a virtual three-dimensional object (e.g., an object visible in the augmented reality view and / or an object visible from different angles). Instant messaging user interface 5008 also includes message bubble 6005 and message bubble 7018, the former including the sent text message and the latter including the received text message including emoticon 7020. Emoticon 7020 is a two-dimensional image that does not correspond to a virtual three-dimensional object. For this reason, emoticon 7020 is displayed without a virtual object indicator.

[0263] Figure 7N Map user interface 7022 is shown, which includes a map 7024, a point of interest information area 7026 for a first point of interest, and a point of interest information area 7032 for a second point of interest. For example, the first point of interest and the second point of interest are search results within or near the area shown on map 7024 corresponding to the search term "Apple" in search input area 7025. In first point of interest information area 7026, a first point of interest object 7028 is displayed with a virtual object indicator 7030, which indicates that first point of interest object 7028 is a virtual three-dimensional object. In second point of interest information area 7032, a second point of interest object 7034 is displayed without a virtual object indicator because second point of interest object 7034 does not correspond to a virtual three-dimensional object visible in the augmented reality view.

[0264] Figure 7O A file management user interface 7036 is shown, which includes file management controls 7038, a file management search input area 7040, a file information area 7042 for a first file (e.g., a portable document format (PDF) file), a file information area 7044 for a second file (e.g., a photo file), a file information area 7046 for a third file (e.g., a virtual chair object), and a file information area 7048 for a fourth file (e.g., a PDF file). The third file information area 7046 includes a virtual object indicator 7050 displayed adjacent to a file preview object 7045 in the file information area 7046, indicating that the third file corresponds to a virtual three-dimensional object. The first, second, and fourth file information areas 7042, 7044, and 7048 are displayed without virtual object indicators because the files corresponding to these file information areas do not have corresponding virtual three-dimensional objects visible in the augmented reality environment.

[0265] Figure 7P An email user interface 7052 is shown, which includes email navigation controls 7054, an email information area 7056, and an email content area 7058 including a representation of a first attachment 7060 and a representation of a second attachment 7062. The representation of the first attachment 7060 includes a virtual object indicator 7064, which indicates that the first attachment is a virtual three-dimensional object visible in the augmented reality environment. The displayed second attachment 7062 does not have a virtual object indicator because the second attachment is not a virtual three-dimensional object visible in the augmented reality environment.

[0266] Figures 8A to 8E 8 is a flowchart illustrating a method 800 for displaying a representation of a virtual object when switching from displaying a first user interface area to displaying a second user interface area according to some embodiments. The method 800 is performed on an electronic device (e.g., a display, a touch-sensitive surface, and one or more cameras (e.g., one or more rear-facing cameras on a side of the device opposite the display and the touch-sensitive surface). Figure 3 Device 300 or Figure 1A The method 800 is performed at a portable multifunction device 100 in the embodiment of the present invention. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display and the touch-sensitive surface are separate. Some operations in method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0267] Method 800 involves detecting input made through contact at a touch-sensitive surface of a device for displaying a representation of a virtual object in a first user interface area. In response to the input, the device uses criteria to determine whether to continuously display the representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of one or more cameras of the device. Using criteria to determine whether to continuously display the representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of one or more cameras enables multiple different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to the input (e.g., by replacing the display of at least a portion of the user interface with the field of view of one or more cameras, or by maintaining the display of the first user interface area without replacing the display of at least a portion of the first user interface area with the representation of the field of view of one or more cameras) increases the efficiency with which a user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0268] The device displays (802) a representation of a virtual object (e.g., a graphical representation of a three-dimensional object such as a virtual chair 5020, a virtual lamp 5084, a shoe, furniture, a hand tool, a decoration, a person, an emoticon, a game character, a virtual piece of furniture, etc.) in a first user interface area (e.g., a two-dimensional graphical user interface or a portion thereof (e.g., a browsable list of furniture images, an image containing one or more selectable objects, etc.)) on the display 112. For example, the first user interface area is as follows Figure 5B Instant messaging user interface 5008 as shown or as Figure 5AE Internet browser user interface 5060 is shown. In some embodiments, in addition to the image of the physical environment surrounding the device, the first user interface area also includes a background (e.g., the background of the first user interface area is a preselected background color / pattern or background image that is different from the output image captured simultaneously by the one or more cameras and different from the real-time content in the field of view of the one or more cameras).

[0269] While displaying a first representation of a virtual object in a first user interface area on the display, the device detects (804) a first input by contact at a location on touch-sensitive surface 112 that corresponds to the representation of the virtual object on the display (e.g., contact is detected on the first representation of the virtual object on the touch screen display, or contact is detected on an affordance displayed in the first user interface area concurrently with the first representation of the virtual object, the affordance being configured to trigger display of an AR view of the virtual object when invoked by the contact). For example, the first input is as described with reference to Figures 5C to 5F Input via contact 5020 as described or as referenced Figures 5AF to 5AL Describes input made through contact 5086.

[0270] In response to detecting a first input by contact (806), based on determining that the first input by contact satisfies a first (e.g., AR-trigger) criterion (e.g., an AR-trigger criterion is a criterion configured to recognize a swipe input, a touch-and-hold input, a press input, a tap input, a hard press with an intensity above a predefined intensity threshold, or another type of predefined input gesture, the criterion being associated with triggering activation of a camera, display of an augmented reality (AR) view of a physical environment surrounding the device, placement of a three-dimensional representation of a virtual object within the augmented reality view of the physical environment, and / or a combination of two or more of the above actions): the device displays a second user interface area on the display, which includes replacing display of at least a portion of the first user interface area with a representation of the field of view of the one or more cameras, and the device continuously displays representations of the virtual objects while switching from displaying the first user interface area to displaying the second user interface area. For example, the second user interface area on the display is as described with reference to Figure 5HThe field of view 5034 of the camera in the disk 5030 is described or as described with reference to Figure 5AH Describes the field of view of the camera 5034. Figures 5C to 5I , based on determining that the input through contact 5026 has increased above the deep press intensity threshold IT D When the display of the first user interface area (the instant messaging user interface 5008) is switched to the display of the second user interface area, the virtual chair object 5020 is continuously displayed, wherein the display of the second user interface area is to replace the display of a portion of the instant messaging user interface 5008 with the field of view 5034 of the camera in the disk 5030. Figures 5AF to 5AH , based on determining that the input through contact 5086 has increased above the deep press intensity threshold IT D characteristic intensity, the virtual light object 5084 is continuously displayed when switching from displaying the first user interface area (Internet browser user interface 5060) to displaying the second user interface area, wherein displaying the second user interface area is to replace the display of a portion of the Internet browser user interface 5060 with the camera's field of view 5034.

[0271] In some embodiments, continuously displaying a representation of a virtual object includes maintaining display of a representation of the virtual object or displaying an animated transition in which a first representation of the virtual object changes to a second representation of the virtual object (e.g., a view of the virtual object at a different size, from a different perspective, with a different rendering style, or at a different location on the display). In some embodiments, the field of view 5034 of the one or more cameras displays a real-time image of the physical environment 5002 surrounding the device, which is updated in real-time as the position and orientation of the device relative to the physical environment changes (e.g., as the device changes position and orientation relative to the physical environment). Figures 5K to 5L In some embodiments, the second user interface area completely replaces the first user interface on the display.

[0272] In some embodiments, the second user interface area overlays a portion of the first user interface area (e.g., a portion of the first user interface area is shown along an edge of the display or around a border of the display). In some embodiments, the second user interface area pops up next to the first user interface area. In some embodiments, the background within the first user interface area is replaced by the contents of the camera's field of view 5034. In some embodiments, the device display shows the virtual object moving from the first orientation as shown in the first user interface area and rotating (e.g., as shown in the first user interface area). Figures 5E to 5IIn some embodiments, the animation includes a transition from displaying a two-dimensional representation of a virtual object when the first user interface area is displayed to displaying a three-dimensional representation of the virtual object when the second user interface area is displayed. In some embodiments, the three-dimensional representation of the virtual object has an anchor plane that is predefined based on the shape and orientation of the virtual object as shown in the two-dimensional graphical user interface (e.g., the first user interface area). When transitioning to the augmented reality view (e.g., the second user interface area), the three-dimensional representation of the virtual object is moved, resized, and reoriented so that the virtual object reaches a new position on the display (e.g., to the center of the augmented reality view, or to another predefined position in the augmented reality view) from its original position on the display, and during or at the end of the movement, the three-dimensional representation of the virtual object is reoriented so that the three-dimensional representation of the virtual object is at a predefined position and / or orientation relative to a predefined plane identified in the field of view of the one or more cameras (e.g., a physical surface that can serve as a support plane for the three-dimensional representation of the virtual object, such as a vertical wall or a horizontal floor surface).

[0273] In some embodiments, the first criterion includes (808) a criterion that is satisfied when (e.g., determined as follows) contact is maintained on the touch-sensitive surface at a location corresponding to the representation of the virtual object with a movement less than a threshold amount of movement for at least a predefined amount of time (e.g., a long press time threshold). In some embodiments, based on determining that the contact meets the criteria for recognizing another type of gesture (e.g., a tap), while maintaining display of the virtual object, the device also performs another predefined function in addition to triggering the AR user interface. Determining whether to continue displaying the representation of the virtual object while replacing the display of at least a portion of the first user interface area with the camera's field of view is based on whether the contact is maintained on the touch-sensitive surface at a location corresponding to the representation of the virtual object with a movement less than a threshold amount of movement for at least a predefined amount of time, which enables multiple different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user is able to perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0274] In some embodiments, the first criterion includes (810) determining that the characteristic intensity of the contact increases above a first intensity threshold (e.g., a light press intensity threshold IT) when (e.g., determined as follows) the characteristic intensity of the contact increases above a first intensity threshold (e.g., a light press intensity threshold IT L or deep compression intensity threshold IT D ) is met. For example, if Figures 5C to 5FAs described, when the characteristic strength of contact 5026 increases to above the deep press strength threshold IT D , the criteria are met, as indicated by the intensity level meter 5028. In some embodiments, based on determining that the contact meets the criteria for recognizing another type of gesture (e.g., a tap), while maintaining display of the virtual object, the device also performs another predefined function in addition to triggering the AR user interface. In some embodiments, the first criterion requires that the first input is not a tap input (e.g., the input has a duration between a touch down of the contact and a lift off of the contact that is greater than a tap time threshold). Based on whether the characteristic intensity of the contact increases to above the first intensity threshold, determining whether to continue displaying the representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of the camera enables multiple different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user is able to perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0275] In some embodiments, the first criterion includes (812) a criterion that is satisfied when (e.g., as determined below) the movement of the contact satisfies a predefined movement criterion (e.g., the contact moves beyond a predefined threshold position on the touch-sensitive surface (e.g., a position corresponding to a boundary of the first user interface area, a position that is a threshold distance from the original position of the contact, etc.), the contact moves at a speed greater than a predefined threshold speed, the movement of the contact ends under a press input, etc.). In some embodiments, during an initial portion of the movement of the contact, a representation of the virtual object is dragged by the contact, and when the movement of the contact is about to satisfy the predefined movement criterion, the virtual object stops moving under the contact to indicate that the first criterion is about to be satisfied; and, if the movement of the contact continues and the continued movement of the contact causes the predefined movement criterion to be satisfied, a transition is initiated to display the second user interface area and display the virtual object within the augmented reality view. In some embodiments, while the virtual object is dragged during an initial portion of the first input, the object size and viewing angle do not change, and once the augmented reality view is displayed and the virtual object is dropped to a location in the augmented reality view, the virtual object is displayed with a size and viewing angle that depends on the physical location represented by the drop location of the virtual object in the augmented reality view. Determining whether to continuously display the representation of the virtual object while replacing the display of at least a portion of the first user interface area with the field of view of the camera is based on whether the movement of the contact meets predefined movement criteria, which enables multiple different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user is able to perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0276] In some embodiments, in response to detecting a first input by contact, based on determining that the first input by contact has satisfied a first criterion, a device having one or more tactile output generators 167 outputs (814) a tactile output that indicates that the first input satisfies the first criterion (e.g., as described with reference to FIG. Figure 5F Tactile output 5032 as described or as referenced Figure 5AH (e.g., haptic output 5088 described above) In some embodiments, a tactile sensation is generated before the field of view of the one or more cameras appears on the display. For example, the tactile sensation indicates that a first criterion for triggering activation of the one or more cameras and subsequently triggering detection of a plane in the field of view of the one or more cameras has been met. Because activating the cameras and making the field of view displayable takes time, the tactile sensation serves as a non-visual signal to the user indicating that the device has detected the necessary input and is ready to present the augmented reality user interface.

[0277] Outputting a tactile output indicating that a criterion is met (e.g., replacing the display of at least a portion of a user interface with the camera's field of view) provides feedback to the user indicating that the provided input meets the criterion. Providing improved tactile feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0278] In some embodiments, in response to detecting at least an initial portion of a first input (e.g., including: detecting a contact; or detecting an input by contact that satisfies a corresponding predefined criterion but does not satisfy a first criterion; or detecting an input that satisfies the first criterion), the device analyzes (816) the field of view of the one or more cameras to detect one or more planes (e.g., floor surface 5038, tabletop 5046, wall, etc.) in the field of view of the one or more cameras. In some embodiments, in response to detecting at least an initial portion of the first input, the one or more cameras are activated, and plane detection is initiated while the cameras are activated. In some embodiments, display of the field of view of the one or more cameras is delayed after the one or more cameras are activated (e.g., delayed from the time the one or more cameras are activated to the time at least one plane is detected in the field of view of the camera). In some embodiments, display of the field of view of the one or more cameras is initiated at the time the one or more cameras are activated, and plane detection is completed after the field of view is already visible on the display (e.g., in the second user interface area). In some embodiments, after detecting the corresponding plane in the field of view of the one or more cameras, the device determines a size and / or position of the representation of the virtual object based on the position of the corresponding plane relative to the field of view of the one or more cameras. In some embodiments, as the electronic device moves, the size and / or position of the representation of the virtual object is updated (e.g., with reference to the image) as the field of view of one or more cameras changes position relative to the corresponding plane. Figures 5K to 5L Determining the size and / or position of a representation of a virtual object based on the position of a corresponding plane detected in the camera's field of view (e.g., without requiring further user input to size and / or position the virtual object relative to the camera's field of view) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0279] In some embodiments, in response to detecting a contact on the touch-sensitive surface at a location corresponding to a representation of a virtual object on the display (e.g., in response to detecting contact 5026 at a location on touch screen 112 corresponding to virtual chair 5020), initiating (818) analysis of the field of view of the one or more cameras to detect one or more planes in the field of view of the one or more cameras. For example, before the first input satisfies the first criterion (e.g., before the characteristic intensity of contact 5026 increases to above the deep press intensity threshold IT D Before, as reference Figure 5F ), and before displaying the second user interface area, initiating activation of the camera and detection of planes in the camera's field of view. By initiating detection of planes upon detecting any interaction with the virtual object, plane detection can be completed before the AR trigger criteria are met, so that there is no visual delay for the user when watching the following process: the virtual object transitions into the augmented reality view when the first input meets the AR trigger criteria. In response to detecting contact at the location of the representation of the virtual object, initiating analysis to detect one or more planes in the camera's field of view (e.g., without requiring further user input to initiate analysis of the camera's field of view) improves the efficiency of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0280] In some embodiments, in response to detecting that a first input through contact satisfies a first criterion (e.g., in response to detecting that a characteristic intensity of contact 5026 increases to above a deep press intensity threshold IT D , as referenced Figure 5F ), initiates (820) analysis of the field of view of one or more cameras to detect one or more planes in the field of view of the one or more cameras. For example, when a first input satisfies a first criterion, camera activation and detection of a plane in the field of view of the camera are initiated, and the field of view of the camera is displayed until the plane detection is completed. By initiating camera activation and plane detection when the AR trigger criteria are met, the camera and plane detection are not activated and maintained unnecessarily, which saves battery power and extends battery life and camera life.

[0281] In some embodiments, in response to detecting that an initial portion of the first input meets the plane detection trigger criteria but does not meet the first criteria, an analysis of the field of view of one or more cameras is initiated (822) to detect one or more planes in the field of view of the one or more cameras. For example, when the initial portion of the first input meets some criteria (e.g., criteria that are less stringent than the AR trigger criteria), camera activation and detection of planes in the camera's field of view are initiated, and the camera's field of view is optionally displayed before plane detection is completed. By starting camera activation and plane detection after certain criteria are met rather than when contact is detected, the camera and plane detection are not unnecessarily activated and kept running, which saves battery power and extends battery life and camera life. By starting camera activation and plane detection before the AR trigger criteria are met, delays (caused by camera activation and plane detection) in displaying virtual objects in the augmented reality view when the first input meets the AR trigger criteria are reduced.

[0282] In some embodiments, the device displays (824) a representation of the virtual object in the second user interface area in a corresponding manner such that the virtual object (e.g., a virtual chair 5020) is oriented at a predefined angle relative to a corresponding plane detected in the field of view 5034 of the one or more cameras (e.g., such that there is no distance (or a minimum distance) between the underside of the four legs of the virtual chair 5020 and the floor surface 5038). For example, the orientation and / or position of the virtual object relative to the corresponding plane is predefined based on the shape and orientation of the virtual object as shown in the two-dimensional graphical user interface (e.g., the corresponding plane corresponds to a horizontal physical surface that can serve as a support surface for the three-dimensional representation of the virtual object in the augmented reality view (e.g., a horizontal tabletop for supporting a vase), or the corresponding plane is a vertical physical surface that can serve as a support surface for the three-dimensional representation of the virtual object in the augmented reality view (e.g., a vertical wall for hanging a virtual picture frame)). In some embodiments, the orientation and / or position of the virtual object is defined by a corresponding surface or boundary (e.g., a bottom surface, a bottom boundary point, a side surface, and / or a side boundary point) of the virtual object. In some embodiments, the anchor plane corresponding to the corresponding plane is an attribute in a set of attributes of the virtual object, and the anchor plane is specified according to the properties of the physical object that the virtual object should represent. In some embodiments, the virtual object is placed at a predefined orientation and / or position relative to multiple planes detected in the field of view of one or more cameras (for example, multiple corresponding sides of the virtual object are associated with corresponding planes detected in the field of view of the camera). In some embodiments, if the horizontal bottom plane relative to the virtual object is defined as a predefined orientation and / or position of the virtual object, the bottom plane of the virtual object is displayed on the floor plane detected in the field of view of the camera (for example, the horizontal bottom plane of the virtual object is parallel to the floor plane, and the distance between it and the floor plane is zero). In some embodiments, if the vertical back plane relative to the virtual object is defined as a predefined orientation and / or position of the virtual object, the back surface of the virtual object is placed against the wall plane detected in the field of view of one or more cameras (for example, the vertical back plane of the virtual object is parallel to the wall plane, and the distance between it and the wall plane is zero). In some embodiments, the virtual object is placed at a fixed distance from the corresponding plane or at an angle other than zero angle or right angle relative to the corresponding plane. Displaying representations of virtual objects relative to a plane detected in the camera's field of view (e.g., without requiring further user input to display the virtual objects relative to the plane in the camera's field of view) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0283] In some embodiments, in response to detecting a corresponding plane in the field of view of one or more cameras, a device having one or more tactile output generators 167 outputs (826) a tactile output indicating that a corresponding plane has been detected in the field of view of the one or more cameras. In some embodiments, a corresponding tactile output is generated for each plane detected in the field of view of the camera (e.g., floor surface 5038 and / or desktop 5046). In some embodiments, a tactile output is generated when plane detection is completed. In some embodiments, the tactile output is accompanied by a visual indication of the field of view plane in the field of view in the second user interface portion (e.g., a transient highlighting of the detected field of view plane). Outputting the tactile output indicating that a plane has been detected in the field of view of the camera provides feedback to the user indicating that the plane has been detected. Providing improved tactile feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing unnecessary additional input for placing virtual objects), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0284] In some embodiments, when switching from displaying the first user interface area to displaying the second user interface area, the device displays (828) an animation of the representation of the virtual object transitioning (e.g., moving, rotating, resizing, and / or re-rendering in a different style, etc.) to a predefined position relative to the corresponding plane in the second user interface area (e.g., as Figures 5F to 5I ), and in conjunction with displaying a representation of a virtual object at a predefined angle relative to the corresponding plane (e.g., at a predefined orientation and / or position relative to the corresponding plane, and its size, rotation angle, and appearance in a final state to be shown in the augmented reality view), the device having one or more tactile output generators 167 outputs a tactile output indicating that the virtual object is displayed in the second user interface area at the predefined angle relative to the corresponding plane. For example, as Figure 5I As shown, in conjunction with displaying a virtual chair 5020 at a predefined angle relative to a floor surface 5038, the device outputs a tactile output 5036. In some embodiments, the generated tactile output is configured to have characteristics (e.g., frequency, number of cycles, modulation, amplitude, accompanying audio waves, etc.) that reflect the following properties of the virtual object or the physical object represented by the virtual object: weight (e.g., heavy vs. light), material (e.g., metal, cotton, wood, marble, liquid, rubber, glass), size (e.g., large vs. small), shape (e.g., thin vs. thick, long vs. short, round vs. pointed, etc.), elasticity (e.g., springy vs. rigid), properties (e.g., playful vs. majestic, gentle vs. strong, etc.), and other properties. For example, the tactile output uses Figures 4F to 4KOne or more of the tactile output patterns shown. In some embodiments, a preset distribution including one or more changes of one or more features over time corresponds to a virtual object (e.g., an emoticon). For example, a "bouncing" tactile output distribution is provided for a "smiley" emoticon virtual object. Outputting a tactile output indicating the placement of a representation of the virtual object relative to a corresponding plane provides the user with feedback indicating that the representation of the virtual object has been automatically placed relative to the corresponding plane. Providing improved tactile feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing unnecessary additional input for placing virtual objects), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0285] In some embodiments (830), the tactile output has a tactile output distribution corresponding to characteristics of the virtual object (e.g., simulated physical properties such as size, density, mass, and / or material). In some embodiments, the tactile output distribution has characteristics (e.g., frequency, number of cycles, modulation, amplitude, accompanying audio waves, etc.) that vary based on one or more characteristics of the virtual object (e.g., weight, material, size, shape, and / or elasticity). For example, the tactile output uses Figures 4F to 4K One or more of the tactile output modes shown. In some embodiments, as the size, weight and / or mass of the virtual object increases, the amplitude and / or duration of the tactile output also increases. In some embodiments, the tactile output mode is selected based on the virtual material that constitutes the virtual object. Outputting a tactile output having a distribution corresponding to the characteristics of the virtual object provides the user with feedback indicating information about the characteristics of the virtual object. Providing improved tactile feedback enhances the operability of the device (e.g., by helping the user provide appropriate input; by reducing unnecessary additional input for placing virtual objects; and by providing a user interface that allows the user to perceive the characteristics of the virtual object without cluttering the user interface with displayed information about these characteristics), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0286] In some embodiments, when displaying a representation of a virtual object in the second user interface area, the device detects (832) adjusting the field of view 5034 of one or more cameras (e.g., Figures 5K to 5L5020) in the second user interface area according to a fixed spatial relationship (e.g., orientation and / or position) between the virtual object and a corresponding plane (e.g., floor surface 5038) in the field of view of the one or more cameras while adjusting the field of view of the one or more cameras (e.g., the virtual object is displayed on the display at an orientation and position such that a fixed angle between the representation of the virtual object and the plane is maintained (e.g., the virtual object appears to remain at a fixed position on the plane or to scroll along the plane of the field of view). For example, Figures 5K to 5L , as the device 100 moves, the virtual chair 5020 in the second user interface area including the camera's field of view 5034 maintains a fixed orientation and position relative to the floor surface 5038. In some embodiments, the virtual objects appear stationary and unchanged relative to the surrounding physical environment 5002, that is, as the field of view of one or more cameras changes as the device moves relative to the surrounding physical environment, the size, position and / or orientation of the representation of the virtual objects on the display changes as the device position and / or orientation changes. Adjusting the representation of the virtual objects based on a fixed relationship between the virtual objects and the corresponding planes (e.g., without requiring further user input to maintain the position of the virtual objects relative to the corresponding planes) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0287] In some embodiments, (e.g., at a time corresponding to replacing display of at least a portion of the first user interface area with a representation of the field of view of one or more cameras), the device display (834) continuously displays an animation (e.g., movement, rotation about one or more axes, and / or scaling) of a representation of the virtual object (e.g., virtual chair 5020) while switching from displaying the first user interface area to displaying the second user interface area (e.g., as Figures 5F to 5IAs shown). For example, the animation includes a transition from displaying a two-dimensional representation of the virtual object when the first user interface area is displayed to displaying a three-dimensional representation of the virtual object when the second user interface area is displayed. In some embodiments, the three-dimensional representation of the virtual object has a predefined orientation relative to the current orientation of a portion of the physical environment captured in the field of view of one or more cameras. In some embodiments, when transitioning to the augmented reality view, the representation of the virtual object is moved, resized, and reoriented so that the virtual object reaches a new position on the display (e.g., the center of the augmented reality view or another predefined position in the augmented reality view) from an initial position on the display, and during or at the end of the movement, the virtual object is reoriented so that the virtual object is at a fixed angle relative to a plane detected in the field of view of the camera (e.g., a physical surface that can support the representation of the virtual object, such as a vertical wall or a horizontal floor surface). In some embodiments, when the animated transition occurs, the lighting of the virtual object and / or the shadow cast by the virtual object is adjusted (e.g., to match the ambient lighting detected in the field of view of one or more cameras). Displaying the animation when the representation of the virtual object switches from displaying the first user interface area to displaying the second user interface area provides feedback to the user indicating that the first input meets the first criterion. Providing improved feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0288] In some embodiments, while the second user interface area is displayed on the display, the device detects (836) a second input via a second contact (e.g., contact 5040), wherein the second input includes (optionally, a press or touch input via the second contact to select a representation of the virtual object and) movement of the second contact along the first path on the display (e.g., as Figures 5N to 5P ), and in response to detecting a second input through a second contact, the device moves the representation of the virtual object (e.g., virtual chair 5020) in the second user interface area along a second path that corresponds to the first path (e.g., is the same as the first path or is constrained by the first path). In some embodiments, the second contact is different from the first contact and is detected after the first contact is lifted off (e.g., as shown in FIG. Figures 5N to 5P As shown in the contact 5040, which Figures 5C to 5F5086 is lifted off the touch-sensitive surface). In some embodiments, the second contact is the same as the first contact that is continuously maintained on the touch-sensitive surface (e.g., as indicated by input made through contact 5086, which satisfies the AR trigger criteria and is then moved on touch screen 112 to move virtual light 5084). In some embodiments, a swipe input on a virtual object causes the virtual object to rotate, with the movement of the virtual object optionally being constrained by a plane in the camera's field of view (e.g., a swipe input causes the representation of a chair to rotate on the plane of the floor in the camera's field of view). Moving the representation of the virtual object in response to detecting the input provides feedback to the user indicating that the position of the displayed virtual object can be moved in response to the user input. Providing improved feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0289] In some embodiments, as the representation of the virtual object moves along the second path based on the movement of the contact and the corresponding plane corresponding to the virtual object, the device adjusts (838) the size of the representation of the virtual object (e.g., based on the virtual distance from the representation of the virtual object to the user to maintain an accurate perspective of the virtual object in the field of view). Figures 5N to 5P , the size of the virtual chair 5020 decreases as the virtual chair moves deeper into the camera's field of view 5034, away from the device 100, and toward the table 5004. Adjusting the size of the representation of the virtual object as the representation of the virtual object moves along the second path based on the movement of the contact and the plane corresponding to the virtual object (e.g., without requiring further user input to adjust the size of the representation of the virtual object to maintain the representation of the virtual object at a realistic size relative to the environment in the camera's field of view) enhances the operability of the device and, in turn, reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0290] In some embodiments, the device maintains (840) the first size (e.g., as shown in FIG. 5A ) of the representation of the virtual object (e.g., virtual light 5084) as the representation of the virtual object moves along the second path. Figures 5AI to 5AL ), the device detects termination of the second input via the second contact (e.g., including detecting lift-off of the second contact, as shown). Figures 5AL to 5AM 5046 ), and in response to detecting termination of the second input by the second contact, the device places the representation of the virtual object at a drop location in the second user interface area (e.g., on the desktop 5046 ), and displays the representation of the virtual object having a second size at the drop location in the second user interface area, the second size being different from the first size (e.g., Figure 5AMThe size of the virtual lamp 5084 after the input termination by the contact 5086 is different from Figure 5AL In one embodiment, the virtual light 5084 is displayed as being the size of the virtual light 5084 before the input by contact 5086 is terminated. For example, while being contact-dragged, the size and viewing angle of the object do not change, and when the object lands at a final location in the augmented reality view, the object is displayed with a size and viewing angle determined based on a physical location in the physical environment corresponding to the drop location of the virtual object shown in the field of view of the camera, such that based on determining the drop location to be a first location in the field of view of the camera, the object has a second size, and based on determining the drop location to be a second location in the field of view of the camera, the object has a third size different from the second size, wherein the second size and the third size are selected based on a distance between the drop location and one or more cameras. Displaying a representation of the virtual object with a changed size in response to detecting termination of the second input to move the virtual object (e.g., without requiring further user input to adjust the size of the virtual object to maintain the virtual object at a realistic size relative to the environment in the field of view of the camera) enhances the operability of the device, which in turn reduces power usage and extends battery life of the device by enabling the user to use the device more quickly and efficiently.

[0291] In some embodiments, based on determining that movement of the second contact along the first path on the display satisfies a second criterion (e.g., at the end of the first path, the contact is within a threshold distance, or outside an edge (e.g., a bottom edge, a top edge, and / or a side edge) of the display or an edge of the second user interface area), the device (842): stops displaying the second user interface area including a representation of the field of view of one or more cameras, and redisplays the (complete) first user interface area with a representation of the virtual object (e.g., if a portion of the first user interface area was previously displayed simultaneously with the second user interface area, then after the second user interface area is no longer displayed, the device displays the complete first user interface area). For example, in response to movement of contact 5054 to drag the virtual chair 5054 to the edge of the touch screen 112, as Figure 5V to Figure 5X As shown, the camera's field of view 5034 is stopped from being displayed, and the complete instant messaging user interface 5008 is redisplayed, as shown in FIG. Figures 5Y to 5AD In some embodiments, as the contact approaches the edge of the display or the edge of the second user interface area, the second user interface area fades out (e.g., Figure 5X to Figure 5Y ), and / or the first user interface area (the portion thereof that is not displayed or blocked) fades in (e.g., as Figures 5Z to 5AAIn some embodiments, the gesture for transitioning from a non-AR view (e.g., a first user interface area) to an AR view (e.g., a second user interface area) is the same as the gesture for transitioning from an AR view to a non-AR view. For example, a drag gesture on a virtual object that exceeds a threshold position in the currently displayed user interface (e.g., within a threshold distance of the boundary of the currently displayed user interface area, or exceeds the boundary of the currently displayed user interface area) causes a transition from the currently displayed user interface area to the corresponding user interface area (e.g., from displaying the first user interface area to displaying the second user interface area, or alternatively, from displaying the second user interface area to displaying the first user interface area). In some embodiments, a visual indication is shown before the first criterion / second criterion is met (e.g., fading out the currently displayed user interface area and fading in the corresponding user interface), and the visual indication is reversible if the input continues and the first criterion / second criterion is not met before the termination of the input is detected (e.g., lift-off of the contact). Redisplaying the first user interface in response to detecting input that meets the input criteria provides additional control options without cluttering the second user interface with additionally displayed controls (e.g., controls for displaying the first user interface from the second user interface). Providing additional control options without cluttering the second user interface with additional displayed controls enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0292] In some embodiments, at a time corresponding to redisplaying the first user interface area, the device displays (844) an animated transition (e.g., movement, rotation about one or more axes, and / or scaling) from displaying a representation of the virtual object in the second user interface area to displaying a representation of the virtual object in the first user interface area (e.g., as Figures 5AB to 5AD Displaying an animated transition from displaying a representation of a virtual object in the second user interface to displaying a representation of the virtual object in the first user interface (e.g., without requiring further user input to reposition the virtual object in the first user interface) enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0293] In some embodiments, as the second contact moves along the first path, the device changes (846) the visual appearance of (e.g., highlights, marks, outlines, and / or otherwise visually changes the appearance of) one or more corresponding planes identified in the field of view of the one or more cameras, the one or more corresponding planes corresponding to the current location of the contact. For example, when contact 5042 moves along the first path, the device changes (e.g., highlights, marks, outlines, and / or otherwise visually changes the appearance of the one or more planes) the one or more corresponding planes identified in the field of view of the one or more cameras, the one or more corresponding planes corresponding to the current location of the contact. Figures 5O to 5PAs the virtual chair 5020 is dragged along the path indicated by arrows 5042 and 5044 in FIG. 5 , the floor surface 5038 is highlighted (e.g., with Figure 5M In some embodiments, based on determining that the contact is at a position corresponding to the first plane detected in the field of view of the camera, the first plane is highlighted. Based on determining that the contact has moved to a position corresponding to the second plane detected in the field of view of the camera (e.g., as Figures 5S to 5U 5046). In some embodiments, multiple planes are highlighted simultaneously. In some embodiments, a first plane of a plurality of visually altered planes is visually altered in a manner different from the manner in which the other planes are visually altered to indicate that contact is at a location corresponding to the first plane. Changing the visual appearance of one or more corresponding planes identified in the camera's field of view provides feedback to the user indicating that the plane has been identified (e.g., a virtual object can be positioned relative to the plane). Providing improved visual feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0294] In some embodiments, in response to detecting the first input by contact, based on determining that the first input by contact meets third (e.g., staging user interface display) criteria (e.g., the staging user interface display criteria is a criterion configured to identify a swipe input, a touch-and-hold input, a press input, a tap input, or a force press with an intensity greater than a predefined intensity threshold), the device displays (848) a third user interface area on the display, which includes replacing the display of at least a portion of the first user interface area (e.g., including a 3D model of the virtual object replacing the 2D image of the virtual object). In some embodiments, before displaying the staging user interface (e.g., as described with reference to Figure 6IIn some embodiments, when another input is detected while the virtual object is displayed in the staging user interface and the input meets the criteria for transitioning to displaying the second user interface area, the device replaces the display of the staging user interface with the second user interface area while continuing to display the virtual object. Further details are described with respect to method 900. Displaying a third user interface based on determining that the first input meets the third criteria provides additional control options without cluttering the first user interface with the additional displayed controls (e.g., controls for displaying the third user interface from the first user interface). Providing additional control options without cluttering the second user interface with the additional displayed controls enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0295] In some embodiments, based on determining that the first input by contact (e.g., a swipe input corresponding to scrolling the first user interface area or a tap input corresponding to a request to display a web page or email corresponding to content in the first user interface area) does not meet the first (e.g., AR trigger) criteria, the device maintains (850) display of the first user interface area without replacing display of at least a portion of the first user interface area with a representation of the field of view of one or more cameras (e.g., as described with reference to FIG. Figures 6B to 6C Using a first criterion to determine whether to maintain display of the first user interface area or whether to continue displaying representations of virtual objects when replacing display of at least a portion of the first user interface area with the field of view of one or more cameras enables multiple different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to the input (e.g., by replacing display of at least a portion of the user interface with the field of view of one or more cameras, or by maintaining display of the first user interface area without replacing display of at least a portion of the first user interface area with representations of the field of view of one or more cameras) increases the efficiency with which a user is able to perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends battery life of the device by enabling the user to use the device more quickly and efficiently.

[0296] It should be understood that Figures 8A to 8EThe specific order in which the operations have been described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 900 and 1000) also apply in a similar manner to the details of the processes described above with respect to Figures 8A to 8E Method 800 is described. For example, the contacts, inputs, virtual objects, user interface areas, intensity thresholds, tactile outputs, fields of view, movements, and / or animations described above with reference to method 800 optionally have one or more of the features of the contacts, inputs, virtual objects, user interface areas, intensity thresholds, tactile outputs, fields of view, movements, and / or animations described herein with reference to other methods described herein (e.g., methods 900, 1000, 16000, 17000, 18000, 19000, and 20000). For the sake of brevity, these details are not repeated here.

[0297] 9A to 9D is a flowchart illustrating method 900 for displaying a first representation of a virtual object in a first user interface area, a second representation of the virtual object in a second user interface area, and a third representation of the virtual object having a representation of a field of view of one or more cameras, according to some embodiments. Method 900 is performed on an electronic device (e.g., a device having a display, a touch-sensitive surface, and one or more cameras (e.g., one or more rear-facing cameras on a side of the device opposite the display and the touch-sensitive surface). Figure 3 Device 300 or Figure 1A The method 900 is performed at a portable multifunction device 100 in the embodiment of the present invention. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on or integrated with the display. In some embodiments, the display and the touch-sensitive surface are separate. Some operations in method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0298] As described below, method 900 involves detecting input via contact at a touch-sensitive surface of a device for displaying a representation of a virtual object in a first user interface (e.g., a two-dimensional graphical user interface). In response to the first input, the device uses criteria to determine whether to display a second representation of the virtual object in a second user interface (e.g., a staging user interface in which the three-dimensional representation of the virtual object can be moved, resized, and / or reoriented). While the second representation of the virtual object is displayed in the second user interface, in response to the second input, the device changes display properties of the second representation of the virtual object based on the second input or displays a third representation of the virtual object in a third user interface that includes a field of view of one or more cameras of the device. Enabling multiple different types of operations to be performed in response to the input (e.g., by changing display properties of the virtual object or displaying the virtual object in the third user interface) increases the efficiency with which a user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0299] The device displays (902) a first representation of a virtual object (e.g., a graphical representation of a three-dimensional object, such as a virtual chair 5020, a virtual lamp 5084, a shoe, furniture, a hand tool, a decoration, a person, an emoticon, a game character, a virtual piece of furniture, etc.) in a first user interface area (e.g., a two-dimensional graphical user interface or a portion thereof (e.g., a browsable list of furniture images, an image containing one or more selectable objects, etc.)) on the display 112. For example, the first user interface area is as follows Figure 6A Instant messaging user interface 5008 is shown. In some embodiments, in addition to the image of the physical environment surrounding the device, the first user interface area also includes a background (e.g., the background of the first user interface area is a preselected background color / pattern or background image that is different from the output image captured simultaneously by the one or more cameras and different from the real-time content in the field of view of the one or more cameras).

[0300] While a first representation of a virtual object is displayed in a first user interface area on the display, the device detects (904) a first input made by a first contact at a location on the touch-sensitive surface that corresponds to the first representation of the virtual object on the display (e.g., the first contact is detected on the first representation of the virtual object on the touch screen display, or the first contact is detected on an affordance (e.g., a toggle control 6018) displayed in the first user interface area concurrently with the first representation of the virtual object, the affordance being configured to trigger display of an AR view (e.g., the camera's field of view 6036) and / or a staging user interface 6010 including a representation of the virtual object (e.g., the virtual chair 5020) when invoked by the first contact). For example, the first input is as described with reference to Figures 6E to 6I Input performed via contact 6006 is described.

[0301] In response to detecting a first input made through a first contact, and based on determining that the first input made through the first contact satisfies a first (e.g., staging trigger) criteria (e.g., the staging trigger criteria is configured to recognize a swipe input, a touch-and-hold input, a press input, a tap input, a downward touch of the contact, an initial movement of the contact, or another type of predefined input gesture, the staging trigger criteria being associated with triggering activation of a camera and / or triggering detection of a field of view plane in the field of view of the camera), the device displays (906) a second representation of the virtual object in a second user interface area that is different from the first user interface area (e.g., the second user interface area is a staging user interface 6010 that does not include the field of view of the camera and includes a simulated three-dimensional space in which the three-dimensional representation of the virtual object can be manipulated (e.g., rotated or moved) in response to the user input). For example, in Figures 6E to 6H , based on determining that the input through contact 6006 has increased to above the deep press intensity threshold IT D The virtual chair object 5020 is displayed in the staging user interface 6010 (e.g., as Figure 6I ), which is different from the instant messaging user interface 5008 (e.g., Figure 6E shown).

[0302] In some embodiments, in response to detecting the first input, and based on determining that the first input satisfies the staging trigger criteria, the device displays a first animated transition that illustrates a transition from the first orientation as shown in the first user interface area (e.g., as shown in the first user interface area). Figure 6E The first orientation of the virtual chair 5020 shown in the instant messaging user interface 5008 in FIG. 5 is moved and reoriented to a second orientation (e.g., the second orientation of the virtual chair 5020 determined based on the gantry plane 6014, as shown in FIG. 5 ). Figure 6I6014).

[0303] While displaying the second representation of the virtual object in the second user interface area, the device detects (908) a second input (e.g., Figures 6Q to 6T In some embodiments, detecting the second input includes: detecting one or more second contacts at locations on the touch screen corresponding to a second representation of the virtual object; detecting a second contact on an affordance that is configured to trigger display of an augmented reality view of the physical environment surrounding the device when invoked by the second contact; detecting movement of the second contact; and / or detecting lift-off of the second contact. In some embodiments, the second input is a continuation of the first input made by the same contact (e.g., the second input is a continuation of the first input made by the same contact). Figures 6E to 6I After the first input by contact 6006 is shown Figures 6Q to 6T The input is made through contact 6034 as shown (e.g., the contact is not lifted)), or it is a separate input made through an entirely different contact (e.g., the second input is made in the Figures 6E to 6I After the first input by contact 6006 is shown Figures 6Q to 6T The input made by contact 6034 as shown (e.g., contact lift-off)), or the continuation of the input made by another contact (e.g., the second input is made in the Figures 6E to 6I After the first input by contact 6006 is shown Figures 6J to 6L For example, the second input may be a continuation of a swipe input, a second tap input, a second press input, a press input subsequent to the first input, a second touch-and-hold input, a continuous touch continuing from the first input, etc.

[0304] In response to detecting a second input (910): based on determining that the second input corresponds to a request to manipulate a virtual object in a second user interface area (e.g., without transitioning to an augmented reality view), the device changes display properties of a second representation of the virtual object within the second user interface area based on the second input, and based on determining that the second input corresponds to a request to display the virtual object in an augmented reality environment, the device displays a third representation of the virtual object having a representation of a field of view of one or more cameras (e.g., the device displays a third user interface including a field of view 6036 of one or more cameras and places a three-dimensional representation of the virtual object (e.g., virtual chair 5020) on a virtual plane (e.g., floor surface 5038) detected within the field of view of the camera that corresponds to a physical plane (e.g., floor) in the physical environment 5002 surrounding the device).

[0305] In some embodiments, the second input corresponding to the request to manipulate the virtual object in the second user interface area is a pinch or swipe made by a second contact on the touch-sensitive surface at a location corresponding to the second representation of the virtual object in the second user interface area. Figures 6J to 6L Input via contact 6006 as shown or as Figures 6N to 6O Input is shown via contacts 6026 and 6030.

[0306] In some embodiments, the second input corresponding to the request to display the virtual object in the augmented reality environment is a tap input, a press input, or a touch-hold or press input followed by a drag input on the touch-sensitive surface at or from a location on the touch-sensitive surface corresponding to the representation of the virtual object in the second user interface area. For example, the second input is Figures 6Q to 6T A deep press input through contact 6034 is shown.

[0307] In some embodiments, changing the display properties of the second representation of the virtual object within the second user interface area based on the second input includes rotating about one or more axes (e.g., by swiping vertically and / or horizontally), resizing (e.g., pinching to resize), tilting about one or more axes (e.g., by tilting the device), changing the viewing angle (e.g., by moving the device horizontally, which in some embodiments is used to analyze the field of view of one or more cameras to detect one or more field of view planes), and / or changing the color of the representation of the virtual object. For example, changing the display properties of the second representation of the virtual object includes, in response to the second input, rotating the second representation of the virtual object. Figures 6J to 6K The horizontal swipe gesture made by contact 6006 shown rotates the virtual chair 5020; in response to Figures 6K to 6L A diagonal swipe gesture made by contact 6006 as shown rotates the virtual chair 5020; or in response to Figures 6N to 6O The illustrated separation gesture performed by contacts 6026 and 6030 increases the size of the virtual chair 5020. In some embodiments, the amount of change in the displayed property of the second representation of the virtual object is associated with the amount of change in the property of the second input (e.g., the distance or speed of movement of the contact, the strength of the contact, the duration of the contact, etc.).

[0308] In some embodiments, the second input is determined to be consistent with the image in the augmented reality environment (e.g., in the field of view 6036 of one or more cameras, as shown in FIG. Figure 6T In response to a request to display a virtual object as described above, the device displays a second animated transition that shows the virtual object being displayed from a corresponding orientation relative to the virtual plane on the display (e.g., Figure 6R The virtual chair 5020 shown in FIG. 5 is reoriented to a third orientation (eg, Figure 6T The three-dimensional representation of the virtual object is reoriented to a third orientation that is based on a current orientation of a portion of the physical environment captured in the field of view of the one or more cameras. For example, the three-dimensional representation of the virtual object is reoriented so that the three-dimensional representation of the virtual object is at a fixed angle relative to a predefined plane (e.g., floor surface 5038) identified in a real-time image of the physical environment 5002 captured in the field of view of the camera (e.g., a physical surface that can support the three-dimensional representation of the virtual object, such as a vertical wall or a horizontal floor surface). In some embodiments, in at least one aspect, the orientation of the virtual object in the augmented reality view is constrained by the orientation of the virtual object in the staging user interface. For example, when the virtual object is transitioned from the staging user interface to the augmented reality view, the rotation angle of the virtual object about at least one axis of the three-dimensional coordinate system is maintained (e.g., as shown with reference to Figures 6Q to 6U Described, as referenced Figures 6J to 6K In some embodiments, the rotation of the virtual chair 5020 is maintained. In some embodiments, the light source projected on the representation of the virtual object in the second user interface area is a virtual light source. In some embodiments, the third representation of the virtual object in the third user interface area is illuminated by a real-world light source (e.g., as detected in and / or determined by the field of view of the one or more cameras).

[0309] In some embodiments, the first criterion includes (912) a criterion that is satisfied when (e.g., as determined below) the first input includes a tap input made by a first contact at a location on the touch-sensitive surface corresponding to the virtual object indicator 5022 (e.g., an indicator of a representation of a virtual object on an overlapping and / or adjacent display, such as an icon). For example, the virtual object indicator 5022 provides an indication that the virtual object corresponding to the virtual object indicator is visible in the staging view (e.g., the staging user interface 6010) and the augmented reality view (e.g., the camera's field of view 6036) (e.g., as described in more detail below with reference to method 1000). Determining whether to display a second representation of the virtual object in the second user interface area based on whether the first input includes a tap input enables multiple different types of operations to be performed in response to the first input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user is able to perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0310] In some embodiments, the first criterion includes (914) a criterion that is satisfied when (e.g., determined as follows) the first contact is maintained at a location on the touch-sensitive surface corresponding to the first representation of the virtual object with a movement less than a threshold movement amount for at least a predefined threshold amount of time (e.g., a long press time threshold). For example, the first criterion is satisfied by a touch hold input. In some embodiments, the first criterion includes a criterion that requires the first contact to be moved after the first contact is maintained at a location on the touch-sensitive surface corresponding to the representation of the virtual object with a movement less than a threshold movement amount for at least a predefined threshold amount of time in order to satisfy the criterion. For example, the first criterion is satisfied by a touch hold input followed by a drag input. Determining whether to display the second representation of the virtual object in the second user interface area based on whether the contact is maintained at a location on the touch-sensitive surface corresponding to the representation of the virtual object with a movement less than a threshold movement amount for at least a predefined threshold amount of time enables multiple different types of operations to be performed in response to the first input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0311] In some embodiments, the first criterion includes (916) determining that the characteristic intensity of the first contact increases above a first intensity threshold (e.g., a deep press intensity threshold IT D ) is met. For example, if Figures 6Q to 6T As described, when the characteristic intensity of contact 6034 increases to above the deep press intensity threshold IT D, the criterion is met, as indicated by the intensity level meter 5028. In some embodiments, based on determining that the contact meets the criteria for identifying another type of gesture (e.g., a tap), while maintaining the display of the virtual object, the device also performs another predefined function in addition to triggering the second (e.g., staging) user interface. In some embodiments, the first criterion requires that the first input is not a tap input (e.g., a force tap input with an intensity above a threshold intensity detected before the contact is lifted off within a tap time threshold of the initial downward touch of the contact). In some embodiments, the first criterion includes a criterion requiring the first contact to be moved after the intensity of the first contact exceeds the first intensity threshold in order to meet the criterion. For example, the first criterion is met by a press input followed by a drag input. Determining whether to display the virtual object in the second user interface area based on whether the characteristic intensity of the contact increases to above the first intensity threshold enables multiple different types of operations to be performed in response to the first input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0312] In some embodiments, in response to detecting a first input made through a first contact, and based on determining that the first input made through the first contact satisfies a second criterion (e.g., an interface scrolling criterion), the device scrolls (918) the first user interface area (and the representation of the virtual object) in a direction corresponding to the direction of movement of the first contact (e.g., the first criterion is not met, and displaying the representation of the virtual object in the second user interface area is abandoned), wherein the second criterion requires that the first input include movement of the first contact in a direction across the touch-sensitive surface greater than a threshold distance (e.g., the second criterion is satisfied by a swipe gesture, such as a vertical swipe or a horizontal gesture). For example, as referenced Figures 6B to 6CAs described, an upward vertical swipe gesture performed by contact 6002 causes the instant messaging user interface 5008 and the virtual chair 5020 to scroll upward. In some embodiments, the first criterion further requires that the first input include movement of the first contact greater than a threshold distance in order to meet the first criterion, and the device determines whether the first input meets the first criterion (e.g., staging trigger criterion) or the second criterion (e.g., interface scrolling criterion) based on whether the initial portion of the first input (e.g., touch hold or press on the representation of the virtual object) meets the object selection criterion. In some embodiments, a swipe input initiated at a touch position other than the position of the virtual object and the AR icon of the virtual object meets the second criterion. Based on whether the first input meets the second criterion, it is determined in response to the first input whether to scroll the first user interface area, which enables a variety of different types of operations to be performed in response to the first input. Enabling a variety of different types of operations to be performed in response to the input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0313] In some embodiments, in response to detecting the first input through the first contact, and based on determining that the first input through the first contact satisfies third (e.g., AR trigger) criteria, the device displays (920) a third representation of the virtual object having a representation of the field of view of the one or more cameras. For example, as shown in FIG. Figures 6AD to 6AG As depicted, a long touch input through the contact 6044 and a subsequent upward drag input of dragging the virtual chair 5020 through the contact 6044 cause the camera's field of view 6036 to display the virtual chair 5020 .

[0314] In some embodiments, the third criterion includes, for example, a criterion determined to be satisfied based on: one or more cameras is active; the device orientation falls within a defined range (e.g., from a defined original orientation, a defined range of rotation angles about one or more axes); input through contact includes a selection input (e.g., a long touch) followed by a drag input (movement of the contact to move a virtual object on the display) (e.g., moving to within a predetermined distance from an edge of the display); a characteristic intensity of the contact increases above an AR trigger intensity threshold (e.g., a light press threshold IT); L or deep compression threshold IT D); the duration of the contact increases to greater than an AR trigger duration threshold (e.g., a long press threshold); and / or the distance the contact moves increases to greater than an AR trigger distance threshold (e.g., a long swipe threshold). In some embodiments, a control (e.g., a toggle control 6018) for displaying a representation of the virtual object in a second user interface area (e.g., the staging user interface 6010) is displayed in a user interface (e.g., a third user interface area that replaces at least a portion of the second user interface area) that includes a representation of the virtual object and the field of view 6036 of one or more cameras.

[0315] In some embodiments, when transitioning directly from a first user interface area (e.g., a non-AR, non-staged, touchscreen UI view) to a third user interface area (e.g., an augmented reality view), the device displays an animated transition showing a three-dimensional representation of a virtual object being reoriented from a corresponding orientation represented in the touchscreen UI (e.g., a non-AR, non-staged view) on the display to an orientation predefined relative to the current orientation of a portion of the physical environment captured in the field of view of the one or more cameras. For example, Figures 6AD to 6AJ As shown, when the first user interface area (e.g., instant messaging user interface 5008, such as Figure 6AD ) directly transitions to a third user interface area (e.g., an augmented reality user interface including the camera's field of view 6036, as shown Figure 6AJ As shown), the virtual chair 5020 is as follows Figures 6AD to 6AH The first orientation shown is changed to a predefined orientation (e.g., as shown) relative to the floor surface 5038 in the physical environment 5002 as captured in the camera's field of view 6036. Figure 6AJ 5038) relative to a predefined plane identified in the real-time image of the physical environment 5002 (e.g., a physical surface that can support the three-dimensional representation of the virtual object, such as a vertical wall or a horizontal floor surface (e.g., floor surface 5038)). Determining whether to display a third representation of the virtual object with the camera's field of view in response to the first input based on whether the first input meets a third criterion enables multiple different types of operations to be performed in response to the first input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0316] In some embodiments, in response to detecting a first input via a first contact, the device determines (922) a current device orientation of the device (e.g., an orientation relative to the physical environment surrounding the device) via one or more device orientation sensors, and a third criterion (e.g., an AR trigger criterion) requires that the current device orientation be within a first orientation range in order to satisfy the third criterion (e.g., the second criterion is satisfied when the angle between the device and the ground is less than a threshold angle, indicating that the device is sufficiently parallel to the ground (to bypass the gap state)). In some embodiments, the first criterion (e.g., a staging trigger criterion) requires that the current device orientation be within a second orientation range in order to satisfy the first criterion (e.g., the first criterion is satisfied when the angle between the device and the ground is within a threshold and reaches 90 degrees, indicating that the device is sufficiently vertical relative to the ground to enter the gap state in the first place). Based on whether the device orientation is within the orientation range, determining whether to display a third representation of a virtual object with the camera's field of view in response to the first input enables multiple different types of operations to be performed in response to the first input. Enabling multiple different types of operations to be performed in response to input increases the efficiency with which a user can perform those operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0317] In some embodiments, at least one display attribute of the second representation of the virtual object (e.g., size, shape, respective angles about yaw, pitch, and roll axes, etc.) is applied (924) to the third representation of the virtual object. Figures 6Q to 6U As described, when the third representation of the virtual chair 5020 is displayed in the augmented reality view that includes the camera's field of view 6036 (e.g., as Figure 6U As shown), as reference Figures 6J to 6KThe rotation of the second representation of the virtual chair 5020 described as applied in the staging user interface 6010 is maintained. In some embodiments, in at least one aspect, the orientation of the virtual object in the augmented reality view is constrained by the orientation of the virtual object in the staging user interface. For example, when the virtual object is transitioned from the staging view to the augmented reality view, the rotation angle of the virtual object around at least one axis of a predefined three-dimensional coordinate system (e.g., yaw, pitch, and roll axes) is maintained. In some embodiments, if the second representation of the virtual object has been manipulated in some way (e.g., changing size, shape, texture, orientation, etc.) through user input, at least one display attribute of the second representation of the virtual object is only applied to the third representation of the virtual object. In other words, when the object is shown in the augmented reality view or is used to constrain the appearance of the object in the augmented reality view in one or more ways, the changes made in the staging view are maintained. Applying at least one display property of the second representation of the virtual object to the third representation of the virtual object (e.g., without requiring further user input to apply the same display property to the second representation of the virtual object and the third representation of the virtual object) enhances the operability of the device (e.g., by allowing the user to apply a rotation to the second virtual object when a large version of the virtual object is displayed in the second user interface, and apply a rotation to the third representation of the virtual object that is displayed with a representation of the field of view of one or more cameras), which in turn reduces power usage and extends battery life of the device by enabling the user to use the device more quickly and efficiently.

[0318] In some embodiments, in response to detecting at least an initial portion of a first input via a first contact (926) (e.g., including: detecting the first contact; or detecting an input via a first contact that satisfies a corresponding predefined criterion but does not satisfy the first criterion; or detecting an input that satisfies the first criterion): the device activates one or more cameras (e.g., activating the camera without immediately displaying the camera's field of view on the display), and the device analyzes the field of view of the one or more cameras to detect one or more planes in the field of view of the one or more cameras. In some embodiments, displaying the field of view 6036 of the one or more cameras is delayed after activating the one or more cameras (e.g., until a second input corresponding to a request to display a virtual object in the augmented reality environment is detected, until at least one field of view plane is detected, or until a field of view plane corresponding to an anchor plane defined for the virtual object is detected). In some embodiments, the field of view 6036 of the one or more cameras is displayed at a time corresponding to the activation of the one or more cameras (e.g., while the one or more cameras are activated). In some embodiments, the field of view 6036 of the one or more cameras is displayed before a plane is detected in the field of view of the one or more cameras (e.g., in response to detecting the first input via the contact and based on the determination, displaying the field of view of the one or more cameras). Activating the camera in response to detecting an initial portion of the first input (e.g., before displaying a third representation of the virtual object having one or more representations of the field of view of the camera) and detecting one or more field of view planes by analyzing the field of view of the camera improves the efficiency of the device (e.g., by reducing the amount of time required to determine the position and / or orientation of the third representation of the virtual object relative to the corresponding planes in the field of view of the camera), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0319] In some embodiments, in response to detecting a corresponding plane (e.g., floor surface 5038) in the field of view of one or more cameras, a device having one or more tactile output generators 167 outputs (928) a tactile output indicating that a corresponding plane has been detected in the field of view of the one or more cameras. In some embodiments, the field of view 6036 may be shown before the field of view plane is identified. In some embodiments, after at least one field of view plane is detected or after all field of view planes are identified, additional user interface controls and / or icons are overlaid on the real-world image in the field of view. Outputting t...

Claims

1. A method for interacting with a virtual three-dimensional object, comprising: At a device having a display generating component and one or more input devices: displaying, by the display generation component, a representation of a first perspective of a virtual three-dimensional object in a first user interface area visible via the display generation component; while displaying the representation of the first perspective of the virtual three-dimensional object in the first user interface area, detecting a first input that includes a movement and corresponds to a request to rotate the virtual three-dimensional object in accordance with the movement to display a portion of the virtual three-dimensional object that is not visible from the first perspective of the virtual three-dimensional object; In response to detecting the first input comprising the movement: Based on determining that the first input corresponds to a request to rotate the virtual three-dimensional object about a first axis, rotating the virtual three-dimensional object relative to the first axis by an amount determined based on a magnitude of the movement of the first input, and the amount is constrained by a restriction on the movement that restricts rotation of the virtual three-dimensional object relative to the first axis beyond a threshold rotation amount; as well as Based on determining that the first input corresponds to a request to rotate the virtual three-dimensional object about a second axis different from the first axis, the device rotates the virtual three-dimensional object relative to the second axis by an amount determined based on the magnitude of the movement of the first input, wherein for inputs having a magnitude of movement above a corresponding threshold, the device rotates the virtual three-dimensional object relative to the second axis by more than the threshold rotation amount.

2. The method according to claim 1, comprising: In response to detecting the first input: determining that the first input corresponds to a request to rotate the virtual three-dimensional object about the first axis based on determining that the first input includes a first movement of a contact across a touch-sensitive surface in a first direction and determining that the first movement of the contact in the first direction satisfies a first criterion for rotating the representation of the virtual three-dimensional object relative to the first axis, wherein the first criterion includes a requirement that the first input includes a movement in the first direction that is greater than a first threshold amount of movement in order to satisfy the first criterion; as well as Based on determining that the first input includes a second movement of the contact across the touch-sensitive surface in a second direction, and determining that the second movement of the contact in the second direction satisfies a second criterion for rotating the representation of the virtual three-dimensional object relative to the second axis, wherein the second criterion includes that the first input includes a movement in the second direction greater than a second threshold movement amount in order to satisfy the second criterion, determining that the first input corresponds to a request to rotate the virtual three-dimensional object about the second axis, wherein the first threshold movement amount is greater than the second threshold movement amount.

3. The method according to any one of claims 1 to 2, wherein: The rotation of the virtual three-dimensional object relative to the first axis occurs with a first degree of correspondence between a characteristic value of a first input parameter of the first input and an amount of rotation applied to the virtual three-dimensional object about the first axis; and the rotation of the virtual three-dimensional object about the second axis occurs with a second degree of correspondence between the characteristic value of the first input parameter of the first input and the amount of rotation applied to the virtual three-dimensional object about the second axis; and The first degree of correspondence involves a smaller rotation of the virtual three-dimensional object relative to the first input parameter than the second degree of correspondence.

4. The method according to any one of claims 1 to 2, comprising: detecting an end of the first input; as well as After detecting the end of the first input, continuing to rotate the virtual three-dimensional object based on a magnitude of the movement of the first input before detecting the end of the first input, comprises: based on determining that the virtual three-dimensional object is rotating relative to the first axis, slowing the rotation of the virtual three-dimensional object relative to the first axis by a first amount, the first amount being proportional to a magnitude of the rotation of the virtual three-dimensional object relative to the first axis; as well as Based on determining that the virtual three-dimensional object is rotating relative to the second axis, the rotation of the virtual three-dimensional object relative to the second axis is slowed down by a second amount, the second amount being proportional to the magnitude of the rotation of the virtual three-dimensional object relative to the second axis, wherein the second amount is different from the first amount.

5. The method according to any one of claims 1 to 2, comprising: detecting an end of the first input; as well as After detecting said end of said first input: inverting at least a portion of the rotation of the virtual three-dimensional object relative to the first axis based on determining that the virtual three-dimensional object has been rotated relative to the first axis beyond a corresponding rotation threshold; as well as Based on determining that the rotation of the virtual three-dimensional object relative to the first axis does not exceed the corresponding rotation threshold, reversing the rotation of the virtual three-dimensional object relative to the first axis is abandoned.

6. The method according to any one of claims 1 to 2, comprising: In response to detecting the first input comprising movement, based on determining that the first input corresponds to a request to rotate the virtual three-dimensional object about a third axis different from the first axis and the second axis, abandoning rotating the virtual three-dimensional object relative to the third axis.

7. The method according to any one of claims 1 to 2, comprising: displaying a representation of a shadow cast by the virtual three-dimensional object while displaying the representation of the first perspective of the virtual three-dimensional object in the first user interface area; as well as The shape of the representation of the shadow is changed depending on the rotation of the virtual three-dimensional object relative to the first axis and / or the second axis.

8. The method according to claim 7, comprising: When rotating the virtual three-dimensional object in the first user interface area: Based on determining to display the virtual three-dimensional object at a second viewing angle revealing a predefined bottom of the virtual three-dimensional object, displaying the representation of the shadow at the second viewing angle of the virtual three-dimensional object is abandoned.

9. The method according to any one of claims 1 to 2, comprising: After rotating the virtual three-dimensional object in the first user interface area, detecting a second input corresponding to a request to reposition the virtual three-dimensional object in the first user interface area; as well as In response to detecting the second input, a representation of the predefined original perspective of the virtual three-dimensional object is displayed in the first user interface area.

10. The method according to any one of claims 1 to 2, comprising: while displaying the virtual three-dimensional object in the first user interface area, detecting a third input corresponding to a request to resize the virtual three-dimensional object; as well as In response to detecting the third input, a size of the representation of the virtual three-dimensional object in the first user interface area is adjusted according to a magnitude of movement of the third input.

11. The method according to claim 10, comprising: while adjusting the size of the representation of the virtual three-dimensional object in the first user interface area, detecting that the size of the representation of the virtual three-dimensional object has reached a predefined default display size of the virtual three-dimensional object; as well as In response to detecting that the size of the representation of the virtual three-dimensional object has reached the predefined default display size of the virtual three-dimensional object, a tactile output is generated to indicate that the virtual three-dimensional object is displayed at the predefined default display size.

12. The method of any one of claims 1 to 2, wherein the device comprises one or more cameras, and the method comprises: while displaying the representation of the third perspective of the virtual three-dimensional object in the first user interface area, detecting a fourth input corresponding to a request to display the virtual three-dimensional object in a second user interface area, the second user interface area including a field of view of one or more cameras; as well as In response to detecting the fourth input, displaying, via the display generation component, a representation of the virtual three-dimensional object on at least a portion of the field of view of the one or more cameras included in the second user interface area, wherein the field of view of the one or more cameras is a view of a physical environment in which the one or more cameras are located, and wherein displaying the representation of the virtual three-dimensional object comprises: Rotating the virtual three-dimensional object around the first axis to a predefined angle; as well as The current angle of the virtual three-dimensional object relative to the second axis is maintained.

13. The method according to any one of claims 1 to 2, comprising: while displaying the representation of the virtual three-dimensional object from a fourth perspective in the first user interface area, detecting a fifth input corresponding to a request to return to a two-dimensional user interface including the two-dimensional representation of the virtual three-dimensional object; as well as In response to detecting the fifth input: rotating the virtual three-dimensional object to illustrate a perspective of the virtual three-dimensional object corresponding to the two-dimensional representation of the virtual three-dimensional object; and The two-dimensional representation of the virtual three-dimensional object is displayed after the virtual three-dimensional object is rotated to show a respective viewing angle corresponding to the two-dimensional representation of the virtual three-dimensional object.

14. The method according to any one of claims 1 to 2, comprising: prior to displaying the representation of the first perspective of the virtual three-dimensional object, displaying a user interface including a representation of the virtual three-dimensional object, the user interface including representations of views of the virtual three-dimensional object from respective perspectives; while displaying the representation of the virtual three-dimensional object, detecting a request to display the virtual three-dimensional object; and, In response to detecting the request to display the virtual three-dimensional object, display of the representation of the virtual three-dimensional object is replaced with the virtual three-dimensional object rotated to match the corresponding viewing angle of the representation of the virtual three-dimensional object.

15. The method according to any one of claims 1 to 2, comprising: Before displaying the first user interface, displaying a two-dimensional user interface including a two-dimensional representation of the virtual three-dimensional object; while displaying the two-dimensional user interface including the two-dimensional representation of the virtual three-dimensional object, detecting a first portion of an input, the first portion satisfying a preview criterion for the two-dimensional representation of the virtual three-dimensional object; as well as In response to detecting the first portion of the input satisfying the preview criteria, displaying a preview of the virtual three-dimensional object that is larger than the two-dimensional representation of the virtual three-dimensional object.

16. The method according to claim 15, comprising: while displaying the preview of the virtual three-dimensional object, detecting a second portion of the input; as well as In response to detecting the second portion of the input: displaying a plurality of selectable options corresponding to a plurality of operations associated with the virtual three-dimensional object based on determining that the second portion of the input satisfies a menu display criterion; as well as Based on determining that the second portion of the input satisfies staging criteria, display of the two-dimensional user interface including the two-dimensional representation of the virtual three-dimensional object is replaced with the first user interface including the virtual three-dimensional object.

17. The method according to any one of claims 1 to 2, wherein the first user interface comprises a plurality of controls, and the method comprises: Before displaying the first user interface, displaying a two-dimensional user interface including a two-dimensional representation of the virtual three-dimensional object; as well as In response to detecting a request to display the virtual three-dimensional object in the first user interface: displaying the virtual three-dimensional object in the first user interface without displaying a set of one or more controls associated with the virtual three-dimensional object; as well as After displaying the virtual three-dimensional object in the first user interface, the set of one or more controls is displayed.

18. A computer system comprising: Display generated components; one or more input devices; one or more processors; and A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for executing any of the methods according to claims 1 to 17.

19. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by a computer system having a display generation component and one or more input devices, cause the computer system to perform any of the methods according to claims 1 to 17.

Images