Device, method and graphical user interface for displaying objects in a 3D scene

By combining display generation components, input devices and cameras in electronic devices, intuitive interaction and display of virtual objects is achieved, and the problems of insufficient feedback on virtual objects and cumbersome input in the prior art are solved, efficiency and user satisfaction are improved, and Save energy.

CN114327096BActive Publication Date: 2024-11-26APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111476740.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-09-24
Filing Date
2020-04-01
Publication Date
2024-11-26
Estimated Expiration
2040-04-01

AI Technical Summary

Technical Problem

In the prior art, the methods and interfaces used for virtual object interaction in augmented reality environments have insufficient feedback, cumbersome input, and time-consuming, resulting in a large cognitive burden on users and wasted energy, especially in battery-driven devices.

Method used

By adopting a combination of display generation components, input devices and cameras in electronic devices, the intuitive display and interaction of virtual objects is realized, the user input is reduced, and a more effective human-machine interface is provided, including detecting input actions and displaying visual indications according to standard responses, and adjust the properties of the virtual object.

Benefits of technology

Improves interaction efficiency and user experience in virtual/augmented reality environments, reduces energy consumption and extends battery life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114327096B_ABST
    Figure CN114327096B_ABST
Patent Text Reader

Abstract

The present disclosure provides a device, method and graphical user interface for displaying an object in a 3D scene. The electronic device displays an environment including a virtual object, the virtual object being associated with a first action triggered based on satisfying a first set of criteria. In response to a first input detected while displaying the environment: performing the first action based on determining that the first input satisfies the first set of criteria, and based on determining that the first input does not satisfy the first set of criteria but satisfies a second set of criteria, not performing the first action and instead displaying a first visual indication of one or more inputs, the one or more inputs, if executed, would cause the first set of criteria to be satisfied.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application with application number 202080034059.6, application date April 1, 2020, and invention name “Device, method and graphical user interface for displaying objects in 3D scenes”. Technical Field

[0002] The present invention generally relates to electronic devices that display virtual objects, including but not limited to electronic devices that display virtual objects in various scenarios. Background Art

[0003] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as touch-sensitive surfaces for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Example touch-sensitive surfaces include trackpads, touch-sensitive remote controls, and touch screen displays. Such surfaces are used to manipulate user interfaces on displays and objects therein. Exemplary user interface objects include digital images, videos, text, icons, and control elements (such as buttons) and other graphics.

[0004] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to generate virtual objects suitable for display in an augmented reality environment, and systems that require lack of processing for manipulating groups of virtual objects are cumbersome, impose a significant cognitive burden on the user, and detract from the experience of the virtual / augmented reality environment. In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the invention

[0005] Therefore, there is a need for a computer system with an improved method and interface for interacting with a virtual object. Such methods and interfaces optionally supplement or replace conventional methods for interacting with a virtual object. Such methods and interfaces reduce the amount, degree and / or nature of input from a user and produce a more efficient human-computer interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0006] The computer system of the present disclosure reduces or eliminates the above defects and other problems associated with interfaces for interacting with virtual objects (e.g., user interfaces for augmented reality (AR) and related non-AR interfaces). In some embodiments, the computer system includes a desktop computer. In some embodiments, the computer system is portable (e.g., a laptop, a tablet, or a handheld device). In some embodiments, the computer system includes a personal electronic device (e.g., a wearable electronic device, such as a watch). In some embodiments, the computer system has a touchpad (and / or communicates with a touchpad). In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display") (and / or communicates with a touch-sensitive display). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI in part by stylus and / or finger contact and gestures on a touch-sensitive surface. In some embodiments, these functions optionally include playing games, image editing, drawing, presentation, word processing, electronic spreadsheet making, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note taking and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0007] According to some embodiments, a method is performed at an electronic device including a display generation component and one or more input devices. The method includes displaying at least a portion of an environment including a virtual object via the display generation component, the virtual object being associated with a first action triggered based on satisfying a first set of criteria. The method also includes detecting a first input when the portion of the environment is displayed by the display generation component. The method also includes, in response to detecting the first input: performing a first action based on determining that the first input satisfies the first set of criteria; and abandoning execution of the first action and instead displaying a first visual indication of one or more inputs based on determining that the first input does not satisfy the first set of criteria but satisfies the second set of criteria, which, when executed, will cause the first set of criteria to be satisfied.

[0008] According to some embodiments, a method is performed at an electronic device including a display generation component, one or more input devices, and one or more cameras. The method includes displaying, by the display generation component: a first user interface area, a first representation of a first media item in the first user interface area, and a first user interface object for displaying a shared user interface. The method also includes detecting a first input corresponding to the shared user interface object when displaying the first user interface area. The method also includes, in response to detecting the first input corresponding to the first user interface object, displaying a shared user interface including a second user interface object to initiate a process for displaying a second representation of the first media item in an augmented reality environment. The method also includes, when displaying the shared user interface, detecting a sequence of one or more inputs including a selection of the second user interface object. The method also includes, in response to detecting a sequence of one or more inputs including a selection of the second user interface object, displaying a second representation of the first media item in a second user interface including content of at least a portion of the field of view of the one or more cameras.

[0009] According to some embodiments, a method is performed at an electronic device including a display generation component, one or more input devices, and one or more cameras. The method includes displaying a plurality of virtual objects by the display generation component. The method also includes detecting a first manipulation input pointing to a corresponding virtual object among the plurality of virtual objects when the plurality of virtual objects are displayed. The method also includes, in response to detecting a first manipulation input pointing to a corresponding virtual object among the plurality of virtual objects: changing the display properties of the corresponding virtual object based on the first manipulation input according to determining that the plurality of virtual objects are displayed together with at least a portion of the field of view of the one or more cameras; and changing the display properties of the plurality of virtual objects based on the first manipulation input according to determining that the plurality of virtual objects are displayed in a user interface that does not include the field of view of the one or more cameras.

[0010] According to some embodiments, a method is performed at an electronic device including a display generation component, one or more input devices, and one or more cameras. The method includes displaying, by the display generation component, a representation of a media item. The method also includes detecting an input corresponding to a request to display the media item in an augmented reality environment. The method also includes, in response to detecting the input corresponding to the request to display the media item in the augmented reality environment: displaying a prompt for changing the properties of the media item based on determining that the media item has properties that do not meet compatibility criteria for display in the augmented reality environment; and displaying a virtual object corresponding to the media item together with content of at least a portion of the field of view of the one or more cameras based on determining that the media item meets the compatibility criteria.

[0011] According to some embodiments, a method is performed at an electronic device including a display generation component and one or more input devices. The method includes displaying a first user interface having a first representation of content via the display generation component. The method also includes receiving a request to display a virtual model corresponding to the content via the one or more input devices when displaying the first user interface having a first representation of the content. The method also includes, in response to receiving a request to display a virtual model corresponding to the content: based on determining that the first user interface is configured to perform an operation associated with the content, displaying a virtual model of the content and a selectable user interface object for performing an operation associated with the content simultaneously; and based on determining that the first user interface is not configured to perform an operation associated with the content, displaying a virtual model of the content without displaying a selectable user interface object for performing an operation associated with the content.

[0012] According to some embodiments, an electronic device includes a display generation component, one or more input devices, optionally one or more cameras, one or more processors, and a memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, a computer-readable storage medium has instructions stored therein, which, when executed by an electronic device having a display generation component, one or more input devices, and optionally one or more cameras, causes the device to perform or cause the execution of the operations of any of the methods described herein. According to some embodiments, a graphical user interface on an electronic device having a display generation component, one or more input devices, optionally one or more cameras, a memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements of the elements displayed in any of the methods described herein, which are updated in response to the input, as described in any of the methods described herein. According to some embodiments, an electronic device includes: a display generation component, one or more input devices, and optionally one or more cameras; and a device for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, an information processing device used in an electronic device having a display generation component, one or more input devices, and optionally one or more cameras includes a device for performing or causing the execution of the operations of any of the methods described herein.

[0013] Thus, an improved method and interface for displaying virtual objects in various scenarios is provided for an electronic device having a display generation component, one or more input devices, and optionally one or more cameras, thereby increasing the effectiveness, efficiency, and user satisfaction with such devices. Such methods and interfaces can supplement or replace conventional methods for displaying virtual objects in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0015] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0016] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0017] Figure 1C is a block diagram showing a tactile output module according to some embodiments.

[0018] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0019] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0020] Figure 4A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0021] Figure 4B An exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0022] FIG. 4C to FIG. 4E Examples of dynamic intensity thresholds are shown according to some embodiments.

[0023] Figure 4F to Figure 4K A set of sample tactile output patterns according to some embodiments are shown.

[0024] FIG. 5A to FIG. 5K , Figure 5L-1 , Figure 5L-2 , Figure 5M-1 , Figure 5M-2 , Figure 5N-1 Figure 5N-2 Figure 5O-1 , Figure 5O-2 and Figures 5P to 5AK An exemplary user interface is shown for displaying a visual indication of one or more inputs that, when executed, would cause criteria for performing an action to be satisfied, according to some embodiments.

[0025] Figures 6A to 6AI An exemplary user interface for displaying media items in a user interface that includes content of at least a portion of a field of view of one or more cameras is shown according to some embodiments.

[0026] FIG. 7A to FIG. 7AQ An exemplary user interface for responding to input directed to a corresponding virtual object among multiple displayed virtual objects is shown according to some embodiments.

[0027] FIG. 8A to FIG. 8K An exemplary user interface is shown for displaying a prompt to change properties of a media item that does not meet compatibility standards for display in an augmented reality environment according to some embodiments.

[0028] 9A to 9C is a flow chart of a process for displaying a visual indication of one or more inputs that, when executed, would cause criteria for performing an action to be satisfied.

[0029] FIG. 10A to FIG. 10C is a flowchart of a process for displaying media items in a user interface that includes content of at least a portion of a field of view of one or more cameras according to some embodiments.

[0030] FIG. 11A to FIG. 11D is a flowchart of a process for responding to input directed to a corresponding virtual object among multiple displayed virtual objects according to some embodiments.

[0031] FIG. 12A to FIG. 12B is a flowchart of a process for displaying a prompt to change properties of a media item that does not meet compatibility standards for display in an augmented reality environment, according to some embodiments.

[0032] FIG. 13A to FIG. 13J An exemplary user interface is shown for simultaneously displaying a virtual model of content and selectable user interface objects for performing operations associated with the content according to some embodiments.

[0033] FIG. 14A to FIG. 14B is a flowchart of a process for simultaneously displaying a virtual model of content and a selectable user interface object for performing operations associated with the content, according to some embodiments. DETAILED DESCRIPTION

[0034] A virtual object is a graphical representation of a three-dimensional object in a virtual environment. Conventional methods of interacting with virtual objects typically lack sufficient feedback to indicate to the user the input required to achieve the desired result (e.g., perform an action associated with the virtual object). In addition, conventional methods require a series of cumbersome inputs (inputs for opening a conversion application, importing the converted items into an application for viewing the converted items in an augmented reality environment, etc.) to make the media items suitable for viewing in an augmented reality environment (e.g., an environment in which the view of the physical world is enhanced with supplementary information that provides the user with additional information not available in the physical world). The embodiments herein provide users with an intuitive way to display virtual objects in a variety of scenarios.

[0035] The systems, methods, and GUIs described herein improve user interface interactions with virtual / augmented reality environments in a variety of ways. For example, they make it easier to display virtual objects in an augmented reality environment and to adjust the appearance of virtual objects displayed in an augmented reality environment in response to different inputs.

[0036] under, Figures 1A to 1C , Figure 2 and Figure 3 A description of an exemplary apparatus is provided. Figure 4A to Figure 4B , FIG. 5A to FIG. 5AK , Figures 6A to 6AI , FIG. 7A to FIG. 7AQ , FIG. 8A to FIG. 8K as well as FIG. 13A to FIG. 13J Exemplary user interfaces for displaying virtual objects in various scenarios are shown. 9A to 9C A flow chart of a method of displaying a visual indication of one or more inputs that, when executed, would cause criteria for performing an action to be satisfied is shown. FIG. 10A to FIG. 10C A flowchart of a method for displaying media items in a user interface that includes content of at least a portion of a field of view of one or more cameras is shown according to some embodiments. FIG. 11A to FIG. 11D A flowchart of a method of responding to input directed to a corresponding virtual object among multiple displayed virtual objects is shown according to some embodiments. FIG. 12A to FIG. 12B A flowchart of a method for displaying prompts for changing properties of a media item that does not meet compatibility standards for display in an augmented reality environment is shown according to some embodiments. FIG. 5A to FIG. 5AK , Figures 6A to 6AI , FIG. 7A to FIG. 7AQ , FIG. 8A to FIG. 8K and FIG. 13A to FIG. 13J The user interface in 9A to 9C , Figure 10A to Fig. 10C , FIG. 11A to FIG. 11D , FIG. 12A to FIG. 12B and FIG. 14A to FIG. 14B process.

[0037] Exemplary Devices

[0038] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0039] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0040] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0041] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0042] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). and Device. Other portable electronic devices are optionally used, such as laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads).

[0043] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0044] The device typically supports a variety of applications, such as one or more of the following applications: a note-taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a game application, a telephone application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital camcorder application, a web browsing application, a digital music player application, and / or a digital video player application.

[0045] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0046] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0047] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0048] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The access of other components of the device 100 (such as the CPU 120 and the peripheral device interface 118) to the memory 102 is optionally controlled by the memory controller 122.

[0049] Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0050] In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0051] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data-only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11d). or IEEE802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0052] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0053] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0054] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some visual outputs or all of the visual outputs correspond to user interface objects. As used herein, the term "indicator" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0055] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0056] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In some embodiments, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0057] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0058] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0059] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0060] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some embodiments, an optical sensor is located on the rear portion of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device, so as to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user is watching other video conference participants on the touch screen, etc.).

[0061] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to a force sensor controller 159 in the I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 located on the front of the device 100.

[0062] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0063] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1AA tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. In some embodiments, the tactile output generator 167 includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, a solenoid, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). The tactile output generator 167 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates a tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front of the device 100.

[0064] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on an analysis of data received from the one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0065] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in FIG. 1A and FIG. Figure 3. The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0066] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0067] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device compatible with some of Apple Inc. (Cupertino, California). iPod and In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in Apple Inc. (Cupertino, California). iPod and A Lightning connector that is the same as or similar to and / or compatible with the Lightning connector used in the device.

[0068] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or stylus contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multi-finger contact). In some embodiments, contact / motion module 130 and display controller 156 detect contact on a touch pad.

[0069] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0070] In some embodiments, detecting a finger tap gesture depends on detecting the length of time between a finger press event and a finger lift event, but is independent of the strength of the finger contact between the finger press event and the finger lift event. In some embodiments, a tap gesture is detected based on determining that the length of time between the finger press event and the finger lift event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the strength of the finger contact during the tap reaches a given strength threshold (greater than a nominal contact detection strength threshold), such as a light press or deep press strength threshold. Therefore, a finger tap gesture can meet a specific input criterion that does not require the characteristic strength of the contact to meet a given strength threshold to meet the specific input criterion. For clarity, the finger contact in a tap gesture is typically required to meet a nominal contact detection strength threshold to detect a finger press event, below which the contact is not detected. A similar analysis applies to detecting a tap gesture by a stylus or other contact. In the case where the device is capable of detecting contact with a finger or stylus hovering over the touch-sensitive surface, the nominal contact detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0071] The same concepts apply in a similar manner to other types of gestures. For example, a swipe gesture, a pinch gesture, an expand gesture, and / or a long press gesture may be optionally detected based on satisfying criteria that are independent of the strength of the contacts included in the gesture or that do not require one or more contacts performing the gesture to reach a strength threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; a zoom gesture is detected based on the movement of two or more contacts toward each other; a zoom gesture is detected based on the movement of two or more contacts away from each other; and a long press gesture is detected based on the duration of a contact on a touch-sensitive surface having less than a threshold amount of movement. Thus, the statement that a particular gesture recognition criterion does not require the strength of the contacts to satisfy a corresponding strength threshold in order to satisfy the particular gesture recognition criterion means that the particular gesture recognition criterion can be satisfied when the contacts in the gesture do not reach the corresponding strength threshold, and can also be satisfied when one or more contacts in the gesture reach or exceed the corresponding strength threshold. In some embodiments, a tap gesture is detected based on determining that a finger down event and a finger up event are detected within a predefined time period, regardless of whether the contact is above or below a corresponding strength threshold during the predefined time period, and a swipe gesture is detected based on determining that the contact moves greater than a predefined amount, even if the contact is above the corresponding strength threshold at the end of the contact movement. Even in specific implementations where detection of gestures is affected by the strength of the contact performing the gesture (e.g., the device detects a long press more quickly when the strength of the contact is above the strength threshold, or the device delays detection of a tap input when the strength of the contact is higher), detection of these gestures does not require the contact to reach a particular strength threshold (e.g., even if the amount of time required to recognize the gesture varies), as long as the criteria for recognizing the gesture can be met without the contact reaching the particular strength threshold.

[0072] In some cases, the contact intensity threshold, duration threshold, and movement threshold are combined in various combinations to create a heuristic algorithm to distinguish between two or more different gestures for the same input element or area, so that multiple different interactions with the same input element can provide a richer set of user interactions and responses. The statement that a particular set of gesture recognition criteria does not require the intensity of one or more contacts to meet the corresponding intensity threshold in order to meet the particular gesture recognition criteria does not exclude the simultaneous evaluation of other intensity-related gesture recognition criteria to identify other gestures with criteria that are met when the gesture includes a contact with an intensity above the corresponding intensity threshold. For example, in some cases, a first gesture recognition criterion for a first gesture (which does not require the intensity of the contact to meet the corresponding intensity threshold to meet the first gesture recognition criterion) competes with a second gesture recognition criterion for a second gesture (which depends on the contact reaching the corresponding intensity threshold). In such a competition, if the second gesture recognition criterion for the second gesture is met first, the gesture is optionally not recognized as satisfying the first gesture recognition criterion for the first gesture. For example, if the contact reaches the corresponding intensity threshold before the contact moves a predefined amount of movement, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contact moves a predefined amount of movement before the contact reaches the corresponding intensity threshold, a swipe gesture is detected instead of a deep press gesture. Even in such cases, the first gesture recognition criteria for the first gesture still does not require the intensity of the contact to satisfy the corresponding intensity threshold to satisfy the first gesture recognition criteria, because if the contact remains below the corresponding intensity threshold until the gesture ends (e.g., a swipe gesture with an intensity of the contact that does not increase to above the corresponding intensity threshold), the gesture will be recognized as a swipe gesture by the first gesture recognition criteria. Therefore, a particular gesture recognition criterion that does not require the intensity of the contact to satisfy the corresponding intensity threshold to satisfy the particular gesture recognition criterion will (A) in some cases ignore the intensity of the contact relative to the intensity threshold (e.g., for a tap gesture) and / or (B) in some cases fail to satisfy the particular gesture recognition criterion (e.g., for a long press gesture) if a set of competing intensity-related gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-related gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input, and in this sense still depends on the intensity of the contact relative to the intensity threshold (e.g., for a long press gesture that competes with the deep press gesture for recognition).

[0073] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0074] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0075] Haptic feedback module 133 includes various software components for generating instructions (e.g., instructions used by haptic feedback controller 161) to generate tactile output at one or more locations on device 100 using tactile output generator 167 in response to user interaction with device 100.

[0076] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0077] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0078] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0079] Contacts module 137 (sometimes called address book or contact list);

[0080] Telephone module 138;

[0081] Video conferencing module 139;

[0082] Email client module 140;

[0083] Instant messaging (IM) module 141;

[0084] Fitness support module 142;

[0085] A camera module 143 for still images and / or video images;

[0086] Image management module 144;

[0087] Browser module 147;

[0088] Calendar module 148;

[0089] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0090] A widget creator module 150 for forming a user-created widget 149-6;

[0091] Search module 151;

[0092] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0093] Memo module 153;

[0094] Map module 154; and / or

[0095] Online video module 155.

[0096] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0097] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and the like.

[0098] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0099] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0100] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0101] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting the corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs or IMPS).

[0102] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, fitness support module 142 includes executable instructions for creating workouts (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0103] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0104] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0105] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0106] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0107] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0108] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0109] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0110] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0111] In conjunction with the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the memo module 153 includes executable instructions for creating and managing memos, to-do items, etc. according to user instructions.

[0112] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0113] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch-sensitive display system 112 or on an external display connected wirelessly or via external port 124), send emails with links to specific online videos, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a specific online video.

[0114] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores subsets of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0115] In some embodiments, the device 100 is a device on which operation of a predefined set of functions is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0116] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the trackpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using a trackpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0117] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (eg, in the operating system 126) and a corresponding application 136-1 (eg, any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0118] Event classifier 170 receives event information and determines the application 136-1 and the application view 191 of application 136-1 to which the event information is to be delivered. Event classifier 170 includes event monitor 171 and event distributor module 174. In some embodiments, application 136-1 includes application internal state 192, which indicates one or more current application views displayed on touch-sensitive display system 112 when the application is active or executing. In some embodiments, device / global internal state 157 is used by event classifier 170 to determine which application(s) is currently active, and application internal state 192 is used by event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0119] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0120] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0121] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0122] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0123] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0124] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0125] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0126] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0127] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizer 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizer determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0128] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0129] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit (not shown) or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views 191 include one or more corresponding event handlers 190. Additionally, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0130] The corresponding event identifier 180 receives event information (e.g., event data 179) from the event classifier 170 and identifies the event from the event information. The event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, the event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0131] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0132] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 contains definitions of events (e.g., predefined sub-event sequences), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0133] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object that triggered the hit test.

[0134] In some embodiments, the definition of the corresponding event 187 also includes delay actions that delay the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0135] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0136] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0137] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0138] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.

[0139] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0140] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0141] It should be understood that the above discussion of event processing for user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, not all of which are initiated on a touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touchpad, such as taps, drags, scrolls, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining an event to be recognized.

[0142] Figure 1C 1 is a block diagram showing a tactile output module according to some embodiments. In some embodiments, the I / O subsystem 106 (e.g., the tactile feedback controller 161 ( Figure 1A ) and / or one or more other input controllers 160 ( Figure 1A ))include Figure 1C At least some of the example components shown. In some embodiments, the peripheral device interface 118 includes Figure 1C At least some of the example components shown.

[0143] In some embodiments, the tactile output module includes a tactile feedback module 133. In some embodiments, the tactile feedback module 133 aggregates and combines tactile outputs from software applications on the electronic device for user interface feedback (e.g., feedback in response to user input corresponding to the displayed user interface and prompts and other notifications indicating the execution of operations or the occurrence of events in the user interface of the electronic device). The tactile feedback module 133 includes one or more of a waveform module 123 (for providing a waveform for generating tactile output), a mixer 125 (for mixing waveforms, such as waveforms in different channels), a compressor 127 (for reducing or compressing the dynamic range of the waveform), a low-pass filter 129 (for filtering out high-frequency signal components in the waveform), and a thermal controller 131 (for adjusting the waveform according to thermal conditions). In some embodiments, the tactile feedback module 133 is included in a tactile feedback controller 161 ( Figure 1A In some embodiments, a separate unit of the tactile feedback module 133 (or a separate specific implementation of the tactile feedback module 133) is also included in the audio controller (e.g., the audio circuit 110, Figure 1A ) and used to generate an audio signal. In some embodiments, a single tactile feedback module 133 is used to generate an audio signal and to generate a waveform for a tactile output.

[0144] In some embodiments, the tactile feedback module 133 also includes a trigger module 121 (e.g., a software application, operating system, or other software module that determines that a tactile output is to be generated and triggers a process for generating a corresponding tactile output). In some embodiments, the trigger module 121 generates a trigger signal for triggering (e.g., by the waveform module 123) the generation of a waveform. For example, the trigger module 121 generates the trigger signal based on a preset timing standard. In some embodiments, the trigger module 121 receives a trigger signal from outside the tactile feedback module 133 (e.g., in some embodiments, the tactile feedback module 133 receives a trigger signal from a hardware input processing module 146 located outside the tactile feedback module 133) and relays the trigger signal to other components within the tactile feedback module 133 (e.g., the waveform module 123) or a software application that triggers an operation (e.g., using the trigger module 121) based on the activation of a user interface element (e.g., an application icon or an affordance within an application) or a hardware input device (e.g., a primary button or an intensity-sensitive input interface, such as an intensity-sensitive touch screen). In some embodiments, the trigger module 121 also (e.g., from the tactile feedback module 133, FIG. 1A and Figure 3 ) receives a tactile feedback generation instruction. In some embodiments, the trigger module 121 responds to the tactile feedback module 133 (or the trigger module 121 in the tactile feedback module 133) (e.g., from the tactile feedback module 133, Figure 1A and Figure 3 ) receives tactile feedback instructions and generates a trigger signal.

[0145] Waveform module 123 receives the trigger signal as input (e.g., from trigger module 121) and provides a waveform for generating one or more tactile outputs in response to receiving the trigger signal (e.g., a waveform selected from a predefined set of waveforms assigned for use by waveform module 123, such as those described below with reference to Figures 4F to 4G waveform described in more detail).

[0146] The mixer 125 receives waveforms as input (e.g., from the waveform module 123) and mixes the waveforms together. For example, when the mixer 125 receives two or more waveforms (e.g., a first waveform in a first channel and a second waveform in a second channel that at least partially overlaps the first waveform), the mixer 125 outputs a combined waveform corresponding to the sum of the two or more waveforms. In some embodiments, the mixer 125 also modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the rest of the two or more waveforms (e.g., by increasing the scale of the particular waveform and / or reducing the scale of the other waveforms in the waveforms). In some cases, the mixer 125 selects one or more waveforms to remove from the combined waveform (e.g., when waveforms from more than three sources have been requested to be output simultaneously by the tactile output generator 167, the waveform from the oldest source is discarded).

[0147] Mixer 127 receives waveforms (e.g., combined waveforms from mixer 125) as input and modifies these waveforms. In some embodiments, compressor 127 reduces these waveforms (e.g., based on tactile output generator 167 ( Figure 1A ) or 357( Figure 3 ) such that the tactile output corresponding to these waveforms is reduced. In some embodiments, compressor 127 limits the waveforms, such as by imposing a predefined maximum amplitude on the waveforms. For example, compressor 127 reduces the amplitude of the waveform portion that exceeds the predefined amplitude threshold, while maintaining the amplitude of the waveform portion that does not exceed the predefined amplitude threshold. In some embodiments, compressor 127 reduces the dynamic range of the waveform. In some embodiments, compressor 127 dynamically reduces the dynamic range of the waveform so that the combined waveform remains within the performance specifications of tactile output generator 167 (e.g., force and / or movable mass displacement limits).

[0148] Low-pass filter 129 receives a waveform (e.g., a compressed waveform from compressor 127) as input and filters (e.g., smoothes) the waveform (e.g., removes or reduces high-frequency signal components in the waveform). For example, in some cases, compressor 127 includes extraneous signals (e.g., high-frequency signal components) in the compressed waveform that interfere with tactile output generation and / or exceed the performance specifications of tactile output generator 167 when generating tactile output based on the compressed waveform. Low-pass filter 129 reduces or removes such extraneous signals in the waveform.

[0149] Thermal controller 131 receives a waveform (e.g., a filtered waveform from low pass filter 129) as input and adjusts the waveform based on a thermal condition of device 100 (e.g., based on an internal temperature detected within device 100, such as the temperature of haptic feedback controller 161, and / or an external temperature detected by device 100). For example, in some cases, the output of haptic feedback controller 161 varies based on temperature (e.g., in response to receiving the same waveform, haptic feedback controller 161 generates a first tactile output when haptic feedback controller 161 is at a first temperature and generates a second tactile output when haptic feedback controller 161 is at a second temperature different from the first temperature). For example, the magnitude (or amplitude) of the tactile output may vary based on temperature. To reduce the effects of temperature changes, the waveform is modified (e.g., the amplitude of the waveform is increased or decreased based on temperature).

[0150] In some embodiments, the tactile feedback module 133 (e.g., trigger module 121) is coupled to the hardware input processing module 146. In some embodiments, Figure 1A The one or more other input controllers 160 in the embodiment include a hardware input processing module 146. In some embodiments, the hardware input processing module 146 receives input from a hardware input device 145 (e.g. Figure 1A In some embodiments, the hardware input device 145 is any input device described herein, such as a touch-sensitive display system 112 ( Figure 1A )、Keyboard / Mouse 350( Figure 3 ), touch pad 355 ( FIG. 3 ), one of other input or control devices 116 ( Figure 1A ) or intensity-sensitive primary buttons. In some embodiments, the hardware input device 145 is composed of intensity-sensitive primary buttons rather than touch-sensitive display system 112 ( Figure 1A )、Keyboard / Mouse 350( Figure 3 ) or touchpad 355( Figure 3 ). In some embodiments, in response to input from a hardware input device 145 (e.g., an intensity-sensitive home button or a touch screen), the hardware input processing module 146 provides one or more trigger signals to the tactile feedback module 133 to indicate that a user input that meets predefined input criteria has been detected, such as an input corresponding to a home button "click" (e.g., a "press click" or a "release click"). In some embodiments, the tactile feedback module 133 provides a waveform corresponding to a home button "click" in response to an input corresponding to a home button "click", thereby simulating the tactile feedback of pressing a physical home button.

[0151] In some embodiments, the tactile output module includes a tactile feedback controller 161 (e.g., tactile feedback controller 161 in FIG. 1A ), which controls the generation of tactile output. In some embodiments, the tactile feedback controller 161 is coupled to a plurality of tactile output generators, and selects one or more tactile output generators from the plurality of tactile output generators and sends a waveform to the selected one or more tactile output generators for generating tactile output. In some embodiments, the tactile feedback controller 161 coordinates a tactile output request corresponding to activating a hardware input device 145 and a tactile output request corresponding to a software event (e.g., a tactile output request from the tactile feedback module 133), and modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the rest of the two or more waveforms (e.g., by increasing the size of a particular waveform and / or reducing the size of the rest of the waveforms in these waveforms, to prioritize the tactile output corresponding to activating the hardware input device 145 over the tactile output corresponding to the software event).

[0152] In some embodiments, such as Figure 1C As shown, the output of the tactile feedback controller 161 is coupled to the audio circuit of the device 100 (eg, the audio circuit 110, Figure 1A ), and provides the audio signal to the audio circuit of the device 100. In some embodiments, the tactile feedback controller 161 provides both a waveform for generating a tactile output and an audio signal for providing an audio output together with the generated tactile output. In some embodiments, the tactile feedback controller 161 modifies the audio signal and / or the waveform (for generating the tactile output) so that the audio output and the tactile output are synchronized (for example, by delaying the audio signal and / or the waveform). In some embodiments, the tactile feedback controller 161 includes a digital-to-analog converter for converting a digital waveform into an analog signal, which is received by the amplifier 163 and / or the tactile output generator 167.

[0153] In some embodiments, the tactile output module includes an amplifier 163. In some embodiments, the amplifier 163 receives a waveform (e.g., from the tactile feedback controller 161), amplifies the waveform, and then sends the amplified waveform to the tactile output generator 167 (e.g., the tactile output generator 167 ( Figure 1A ) or 357( Figure 3)). For example, amplifier 163 amplifies the received waveform to a signal level that meets the physical specifications of tactile output generator 167 (e.g., amplifies to the voltage and / or current required by tactile output generator 167 to generate a tactile output so that the signal sent to tactile output generator 167 generates a tactile output corresponding to the waveform received from tactile feedback controller 161) and sends the amplified waveform to tactile output generator 167. In response, tactile output generator 167 generates a tactile output (e.g., by shifting the movable mass back and forth in one or more dimensions relative to the neutral position of the movable mass).

[0154] In some embodiments, the tactile output module includes a sensor 169 coupled to the tactile output generator 167. The sensor 169 detects the state or state change (e.g., mechanical position, physical displacement, and / or movement) of the tactile output generator 167 or one or more components of the tactile output generator 167 (e.g., one or more moving components for generating tactile output, such as a membrane). In some embodiments, the sensor 169 is a magnetic field sensor (e.g., a Hall effect sensor) or other displacement and / or motion sensor. In some embodiments, the sensor 169 provides information (e.g., the position, displacement, and / or movement of one or more components in the tactile output generator 167) to the tactile feedback controller 161, and based on the information about the state of the tactile output generator 167 provided by the sensor 169, the tactile feedback controller 161 adjusts the waveform output from the tactile feedback controller 161 (e.g., the waveform sent to the tactile output generator 167, optionally via the amplifier 163).

[0155] Figure 2 A device having a touch screen (e.g., Figure 1AThe portable multifunction device 100 of the present invention is a touch-sensitive display system 112 (such as a touch-sensitive display system 112). The touch screen optionally displays one or more graphics in a user interface (UI) 200. In these embodiments and in other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, when the user interrupts contact with the one or more graphics, selection of the one or more graphics will occur. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0156] The device 100 optionally also includes one or more physical buttons, such as a "home button" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0157] In some embodiments, the device 100 includes a touch screen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for powering the device on / off and for locking the device, a volume adjustment button 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button pressed for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the strength of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.

[0158] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. Communication bus 320 optionally includes circuits (sometimes referred to as chipsets) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 167 described above), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or sensors similar to those described above) Figure 1A Memory 370 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 370 optionally includes one or more storage devices located away from CPU 310. In some embodiments, memory 370 stores information related to portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0159] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to the instruction set for performing the above-mentioned functions. The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0160] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0161] Figure 4A An example user interface 400 of an application menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0162] One or more signal strength indicators of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0163] ·time;

[0164] Bluetooth indicator;

[0165] Battery status indicator;

[0166] A tray 408 with icons for commonly used applications, such as:

[0167] o an icon 416 of the phone module 138 labeled “Phone”, which optionally includes an indicator 414 of the number of missed calls or voicemails;

[0168] o an icon 418 of the email client module 140 labeled “Mail,” which optionally includes an indicator 410 of the number of unread emails;

[0169] o An icon 420 labeled “Browser” of the browser module 147; and

[0170] o An icon 422 labeled "Music" of the video and music player module 152; and

[0171] Icons for other apps, such as:

[0172] o Icon 424 of IM module 141 labeled “Message”;

[0173] o Icon 426 labeled “Calendar” of calendar module 148;

[0174] o Icon 428 labeled “Photos” of the image management module 144;

[0175] o An icon 430 labeled “Camera” of the camera module 143;

[0176] o Icon 432 labeled “Online Video” of the online video module 155;

[0177] o Icon 434 labeled "Stock Market" of the Stock Market widget 149-2;

[0178] o an icon 436 labeled “MAP” of the map module 154;

[0179] o Icon 438 labeled "Weather" of the Weather widget 149-1;

[0180] o Icon 440 labeled as “Clock” of the alarm clock widget 149-4;

[0181] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0182] o An icon 444 labeled "Notepad" of the notepad module 153; and

[0183] o An icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136.

[0184] It should be noted that Figure 4A The icon labels shown in are merely exemplary. For example, other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0185] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touchpad 355) Figure 3 Although many of the examples that follow will be given with reference to input on a touch screen touch-sensitive display system 112 (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4BIn some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects contact with touch-sensitive surface 451 at a location corresponding to a corresponding location on the display (e.g., Figure 4B 460 and 462) (e.g., in Figure 4B 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B 450 in ) is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0186] In addition, although the following examples are given primarily with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0187] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that the touch-sensitive surface 451 in the display includes a touch-screen display (e.g., Figure 1AA touch-sensitive display system 112 or Figure 4A In some implementations of a touch screen in a touch screen display, a contact detected on the touch screen acts as a "focus selector" such that when an input (e.g., a press input by a contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form that the focus selector takes, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to convey the user's desired interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a trackpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display).

[0188] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). The intensity of the contact is used as an attribute of the user input, thereby allowing the user to access additional device functions that the user cannot easily access on a smaller device with limited real estate for displaying indications (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical control / mechanical control such as a knob or button).

[0189] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, the user of the device is provided with a software setting for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0190] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact begins to move, before contact ends, before or after contact is detected to increase in intensity, and / or before or after contact is detected to decrease in intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, the value generated by low-pass filtering the contact intensity over a predefined time period or starting from a predefined time, etc. In some embodiments, the duration of the contact is used in determining the feature strength (e.g., when the feature strength is the average of the strength of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first strength threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second strength threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more strength thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding option or give up performing the corresponding operation), rather than to determine whether to perform the first operation or the second operation.

[0191] In some embodiments, a portion of a gesture is identified for determining a characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position and reaches an end position (e.g., a drag gesture) where the strength of the contact increases. In this example, the characteristic strength of the contact at the end position may be based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm may be applied to the strength of the swipe gesture before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the strength of the swipe contact to achieve the purpose of determining the characteristic strength.

[0192] The user interface diagrams described herein optionally include various intensity diagrams that illustrate contacts on the touch-sensitive surface relative to one or more intensity thresholds (e.g., contact detection intensity threshold IT ). 0 , Light press intensity threshold IT L , Deep compression intensity threshold IT D (For example, at least initially above IT L ) and / or one or more other intensity thresholds (e.g., L Low intensity threshold IT H )). The intensity graph is not typically part of the displayed user interface, but is provided to help interpret the graph. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from that typically associated with clicking a button of a physical mouse or trackpad. In some embodiments, when a characteristic intensity is detected that is below the light press intensity threshold (e.g., and above the nominal contact detection intensity threshold IT 0 , contacts below the nominal contact detection intensity threshold are no longer detected), the device moves the focus selector based on the movement of the contact on the touch-sensitive surface, without performing the actions associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0193] In some embodiments, the response of the device to the input detected by the device depends on a criterion based on the contact strength during the input. For example, for some "light press" inputs, the strength of the contact exceeding the first strength threshold during the input triggers a first response. In some embodiments, the response of the device to the input detected by the device depends on a criterion including both the contact strength during the input and a time-based criterion. For example, for some "deep press" inputs, as long as a delay time passes between satisfying the first strength threshold and satisfying the second strength threshold, the strength of the contact exceeding the second strength threshold greater than the first strength threshold of the light press during the input triggers a second response. The duration of the delay time is typically less than 200 ms (milliseconds) (e.g., 40 ms, 100 ms, or 120 ms, depending on the magnitude of the second strength threshold, where the delay time increases as the second strength threshold increases). The delay time helps avoid accidentally identifying a deep press input. As another example, for some "deep press" inputs, a period of reduced sensitivity will occur after reaching the first strength threshold. During this period of reduced sensitivity, the second strength threshold increases. This temporary increase in the second intensity threshold also helps avoid accidental deep press inputs.For other deep press inputs, the response to detecting the deep press input does not depend on time-based criteria.

[0194] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Example factors are described in U.S. patent applications 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0195] For example, Figure 4C A dynamic intensity threshold 480 is shown that changes over time based in part on the intensity of touch input 476 over time. Dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predefined delay time p1 from when touch input 476 is initially detected, and a second component 478 that tracks the intensity of touch input 476 over time. The initial high intensity threshold of first component 474 reduces accidental triggering of a “deep press” response while still allowing an immediate “deep press” response if touch input 476 provides sufficient intensity. Second component 478 reduces unintentional triggering of a “deep press” response through gradual intensity fluctuations of the touch input. In some embodiments, when touch input 476 satisfies dynamic intensity threshold 480 (e.g., at Figure 4C ), triggering a "deep press" response.

[0196] Figure 4D Another dynamic intensity threshold 486 (eg, intensity threshold I D ). Figure 4D Two other intensity thresholds are also shown: a first intensity threshold I H and the second intensity threshold I L .exist Figure 4D Although the touch input 484 satisfies the first intensity threshold I before time p2, H and the second intensity threshold I L , but no response is provided until a delay time p2 has elapsed at time 482. Similarly, Figure 4D , the dynamic intensity threshold 486 decays over time, wherein the decay is from time 482 (triggering the second intensity threshold I L This type of dynamic intensity threshold reduction begins immediately after the trigger is triggered with a lower threshold intensity (such as the first intensity threshold I H Or the second intensity threshold I L ) is triggered after or simultaneously with the response associated with the dynamic intensity threshold I D The associated response.

[0197] Figure 4E Another dynamic intensity threshold 492 (eg, intensity threshold I D ).exist Figure 4E In the example, after a delay time p2 has passed since the touch input 490 was initially detected, the trigger is triggered with the intensity threshold I L At the same time, the dynamic intensity threshold 492 decays after a predefined delay time p1 has passed since the touch input 490 was initially detected. L The associated response then decreases the intensity of the touch input 490, and then increases the intensity of the touch input 490 without releasing the touch input 490, which may trigger the touch input 490 to be in accordance with the intensity threshold I. D The associated response (e.g., at time 494) may be performed even when the intensity of touch input 490 is below another intensity threshold (e.g., intensity threshold I L ) is also the case.

[0198] The contact characteristic strength is below the light press strength threshold IT L The intensity increases to between the light pressure intensity threshold IT L and deep compression intensity threshold IT D The intensity between IT and the deep press intensity threshold is sometimes referred to as a "light press" input.D The intensity of the deep compression increases to above the deep compression intensity threshold IT D The strength of the contact feature is sometimes called a "deep press" input. The contact feature strength is below the contact detection strength threshold IT 0 The intensity increases to between the contact detection intensity threshold IT 0 With light pressure intensity threshold IT L The strength between is sometimes referred to as detecting contact on the touch surface. The characteristic strength of the contact is above the contact detection strength threshold IT 0 The intensity decreases to below the contact detection intensity threshold IT 0 The strength of IT is sometimes referred to as detecting a contact lift-off from the touch surface. 0 In some embodiments, IT 0 In some examples, a shaded circle or ellipse is used to represent the intensity of a contact on the touch-sensitive surface. In some examples, an unshaded circle or ellipse is used to represent a corresponding contact on the touch-sensitive surface without specifying the intensity of the corresponding contact.

[0199] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the strength of the contact (or multiple contacts) increases to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting that the strength of the corresponding contact increases to above the press input strength threshold (e.g., the corresponding operation is performed on a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and the strength of the contact subsequently decreases to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the press input threshold (e.g., the corresponding operation is performed on an "up stroke" of the corresponding press input).

[0200] In some embodiments, the device employs intensity hysteresis to avoid unexpected input, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., performing the corresponding operation on an "up stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact strength increases from an intensity equal to or below a hysteresis strength threshold to an intensity equal to or above a press input strength threshold and, optionally, the contact strength subsequently decreases to an intensity equal to or below the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact strength or a decrease in contact strength, depending on the circumstances).

[0201] For ease of explanation, descriptions of operations performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input are optionally triggered in response to detecting that the intensity of the contact increases above the press input intensity threshold, the intensity of the contact increases from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, the intensity of the contact decreases below the press input intensity threshold, or the intensity of the contact decreases below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where operations are described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operations are optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold. As described above, in some embodiments, triggering of these operations is also dependent on satisfying a time-based criterion (e.g., a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold).

[0202] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, the movement of the touch-sensitive surface may be optionally interpreted or sensed by the user as the "roughness" of the touch-sensitive surface. Although such interpretation of touch by the user will be limited by the user's individualized sensory perception, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or ordinary) user. Providing tactile feedback to the user using tactile output enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0203] In some embodiments, the tactile output pattern specifies characteristics of the tactile output, such as the amplitude of the tactile output, the shape of the motion waveform of the tactile output, the frequency of the tactile output, and / or the duration of the tactile output.

[0204] When a device (e.g., one or more tactile output generators that generate tactile output via moving a movable mass) generates tactile output with different tactile output patterns, the tactile output can produce different tactile sensations in a user holding or touching the device. Although the user's senses are based on the user's perception of the tactile output, most users will be able to recognize changes in the waveform, frequency, and amplitude of the tactile output generated by the device. Therefore, the waveform, frequency, and amplitude can be adjusted to indicate to the user that different operations have been performed. In this way, a tactile output with a tactile output pattern designed, selected, and / or arranged to simulate the characteristics of an object in a given environment (e.g., a user interface including graphic features and objects, a simulated physical environment with virtual boundaries and virtual objects, a real physical environment with physical boundaries and physical objects, and / or a combination of any of the above) (e.g., size, material, weight, stiffness, smoothness, etc.); behavior (e.g., oscillation, displacement, acceleration, rotation, extension, etc.); and / or interaction (e.g., collision, adhesion, repulsion, attraction, friction, etc.) will provide helpful feedback to the user in some cases, which reduces input errors and improves the efficiency of the user's operation of the device. In addition, tactile output is optionally generated to correspond to feedback that is independent of simulated physical properties (such as input threshold or object selection). Such tactile output will provide helpful feedback to the user in some cases, which reduces input errors and improves the efficiency of the user's operation of the device.

[0205] In some embodiments, the tactile output with a suitable tactile output mode serves as a prompt for an event of interest in the user interface or behind the screen in the device. Examples of events of interest include activation of affordances (such as real or virtual buttons or toggle switches) provided on the device or in the user interface, the success or failure of the requested operation, reaching or crossing a boundary in the user interface, entering a new state, switching input focus between objects, activating a new mode, reaching or crossing an input threshold, detecting or identifying a type of input or gesture, etc. In some embodiments, a tactile output is provided to serve as a warning or prompt for an upcoming event or result that will occur unless a change in direction or interruption input is detected in time. Tactile output is also used in other contexts to enrich user experience, improve accessibility of devices for users with visual or motor difficulties or other accessibility needs, and / or improve the efficiency and functionality of user interfaces and / or devices. Optionally, the tactile output is compared with audio input and / or visual user interface changes, which further enhances the user's experience when the user interacts with the user interface and / or device, and is conducive to better transmission of information about the state of the user interface and / or device, and this reduces input errors and improves the efficiency of user operations on the device.

[0206] FIG. 4F to FIG. 4HA set of sample tactile output patterns are provided that can be used, individually or in combination, as is or through one or more transformations (e.g., modulation, amplification, truncation, etc.) to form suitable tactile feedback in various scenarios for various purposes (such as those described above and those described with respect to the user interfaces and methods discussed herein). This example of a control panel for tactile output shows how a set of three waveforms and eight frequencies can be used to generate an array of tactile output patterns. In addition to the tactile output patterns shown in these figures, each of these tactile output patterns is optionally adjusted in amplitude by changing the gain value of the tactile output pattern, as shown, for example, for Fig. 4I to Figure 4K 80Hz, FullTap 200Hz, MiniTap 80Hz, MiniTap 200Hz, MicroTap 80Hz, and MicroTap 200Hz, each of which is shown as a variant with a gain of 1.0, 0.75, 0.5, and 0.25. Figures 4I to 4K As shown, changing the gain of a tactile output pattern changes the amplitude of the pattern without changing the frequency of the pattern or changing the shape of the waveform. In some embodiments, changing the frequency of the tactile output pattern also results in a lower amplitude because some tactile output generators are limited by how much force can be applied to the movable mass, so the higher frequency movement of the mass is constrained to a lower amplitude to ensure that the acceleration required to form the waveform does not require forces outside the operating force range of the tactile output generator (e.g., the peak amplitude of FullTap at 230 Hz, 270 Hz, and 300 Hz is lower than the amplitude of FullTap at 80 Hz, 100 Hz, 125 Hz, and 200 Hz).

[0207] Figure 4F to Figure 4K A tactile output pattern having a particular waveform is shown. The waveform of the tactile output pattern represents a pattern of physical displacements relative to a neutral position (e.g., xzero) versus time through which the movable mass passes to generate a tactile output having the tactile output pattern. For example, Figure 4F The first set of tactile output patterns shown in (eg, the “FullTap” tactile output pattern) each have a waveform that includes oscillations having two complete cycles (eg, oscillations that start and end at a neutral position and pass through the neutral position three times). Figure 4G The second set of tactile output patterns shown in (eg, the “MiniTap” tactile output pattern) each have a waveform that includes an oscillation that comprises one complete cycle (eg, an oscillation that starts and ends at a neutral position and passes through the neutral position once). Figure 4HThe third set of tactile output patterns shown in (e.g., the "MicroTap" tactile output patterns) each have a waveform that includes an oscillation that includes half a complete cycle (e.g., an oscillation that starts and ends at a neutral position and does not pass through the neutral position). The waveform of the tactile output pattern also includes a start buffer and an end buffer that represent the gradual acceleration and deceleration of the movable mass at the beginning and end of the tactile output. Figure 4F to Figure 4K The exemplary waveform shown includes xmin and xmax values ​​representing the maximum and minimum extents of movement of the movable mass. For larger electronic devices with larger movable masses, the minimum and maximum extents of movement of the mass may be larger or smaller. Figure 4F to Figure 4K The examples shown in describe the movement of a mass in 1 dimension, but similar principles will also apply to the movement of a movable mass in two or three dimensions.

[0208] like Figure 4F to Figure 4K As shown, each tactile output pattern also has a corresponding characteristic frequency, which affects the "pitch" of the tactile sensation felt by the user from the tactile output having the characteristic frequency. For continuous tactile output, the characteristic frequency represents the number of cycles (e.g., cycles per second) that the movable mass of the tactile output generator completes in a given time period. For discrete tactile output, a discrete output signal is generated (e.g., having 0.5, 1, or 2 cycles), and the characteristic frequency value specifies how fast the movable mass needs to move to generate a tactile output having the characteristic frequency. FIG. 4F to FIG. 4H As shown, for each type of tactile output (e.g., defined by a corresponding waveform, such as FullTap, MiniTap, or MicroTap), higher frequency values ​​correspond to faster movement of the movable mass and, therefore, generally speaking, to shorter tactile output completion times (e.g., including the time required to complete the number of cycles of the discrete tactile output plus the start and end buffer times). For example, a FullTap with a characteristic frequency of 80 Hz takes longer to complete (e.g., in 100 Hz) than a FullTap with a characteristic frequency of 100 Hz. Figure 4F, 35.4ms vs. 28.3ms). In addition, for a given frequency, a tactile output having more cycles in its waveform at the corresponding frequency takes longer to complete than a tactile output having fewer cycles in its waveform at the same corresponding frequency. For example, a FullTap at 150Hz takes longer to complete than a MiniTap at 150Hz (e.g., 19.4ms vs. 12.8ms), and a MiniTap at 150Hz takes longer to complete than a MicroTap at 150Hz (e.g., 12.8ms vs. 9.4ms). However, for tactile output modes with different frequencies, this rule may not apply (e.g., a tactile output with more cycles but a higher frequency may take a shorter amount of time to complete than a tactile output with fewer cycles but a lower frequency, and vice versa). For example, at 300Hz, FullTap takes the same amount of time as MiniTap (e.g., 9.9ms).

[0209] like Figure 4F to Figure 4K As shown, the tactile output pattern also has a characteristic amplitude, which affects the amount of energy contained in the tactile signal or the "intensity" of the tactile sensation that the user can feel through the tactile output having the characteristic amplitude. In some embodiments, the characteristic amplitude of the tactile output pattern refers to the absolute value or normalized value representing the maximum displacement of the movable mass relative to the neutral position when generating the tactile output. In some embodiments, the characteristic amplitude of the tactile output pattern can be adjusted according to various conditions (e.g., customized based on user interface context and behavior) and / or preconfigured metrics (e.g., input-based metrics, and / or user interface-based metrics), such as by a fixed or dynamically determined gain factor (e.g., a value between 0 and 1). In some embodiments, an input-based metric (e.g., an intensity change metric or an input speed metric) measures the characteristics of the input during the input that triggers the generation of the tactile output (e.g., the rate of change of the characteristic intensity of the contact in the press input or the rate of movement of the contact on the touch-sensitive surface). In some embodiments, a user interface-based metric (e.g., a cross-border speed metric) measures a characteristic of a user interface element (e.g., the speed at which the element moves across a hidden or visible boundary in the user interface) during a user interface change that triggers the generation of a tactile output. In some embodiments, a characteristic amplitude of a tactile output pattern may be "envelope" modulated, and the peaks of adjacent cycles may have different amplitudes, wherein one of the waveforms shown above is further modified by multiplying an envelope parameter that changes over time (e.g., from 0 to 1) to gradually adjust the amplitude of a portion of the tactile output over time when the tactile output is generated.

[0210] Although in Figure 4F to Figure 4KFor the purpose of illustration, only specific frequencies, amplitudes, and waveforms are shown in the sample tactile output patterns, but tactile output patterns with other frequencies, amplitudes, and waveforms can also be used for similar purposes. For example, a waveform with between 0.5 and 4 cycles can be used. Other frequencies in the range of 60 Hz-400 Hz can also be used.

[0211] User interface and associated processes

[0212] Attention is now directed to embodiments of a user interface ("UI") and associated processes that may be implemented on an electronic device such as portable multifunction device 100 or device 300 having display generating components, one or more input devices, and (optionally) one or more cameras.

[0213] FIG. 5A to FIG. 5AK An exemplary user interface for displaying a visual indication of one or more inputs that, when executed, would cause a criterion for performing an action (e.g., displaying an animation sequence associated with a virtual object) to be satisfied is shown according to some embodiments. For example, in response to detecting an input that does not satisfy a criterion for triggering an animation associated with a virtual object, a visual indication of an input that would satisfy the criterion for triggering an animation is displayed. The user interfaces in these figures are used to illustrate the processes described below, including 9A to 9C , FIG. 10A to FIG. 10C , FIG. 11A to FIG. 11D , FIG. 12A to FIG. 12B and FIG. 13A to FIG. 13B For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device with touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, in response to detecting a contact on touch-sensitive surface 451 when the user interface shown in the figure is displayed on display 450 together with the focus selector, similar operations are optionally performed on a device with display 450 and a separate touch-sensitive surface 451.

[0214] exist Figure 5AIn some embodiments, a virtual object 5002 (virtual box) is displayed in an object staging user interface 5004. In some embodiments, while the object staging user interface 5004 is displayed, user input for manipulating the virtual object 5002 (e.g., input by contact at a location corresponding to the virtual object 5002 displayed on the touch-sensitive display system 112) causes a feature of the virtual object 5002 to change (e.g., the virtual object rotates around one or more axes). This enables the user to view the virtual object 5002 from various angles. In some embodiments, when the virtual object 5002 is displayed in a user interface (e.g., an augmented reality view) that includes at least a portion of the field of view of one or more cameras of the device 100, changes made to the features of the virtual object 5002 when the virtual object is displayed in the object staging user interface 5004 (e.g., a rotation applied to the virtual object 5002 around a first axis) are applied to the virtual object 5002. This enables the user to set the orientation that the virtual object 5002 will have when displayed in the augmented reality view.

[0215] The object staging user interface 5004 includes a toggle control 5008 that indicates a current display mode (e.g., the current display mode is the object staging user interface mode, as indicated by the highlighted “object” indicator) and that, when activated, causes a transition to a selected display mode. For example, when the object staging user interface 5004 is displayed, a tap input by contact at a location corresponding to the toggle 5008 (e.g., a location corresponding to a portion of the toggle control 5008 that includes the text “AR”) causes the object staging user interface 5008 to be replaced with at least a portion of the field of view of one or more cameras of the device 100. The object staging user interface 5004 also includes a cancel control 5006 (e.g., for replacing the display of the object staging user interface 5004 with a different user interface such as a previously displayed user interface) and a share control 5010 (e.g., for displaying a sharing interface).

[0216] FIG. 5B to FIG. 5C An input is shown that causes the virtual object 5002 to rotate while the object staging user interface 5004 is displayed. Figure 5B , an input (e.g., a right swipe gesture) by contact 5012 is detected at a position corresponding to virtual object 5002. When the contact moves along the path indicated by arrow 5014, virtual object 5002 rotates around an axis perpendicular to the movement of the contact.

[0217] exist Figure 5D, an input (e.g., a tap input) is detected by contact 5016 at a location on touch-sensitive display system 112 corresponding to the “AR” region of toggle control 5008. In response to the input, display of object staging user interface 5004 is replaced with display of virtual object 5002 in an augmented reality view (e.g., displaying virtual object 5002 in a physical environment captured by one or more cameras of device 100), as shown. FIG. 5E to FIG. 5F To indicate that the current display mode is the augmented reality user interface mode, Fig. 5F The "AR" area of ​​the toggle control 5008 is highlighted.

[0218] Figure 5E A physical environment 5018 is shown in which the device 100 is operating. Table 5020 is a physical object located in the physical environment 5018. The display of the device 100 shows a virtual object 5002 placed on an upper surface of the table 5020 in a view of the physical environment 5018 as captured by one or more cameras of the device 100.

[0219] FIG. 5F to FIG. 5I An input that causes animation of virtual object 5002 to occur is shown. Fig. 5F In FIG. 5 , a virtual object 5002 is shown in an augmented reality user interface 5022 that includes a view of a physical environment 5018 as captured by one or more cameras of device 100. Figure 5G In FIG. 5 , an input (e.g., a tap input) is detected by contact 5024 at a location corresponding to virtual object 5002 on touch-sensitive display system 112. In response to the input, virtual object 5002 is displayed in an animated manner (e.g., an animation sequence shows that the lid of the box is gradually opened), such as FIG. 5H to FIG. 5I In some embodiments, the animation of the virtual object 5002 continues while the contact 5024 remains at a location corresponding to the virtual object 5002. In some embodiments, when the animation of the virtual object 5002 has been initiated, the animation continues after the contact 5024 is lifted off the touch-sensitive display system 112 (e.g., the animation sequence is completed and / or the animation sequence continues to loop).

[0220] Figure 5J to Figure 5K An input is shown that causes the virtual object 5002 to move while the augmented reality user interface 5022 is displayed. Figure 5J 5024 is detected at a position corresponding to virtual object 5002. When the contact moves along the path indicated by arrow 5026, virtual object 5002 moves along the surface of table 5020 in the direction of movement of the contact, as shown in FIG. Figure 5K as shown in .

[0221] exist Figure 5K , because the input that caused the virtual object 5002 to move does not meet the criteria for causing the animation of the virtual object 5002 to occur, a visual indication 5028 of the input required to cause the animation to occur is displayed. The visual indication 5028 includes the text "Tap on the box to see the animation!" In this way, the user is provided with feedback that provides information about the input that will trigger the playback of the animation sequence.

[0222] 5L-50 illustrate gaze input that causes animation of virtual object 5002. In FIG5L-1 , user 5030 is gazing at a lower portion of the display of device 100, as indicated by gaze indicator 5034 indicating the gaze target of user 5030's eye 5032. Figure 5L-2 An augmented reality user interface 5022 is shown displayed by device 100. Figure 5L-2 , the target of gaze 5034 is a location in augmented reality user interface 5022 that does not correspond to virtual object 5002. Visual indication 5036 is displayed, indicating an input that will cause animation of virtual object 5002 to occur (e.g., "Look at the box to see the animation!").

[0223] exist Figure 5M-1 , user 5030 is looking at the upper portion of the display of device 100, as indicated by gaze indicator 5038. Figure 5M-2 In FIG. 5 , the target of gaze 5038 is a position in augmented reality user interface 5022 corresponding to virtual object 5002. Figure 5N-2 and Figure 5O-2 As shown in FIG. 5034 , based on determining that the target of gaze 5038 is virtual object 5002, visual indication 5036 stops being displayed and animation of virtual object 5002 occurs. In some embodiments, the animation of virtual object 5002 continues while gaze 5038 continues to target virtual object 5002, such as Figure 5N-1 to Figure 5N-2 and Figure 5O-1 to Figure 5O-2 In some embodiments, based on determining that the target of the gaze is not the virtual object 5002, the animation of the virtual object 5002 stops. In some embodiments, when the animation of the virtual object 5002 has been initiated, the animation continues after the gaze moves away from the virtual object 5002 (e.g., the animation sequence is completed and / or the animation sequence continues to loop).

[0224] Figure 5P to Figure 5R An input is shown that attempts to place virtual object 5002 on a surface having dimensions that are incompatible with virtual object 5002. Figure 5P, input by contact 5040 is detected at a location corresponding to virtual object 5002. As contact 5040 moves along the path indicated by arrow 5042, virtual object 5002 moves along floor 5046 (e.g., the floor of physical environment 5018 captured in the field of view of the one or more cameras of device 100) in the direction of movement of the contact, as shown in FIG. Figure 5P to Figure 5Q As shown in Figure 5Q to Figure 5R , when the user attempts to place virtual object 5002 on the upper surface of lamp 5044, contact 5040 continues to move along the path indicated by arrow 5048. Figure 5R In the present embodiment, because the dimensions of the upper surface of the light 5044 are incompatible with the dimensions of the virtual object 5002 (e.g., including dimensions corresponding to an animation sequence associated with the virtual object 5002), a visual indication 5050 is displayed indicating that the dimensions of the upper surface of the light 5044 are incompatible with the virtual object 5002 (e.g., “That object cannot be placed on this surface!”).

[0225] Figure 5S to Figure 5W The positioning of the virtual object 5002 in the displayed field of view of the camera of the device 100 is shown so that animation of the virtual object 5002 occurs. Figure 5S , a virtual object 5002 has been placed on the upper surface of a physical table 5020 (e.g., such that the virtual object 5002 is displayed at a fixed position relative to the physical table 5020 as the field of view of the camera of the device 100 changes). The virtual object 5002 and the physical table 5020 are only partially in the displayed field of view of the camera of the device 100. Because the virtual object 5002 is not fully displayed in the field of view of the camera of the device 100, a visual indication 5054 is displayed indicating an input that will cause animation of the virtual object 5002 to occur (e.g., "Move the box to the center to see the animation!"). Circle 5052 indicates the location to which the virtual object will be moved in order for animation of the virtual object 5002 to occur. In some embodiments, the device 100 does not display the circle 5052 (as indicated by the dotted line). In some embodiments, the circle 5052 is a prompt displayed by the device 100. In some embodiments, animation of virtual object 5002 occurs when virtual object 5002 is fully displayed in the displayed portion of the field of view of the camera of device 100.

[0226] exist Figure 5T , device 100 has moved relative to physical environment 5018 such that virtual object 5002 is partially displayed within circle 5052. Because virtual object 5002 is not fully displayed within circle 5052, animation of virtual object 5002 has not occurred and visual indication 5054 continues to be displayed. Figure 5U, the device 100 has moved relative to the physical environment 5018 so that the virtual object 5002 is fully displayed within the circle 5052. Since the virtual object 5002 is fully displayed within the circle 5052, an animation of the virtual object 5002 occurs (e.g., Figure 5U , Figure 5V and Figure 5W ) and visual indication 5054 stops being displayed.

[0227] Figures 5X to 5AC An embodiment is shown in which multiple criteria must be met in order for an animation sequence of a virtual object 5002 to be played back. Figure 5X , the virtual object 5002 and the physical table 5020 are only partially in the displayed field of view of the camera of the device 100. Until these two criteria are met, a visual indication 5056 is displayed indicating an input that will cause the animation of the virtual object 5002 to occur (e.g., "Move the box to the center and tap on the box to see the animation!"). The circle 5052 indicates the location where the virtual object will be moved to cause the animation of the virtual object 5002 to occur.

[0228] exist Figure 5Y , device 100 has moved relative to physical environment 5018 such that virtual object 5002 is partially displayed within circle 5052. Since the plurality of criteria have not been met, animation of virtual object 5002 has not occurred and visual indication 5054 continues to be displayed. Figure 5Z , device 100 has moved relative to physical environment 5018 so that virtual object 5002 is fully displayed within circle 5052, thereby satisfying a first criterion of multiple criteria that must be met in order for the animation sequence of virtual object 5002 to play back. Because the first criterion has been met but the second criterion has not yet been met, visual indication 5056 is updated to indicate the input that must be provided to satisfy the second criterion (e.g., "Tap on the box to see the animation!"). Figure 5AA In the embodiment, an input (e.g., a tap input) is detected by contact 5058 at a position corresponding to virtual object 5002. In response to the input, animation of virtual object 5002 occurs (e.g., Figure 5AA , Figure 5AB and Figure 5AC ) and visual indication 5054 stops being displayed.

[0229] Figures 5AD to 5AH Movement of the device 100 that causes animation playback of the virtual object 5002 to occur is shown. Figure 5AD, the distance between the displayed location of the virtual object 5002 and the device 100 is greater than the threshold distance at which animation of the virtual object 5002 occurs. A visual indication 5060 is displayed to indicate an input that will cause animation of the virtual object 5002 to occur (e.g., "Move closer to the box to see the animation!"). Figure 5AE , device 100 has moved closer to the displayed location of virtual object 5002. Because the distance between the displayed location of virtual object 5002 and device 100 is still greater than the threshold distance at which animation of virtual object 5002 occurs, animation of virtual object 5002 has not occurred and visual indication 5060 continues to be displayed. Figure 5AF In the example, the device 100 has moved closer to the displayed position of the virtual object 5002, so that the distance between the displayed position of the virtual object 5002 and the device 100 has decreased below the threshold distance at which the animation of the virtual object 5002 occurs. Since the distance between the displayed position of the virtual object 5002 and the device 100 has decreased below the threshold distance, the animation of the virtual object 5002 occurs (e.g., Figure 5AF , Figure 5AG and Figure 5AH ) and visual indication 5060 stops being displayed.

[0230] Figures 5AI to 5AK A playhead control is shown for adjusting the playback of an animation sequence that animates the virtual object 5002. Figure 5AI , a timeline 5062 corresponding to playback of an animation sequence of a virtual object 5002 is displayed. A playhead 5064 indicates the position of the currently displayed frame of the animation sequence within the timeline 5062. An input via contact 5066 is detected at a position corresponding to the playhead 5064. As contact 5066 moves along the path indicated by arrow 5068, the playhead 5064 moves along the timeline 5062 in the direction of the movement of the contact (e.g., the playhead 5064 is "dragged" by the movement of contact 5066) and the animation sequence of the virtual object 5002 advances according to the movement of the playhead 5064, as shown. Figure 5AI to Figure 5AJ As shown in Figures 5AJ to 5AK , contact 5066 continues to move along the path indicated by arrow 5070, causing the playhead 5064 to move further and the animation sequence of virtual object 5002 to advance further.

[0231] Figures 6A to 6AI An exemplary user interface for displaying a media item (e.g., displaying a photo in an augmented reality environment) in a user interface that includes content of at least a portion of a field of view of one or more cameras according to some embodiments is shown. The user interfaces in these figures are used to illustrate the processes described below, including 9A to 9C , FIG. 10A to FIG. 10C , FIG. 11A to FIG. 11D , FIG. 12A to FIG. 12B and FIG. 13A to FIG. 13B For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device with touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, in response to detecting a contact on touch-sensitive surface 451 when the user interface shown in the figure is displayed on display 450 together with the focus selector, similar operations are optionally performed on a device with display 450 and a separate touch-sensitive surface 451.

[0232] exist Fig. 6A , the currently selected photo 6002 is displayed in the photo viewing interface 6004 of the photo management application. The photo management application user interface includes a photo thumbnail navigation bar 6006 that displays thumbnails of photos (e.g., photos captured by the device 100). In some embodiments, a left or right swipe input on the photo thumbnail navigation bar 6006 causes the displayed set of photo thumbnails to scroll in the direction of the swipe input so that additional photo thumbnails are displayed. In some embodiments, an input (e.g., a tap input) on a photo thumbnail displayed in the photo thumbnail navigation bar 6006 causes an enlarged version of the photo to be selected for display in the photo viewing interface 6004. For example, since the photo 6002 is currently selected (e.g., as indicated by the frame 6003 surrounding the photo thumbnail corresponding to the currently selected photo 6002), the enlarged version of the photo 6002 is displayed in the photo viewing interface 6004. Photo viewing interface 6004 includes controls 6008 for displaying a sharing user interface, controls 6010 for applying metadata to the selected photo (eg, metadata indicating that the selected photo is a favorite photo), and controls 6012 for deleting the selected photo.

[0233] exist Figure 6B In the example of FIG. 6 , an input (e.g., a tap input) is detected by contact 6014 at a position corresponding to control 6008 for displaying a sharing user interface. In response to the input, a sharing user interface 6016 is displayed that covers the photo viewing interface 6004, as shown in FIG. Figure 6C. The sharing user interface 6016 includes an indication 6018 of contacts with whom the currently selected photo 6002 may be shared, an indication 6020 of a communication mode via which the currently selected photo 6002 may be shared, and additional functions 6022 applicable to the currently selected photo 6002. While the sharing user interface 6016 is displayed, a control 6028 for stopping displaying the sharing user interface 6016 is displayed.

[0234] exist Figure 6C In the embodiment of the present invention, an input (e.g., a tap input) by contact 6026 is detected at a location corresponding to a control 6024 for displaying one or more selected objects in an object three-dimensional viewing mode (e.g., object management user interface 5004 or augmented reality user interface 5022). In response to the input, sharing user interface 6016 and photo viewing interface 6004 cease to be displayed and the currently selected photo 6002 is displayed together with the field of view of one or more cameras of device 100, as shown in FIG. Fig.6D to that shown in FIG6E .

[0235] Fig.6D A physical environment 5018 is shown in which the device 100 is operated. A physical table 5020 appears in the physical environment 5018 and in a view of the physical environment 5018 captured by one or more cameras of the device 100 (as displayed by the device 100). A photograph 6002 displayed by the device 100 appears to be hanging on a wall 6032 of the physical environment 5018.

[0236] FIG. 6E to FIG. 6G 1 shows adjustments to the displayed portion of the field of view of the camera of device 100 that occur as device 100 moves in the physical environment. Fig. 6E , the photo 6002 is shown in an augmented reality user interface 5022 that includes a view of the physical environment. In some embodiments, when displayed in the augmented reality user interface 5022, the photo 6002 appears to have a three-dimensional appearance (e.g., a simulated backing 6034 of the photo 6002 creates a space between the wall 6032 and the displayed location of the photo 6002). FIG. 6E to FIG. 6F , when device 100 moves from a first location in the physical environment to a second location in the physical environment, the displayed locations of photo 6002 and table 5020 change, thereby indicating that photo 6002 is displayed at a fixed location relative to the physical environment. Figure 6F to Figure 6G , device 100 continues to move from the second location in the physical environment to the third location in the physical environment, and the displayed locations of photo 6002 and table 5020 continue to change according to the movement of device 100.

[0237] FIG. 6H to FIG. 6LA sequential selection of multiple photos is shown.

[0238] exist Figure 6H , a plurality of photos are displayed in a photo selection interface 6036 of a photo management application. An input (e.g., a tap input) by contact 6040 is detected at a location on the touch-sensitive display system 112 corresponding to a “select” control for enabling a selection mode, in which input received at a corresponding photo will cause selection of the corresponding photo. In response to the input, the selection mode is enabled.

[0239] exist Fig.6I , an input (e.g., a tap input) by contact 6042 is detected at a position corresponding to photo 6002. In response to the input, photo 6002 is selected (e.g., as marked by selection indicator 6044 displayed near photo 6002).

[0240] exist Figure 6J , an input (e.g., a tap input) by contact 6048 is detected at a position corresponding to photo 6046. In response to the input, photo 6046 is selected (e.g., as marked by selection indicator 6050 displayed near photo 6046).

[0241] exist Figure 6K , an input (e.g., a tap input) by contact 6054 is detected at a position corresponding to photo 6052. In response to the input, photo 6052 is selected (e.g., as marked by selection indicator 6056 displayed near photo 6052).

[0242] exist Figure 6L , when photos 6002, 6046, and 6052 are selected, input (e.g., a tap input) by contact 6058 is detected at a location corresponding to share control 6008. In response to the input, sharing user interface 6016 (described above with respect to Figure 6C described), such as Figure 6M When sharing user interface 6016 is displayed, currently selected photos 6002, 6046, and 6052 are displayed (e.g., the group of multiple photos (including the photos displayed in photo selection interface 6036) is filtered so that only the currently selected photos are displayed when sharing user interface 6016 is displayed).

[0243] Figure 6M to Figure 6Z Inputs for manipulating multiple photos are shown.

[0244] exist Figure 6MIn the embodiment of the present invention, an input (e.g., a tap input) by contact 6060 is detected at a position corresponding to control 6024 for displaying one or more selected photos in the object three-dimensional viewing mode. In response to the input, sharing user interface 6016 and photo selection interface 6036 stop being displayed and currently selected photos 6002, 6046, and 6052 are displayed in augmented reality user interface 5022, as shown in FIG. FIG. 6N to FIG. 6O as shown in .

[0245] Figure 6N The physical environment 5018 in which the device 100 is operated is shown. The photos 6002, 6046, and 6052 displayed by the device 100 appear to be hanging on the back wall 6032 of the physical environment 5018.

[0246] exist Fig.6O 60, an input (e.g., a left swipe input) is detected by contact 6062 at a position corresponding to photo 6002. When contact 6062 moves along the path indicated by arrow 6064, photo 6002 moves along rear wall plane 6032 detected in the physical environment. Figure 6P , photo 6002 moves from rear wall plane 6032 to side wall plane 6031.

[0247] exist Figure 6Q , an input (e.g., a left swipe input) through contact 6066 is detected at a position corresponding to photo 6046. Figure 6Q to Figure 6R , when contact 6066 moves along the path indicated by arrow 6068, photo 6046 moves to the left along rear wall plane 6032 in the direction indicated by arrow 6068 to the left. Figure 6R The new location is shown in .

[0248] exist Figure 6R , an input (e.g., a left swipe input) by contact 6070 is detected at a position corresponding to photo 6052. Figure 6R to Figure 6S When the contact 6070 moves along the path indicated by the arrow 6072, the photo 6052 moves to the left along the rear wall plane 6032 in the direction indicated by the arrow 6072 to the Figure 6S The new location is shown in .

[0249] exist Figure 6S , an input (e.g., an upward swipe input) is detected by contact 6074 at a position corresponding to photo 6052. FIG. 6S to FIG. 6TWhen the contact 6074 moves along the path indicated by the arrow 6076, the photo 6052 moves upward along the rear wall plane 6032 in the direction indicated by the arrow 6076 to the position shown in FIG. Figure 6T The new location is shown in .

[0250] exist Figure 6U 6046, an input (e.g., a pinch gesture) is detected through contacts 6086 and 6088 for changing the simulated physical size of photo 6046. When contact 6086 moves along the path indicated by arrow 6090 and contact 6088 moves along the path indicated by arrow 6092, the size of photo 6046 decreases, as shown in FIG. Figure 6U to Figure 6V as shown in .

[0251] exist Figure 6W 6, an input through contacts 6086 and 6088 is detected for rotating photo 6046. When contact 6086 moves along the path indicated by arrow 6094 and contact 6088 moves along the path indicated by arrow 6096, photo 6046 rotates around an axis perpendicular to rear wall plane 6032, as shown in FIG. Figure 6W to Figure 6X As shown in . Figure 6X to Figure 6Y , as contact 6086 moves along the path indicated by arrow 6098 and contact 6088 moves along the path indicated by arrow 6100, photo 6046 continues to rotate.

[0252] exist Figure 6Z , input via contacts 6102 and 6104 is detected for rotating photo 6046 about axis 6016 parallel to back wall plane 6032, as indicated by rotation arrow 6108. Because rotation about an axis parallel to back wall plane 6032 is not permitted, a visual indication 6098 of the disallowed activity is displayed (e.g., including text indicating “Cannot rotate in this direction!”).

[0253] exist Figure 6AA , currently selected photo 6110 is displayed in photo viewing interface 6004 of the photo management application (e.g., as indicated by frame 6003 surrounding the photo thumbnail corresponding to currently selected photo 6110). Input (e.g., a tap input) by contact 6112 is detected at a location corresponding to control 6008 for displaying sharing user interface 6016. In response to the input, sharing user interface 6016 is displayed overlaying photo viewing interface 6004, as shown in FIG. Figure 6AB as shown in .

[0254] exist Figure 6ABIn the embodiment of the present invention, an input (e.g., a long press or a deep press input) by contact 6114 is detected at a position corresponding to a control 6024 for displaying one or more selected objects in an object three-dimensional viewing mode (e.g., object management user interface 5004 or augmented reality user interface 5022). In response to the input, a menu 6118 is displayed to indicate a display mode (e.g., "object" mode or "clip art" mode) for displaying one or more currently selected photos in the object three-dimensional viewing mode, such as Figure 6AC as shown in .

[0255] exist Figure 6AC , an input (e.g., a tap input) is detected by contact 6116 at a location corresponding to the “Clip Art” area of ​​menu 6118. In response to the input, sharing user interface 6016 and photo viewing interface 6004 cease to be displayed and a clip art version of image 6110 is displayed along with the field of view of one or more cameras of device 100, as shown in FIG. Figures 6AD to 6AE as shown in .

[0256] Figure 6AD A physical environment 5018 is shown in which the device 100 is operated. A physical table 5020 appears in the physical environment 5018 and in a view of the physical environment 5018 captured by one or more cameras of the device 100 (as displayed by the device 100). A clip art version of the photo 6110, as displayed by the device 100, appears in front of the table 5020 in the physical environment 5018.

[0257] Figure 6AE to Figure 6AI 1 shows adjustments to the displayed portion of the field of view of the device's camera that occur as the device 100 moves in the physical environment. Figure 6AE , a clip art version of the photo 6110 is shown in an augmented reality user interface 5022 that includes a view of the physical environment. Figure 6AE arrive Figure 6AF And from Figures 6AF to 6AG , device 100 moves from a position in front of the clip art version of image 6110 to a position to the side of the clip art version of image 6110, thereby displaying the clip art version of image 6110 with a simulated backing 6120 (e.g., having a simulated thickness that is small relative to the size of the clip art version of image 6110, such as a simulated thickness of 1 cm to 10 cm). In some embodiments, the clip art version of image 6110 displayed in augmented reality user interface 5022 is displayed without a backing (and thus from a simulated thickness of 1 cm to 10 cm). Figure 6AH A 2D image showing the case where the lateral view is not visible. Figure 6AG to Figure 6AH And from Figure 6AH to Figure 6AI, device 100 continues to move from a position to the side of the clip art version of image 6110 to a position behind the clip art version of image 6110, and the displayed positions of image 6110 and table 5020 continue to change according to the movement of device 100. Figure 6AI , the clip art version of image 6110 appears as if the clip art version of image 6110 is viewed from a position in front of the clip art version of image 6110 (e.g. Figure 6AE ) is a reverse version of .

[0258] FIG. 7A to FIG. 7AQ An exemplary user interface for responding to input directed to a corresponding virtual object in a plurality of displayed virtual objects according to some embodiments is shown. The user interfaces in these figures are used to illustrate the processes described below, including 9A to 9C , Fig. 10A To Figure 10C, FIG. 11A to FIG. 11D , FIG. 12A to FIG. 12B and FIG. 13A to FIG. 13B . For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device having a touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on the touch-sensitive display system 112. However, in response to detecting a contact on the touch-sensitive surface 451 when the user interface shown in the figure is displayed on the display 450 together with the focus selector, similar operations are optionally performed on a device having a display 450 and a separate touch-sensitive surface 451.

[0259] exist Fig. 7A In the object staging user interface 5004 (such as for Figure 5A 2 (further description) displays a virtual object 7002 (virtual refrigerator), a virtual object 7004 (virtual washing machine), and a virtual object 7006 (virtual television).

[0260] FIG. 7B to FIG. 7C Rotation of virtual objects 7002, 7004, and 7006 is shown in response to input detected while object staging user interface 5004 is displayed.

[0261] exist Figure 7B, an input (e.g., an upward vertical swipe input) by contact 7008 is detected at a position corresponding to virtual object 7002. As contact 7008 moves along the path indicated by arrow 7010, virtual objects 7002, 7004, and 7006 rotate in the direction indicated by arrow 7010 (e.g., tilt upward). A virtual boundary 7012 (not shown, as indicated by a dotted line) surrounds virtual objects 7002, 7004, and 7006. Figure 7C As shown in , in response to a vertical swipe input made through contact 7008, virtual objects 7002, 7004, and 7006 rotate about a horizontal center line 7014 of a virtual boundary 7012. For example, horizontal center line 7014 is a center line perpendicular to the direction of arrow 7016, which indicates the direction of the detected upward vertical swipe input. In some embodiments, in response to a left or right horizontal swipe input, virtual objects 7002, 7004, and 7006 rotate collectively about a vertical center line (not shown) that bisects boundary 7012 in the direction of the horizontal swipe input. In some embodiments, in response to a left or right horizontal swipe input, virtual objects 7002, 7004, and 7006 each rotate in the direction of the horizontal swipe input about a vertical center line that bisects each virtual object. Fig.7D , contact 7008 has been lifted off of touch-sensitive display system 112.

[0262] exist Fig. 7E , an input (e.g., a tap input) is detected by contact 7018 at a location corresponding to the “AR” region of toggle control 5008. In response to the input, display of object staging user interface 5004 is replaced with display of virtual objects 7002, 7004, and 7006 in an augmented reality view (e.g., displaying virtual objects 7002, 7004, and 7006 in a physical environment captured by one or more cameras of device 100), as shown. FIG. 7F to FIG. 7G In some embodiments, rotations applied to virtual objects 7002, 7004, and 7006 in object staging user interface 5004 (e.g., rotations about horizontal centerline 7014, as for virtual objects 7002, 7004, and 7006) are not maintained when displaying virtual objects in augmented reality user interface 5022. FIG. 7B to FIG. 7C 5022). For example, virtual objects 7002, 7004, and 7006 are configured to be placed at fixed positions relative to a horizontal plane and are thus provided with a horizontal surface that is substantially parallel to the detected horizontal plane. In some embodiments, rotations applied to virtual objects 7002, 7004, and 7006 in the object staging user interface 5004 (e.g., rotations about a vertical centerline) are maintained when displaying the virtual objects in the augmented reality user interface 5022.

[0263] like FIG. 7F to FIG. 7G As shown in , when the field of view of the camera is initially displayed, a semi-transparent representation of the virtual objects may be displayed (e.g., when one or more planes corresponding to one or more of the virtual objects have not yet been detected in the field of view of the camera of device 100). In some embodiments, in response to detecting that the object placement criteria are met (e.g., one or more planes corresponding to the one or more virtual objects have been detected in the field of view of the camera), the device displays an animated transition that shows one or more of the virtual objects being moved from a first orientation (e.g., as Figure 7G ) to a second orientation (e.g., as Figure 7H ) and from having a first set of visual attributes (e.g., virtual objects 7002, 7004, and 7006 in Figure 7G ) to have a second set of visual attributes (e.g., virtual objects 7002, 7004, and 7006 are shown in a semi-transparent state in FIG. Figure 7H For example, once one or more planes for placing virtual objects 7002, 7004, and 7006 are identified in the camera's field of view, virtual objects 7002, 7004, and 7006 are placed on the planes while visually adjusting their orientation, size, and translucency (etc.).

[0264] exist FIG. 7F to FIG. 7G , semi-transparent representations of virtual objects 7002, 7004, and 7006 are displayed in an augmented reality user interface 5022 that includes a view of the physical environment as captured by one or more cameras of device 100. Semi-transparent representations of virtual objects 7002, 7004, and 7006 are displayed at a fixed position relative to touch-sensitive display system 112. For example, from Figure 7F to Figure 7G , as the device 100 moves relative to the physical environment (as indicated, for example, by the changing position of the physical table 7020 in the camera's field of view), the virtual objects 7002, 7004, and 7006 remain in fixed positions relative to the touch-sensitive display system 112.

[0265] In 7H, a plane 7022 corresponding to virtual objects 7002, 7004, and 7006 has been detected in the field of view of the camera (e.g., a horizontal plane corresponding to the floor in the physical environment) and the virtual objects 7002, 7004, and 7006 are placed on the detected plane in the augmented reality user interface 5022. When the virtual objects 7002, 7004, and 7006 have been placed on the detected plane, as the device 100 moves in the physical environment and the field of view of the one or more cameras changes, the virtual objects remain at a fixed position relative to the detected plane (as described below for Figures 7R to 7S described).

[0266] FIG. 7H to FIG. 7R Inputs that change the position and orientation of virtual objects 7002, 7004, and 7006 displayed in augmented reality user interface 5022 are shown. Figure 7H , an input (e.g., an upward vertical swipe input) through contact 7022 is detected at a position corresponding to virtual object 7002. When contact 7024 moves along the path indicated by arrow 7026, virtual object 7002 moves along plane 7022 in the direction indicated by arrow 7026, as shown in FIG. FIG. 7H to FIG. 7I As virtual object 7002 moves along plane 7022, the size of virtual object 7002 is adjusted (eg, based on the virtual distance from the representation of the virtual object to the user, in order to maintain an accurate perspective of the virtual object in the field of view).

[0267] exist Fig.7I , when the contact 7024 continues to move along the path indicated by the arrow 7028, the virtual object 7002 moves along the plane 7022 in the direction indicated by the arrow 7028, as shown in FIG. Fig.7I 7J . As virtual object 7002 moves toward the user along plane 7022 , the size of virtual object 7002 increases.

[0268] exist Figure 7J , an input (e.g., an expand gesture) is detected through contacts 7030 and 7032 for changing the simulated physical size of virtual object 7002. When contact 7030 moves along the path indicated by arrow 7034 and contact 7032 moves along the path indicated by arrow 7036, the size of virtual object 7002 increases, as shown in FIG. Figures 7J to 7K as shown in (e.g., based on the virtual distance from the representation of the virtual object to the user, with the goal of maintaining an accurate perspective of the virtual object in the field of view).

[0269] exist Figure 7K, an input (e.g., a downward and leftward swipe input) by contact 7038 is detected at a position corresponding to virtual object 7006. When contact 7038 moves along the path indicated by arrow 7040, virtual object 7006 moves along plane 7022 in the direction indicated by arrow 7040, as shown in FIG. Figures 7K to 7L As virtual object 7006 moves toward the user along plane 7022, the size of virtual object 7006 increases.

[0270] exist Figure 7L , an input (e.g., a downward and rightward swipe input) by contact 7042 is detected at a position corresponding to virtual object 7004. When contact 7042 moves along the path indicated by arrow 7044, virtual object 7004 moves along plane 7022 in the direction indicated by arrow 7044, as shown in FIG. Figures 7L to 7M As the virtual object 7004 moves toward the user along the plane 7022, the size of the virtual object 7004 increases.

[0271] exist Figure 7M , an input (e.g., an expand gesture) is detected through contacts 7046 and 7048 for changing the simulated physical size of virtual object 7004. When contact 7046 moves along the path indicated by arrow 7050 and contact 7048 moves along the path indicated by arrow 7052, the size of virtual object 7004 increases, as shown in FIG. Figures 7M to 7N as shown in .

[0272] exist Figure 7N , an input (e.g., a pinch gesture) is detected through contacts 7054 and 7056 for changing the simulated physical size of virtual object 7002. When contact 7054 moves along the path indicated by arrow 7058 and contact 7056 moves along the path indicated by arrow 7060, the size of virtual object 7002 decreases, as shown in FIG. FIG. 7N to FIG. 7O as shown in .

[0273] exist Fig.7O , an input (e.g., a pinch gesture) is detected through contacts 7062 and 7064 for changing the simulated physical size of virtual object 7004. When contact 7062 moves along the path indicated by arrow 7066 and contact 7064 moves along the path indicated by arrow 7068, the size of virtual object 7002 decreases, as shown in FIG. Figure 7O to Figure 7P as shown in .

[0274] exist Figure 7P, an input (e.g., an upward and left swipe input) by contact 7070 is detected at a position corresponding to virtual object 7004. When contact 7070 moves along the path indicated by arrow 7072, virtual object 7004 moves along plane 7022 in the direction indicated by arrow 7072, as shown in FIG. Figure 7P to Figure 7Q As virtual object 7004 moves away from the user along plane 7022, the size of virtual object 7004 decreases (e.g., based on the virtual distance from the representation of the virtual object to the user, in order to maintain an accurate perspective of the virtual object in the field of view).

[0275] exist Figure 7Q 7006, an input (e.g., a rotation gesture) is detected through contacts 7074 and 7076 for changing the simulated orientation of virtual object 7006. When contact 7074 moves along the path indicated by arrow 7078 and contact 7076 moves along the path indicated by arrow 7080, virtual object 7006 rotates about an axis perpendicular to plane 7022 relative to which virtual object 7006 is positioned, as shown in FIG. Figure 7Q to Figure 7R As shown in

[0276] from Figure 7R to Figure 7S And from Figure 7S to Figure 7T , as the device 100 moves relative to the physical environment (as indicated, for example, by the changing position of the physical table 7020 in the camera's field of view), the virtual objects 7002, 7004, and 7006 remain at fixed positions relative to the plane 7022 in the physical environment (e.g., as the view of the physical environment changes, the positions of the virtual objects 7002, 7004, and 7006 move with the physical environment).

[0277] exist Figure 7T In the embodiment, an input (e.g., a tap input) is detected in the augmented reality user interface 5022 through the contact 7082. In response to the input, the toggle control 5008, the cancel control 5006, and the share control 5010 are displayed, as shown in FIG. Figure 7U as shown in .

[0278] exist Figure 7U , an input (e.g., a tap input) is detected by contact 7084 at a location corresponding to the “object” area of ​​toggle control 5008. In response to the input, the display of virtual objects 7002, 7004, and 7006 in augmented reality user interface 5022 is replaced with the display of virtual objects 7002, 7004, and 7006 in object staging view 5004, as shown in FIG. Figure 7U , Figure 7V , Figure 7W and Figure 7XWhen a transition occurs from displaying virtual objects 7002, 7004, and 7006 in augmented reality user interface 5022 to displaying virtual objects 7002, 7004, and 7006 in object staging view 5004, the device displays an animated transition showing virtual objects 7002, 7004, and 7006 moving (e.g., rotating, scaling, translating, and / or a combination of the above) back to a predefined configuration (e.g., a predefined order, position, orientation, and / or size) of virtual objects 7002, 7004, and 7006 (as previously described in Fig. 7A 5004). For example, from Figure 7U to Figure 7X :Virtual object 7002 moves downward and leftward to the position previously Fig. 7A The position of virtual object 7002 is shown in FIG. 7004 ; virtual object 7004 moves downward and rightward to the position previously located in FIG. Fig. 7A The position of the virtual object 7004 is displayed in the figure; and the virtual object 7006 moves upward, rightward and rotates counterclockwise to the position previously in the figure. Fig. 7A 5004. In some embodiments, when virtual objects 7002, 7004, and 7006 are displayed in object staging view 5004, the virtual objects are displayed in order of height (e.g., in order of decreasing height) and / or with equal spacing between displayed virtual objects.

[0279] exist Figure 7X , an input (e.g., a tap input) by contact 7086 is detected at a location corresponding to the “object” area of ​​toggle control 5008. In response to the input, the display of virtual objects 7002, 7004, and 7006 in object staging view 5004 is replaced with the display of virtual objects 7002, 7004, and 7006 in augmented reality user interface 5022, as shown in FIG. Figure 7X to Figure 7Y As shown in Figure 7Y In the example, according to the adjustments previously made to the virtual objects 7002, 7004 and 7006 (such as for FIG. 7H to FIG. 7R Virtual objects 7002, 7004 and 7006 are displayed using the position, orientation and size obtained by the controller (described above).

[0280] Figure 7Z to Figure 7AC The boundaries of virtual objects are shown for setting the spacing between virtual objects in the object staging user interface 5004. Figure 7Z , boundary 7094 corresponds to virtual object 7088 and boundary 7098 corresponds to virtual object 7092. Boundary 7096 corresponds to the stationary state of virtual object 7090.

[0281] An animation sequence is associated with virtual object 7090. Timeline 7100 indicates a playback position within the animation sequence associated with virtual object 7090. Play controls 7102 for initiating playback of the animation sequence associated with virtual object 7090 and volume controls 7104 for adjusting the volume of sounds associated with the animation sequence are shown. Figure 7Z In the example, an input (e.g., a tap input) is detected by contact 7106 at a position corresponding to playback control 7102. In response to the input, an animation sequence associated with virtual object 7090 is activated, such as Figure 7Z to Figure 7AC The movement of the virtual object 7090 in FIG. 7090 extends beyond the first boundary 7096. The boundary 7108 corresponds to the extension range of the virtual object 7090 during the duration of the animation sequence. Figure 7AC The boundaries 7094, 7108, and 7098 indicated in are used to set equal spacing between virtual objects 7088, 7090, and 7092 in the object staging user interface 5004.

[0282] Figures 7AD to 7AL A selection of multiple virtual objects to be placed in the augmented reality user interface 5022 is shown.

[0283] Figure 7AD An object management user interface 7099 (e.g., a website, a photo management application user interface, and / or a file management user interface) is shown that displays two-dimensional images (such as image 7118) and / or two-dimensional representations of three-dimensional virtual objects (such as virtual object 7110).

[0284] exist Figure 7AE , an input (eg, a tap input) by contact 7112 is detected at a position corresponding to virtual object 7110. In response to the input, virtual object 7110 is selected (eg, as marked by selection indicator 7114 displayed near virtual object 7112).

[0285] exist Figure 7AF , an input (e.g., a tap input) by contact 7116 is detected at a location corresponding to image 7118. In response to the input, image 7118 is selected (e.g., as marked by selection indicator 7120 displayed overlaying image 7118).

[0286] exist Figure 7AG , an input (e.g., a tap input) by contact 7124 is detected at a position corresponding to virtual object 7122. In response to the input, virtual object 7122 is selected (e.g., as marked by selection indicator 7126 displayed near virtual object 7122).

[0287] exist Figure 7AH , when virtual objects 7110, 7118, and 7122 are selected, input (e.g., tap input) by contact 7128 is detected at a location corresponding to sharing control 7130. In response to the input, sharing user interface 6016 (described above with respect to Figure 6C 7AI. When the sharing user interface 6016 is displayed, the currently selected virtual objects 7110, 7118, and 7122 are displayed (e.g., the collection of objects and / or images (including the objects and / or images displayed in the object management user interface 7099) is filtered so that only the currently selected virtual object is displayed when the sharing user interface 6016 is displayed).

[0288] exist Figure 7AI In the present embodiment, an input (e.g., a tap input) by contact 7128 is detected at a position corresponding to control 6024 for displaying one or more selected objects in object three-dimensional viewing mode. In response to the input, sharing user interface 6016 and object management user interface 7099 stop being displayed and currently selected virtual objects 7110, 7118, and 7122 are displayed in object staging user interface 5004, as shown in FIG. Figure 7AJ (In some embodiments, in response to input for displaying the one or more selected objects in the object three-dimensional viewing mode, the one or more selected objects are displayed in the augmented reality user interface 5022, such as for Figure 7AK described.)

[0289] exist Figure 7AJ , an input (e.g., a tap input) by contact 7130 is detected at a location corresponding to the “AR” region of toggle control 5008. In response to the input, the display of virtual objects 7110, 7118, and 7122 in object staging view 5004 is replaced with the display of virtual objects 7110, 7118, and 7122 in augmented reality user interface 5022.

[0290] In some embodiments, a plane type is defined for a virtual object, and when the virtual object is placed in an augmented reality user interface, the virtual object is displayed at a fixed position relative to the plane with the defined plane type. For example, the plane type corresponding to virtual object 7110 (virtual chair) is a horizontal physical surface that can be used as a support surface for the three-dimensional representation of the virtual object in the augmented reality view (e.g., a floor surface for supporting a chair). The plane type corresponding to virtual object 7118 (photographic image) is a vertical physical surface that can be used as a support surface for the three-dimensional representation of the virtual object in the augmented reality view (e.g., a vertical wall for hanging a photographic image). The plane type corresponding to virtual object 7122 (wall sconce) is a vertical physical surface that can be used as a support surface for the three-dimensional representation of the virtual object in the augmented reality view (e.g., a vertical wall for hanging a wall sconce). Figure 7AK In response to Figure 7AJ The detected input described displays virtual objects 7110, 7118, and 7122 in the augmented reality user interface 5022 according to the plane type defined for the corresponding virtual object. For example, the virtual chair object 7110 is displayed so that it appears to be supported by the floor plane 5046 detected in the physical environment, the virtual photographic image object 7118 is displayed so that it appears to be suspended from the back wall plane 7132 detected in the physical environment, and the virtual sconce object 7122 is displayed so that it appears to be suspended from the side wall plane 7134 detected in the physical environment.

[0291] In some embodiments, no plane type is defined for a selected virtual object and the virtual object is displayed at a fixed position relative to a single plane (e.g., a default plane type, a first horizontal plane detected in the physical environment, and / or a first plane of any type detected in the physical environment). Figure 7AL In response to Figure 7AJ The detected input described displays virtual objects 7110 , 7118 , and 7122 at fixed positions relative to floor plane 5046 , such that the virtual objects appear to be supported by floor plane 5046 .

[0292] Figure 7AM to Figure 7AQ Manipulation of a virtual object to be placed in augmented reality user interface 5022 is shown.

[0293] exist Figure 7AM, an input (e.g., a rotation gesture) is detected through contacts 7136 and 7138 for changing the simulated orientation of virtual object 7118. When contact 7136 moves along the path indicated by arrow 7140 and contact 7138 moves along the path indicated by arrow 7142, virtual object 7118 rotates about an axis perpendicular to the vertical wall plane 7132 relative to which virtual object 7118 is positioned, as shown in FIG. Figure 7AM to Figure 7AN As shown in Figures 7AN to 7AO , contact 7136 continues to move along the path indicated by arrow 7141, contact 7138 continues to move along the path indicated by arrow 7143, and virtual object 7110 continues to rotate.

[0294] exist Figure 7AO , input (e.g., a rotation gesture) is detected through contacts 7144 and 7146 for changing the simulated orientation of virtual object 7110. When contact 7144 moves along the path indicated by arrow 7148 and contact 7146 moves along the path indicated by arrow 7150, virtual object 7110 rotates about an axis perpendicular to horizontal floor plane 5046 relative to which virtual object 7110 is positioned, as shown in FIG. Figure 7AO to Figure 7AP As shown in Figure 7AP to Figure 7AQ , contact 7144 continues to move along the path indicated by arrow 7152, contact 7146 continues to move along the path indicated by arrow 7154, and virtual object 7110 continues to rotate.

[0295] FIG. 8A to FIG. 8K An exemplary user interface for displaying a prompt to change the properties of a media item that does not meet compatibility criteria for display in an augmented reality environment according to some embodiments is shown. The user interfaces in these figures are used to illustrate the process described below, including 9A to 9C , Figure 10A to Fig. 10C , FIG. 11A to FIG. 11D , FIG. 12A to FIG. 12B and FIG. 13A to FIG. 13B For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device with touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, in response to detecting a contact on touch-sensitive surface 451 when the user interface shown in the figure is displayed on display 450 together with the focus selector, similar operations are optionally performed on a device with display 450 and a separate touch-sensitive surface 451.

[0296] FIG. 8A to FIG. 8E Handling of media items that do not meet compatibility criteria for display in an augmented reality environment is shown.

[0297] exist Fig. 8A , file management user interface 8000 displays media items including AR-incompatible media item 8002. Input (e.g., a tap input) via contact 8004 is detected at a location corresponding to AR-incompatible media item 8002. In response to the input, AR-incompatible media item 8002 is selected (e.g., as marked by selection indicator 8006).

[0298] exist Figure 8B , when AR-incompatible media item 8002 is selected, input (e.g., tap input) by contact 8007 is detected at a location corresponding to sharing control 7130. In response to the input, sharing user interface 6016 (described above with respect to Figure 6C ) is displayed together with the currently selected AR incompatible media item 8002, such as Figure 8C as shown in .

[0299] exist Figure 8C , an input (e.g., a tap input) made by contact 8008 is detected at a location corresponding to a control 6024 for displaying one or more selected objects (such as a selected AR-incompatible media item 8002) in a three-dimensional viewing mode. In response to detecting the input, the device determines whether the selected AR-incompatible media item 8002 has attributes that do not meet the compatibility criteria for display in the augmented reality environment. Because the selected AR-incompatible media item 8002 has attributes that do not meet the compatibility criteria for display in the augmented reality environment, the device displays a prompt panel 8012, which includes a prompt for changing the attributes of the media item (e.g., “The selected item is incompatible with AR. Would you like to convert the item to an AR-compatible format?”), such as Fig.8D The prompt panel 8012 includes a control 8014 for converting the selected AR-incompatible media item 8002 and a control 8016 for dismissing the prompt panel 8012 .

[0300] exist Fig.8D In the example of FIG. 1 , an input (e.g., a tap input) is detected by contact 8018 at a location corresponding to control 8014 for converting a selected media item to an AR-compatible format. In response to the input, AR-incompatible media item 8002 is converted to an AR-compatible format and displayed in the example of FIG. Fig.8F In the augmented reality user interface 5022 shown in (or in Fig. 8E8002 (e.g., virtual object 8003 corresponding to the transformed version of media item 8002). While virtual object 8003 corresponding to the transformed version of media item 8002 is displayed in object staging user interface 5004, input (e.g., tap input) via contact 8020 is detected at a location corresponding to the “AR” region of toggle control 5008. In response to the input, display of object staging user interface 5004 is replaced with display of virtual object 8003 corresponding to the transformed version of media item 8002 in augmented reality user interface 5022, as shown in FIG. Fig. 8E to Fig.8F as shown in .

[0301] Figure 8G to Figure 8K Processing of media items that meet compatibility criteria for display in an augmented reality environment is shown.

[0302] exist Figure 8G , file management user interface 8000 displays media items including AR-compatible media item 8022. Input (e.g., a tap input) via contact 8028 is detected at a location corresponding to AR-compatible media item 8022. In response to the input, AR-compatible media item 8022 is selected (e.g., as marked by selection indicator 8026).

[0303] exist Figure 8H , when AR-compatible media item 8022 is selected, input (e.g., tap input) by contact 8028 is detected at a location corresponding to sharing control 7130. In response to the input, sharing user interface 6016 (described above with respect to Figure 6C ) is displayed together with the currently selected AR-compatible media item 8022, such as Fig.8I as shown in .

[0304] exist Fig.8I In the embodiment of the present invention, an input (e.g., a tap input) is detected by contact 8028 at a location corresponding to a control 6024 for displaying one or more selected objects (such as a selected AR-compatible media item 8022) in a three-dimensional viewing mode. In response to detecting the input, the device determines whether the selected AR-compatible media item 8022 has an attribute that does not meet the compatibility criteria for display in the augmented reality environment. Because the selected AR-compatible media item 8022 meets the compatibility criteria for display in the augmented reality environment, the device displays the selected AR-compatible media item 8022 in the augmented reality environment. Figure 8K In the augmented reality user interface 5022 shown in (or in Figure 8J8022). While the virtual object 8023 corresponding to the AR-compatible media item 8022 is displayed in the object staging user interface 5004, an input (e.g., a tap input) by contact 8030 is detected at a location corresponding to the “AR” region of the toggle control 5008. In response to the input, the display of the object staging user interface 5004 is replaced with the display of the virtual object 8023 corresponding to the AR-compatible media item 8022 in the augmented reality user interface 5022, as shown in FIG. Figure 8J to Figure 8K as shown in .

[0305] 9A to 9C 1 is a flow diagram of a method 900 for displaying a visual indication of one or more inputs that, when executed, would cause criteria for performing an action to be satisfied, according to some embodiments. The method 900 is performed on an electronic device (e.g., a display generating component (e.g., a display, a projector, a heads-up display, etc.) and one or more input devices (e.g., a touch screen display that functions as both a display and a touch-sensitive surface, a camera, a controller, a joystick, buttons, etc.) having a display generating component (e.g., a display, a projector, a heads-up display, etc.) and one or more input devices (e.g., a touch screen display that functions as both a display and a touch-sensitive surface, a camera, a controller, a joystick, buttons, etc.) Figure 3 Device 300 or Figure 1A The method 900 is performed at a portable multifunction device 100 of the embodiment of the present invention. In some embodiments, the display generation component is a touch screen display, and the touch sensitive surface is an input device on or integrated with the display generation component. In some embodiments, the display generation component is separate from one or more input devices. Some operations in method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0306] The method involves detecting an input while displaying an environment including a virtual object. The virtual object is associated with an action (e.g., an animation). In response to the input, the device determines whether to perform the action or displays an indication of the input that will cause the action to be performed. Displaying an indication of the input that will cause the action associated with the virtual object to be performed provides improved visual feedback to the user (e.g., by determining when the user does not provide an input that meets the criteria for causing the animation of the virtual object and providing information about the required input to the user). Providing improved visual feedback to the user increases the efficiency with which the user is able to cause the action associated with the virtual object to be performed, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0307] The device displays, at (902), via a display generation component (e.g., touch-sensitive display system 112), at least a portion of an environment (e.g., a virtual reality environment, an augmented reality environment, or an environment that allows objects to be placed in the augmented reality environment by manipulating the object), the at least a portion including a virtual object associated with a first action (e.g., an animation sequence that animates display of at least a portion of the virtual object) that is triggered based on satisfying a first set of criteria. For example, Fig. 5F 5002 is shown as a virtual object displayed in an augmented reality user interface 5022. FIG. 5E to FIG. 5F As shown in FIG. 1 , the augmented reality user interface 5022 displays an augmented reality environment that includes a virtual object 5002 and a physical environment as captured by one or more cameras of the device 100 (e.g., one or more cameras of the camera module 143). The virtual object 5002 is associated with an animation sequence, such as Figure 5H to Fig.5I as shown in .

[0308] While the portion of the environment is displayed by the display generating component, the device detects a first input at (904). Figure 5G In FIG. 5 , a tap input is detected at a position corresponding to virtual object 5002. Figure 5J to Figure 5K , a swipe input is detected at a location corresponding to virtual object 5002.

[0309] In response to detecting a first input at (906): based on determining that the first input satisfies a first set of criteria (e.g., the first set of criteria is satisfied by a tap input, a press input, detection of a gaze of a user of the electronic device pointing to a location corresponding to the virtual object, an input causing the virtual object to be displayed at a specified portion of a field of view of one or more cameras, and / or movement of at least a portion of the electronic device to within a defined distance of the displayed location of the virtual object), the device performs a first action (e.g., animating at least a portion of the virtual object according to an animation sequence associated with the virtual object), and based on determining that the first input does not satisfy the first set of criteria but satisfies a second set of criteria (e.g., a swipe input for moving the virtual object, a swipe input for rotating the virtual object, A swipe input for a virtual object, a pinch input for reducing the displayed size of a virtual object, an expand input for increasing the displayed size of a virtual object, detection of a user's gaze of an electronic device directed at a location that does not correspond to a virtual object, an input that causes a virtual object to be displayed at a location that is not within a specified portion of a field of view of one or more cameras, and / or a determination that the electronic device is not within a defined distance of a displayed location of a virtual object to satisfy a second set of criteria), the device abandons performing the first action and instead displays a first visual indication of one or more inputs (e.g., a text prompt indicating the one or more inputs and / or an animation showing the one or more inputs), which, when executed, would cause the first set of criteria to be satisfied.

[0310] For example, in response to an input satisfying a first set of criteria (e.g., Figure 5G ), a first action is performed (e.g., playback of an animation sequence occurs, such as for FIG. 5H to FIG. 5I In response to an input that does not satisfy the first set of criteria but satisfies the second set of criteria (e.g., for Figure 5J to Figure 5K swipe input described above), the device abandons performing the first action and instead displays a first visual indication of one or more inputs that, when performed, would cause the first set of criteria to be satisfied (e.g., the device displays a visual indication 5028 indicating “Tap on the box to see the animation!”, as in response to Figure 5K In some embodiments, in addition to or in lieu of the visual indication, an audio indication of one or more inputs that, when executed, would cause the first set of criteria to be satisfied is provided.

[0311] In some embodiments, at (908) the first set of criteria includes criteria that are satisfied based on determining that at least a portion of the electronic device (e.g., a corresponding input device of the one or more input devices, a display generating component, and / or one or more cameras) is moved to a position within a defined distance of the displayed position of the virtual object. In some embodiments, a first visual indication of one or more inputs that, when executed, would cause the first set of criteria to be satisfied includes instructions to move the electronic device to a position within a defined distance of the displayed position of the virtual object (e.g., "Move the device closer to the box."). For example, as for Figures 5AD to 5AH As described, a first set of criteria for performing an action is satisfied based on determining that the device 100 has moved to a position within a defined distance of the displayed position of the virtual object 5002. Figures 5AD to 5AE , the device is not within a defined distance of the displayed location of the virtual object 5002, and displays a visual indication 5060 of one or more inputs that, when executed, would cause the first set of criteria to be satisfied. Figure 5AF , the device has moved to within a defined distance of the displayed location of the virtual object 5002 and executed an animation sequence, such as for Figures 5AF to 5AH Described. Determining whether to perform an action associated with a virtual object or displaying an indication of an input that will cause the action to be performed (depending on whether the device is moved to a position within a defined distance of a displayed position of the virtual object) provides improved visual feedback to the user (e.g., by determining when the device is not sufficiently close to a virtual object and providing information indicating that the user must be close to the displayed position of the virtual object to activate an animation of the virtual object). Providing improved visual feedback to the user increases the efficiency with which the user is able to cause actions associated with the virtual object to be performed, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0312] In some embodiments, at (910) the one or more input devices include a touch screen display (e.g., touch-sensitive display system 112) and the first set of criteria includes criteria satisfied based on determining that the first input is a tap input made by contact at a location on the touch screen display that corresponds to the virtual object (e.g., as for Figure 5G to Figure 5IDescribed). In some embodiments, the first visual indication includes a guide to tap a location corresponding to the virtual object (e.g., "Tap on the box."). Determining whether to perform an action associated with the virtual object or displaying an indication of an input that will cause the action to be performed (depending on whether the input is a tap input made by contact at a location corresponding to the virtual object on the touch screen display) provides improved visual feedback to the user (e.g., by determining whether the input is a tap input and providing information indicating that the user must tap the virtual object to activate the animation of the virtual object). Providing improved visual feedback increases the efficiency with which the user is able to cause actions associated with the virtual object to be performed, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0313] In some embodiments, at (912) the one or more input devices include a sensor for detecting a direction of a gaze of a user of the electronic device (e.g., one or more cameras of camera module 143) and the first set of criteria includes criteria that are satisfied based on determining that the user's gaze is directed to a location corresponding to a virtual object (e.g., a location within a specified portion of the virtual object, a location anywhere on the virtual object, and / or a location within a defined distance of the virtual object). For example, as for Figure 5L-1 and Figure 5L-2 , Figure 5M-1 and Figure 5M-2 , Figure 5N-1 and Figure 5N-2 as well as Figure 5O-1 and Figure 5O-2 As described above, a first set of criteria for performing an action is met based on determining that the user's gaze is directed to a position corresponding to the virtual object 5002. Figure 5L-1 and Figure 5L-2 , the gaze 5034 of the user 5030 is not directed at the displayed location of the virtual object 5002, and a visual indication 5036 of one or more inputs is displayed that, when executed, would cause the first set of criteria to be satisfied. Figure 5M-1 and Figure 5M-2 , the gaze 5038 of the user 5030 points to the displayed location of the virtual object 5002 and performs an animation sequence, such as for Figure 5M-1 and Figure 5M-2 , Figure 5N-1 and Figure 5N-2 and Figure 5O-1 and Figure 5O-2Described. In some embodiments, the first visual indication includes a guide that directs the user's gaze to a location corresponding to the virtual object (e.g., "Look at the box."). Determining whether to perform an action associated with the virtual object or displaying an indication of an input that will cause the action to be performed (depending on whether the user's gaze is directed to a location corresponding to the virtual object) enables multiple different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user can perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0314] In some embodiments, the electronic device includes one or more cameras (e.g., one or more rear-facing cameras on a side of the device opposite to a display generating component), the displayed environment includes at least a portion of a field of view of the one or more cameras, the device detects movement of the electronic device at (914) (e.g., lateral movement and / or rotation of the electronic device), the movement adjusts the field of view of the one or more cameras, and in response to detecting the movement of the electronic device, the device adjusts a virtual object according to a fixed spatial relationship (e.g., orientation and / or position) between the virtual object and a corresponding plane in the field of view of the one or more cameras (e.g., the virtual object is displayed on the display with a certain orientation and position so that a fixed angle between the virtual object and the plane is maintained when the field of view of the one or more cameras is adjusted in response to the detected movement of the electronic device (e.g., the virtual object appears to remain at a fixed position on the plane). For example, as for Figure 5E As depicted, augmented reality user interface 5022 displays physical environment 5018 as captured in the field of view of one or more cameras. Figure 5S to Figure 5U As described, when the device moves so that the field of view of the one or more cameras is adjusted, the virtual object 5002 moves according to the fixed spatial relationship between the virtual object 5002 and the plane corresponding to the upper surface of the physical table 5020. Adjusting the virtual object according to the fixed spatial relationship between the virtual object and the plane in the field of view of the one or more cameras in response to the movement of the electronic device provides improved visual feedback to the user (e.g., by providing an indication that the virtual object has been placed at a fixed position relative to the plane). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0315] In some embodiments, at (916) the first set of criteria includes criteria that are satisfied based on determining that the virtual object is displayed within a specified portion of the field of view of the one or more cameras (e.g., the virtual object is fully visible in the displayed portion of the field of view of the one or more cameras, the virtual object is substantially centered in the displayed portion of the field of view of the one or more cameras, the virtual object is displayed within the specified portion of the field of view of the one or more cameras, and / or at least a predetermined portion of the virtual object is displayed in the field of view of the camera). For example, at FIG. 5S to FIG. 5T , the virtual object 5002 is not fully displayed within a designated portion (e.g., circle 5052) of the field of view of the one or more cameras displayed in the augmented reality user interface 5022, and a visual indication 5054 of one or more inputs is displayed that, when executed, would cause the first set of criteria to be satisfied. Figure 5U , the virtual object 5002 is fully displayed within the circle 5052 and performs the animation sequence, such as for Figure 5U to Figure 5W In some embodiments, the first visual indication includes instructions to move the device so that the virtual object is displayed within a specified portion of the field of view of the one or more cameras (e.g., "Move the device to view the box."). In some embodiments, the first set of criteria requires that multiple criteria (e.g., gaze at the virtual object, tap input on the virtual object, proximity to the virtual object, and / or virtual object being in the field of view) be met (e.g., as for Figures 5X to 5AC. In some embodiments, based on determining that a virtual object display criterion is met (e.g., the virtual object is fully visible in the displayed portion of the field of view of the one or more cameras, the virtual object is substantially centered in the displayed portion of the field of view of the one or more cameras, the virtual object is displayed within a specified portion of the field of view of the one or more cameras, and / or at least a predetermined portion of the virtual object is displayed in the field of view of the camera), the first visual indication is not displayed. In some embodiments, the virtual object has a boundary (e.g., an outline of the virtual object 5002 or an ellipse or box surrounding the virtual object 5002), performing the action (e.g., animation of the virtual object) causes at least a portion of the virtual object to extend beyond the first boundary, the second boundary corresponds to the extension range of the first corresponding virtual object during the animation, and the first set of criteria includes a criterion that is met based on determining that the second boundary is fully displayed in the displayed portion of the field of view of the camera of the device. Determining whether to perform an action associated with the virtual object or display an indication of an input that will cause the action to be performed (depending on whether the virtual object is displayed within the specified portion of the field of view of the one or more cameras) enables a variety of different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to input increases the efficiency with which a user can perform those operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0316] In some embodiments, at (918), the second set of criteria includes criteria that are satisfied based on determining that the first input is an input for moving a virtual object (e.g., a swipe input including movement of a contact along a path on a touch screen display). Figure 5J to Figure 5K Inputs for moving a virtual object 5002 are depicted. Displaying an indication of the input that will cause the action to be performed in response to the input for moving the virtual object provides improved visual feedback to the user (e.g., by providing the user with information about the different inputs required to animate the virtual object). Providing visual feedback to the user enhances the operability of the device (e.g., by helping the user provide appropriate inputs and reducing unnecessary additional inputs for interacting with virtual objects), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0317] In some embodiments, at (920) the device receives a request to display a virtual object at a first location in the environment, and in response to the request to display the virtual object at the first location in the environment: based on determining that a surface size corresponding to the first location does not satisfy a size compatibility standard for the virtual object, the device displays a visual indication that provides information indicating that the first location does not satisfy the size compatibility standard (and abandons displaying the virtual object at the first location in the environment), and based on determining that a surface size corresponding to the first location satisfies the size compatibility standard for the virtual object (e.g., the surface size corresponding to the first location is smaller than the size of the virtual object (e.g., smaller by a predefined amount)), the device displays the virtual object at the first location in the environment. For example, as for Figure 5Q to Figure 5R As described, in response to a request to display virtual object 5002 at the upper surface of physical lamp 5004 (e.g., along Figure 5Q ), the visual indication 5050 provides information indicating that the first position does not meet the size compatibility criteria (e.g., information including text indicating "The object cannot be placed on this surface!") because the size of the upper surface of the physical light 5044 does not meet the size compatibility criteria of the virtual object 5002. In some embodiments, the virtual object has a size attribute (e.g., corresponding to the real-world size of the object), and the size attribute indicates the size of at least a portion of the virtual object (e.g., length, width, radius and / or area). For example, the size attribute indicates the size of a portion of the virtual object (e.g., the size of the surface of the virtual object that will be placed at a fixed position relative to (e.g., facing) a surface in the physical environment). The first position is, for example, the upper surface of a table, a wall, a floor, or a ceiling. Displaying a visual indication in response to determining that the position where the user requested to display the virtual object does not meet the size compatibility criteria provides improved visual feedback to the user (e.g., by providing an indication of why the virtual object is not placed at that position and / or how to place the virtual object at a position that meets the size compatibility criteria). Providing improved visual feedback enhances the operability of the device (e.g., by helping the user provide appropriate input and reducing unnecessary additional input for placing virtual objects) and makes the user-device more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0318] In some embodiments, movement of the electronic device is detected while (and / or after) performing the first action at (922), and in response to detecting the movement of the electronic device, the virtual object is adjusted according to the fixed spatial relationship between the virtual object and the corresponding plane so that the virtual object at least partially moves beyond the displayed portion of the field of view of the one or more cameras, and when the virtual object at least partially exceeds the displayed portion of the field of view of the one or more cameras, the device displays a second visual indication (e.g., the same as or different from the first visual indication) of one or more inputs that, when executed, would cause the first set of criteria to be satisfied. Displaying indications of inputs that would cause the first set of criteria to be satisfied when the virtual object exceeds the displayed field of view of the one or more cameras provides improved visual feedback to the user (e.g., to indicate that the user must move the device so that the virtual object is visible (e.g., fully visible) within the field of view of the one or more cameras). Providing improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input, reducing unnecessary additional input to cause the first set of criteria to be satisfied), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0319] In some embodiments, the electronic device includes one or more cameras, and based on determining that the first input corresponds to a request to display a virtual object in an augmented reality environment (e.g., an input at a location corresponding to an “AR” area of ​​toggle control 5008, such as an input for Figure 5D ), the device replaces the display of the environment including the virtual object with a display of at least a portion of the field of view of the one or more cameras at (924) (e.g., as for FIG. 5E to FIG. 5FAs described). In some embodiments, the environment displayed by the display generation component includes a background other than an image of the physical environment surrounding the device (e.g., the background of the first user interface area is a pre-selected background color / pattern, or a background image that is different from the output image captured simultaneously by the one or more cameras and different from the live content in the field of view of the one or more cameras), and in response to a request to display a virtual object in the augmented reality environment, the display of the environment is replaced with a display of the physical environment surrounding the device (e.g., as captured in the field of view of the one or more cameras). Either an action associated with the virtual object is performed, or an indication of an input that will cause the action to be performed is displayed, or in response to the input, the display of the environment including the virtual object is replaced with a display of the virtual object together with the field of view of the one or more cameras, which enables multiple different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to the input increases the efficiency with which the user is able to perform these operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0320] In some embodiments, the second set of criteria at (926) includes criteria that are satisfied based on determining that the first input is an input for changing a feature (e.g., orientation and / or size) of the virtual object. In some embodiments, the input for changing a feature includes a swipe input (e.g., movement of a contact along a path on a touch screen display). For example, the input for changing a feature of the virtual object is an input for rotating the virtual object 5002 detected when the virtual object 5002 is displayed in the object staging user interface 5004, such as for FIG. 5B to FIG. 5CDescribed. In some embodiments, the swipe input changes the orientation of the virtual object (e.g., by rotating the virtual object around one or more axes). In some embodiments, the input for changing the feature includes a pinch and / or spread input (e.g., movement of two contacts along corresponding paths on the touch screen display, the paths bringing the two contacts closer to each other (pinch) or apart from each other (spread)). In some embodiments, the pinch and / or spread input changes the displayed size of the virtual object. In some embodiments, the first input that meets the first set of criteria is an input detected when the virtual object 5002 is displayed in the object staging user interface 5004 (e.g., a tap input at a position corresponding to the virtual object 5002) (e.g., causing an animation sequence associated with the virtual object 5002 to occur when the virtual object 5002 is displayed in the object staging user interface 5004). Either an action associated with the virtual object is performed, or an indication of the input that will cause the action to be performed is displayed, or the display of the environment including the virtual object is replaced with a display of the virtual object together with the field of view of one or more cameras in response to the input, which enables a variety of different types of operations to be performed in response to the input. Enabling multiple different types of operations to be performed in response to input increases the efficiency with which a user can perform those operations, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0321] In some embodiments, the first visual indication of the one or more inputs that, when executed at (928), would cause the first set of criteria to be satisfied includes a text prompt (e.g., a text prompt indicating "Tap on the box to see the animation!", such as for Figure 5K Displaying a text prompt indicating an input that will cause an action associated with a virtual object to be performed provides improved visual feedback to the user (e.g., by providing the user with text indicating the required input). Providing improved visual feedback to the user increases the efficiency with which the user is able to cause an action associated with a virtual object to be performed, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0322] In some embodiments, the device detects a second input that satisfies the first set of criteria while displaying a first visual indication of one or more inputs that, when executed, would cause the first set of criteria to be satisfied at (930), and in response to detecting the second input that satisfies the first set of criteria, the device ceases to display the first visual indication. Figures 5AD to 5AFAs described, an input that satisfies a first set of criteria for performing an action is detected while the visual indication 5060 is displayed, and based on determining that the device 100 has moved to a position within a defined distance of the displayed position of the virtual object 5002 (e.g., for Figure 5AF ), the device ceases to display the visual indication 5060. In response to detecting an input that satisfies the first set of criteria (e.g., without further user input that eliminates the first visual indication), ceasing to display the visual indication of the input that will cause the action to be performed reduces the number of inputs required to perform the operation. Reducing the number of inputs required to perform the operation enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0323] In some embodiments, performing the first action at (932) includes animating at least a portion of the virtual object (e.g., animating the virtual object 5002 so that the lid of the box opens, such as for Figure 5G to Figure 5I ). Animating at least a portion of a virtual object provides improved visual feedback to the user (e.g., to indicate that the input provided meets the criteria for animating the object). Providing improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0324] In some embodiments, the device displays a video playhead control at (934) (e.g., during and / or after performing the first action) for adjusting playback of the animation sequence that animates the portion of the virtual object. Figures 5AI to 5AK As described, the input of moving the video playhead control 5066 adjusts the playback of the animation sequence that animates the cover of the virtual object 5002. Displaying the video playhead control for adjusting the playback of the animation sequence that animates the portion of the virtual object provides the user with improved visual feedback (e.g., to indicate that the playback of the virtual object is adjustable). Providing improved visual feedback enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0325] In some embodiments, at (936) the electronic device includes an audio generation component, performing the first action includes generating a first audio channel and a second audio channel by the audio generation component, and the device adjusts the amplitude of the first audio channel and the amplitude of the second audio channel according to the displayed position of the virtual object in the environment. In some embodiments, spatial audio is used to provide the user with an indication of the distance and / or position of the virtual object in the environment relative to the (real or virtual) position of the user in the environment. For example, when the distance between the virtual object and the user decreases, the amplitude of the first audio channel and the amplitude of the second audio channel increase, and when the virtual object moves from the left side of the user to the right side of the user, the amplitude of the left audio channel decreases and the amplitude of the right audio channel increases. Adjusting the amplitude of the first audio channel and the amplitude of the second audio channel based on the displayed position of the virtual object in the environment provides auditory feedback to the user (e.g., to indicate the distance and position of the virtual object in the environment relative to the position of the user in the environment). Providing auditory feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user locate virtual objects that are not currently within the displayed field of view of the one or more cameras), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0326] It should be understood that 9A to 9C The specific order in which the operations have been described herein is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 1000, 1100, 1200, and 1400) are also applicable in a similar manner to the details of other processes described above with respect to 9A to 9C Method 900 is described. For example, the contacts, inputs, gestures, virtual objects, environments, user interface areas, user interface objects, fields of view, movement, and / or animations described above with reference to method 900 optionally have one or more of the features of the contacts, inputs, gestures, media items, virtual objects, environments, user interface areas, user interface objects, fields of view, movement, and / or animations described herein with reference to other methods described herein (e.g., methods 1000, 1100, 1200, and 1400). For the sake of brevity, these details are not repeated here.

[0327] FIG. 10A to FIG. 10C1 is a flow chart of method 1000 for displaying a media item in a user interface that includes content of at least a portion of a field of view of one or more cameras, according to some embodiments. Method 1000 is performed on an electronic device (e.g., a display generating component (e.g., a display, a projector, a heads-up display, etc.), one or more input devices (e.g., a touch screen display that functions as both a display and a touch-sensitive surface, a camera, a controller, a joystick, buttons, etc.), and one or more cameras (e.g., one or more rear-facing cameras on a side of the device opposite the display and the touch-sensitive surface). Figure 3 Device 300 or Figure 1A The method 1000 is performed at a portable multifunction device 100 of the embodiment of the present invention. In some embodiments, the display generation component is a touch screen display, and the touch sensitive surface is an input device on or integrated with the display generation component. In some embodiments, the display generation component is separate from one or more input devices. Some operations in method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0328] The method involves displaying representations of media items in a user interface that includes content of at least a portion of a field of view of one or more cameras. Displaying media items in a user interface that includes content of at least a portion of a field of view of one or more cameras reduces the number of inputs required to perform an operation (e.g., by providing controls in an existing shared user interface for displaying representations of media items in an interface that includes content of at least a portion of a field of view of one or more cameras). Reducing the number of inputs required to perform an operation enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to display representations of media items in an interface that includes content of at least a portion of a field of view of one or more cameras and / or access an external application to convert media items for display in an interface that includes content of at least a portion of a field of view of one or more cameras without having to provide input), which in turn reduces power usage and extends battery life of the device by enabling the user to use the device more quickly and efficiently.

[0329] The device displays at (1002) a first user interface area (eg, a photo viewing interface 6004 of a photo management application, such as a touch-sensitive display system 112) by a display generating component (eg, a touch-sensitive display system 112). Fig. 6A 6002), a first representation of a first media item in a first user interface area (e.g., a photo (such as photo 6002), an image, a still image corresponding to a video, an album cover corresponding to music content, or a thumbnail representation thereof), and a first user interface object (e.g., control 6008) for displaying a sharing user interface (e.g., an interface providing multiple revelations corresponding to various sharing methods for the first media item and / or other media items).

[0330] While displaying the first user interface area, the device detects a first input corresponding to a shared user interface object at (1004). For example, the device detects an input at a location corresponding to a control 600 for displaying a shared user interface, such as for Figure 6B described.

[0331] In response to detecting the first input corresponding to the first user interface object, the device displays a shared user interface at (1006) (e.g., overlaying on or replacing display of the first user interface), the shared user interface including a second user interface object for initiating a process of displaying a second representation of the first media item in the augmented reality environment (e.g., a control 6024 for displaying one or more selected objects in an object three-dimensional viewing mode). For example, in response to a control for Figure 6B The device displays a shared user interface 6016 including controls 6024, such as for Figure 6C described.

[0332] While displaying the shared user interface, the device detects, at (1008), a sequence of one or more inputs including a selection of a second user interface object. Figure 6C As described, while sharing user interface 6016 is displayed, input is detected at a location corresponding to control 6024.

[0333] In response to detecting a sequence of one or more inputs including a selection of a second user interface object, the device (stops displaying the shared user interface and / or the first user interface area, and) displays, at (1010), a second representation of the first media item in a second user interface (e.g., augmented reality user interface 5022) that includes content of at least a portion of a field of view of the one or more cameras (e.g., one or more cameras of camera module 143). For example, in response to a selection of a second user interface object, the device (stops displaying the shared user interface and / or the first user interface area, and) displays, at (1010), a second representation of the first media item in a second user interface (e.g., augmented reality user interface 5022) that includes content of at least a portion of a field of view of the one or more cameras (e.g., one or more cameras of camera module 143). Figure 6C The device displays the photo 6002 in an augmented reality user interface 5022 that includes at least a portion of a field of view of one or more cameras of the device 100, such as for FIG. 6D to FIG. 6E In some embodiments, the second representation of the first media item is displayed with simulated depth (e.g., giving a flat image the appearance of depth in an environment). Fig. 6E As depicted, image 6002 is displayed with a simulated backing 6034 that adds the appearance of depth in the environment to image 6002. In some embodiments, the second representation of the first media item is modified to include simulated reflectivity (e.g., making the image appear glossy) and / or increased brightness (e.g., making the image appear shiny).

[0334] In some embodiments, at (1012) the displayed second representation of the first media item is overlaid on a corresponding plane (e.g., a wall, floor, or ceiling) in the field of view of the one or more cameras. Fig. 6E , a photograph 6002 is overlaid on a wall 6032 of a physical environment 5018 as detected in the field of view of the one or more cameras of the device 100. Displaying representations of media items overlaid on a plane in the environment provides improved visual feedback to the user (e.g., allowing the user to see the distance and location of media items in the environment). Providing improved visual feedback enhances the operability of the device and makes the user-device interface more efficient (e.g., providing the user with convenient access and manipulation of media items), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0335] [Rotation of image is constrained by wall surface] In some embodiments, while displaying a second representation of a first media item in a second user interface, the device detects at (1014) an input for rotating the second representation of the first media item (e.g., input by two contacts moving along a rotation path (e.g., clockwise or counterclockwise)). In response to the input, the device rotates the second representation of the first media item about an axis that is perpendicular to a corresponding plane in the field of view of the one or more cameras on which the first media item is displayed. For example, as for Figure 6W to Figure 6Y As described, in response to input for rotating the photo 6046 about an axis that is perpendicular to the wall 6032 on which the photo 6046 is displayed, the device rotates the photo 6046 about an axis that is perpendicular to the wall 6032. In some embodiments, based on determining that the input for rotating the second representation of the first media item is a request to rotate the second representation of the first media item about an unavailable axis (e.g., an axis that is not perpendicular to a corresponding plane in the field of view of the one or more cameras on which the first media item is displayed), a visual indication is displayed (e.g., to indicate that rotation about an unavailable axis is not permitted.) For example, as for Figure 6ZAs described, in response to input for rotating the photo 6046 about an axis 6106 parallel to the wall 6032 on which the photo 6046 is displayed, the device displays a visual indication 6098 indicating that rotation about the axis 6106 is not permitted. Rotating the media item representation about an axis perpendicular to the plane in the field of view of the one or more cameras in response to input reduces the number of inputs required to perform the operation (e.g., by allowing the user to interact directly with the displayed representation of the media item as displayed in the augmented reality user interface without requiring the user to indicate a preferred rotation in different scenarios before displaying the media item in the augmented reality user interface). Reducing the number of inputs required to perform the operation enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and improves the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0336] In some embodiments, the device detects at (1016) a first movement of the electronic device (e.g., lateral movement and / or rotation of the electronic device) that adjusts the field of view of the one or more cameras (e.g., FIG. 6E to FIG. 6F 6002) and, in response to detecting movement of the electronic device, the device adjusts a second representation of a first media item (e.g., photo 6002) according to a fixed spatial relationship (e.g., orientation and / or position) between the second representation of the first media item and a corresponding plane (e.g., wall 6032) in the field of view of the one or more cameras. Adjusting the media item representation according to the fixed spatial relationship relative to the field of view of the camera provides improved visual feedback to the user (e.g., to indicate the fixed spatial relationship of the media item representation relative to the detected plane in the physical environment). Providing improved visual feedback enhances the operability of the device and makes the user-device interface more effective (e.g., by helping the user provide appropriate input and reducing usage errors when interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0337] In some embodiments, the device detects a selection of a first representation of a second media item at (1018) (e.g., when displaying a first user interface area or when displaying a shared user interface), and in response to detecting a sequence of one or more inputs including a selection of a second user interface object, the device displays a second representation of the second media item in a second user interface including content of at least a portion of a field of view of one or more cameras (e.g., the second representation of the first media item and the second representation of the second media item are displayed simultaneously in a portion of the field of view of one or more cameras). For example, as described for FIG. 6I to FIG. 6JAs described above, input for selecting a first media item (photo 6002) and a second media item (photo 6046) is detected. In response to detecting input including input for selecting photos 6002 and 6046 (and input for displaying photos 6002 and 6046 in augmented reality user interface 5022, as for Figures 6L to 6M ) including an input sequence including the one or more cameras, displaying photos 6002 and 6046 in an augmented reality user interface 5022 including content of at least a portion of the field of view of the one or more cameras, such as for FIG. 6N to FIG. 6O Described. In some embodiments, a second representation of a first media item and a second representation of a second media item are displayed in a fixed spatial relationship to a plane in the field of view of the one or more cameras (e.g., overlaid on the same plane). Displaying multiple media items in a user interface that includes the field of view of one or more cameras in response to detecting a sequence of one or more inputs reduces the number of inputs required to perform an operation (e.g., by allowing a user to select multiple media items using a single selection user interface and place multiple selected items simultaneously). Reducing the number of inputs required to perform an operation enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0338] In some embodiments, a selection of a second media item is detected while displaying a shared user interface at (1020). For example, the first media item is selected by a first input corresponding to a shared user interface object (e.g., when the first input is received while the first media item is displayed in a first user interface area) and the second media item is selected by an additional input received at a location corresponding to the second media item while displaying the shared user interface (e.g., belonging to a sequence of one or more inputs that includes a selection of the second user interface object). Displaying multiple media items in a user interface that includes a field of view of one or more cameras in response to detecting a sequence of one or more inputs reduces the number of inputs required to perform an operation (e.g., by allowing a user to provide a single input at a shared user interface object to simultaneously place multiple selected items). Reducing the number of inputs required to perform an operation enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0339] In some embodiments, while displaying the first user interface area, the device displays a first representation of the second media item at (1022) and detects selection of the second media item while displaying the first user interface area. For example, by displaying the first user interface area (e.g., photo selection interface 6036) at a location corresponding to the first media item (e.g., such as a location for the photo selection interface 6036). Fig.6I6002) by detecting a first input (e.g., a tap input) corresponding to a first media item (e.g., photo 6002) and selecting the first media item by displaying the first user interface area (e.g., before detection of the first input corresponding to the shared user interface object) at a location corresponding to the second media item (e.g., as described above). Figure 6J The device 6046 is a device that displays multiple media items in a user interface that includes a field of view of one or more cameras in response to detecting a sequence of one or more inputs. ...

[0340] In some embodiments, selection of the first media item is detected at a first time at (1024) (e.g., by a first input corresponding to a shared user interface object (e.g., when the first input is received while the first media item is displayed in the first user interface area) or by a first selection input detected before the first input corresponding to the shared user interface object (e.g., a tap input at a location corresponding to the first media item)). For example, selection of the first representation of photo 6002 is detected at a first time (e.g., as indicated for Fig.6I A selection of the second media item is detected at a second time (e.g., by a selection input detected while the sharing user interface is displayed or by a second selection input (e.g., a tap input at a location corresponding to the second media item) detected prior to detection of the first input corresponding to the sharing user interface object). For example, a selection of the first representation of photo 6046 is detected at a second time (e.g., as described for Figure 6JDescribed as above). Displaying a second representation of the first media item in the second user interface and displaying a second representation of the second media item in the second user interface includes: displaying the second representation of the first media item and the second representation of the second media item in a first order (e.g., the first media item is displayed to the left of (or above or in front of) the second media item (e.g., covered on a plane in the field of view of the one or more cameras)) based on determining that the first time is before the second time, and displaying the second representation of the first media item and the second representation of the second media item in a second order different from the first order (e.g., the first media item is displayed to the right of (or below or behind) the second media item) based on determining that the second time is before the first time. For example, since the first time when the first representation of photo 6002 is selected is before the second time when the first representation of photo 6046 is selected, Fig.6O 6046. If the first representation of photo 6046 is selected before the first representation of photo 6046, the second representation of photo 6002 will instead be displayed on the right side of the second representation of photo 6046. Sorting the multiple media item representations in the environment according to the selection time of the corresponding media items can perform operations (e.g., operations to determine the order of the media item representations in the environment) based on the selection order without further user input to sort the multiple media items in the environment. Automatically performing operations without further user input enhances the operability of the device, which reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0341] In some embodiments, while a first representation of a first media item is displayed in a second user interface and a second representation of a second media item is displayed in a second user interface, the device detects at (1026) input for manipulating the second representation of the corresponding media item (e.g., input including a swipe gesture (for changing the position of the second representation of the corresponding media item on a corresponding plane in the field of view of the one or more cameras or for moving the second representation of the corresponding media item from a first corresponding plane in the field of view of the one or more cameras to a second corresponding plane in the field of view of the one or more cameras), input including an expand gesture (for increasing the size of the second representation of the corresponding media item), input including a pinch gesture (for decreasing the size of the second representation of the corresponding media item), and / or input including a rotate gesture (for rotating the second representation of the corresponding media item). In response to detecting the input for manipulating the second representation of the corresponding media item: based on determining that the first input points to the second representation of the first media item (e.g., a swipe input pointing to photo 6002, as for Figure 6O to Figure 6P), the device manipulates the second representation of the first media item based on the first input (e.g., the photo 6002 moves according to the swipe input); and according to determining that the first input points to the second representation of the second media item (e.g., the swipe input points to the photo 6046, as for Figure 6Q to Figure 6R ), the device manipulates a second representation of a second media item based on the first input (eg, at this time photo 6046 moves according to a swipe input). Figure 6P to Figure 6Y Examples of inputs for manipulating media items 6002, 6046, and 6052 in augmented reality user interface 5022 are shown. In some embodiments, input for manipulating a second representation of a corresponding media item is detected when only a single media item (e.g., a second representation of a first media item) is displayed and the displayed representation of the media item is manipulated in response to the input. Manipulating the corresponding media item representation in the environment in response to detecting the input reduces the number of inputs required to perform the operation (e.g., by allowing the user to directly manipulate the displayed object without the need for additional input to specify the item to be manipulated). Reducing the number of inputs required to manipulate media item representations enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0342] In some embodiments, the sharing user interface at (1028) includes a third user interface object for transferring the first media item to a remote device (e.g., via email, message (e.g., SMS or MMS), or an inter-device file transfer service (e.g., AirDrop)). Figure 6C As described, the sharing user interface 6016 includes a user interface object for transferring the photo 6002 to a contact via an inter-device file transfer service (labeled with the contact names "John" and "Mary", as indicated in area 6018) and a user interface object for transferring the photo 6002 to a remote device via email or via a message, respectively (labeled "Mail" and "Message" in area 6020). In some embodiments, the sharing user interface includes a fourth user interface object for viewing and / or storing the first media item in an application (e.g., an application different from the application that displays the first user interface object (e.g., a photo management application) (e.g., a note management application). For example, as for Figure 6C As depicted, sharing user interface 6016 includes a user interface object (labeled “Notepad” in area 6018) for storing photo 6002 to a notepad application that is distinct from the photo management application. In some embodiments, the sharing user interface includes a fifth user interface object for copying the first media item. For example, as for Figure 6CAs described, sharing user interface 6016 includes a user interface object for copying photo 6002 (labeled "Copy" in area 6022). In some embodiments, the sharing user interface includes a sixth user interface object for stopping display of the sharing user interface. Figure 6C As described, a user interface object for stopping displaying the sharing user interface 6016 (control 6028 for stopping displaying the sharing user interface 6016) is displayed. Including a user interface object for transmitting the media item to the remote device on the same sharing user interface that includes an option for displaying the media item in the field of view of one or more cameras provides additional control options without cluttering the user interface (e.g., with controls for displaying the additional user interface). Providing additional control options without cluttering the user interface with additional displayed controls enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0343] In some embodiments, at (1030), a second representation of the first media item (e.g., a clip art version of the photo 6110) is displayed in an orientation perpendicular to a plane (e.g., a floor or ceiling) in the field of view of the one or more cameras. Figure 6AE to Figure 6AH As described, the clip art version of the photograph 6110 is displayed in an orientation perpendicular to the floor plane 5046 (eg, such that the floor plane 5046 appears to support the clip art version of the photograph 6110). In some embodiments, the background of the first media item is transparent (e.g., an outline of the subject of the first media item is detected and the background is removed from the second representation of the first media item, including one or more portions of the first media item that extend beyond the outline (e.g., portions of the photo 6110 that extend beyond the clip art version of the photo 6110), such that the physical environment as captured in the field of view of the one or more cameras is visible around the outline and "through" the transparent portions of the second representation of the first media item. For example, the physical table 5020 is visible around the outline of the clip art version of the photo 6110. Displaying the media item representation in an orientation perpendicular to a plane in the field of view of the one or more cameras provides the user with additional control options without cluttering the user interface with the additional displayed controls (e.g., enabling the user to use controls accessed via the same shared user interface to display the media item at a fixed position parallel to a vertical surface or perpendicular to a horizontal surface). Providing additional control options to the user enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0344] In some embodiments, while displaying the first side of the second representation of the first media item, the device detects at (1032) a second movement of the electronic device that adjusts the field of view of the one or more cameras (e.g., as directed to Figure 6AE to Figure 6AI In response to detecting a second movement of the electronic device, the device adjusts the second representation of the first media item (e.g., a clip art version of the photo 6110) according to the fixed spatial relationship (e.g., orientation and / or position) between the second representation of the first media item and the plane in the field of view of the one or more cameras. After the second movement, a second side of the second representation of the first media item is displayed that is opposite to the first side and is a reverse version of the first side of the second representation of the first media item. For example, Figure 6AI The clipart version of the photo 6110 shown in Figure 6AE The second side opposite the first side that displays the second representation of the first media item in response to movement of the electronic device provides improved visual feedback to the user (e.g., indicating the user's position relative to the photo), thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0345] It should be understood that FIG. 10A to FIG. 10C The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 900, 1100, 1200, and 1400) are also applicable in a similar manner to the details of the above with respect to FIGS. 10A to 10B. Fig. 10C Method 1000 is described. For example, the contacts, inputs, gestures, media items, environments, user interface areas, user interface objects, fields of view, and / or movements described above with reference to method 1000 optionally have one or more of the features of the contacts, inputs, gestures, media items, virtual object environments, virtual objects, user interface areas, user interface objects, fields of view, and / or movements described herein with reference to other methods described herein (e.g., methods 900, 1100, 1200, and 1400). For the sake of brevity, these details are not repeated here.

[0346] FIG. 11A to FIG. 11D1 is a flow chart showing a method 1100 for responding to input directed to a corresponding virtual object in a plurality of displayed virtual objects according to some embodiments. The method 1100 is performed in an electronic device (e.g., a display generating component (e.g., a display, a projector, a head-up display, etc.), one or more input devices (e.g., a touch screen display that serves as both a display and a touch-sensitive surface, a camera, a controller, a joystick, buttons, etc.), and one or more cameras (e.g., one or more rear-facing cameras on the side of the device opposite the display and the touch-sensitive surface). Figure 3 Device 300 or Figure 1A The method 1100 is performed at a portable multifunction device 100 of the present invention. In some embodiments, the display generation component is a touch screen display, and the touch-sensitive surface is an input device on or integrated with the display generation component. In some embodiments, the display generation component is separate from one or more input devices. Some operations in method 1100 are optionally combined, and / or the order of some operations is optionally changed.

[0347] The method involves changing display properties of a single virtual object or multiple virtual objects based on a manipulation input, depending on whether the multiple virtual objects are displayed with a portion of the field of view of one or more cameras. Changing the display properties of a single virtual object or changing the display properties of multiple virtual objects based on whether the multiple virtual objects are displayed with a portion of the field of view of the camera provides additional control options without cluttering the user interface with the additional displayed controls (e.g., using the same input to perform different operations without requiring separate controls for different operations). Providing additional control options without cluttering the user interface with the additional displayed controls enhances the operability of the device and makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0348] The device displays a plurality of virtual objects (e.g., graphical representations of three-dimensional objects) at (1102) by a display generating component (e.g., a touch-sensitive display system 112). Fig. 7A As described, virtual objects 7002, 7004, and 7006 are displayed in object staging user interface 5004; and as described for Figure 7H As described, virtual objects 7002, 7004, and 7006 are displayed in an augmented reality user interface 5022 that includes at least a portion of the field of view of the one or more cameras (e.g., one or more cameras of camera module 143).

[0349] While displaying the plurality of virtual objects, the device detects, at (1104), a first manipulation input (e.g., a pinch or swipe at a location on the touch-sensitive surface corresponding to: the first virtual object, the second virtual object, another virtual object in the plurality of virtual objects, or a space adjacent to the corresponding virtual object in the plurality of virtual objects). Figure 7B As described, an upward swipe input is detected at a location corresponding to the virtual object 7002 displayed in the object staging user interface 5004. Figure 7H , an upward swipe input is detected at a location corresponding to the virtual object 7002 displayed in the augmented reality user interface 5022.

[0350] In response to detecting a first manipulation input directed to a corresponding virtual object among the multiple virtual objects at (1106): based on determining that the multiple virtual objects are displayed together with at least a portion of the field of view of the one or more cameras, the device changes display properties of the corresponding virtual objects based on the first manipulation input (e.g., rotates, scales, and / or moves the corresponding virtual objects individually), and based on determining that the multiple virtual objects are displayed in a user interface that does not include the field of view of the one or more cameras (e.g., a staging user interface that includes a simulated three-dimensional space, in which three-dimensional representations of the virtual objects can be manipulated in response to user input (e.g., rotates and / or scales the three-dimensional representations of the virtual objects around an x-axis, a y-axis, and / or a z-axis)), the device changes display properties of the multiple virtual objects based on the first manipulation input (e.g., rotates and / or scales the multiple virtual objects together). For example, in response to detecting a first manipulation input directed to a corresponding virtual object among the multiple virtual objects: based on determining that the multiple virtual objects are displayed together with at least a portion of the field of view of the one or more cameras, the device changes display properties of the multiple virtual objects based on the first manipulation input (e.g., rotates and / or scales the multiple virtual objects together). Figure 7B The input described above changes the display properties of virtual objects 7002, 7004, and 7006 when displaying virtual object 7002 in object staging user interface 5004 that does not include the field of view of the one or more cameras (e.g., virtual objects 7002, 7004, and 7006 all rotate simultaneously, such as FIG. 7B to FIG. 7C In response to Figure 7H The input described above changes the display properties of the virtual object 7002 (e.g., moves the virtual object 7002, such as FIG. 7H to FIG. 7I As shown in , and other virtual objects are not moved (virtual objects 7004 and 7006 are in FIG. 7H to FIG. 7I remains in the same position).

[0351] In some embodiments, when the plurality of virtual objects are displayed in a user interface that does not include a field of view of the one or more cameras, the plurality of virtual objects are displayed according to a first set of display characteristics at (1108). Displaying the virtual objects according to the first set of display characteristics when the virtual objects are displayed in a user interface that includes a field of view of the camera provides improved visual feedback to the user (e.g., by allowing the user to determine how the virtual objects are displayed). Providing improved visual feedback to the user increases the efficiency with which the user is able to manipulate the virtual objects, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0352] In some embodiments, at (1110), the first set of display features includes a height ordering feature and displaying the plurality of objects according to the height ordering feature includes ordering respective virtual objects in the plurality of virtual objects according to respective heights of the respective virtual objects (e.g., displaying a first virtual object having a first height to the left of, above, and / or in front of a second virtual object having a second height). For example, Figure 7X As shown in , virtual objects 7002, 7004, and 7006 are displayed in descending height order. In some embodiments, the multiple objects are displayed in ascending height order. Displaying virtual objects according to the height sorting feature provides improved visual feedback to the user (e.g., by allowing the user to easily understand the number and distribution of virtual objects). Providing improved visual feedback to the user enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0353] In some embodiments, at (1112), the first set of display features includes a spacing feature and displaying the plurality of objects according to the spacing feature includes positioning respective virtual objects in the plurality of virtual objects at uniform distances between respective boundaries of adjacent respective virtual objects. Figure 7X As shown in , virtual objects 7002, 7004, and 7006 are displayed at uniform distances between the boundaries (e.g., outlines) of the respective virtual objects. Displaying virtual objects according to spacing characteristics including the positioning of the respective virtual objects when displaying virtual objects in a user interface that includes the field of view of the camera provides improved visual feedback to the user (e.g., by allowing the user to easily understand the number and distribution of virtual objects). Providing improved visual feedback to the user increases the efficiency with which the user is able to manipulate virtual objects, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0354] In some embodiments, at (1114), a first corresponding virtual object in the plurality of virtual objects has a first boundary corresponding to a stationary state of the first corresponding virtual object (e.g., an outline of the first corresponding virtual object or an ellipse or box surrounding the first corresponding virtual object). Figure 7Z As described, the virtual object 7090 has a first boundary 7096, which is a box around the virtual object 7090 (e.g., its outer edge). The first corresponding virtual object is associated with a first action (e.g., an animation sequence that animates at least a portion of the virtual object), and the first action causes at least a portion of the first virtual object to extend beyond the first boundary (e.g., the virtual object increases in size during the animation). For example, as shown in FIG. Figure 7Z to Figure 7AB As described, the virtual object 7090 has an animation sequence that causes the virtual object 7090 to move beyond the first boundary 7096. The first corresponding virtual object has a second boundary that corresponds to the extension range of the first corresponding virtual object during the first action (e.g., an outline of the first corresponding virtual object extended by the first action or an ellipse or a box surrounding the first corresponding virtual object extended by the first action). Figures 7AB to 7AC As described, the second boundary 7108 of the virtual object 7090 up to the extension range of the virtual object 7090 includes the movement of the virtual object 7090 beyond the boundary 7096 during the animation sequence. The corresponding boundaries of the adjacent corresponding virtual objects include the second boundary of the first corresponding virtual object. For example, the virtual object 7088 has a boundary 7094, which is a box around the virtual object 7088 (e.g., its outer edge), the virtual object 7090 has a second boundary 7108 of the virtual object 7090 up to the extension range of the virtual object 7090, and the virtual object 7092 has a boundary 7098, which is a box around the virtual object 7092 (e.g., its outer edge). The virtual objects 7088, 7090 and 7092 are displayed at a uniform distance between the boundaries (e.g., outlines) of the corresponding virtual objects. A virtual object having a first boundary corresponding to a static state of the virtual object and a second boundary corresponding to an extended range of the virtual object during an associated action provides improved visual feedback to the user (e.g., by allowing the user to determine the range of animation of the virtual object and manipulate the virtual object based on information about the range of animation). Providing improved visual feedback to the user increases the efficiency with which the user is able to interact with the virtual object (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0355] In some embodiments, after displaying the plurality of virtual objects in a user interface (e.g., object staging user interface 5004) that does not include the field of view of the one or more cameras while causing the plurality of virtual objects to be displayed with at least a portion of the field of view of the one or more cameras (e.g., in augmented reality user interface 5022), based on the first manipulation input, the device detects at (1116) a request to display the plurality of virtual objects in a user interface (e.g., object staging user interface 5004) that does not include the field of view of the one or more cameras. For example, after display properties of virtual objects 7002, 7004, and 7006 have been changed in response to input provided when displaying augmented reality user interface 5022 (e.g., for FIG. 7H to FIG. 7R After that, input is provided at a position corresponding to the "object" area of ​​the toggle control 5008 as a request to display virtual objects 7002, 7004 and 7006 in the object staging interface 5004, such as for Figure 7U In response to detecting a request to display the plurality of virtual objects in a user interface that does not include the field of view of the one or more cameras, the device displays the user interface that does not include the field of view of the one or more cameras, including displaying the plurality of virtual objects according to a first set of display characteristics. Figure 7U to Figure 7X As described, virtual objects 7002, 7004, and 7006 are displayed in object staging interface 5004 with the same positions, orientations, and order that virtual objects 7002, 7004, and 7006 had in object staging interface 5004 before a change in display properties occurred in response to input provided when augmented reality user interface 5022 was displayed. For example, Figure 7X The positions, orientations, sizes and order of virtual objects 7002, 7004 and 7006 in Fig. 7A The positions, orientations, sizes, and order of virtual objects 7002, 7004, and 7006 in the display are the same. Displaying the virtual objects according to the first set of display features when displaying a user interface that does not include the field of view of the one or more cameras reduces the number of inputs required to perform operations (e.g., by displaying all virtual objects in easily accessible locations so that the user can view and manipulate the virtual objects without having to reposition the virtual objects). Reducing the number of inputs required to perform operations increases the efficiency with which the user can switch between different displays, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0356] In some embodiments, when the plurality of virtual objects are displayed in a user interface (e.g., object staging user interface 5004) that does not include the field of view of the one or more cameras, the device detects at (1118) a request to redisplay the plurality of virtual objects with at least a portion of the field of view of the one or more cameras (e.g., in augmented reality user interface 5022). For example, display properties of virtual objects 7002, 7004, and 7006 that have been changed in response to input provided when displaying augmented reality user interface 5022 (e.g., for FIG. 7H to FIG. 7R ), and displaying virtual objects 7002, 7004, and 7006 in the object staging interface 5004 (as for Figure 7X After that, an input at a location corresponding to the “AR” region of the toggle control 5008 is provided as a request to redisplay the virtual objects 7002, 7004, and 7006 in the augmented reality user interface 5022 (such as for Figure 7X In response to detecting a request to redisplay the plurality of virtual objects together with at least a portion of the field of view of the one or more cameras, the device redisplays the plurality of virtual objects together with at least a portion of the field of view of the one or more cameras, wherein display properties of the corresponding virtual objects are changed based on the first manipulation input. Figure 7X to Figure 7Y As described, virtual objects 7002, 7004, and 7006 are redisplayed in augmented reality user interface 5022. In FIG. 7Y, in response to the input provided (eg, for FIG. 7H to FIG. 7R The virtual objects 7002, 7004, and 7006 are displayed with changes in position, orientation, size, and order (described in the previous section). Redisplaying the corresponding virtual object with certain display attributes based on the first manipulation input reduces the number of inputs required (e.g., by reapplying the display attributes to the corresponding virtual object without requiring the user to provide input a second time to manipulate the corresponding virtual object). Reducing the number of inputs required to perform an operation increases the efficiency with which the user can interact with the virtual objects, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0357] In some embodiments, at (1120), the first manipulation input is a rotation input and (in a user interface that does not include a field of view of the one or more cameras) changing the display properties of the plurality of virtual objects includes rotating the plurality of virtual objects around a central region (e.g., a center of gravity) of a boundary corresponding to (e.g., surrounding) the plurality of virtual objects. FIG. 7B to FIG. 7CAs described, virtual objects 7002, 7004, and 7006 rotate around a center line 7014 as a center area of ​​a boundary 7012. In some embodiments, the boundary corresponding to the multiple virtual objects is an ellipse or a box surrounding the multiple virtual objects. In some embodiments, at least one of the multiple virtual objects is associated with a first action (e.g., an animation sequence that displays at least a portion of a virtual object in an animated manner), the first action causes at least a portion of the first virtual object to extend beyond a first boundary corresponding to a stationary state of the virtual object (e.g., the virtual object increases in size during the animation), and the boundary corresponding to the multiple virtual objects covers a second boundary corresponding to the extended range of the at least one virtual object. For example, as described for FIG. 7AC, the first boundary 7096 corresponds to the stationary state of the virtual object 7090 and the second boundary 7108 corresponds to the extended range of the virtual object 7090. Rotating the multiple virtual objects around the center area of ​​the boundary corresponding to the multiple virtual objects reduces the number of inputs required to perform an operation (e.g., by rotating the virtual objects collectively in response to a single input (rather than requiring several separate inputs)). Reducing the number of inputs required to perform operations increases the efficiency with which a user can perform those operations, thereby enhancing the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0358] In some embodiments, before displaying the plurality of virtual objects, the device displays a selection user interface at (1122) (e.g., a photo user interface displaying a plurality of selectable images, a file user interface displaying a plurality of selectable files, or a website including a plurality of selectable objects). While displaying the selection user interface, the device detects selection of the plurality of virtual objects. For example, Figure 7AD As described in FIG. 7AH , when displaying the object management user interface 7099, the selection of virtual objects 7110, 7118, and 7122 is detected. When the plurality of virtual objects are selected, the device receives a request to display the plurality of virtual objects. For example, the request to display the plurality of virtual objects includes an input at a location corresponding to the sharing control 7130 (e.g., for Figure 7AH ), and input at a location corresponding to a control 6024 for displaying one or more selected objects in an object three-dimensional viewing mode (such as for Figure 7AI In response to receiving a request to display the plurality of virtual objects, the device displays the plurality of virtual objects in a user interface that does not include the field of view of the one or more cameras. Figures 7AH to 7AI The input described above displays virtual objects 7110, 7118, and 7122 in the object staging view 5004, such as for Figure 7AJ Described. In some embodiments, a selection user interface enables selection of multiple virtual objects using a single selection input (e.g., the selection user interface includes a link for accessing a predefined group of virtual objects). In some embodiments, the anchor plane corresponding to the corresponding plane is an attribute in a set of attributes of the corresponding virtual object in the multiple virtual objects, and the anchor plane is specified according to the properties of the physical object that the corresponding virtual object should represent. In some embodiments, the virtual object is placed at a predefined orientation and / or position relative to multiple planes detected in the field of view of the one or more cameras (e.g., multiple corresponding sides of the virtual object are associated with corresponding planes detected in the field of view of the camera). In some embodiments, if a predefined orientation and / or position of the virtual object is defined relative to a horizontal bottom plane of the virtual object, the bottom plane of the virtual object is displayed on the floor plane detected in the field of view of the camera (e.g., the horizontal bottom plane of the virtual object is parallel to the floor plane, and the distance between it and the floor plane is zero). In some embodiments, if a vertical rear plane relative to the virtual object is defined as a predefined orientation and / or position of the virtual object, the rear surface of the virtual object is placed against a wall plane detected in the field of view of one or more cameras (e.g., the vertical rear plane of the virtual object is parallel to the wall plane, and the distance between it and the wall plane is zero). In some embodiments, the virtual object is placed at a fixed distance from the corresponding plane or at an angle other than zero angle or right angle relative to the corresponding plane. Enabling the selection of multiple virtual objects from a selection user interface reduces the number of inputs required to select multiple objects (e.g., by allowing the user to select multiple objects from a single user interface without requiring the user to select objects from multiple user interfaces separately). Reducing the number of inputs required to perform an operation enhances the operability of the device, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0359] In some embodiments, a first virtual object among the multiple virtual objects has a first defined plane type (e.g., the first virtual object is configured to be displayed at a fixed position relative to a first type of plane such as a vertical (e.g., wall) plane), and a second virtual object among the multiple virtual objects has a second defined plane type (e.g., the second virtual object is configured to be displayed at a fixed position relative to a second type of plane such as a horizontal (e.g., floor, ceiling, or table surface) plane). For example, the defined plane type of virtual object 7118 is a vertical plane, and the defined plane type of virtual object 7110 is a horizontal plane. In response to receiving a request to display the multiple virtual objects at (1124): displaying the first virtual object among the multiple virtual objects at a fixed position relative to a first plane corresponding to the first defined plane type (e.g., displaying the first virtual object at a fixed position relative to the vertical plane), and displaying the second virtual object among the multiple virtual objects at a fixed position relative to a second plane corresponding to a second defined plane type different from the first defined plane type (e.g., displaying the second virtual object at a fixed position relative to the horizontal plane). For example, as for Figure 7AK As described, virtual object 7118 is displayed at a fixed position relative to vertical plane 7132 and virtual object 7110 is displayed at a fixed position relative to horizontal plane 5046. Displaying the first virtual object at a fixed position relative to the first plane and displaying the second virtual object a...

Claims

1. A method for displaying a virtual object, comprising: At an electronic device including a display generating component, one or more input devices, and one or more cameras: Displaying, by the display generating component, a plurality of virtual objects including a first virtual object and a second virtual object; When the plurality of virtual objects are displayed, detecting a first manipulation input directed to the first virtual object among the plurality of virtual objects; as well as In response to detecting the first manipulation input directed to the first virtual object among the plurality of virtual objects: Based on determining that the plurality of virtual objects are displayed in a user interface area that includes at least a portion of a field of view of the one or more cameras, changing a display attribute of the first virtual object based on the first manipulation input without changing a display attribute of the second virtual object based on the first manipulation input; as well as Based on determining that the plurality of virtual objects are displayed in an area of ​​the user interface that does not include the field of view of the one or more cameras: changing the display attribute of the first virtual object based on the first manipulation input; as well as The display attribute of the second virtual object is changed based on the first manipulation input.

2. The method according to claim 1, wherein: When displaying the plurality of virtual objects in the user interface area that does not include the field of view of the one or more cameras, the first virtual object and the second virtual object of the plurality of virtual objects are each displayed according to a first set of display characteristics.

3. The method according to claim 2, wherein: The first set of display features includes highly ranked features; and Displaying the plurality of virtual objects according to the height sorting feature includes sorting the first virtual object and the second virtual object in the plurality of virtual objects according to their respective heights.

4. The method according to any one of claims 2 to 3, wherein: The first set of display features includes spacing features; and Displaying the plurality of virtual objects according to the spacing feature includes positioning the first virtual object and the second virtual object of the plurality of virtual objects at a uniform distance between respective boundaries of the first virtual object and the second virtual object.

5. The method according to any one of claims 2 to 3, comprising: while causing the plurality of virtual objects to be displayed in the user interface area that includes at least a portion of the field of view of the one or more cameras, after changing the display attribute of the first virtual object based on the first manipulation input, detecting a request to display the plurality of virtual objects in the user interface area that does not include the field of view of the one or more cameras; as well as In response to detecting the request to display the multiple virtual objects in the user interface area that does not include the field of view of the one or more cameras, displaying the user interface area that does not include the field of view of the one or more cameras includes displaying the first virtual object and the second virtual object according to the first set of display characteristics.

6. The method according to claim 5, comprising: After changing the display property of the first virtual object and changing the display property of the second virtual object based on the first manipulation input, while displaying the plurality of virtual objects in the user interface area that does not include the field of view of the one or more cameras, detecting a request to cause the plurality of virtual objects to be redisplayed in the user interface area that includes at least a portion of the field of view of the one or more cameras; as well as In response to detecting the request to redisplay the multiple virtual objects in the user interface area that includes at least a portion of the field of view of the one or more cameras, the multiple virtual objects are redisplayed in the user interface area that includes at least a portion of the field of view of the one or more cameras, including redisplaying the first virtual object and the second virtual object according to the display properties of the first virtual object and the display properties of the second virtual object as changed based on the first manipulation input.

7. The method according to any one of claims 1 to 3, wherein: The first manipulation input is a rotation input.

8. The method according to any one of claims 1 to 3, comprising: Before displaying the plurality of virtual objects, displaying a selection user interface; while displaying the selection user interface, detecting a selection of the plurality of virtual objects including the first virtual object and the second virtual object; When the plurality of virtual objects are selected, receiving a request to display the plurality of virtual objects; as well as In response to receiving the request to display the plurality of virtual objects, the plurality of virtual objects are displayed in an area of ​​the user interface that does not include the field of view of the one or more cameras.

9. The method according to claim 8, wherein: The first virtual object of the plurality of virtual objects has a first defined plane type; the second virtual object of the plurality of virtual objects having a second defined plane type different from the first defined plane type; and In response to receiving the request to display the plurality of virtual objects, when displaying the first virtual object and the second virtual object in the user interface area that includes the field of view of the one or more cameras: displaying the first virtual object of the plurality of virtual objects at a fixed position relative to a first plane corresponding to the first defined plane type; and The second virtual object of the plurality of virtual objects is displayed at a fixed position relative to a second plane corresponding to the second defined plane type different from the first defined plane type.

10. The method according to claim 9, comprising: detecting a rotation input while displaying the first virtual object and the second virtual object in the user interface area that includes the field of view of the one or more cameras, displaying the first virtual object at the fixed position relative to the first plane, and displaying the second virtual object at the fixed position relative to the second plane; as well as In response to detecting the rotation input: Rotate the first virtual object around a first axis according to determining that the rotation input points to the first virtual object; as well as Based on determining that the rotation input is directed to the second virtual object, the second virtual object is rotated around a second axis different from the first axis.

11. A computer system comprising: Display generated components; one or more input devices; one or more cameras; one or more processors; and A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: Displaying, by the display generating component, a plurality of virtual objects including a first virtual object and a second virtual object; When the plurality of virtual objects are displayed, detecting a first manipulation input directed to the first virtual object among the plurality of virtual objects; as well as In response to detecting the first manipulation input directed to the first virtual object among the plurality of virtual objects: Based on determining that the plurality of virtual objects are displayed in a user interface area that includes at least a portion of a field of view of the one or more cameras, changing a display attribute of the first virtual object based on the first manipulation input without changing a display attribute of the second virtual object based on the first manipulation input; as well as Based on determining that the plurality of virtual objects are displayed in an area of ​​the user interface that does not include the field of view of the one or more cameras: changing the display attribute of the first virtual object based on the first manipulation input; as well as The display attribute of the second virtual object is changed based on the first manipulation input.

12. The computer system of claim 11, wherein the one or more programs include instructions for executing the method of any one of claims 2 to 10.

13. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system having a display generation component, one or more input devices, and one or more cameras, cause the computer system to: Displaying, by the display generating component, a plurality of virtual objects including a first virtual object and a second virtual object; When the plurality of virtual objects are displayed, detecting a first manipulation input directed to the first virtual object among the plurality of virtual objects; as well as In response to detecting the first manipulation input directed to the first virtual object among the plurality of virtual objects: Based on determining that the plurality of virtual objects are displayed in a user interface area that includes at least a portion of a field of view of the one or more cameras, changing a display attribute of the first virtual object based on the first manipulation input without changing a display attribute of the second virtual object based on the first manipulation input; as well as Based on determining that the plurality of virtual objects are displayed in an area of ​​the user interface that does not include the field of view of the one or more cameras: changing the display attribute of the first virtual object based on the first manipulation input; as well as The display attribute of the second virtual object is changed based on the first manipulation input.

14. The computer-readable storage medium of claim 13, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to perform the method of any one of claims 2 to 10.