Devices, methods, and graphical user interfaces for depth-based annotation

Abstract

The present disclosure provides devices, methods, and graphical user interfaces for depth-based annotation. The computer system displays a virtual object overlaid on a view of the physical environment as a virtual effect. The computer system displays a respective animated movement of the virtual object on a view of the physical environment, wherein the respective animated movement is constrained according to a direction of the simulated gravity associated with the view of the physical environment. If the current position of the virtual object corresponds to different surfaces at different heights detected in the view of the physical environment during the respective animated movement of the virtual object, the computer constrains the respective animated movement of the virtual object according to the different surfaces detected in the view of the physical environment.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of September 20, 2019, the invention name of which is “Device, method and graphical user interface for depth-based annotation” and the application number is: 201980056942.2. Technical Field

[0002]

[0013] Generally, electronic devices are directed to displaying an image of a physical environment, including but not limited to displaying annotations at spatial locations in the image corresponding to spatial locations in the physical environment captured in the image. Background Art

[0003] In recent years, the development of computer systems for augmented media has increased significantly. Examples of augmented media include augmented reality environments that include at least some virtual elements that replace or enhance the physical world and augmented storage media that include at least some virtual elements that replace or enhance storage media such as images and video content. Input devices such as touch-sensitive surfaces for computer systems and other electronic computing devices are used to augment media. Example touch-sensitive surfaces include trackpads, touch-sensitive remote controls, and touch screen displays. Such surfaces are used to manipulate user interfaces on displays and objects therein. Exemplary user interface objects include digital images, videos, text, icons, and control elements (such as buttons) and other graphics.

[0004] However, methods and interfaces for augmented media are cumbersome, inefficient, and limited. For example, when the current camera view of the user's device does not correspond to a portion of the physical environment, augmentation such as user input annotations with a fixed spatial position relative to that portion of the physical environment may be difficult for the user to locate. Search augmentation places a significant cognitive burden on the user and impairs the experience of using augmented media. In addition, providing augmented input for storage media is time consuming when augmented input must be provided separately for each portion of the storage media (e.g., previously captured video). In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the invention

[0005] Therefore, there is a need for a computer system with an improved method and interface for enhancing media data. Such methods and interfaces optionally supplement or replace conventional methods for enhancing media data. Such methods and interfaces reduce the amount, degree, and / or nature of input from a user and produce a more efficient human-computer interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0006] The disclosed computer system reduces or eliminates the above-mentioned defects and other problems associated with the interface for augmenting media data with virtual objects and / or annotation input. In some embodiments, the computer system includes a desktop computer. In some embodiments, the computer system is portable (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the computer system includes a personal electronic device (e.g., a wearable electronic device, such as a watch). In some embodiments, the computer system has a touchpad (and / or communicates with the touchpad). In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display") (and / or communicates with a touch-sensitive display). In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI in part by stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include playing games, image editing, drawing, presentation, word processing, spreadsheet making, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note taking and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0007] According to some embodiments, a method is performed at a computer system having a display generation component, one or more input devices, and one or more cameras. The method includes displaying a first user interface area via the display generation component, the first user interface area including a representation of a field of view of one or more cameras, the representation being updated as the field of view of the one or more cameras changes over time. The method also includes: while displaying the first user interface area including the representation of the field of view of the one or more cameras, receiving via the one or more input devices a first request to add an annotation to the displayed representation of the field of view of the one or more cameras. The method also includes: in response to the first request to add the annotation to the displayed representation of the field of view of the one or more cameras: replacing the display of the representation of the field of view of the one or more cameras with a still image of the field of view of the one or more cameras in the first user interface area, the still image being captured at a time corresponding to receiving the first request to add the annotation. The method also includes: while displaying the still image in the first user interface area, receiving via the one or more input devices a first annotation on a first portion of the still image, wherein the first portion of the still image corresponds to a first portion of a physical environment captured in the still image. The method also includes: while displaying a first annotation on a first portion of the still image in the first user interface area, receiving, via the one or more input devices, a first request to redisplay a representation of the field of view of the one or more cameras in the first user interface area. The method also includes: in response to receiving the request to redisplay the representation of the field of view of the one or more cameras in the first user interface area: replacing display of the still image in the first user interface area with the representation of the field of view of the one or more cameras. The method also includes: based on determining that the first portion of the physical environment captured in the still image is currently outside the field of view of the one or more cameras, simultaneously displaying an indication of the current spatial relationship of the one or more cameras relative to the first portion of the physical environment captured in the still image and the representation of the field of view of the one or more cameras; and based on determining that the first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras, forgoing displaying the indication.

[0008] According to some embodiments, a method is performed on a computer system having a display generation component and one or more input devices. The method includes displaying a user interface including a video playback area via the display generation component. The method also includes: while displaying the playback of the first portion of the video in the video playback area, receiving a request to add an annotation to the video playback via the one or more input devices. The method also includes: in response to receiving the request to add the annotation: pausing the playback of the video at a first position in the video; and displaying a still image corresponding to the first pause position of the video. The method also includes: while displaying the still image, receiving an annotation on a first portion of the physical environment captured in the still image via the one or more input devices. The method also includes: after receiving the annotation, displaying a second portion of the video corresponding to a second position in the video in the video playback area, the second position being different from the first position in the video, wherein the first portion of the physical environment is captured in the second portion of the video, and the annotation is displayed in the second portion of the video.

[0009] According to some embodiments, a method is performed on a computer system having a display generation component and one or more input devices. The method also includes: displaying, via the display generation component, a first previously captured media object including one or more first images, wherein the first previously captured media object and first depth data corresponding to a first physical environment captured in each of the one or more first images are recorded and stored together. The method also includes: while displaying the first previously captured media object, receiving, via the one or more input devices, a first user request to add a first virtual object to the first previously captured media object. The method also includes: in response to the first user request to add the first virtual object to the first previously captured media object, displaying the first virtual object above at least a portion of the corresponding image in the first previously captured media object, wherein the first virtual object is displayed as having at least a first position or orientation, the first position or orientation being determined using the first depth data corresponding to the corresponding image in the first previously captured media object.

[0010] According to some embodiments, a method is performed at a computer system having a display generation component, a first set of one or more input devices, and a first set of one or more cameras. The method includes: sending a request to a remote device to initiate a shared annotation session with a second device, the second device including a second display generation component, a second set of one or more input devices, and a second set of one or more cameras. The method also includes: in response to sending the request to initiate a shared annotation session with the second device, receiving an indication of accepting the request to initiate the shared annotation session. The method also includes: in response to receiving the indication of accepting the request to initiate the shared annotation session, displaying a first prompt to move the first device toward the second device via the first display generation component. The method also includes: after displaying the first prompt, based on determining that the connection criteria of the first device and the second device are met, displaying a representation of the field of view of the first set of cameras in the shared annotation session with the second device, wherein: the connection criteria require that at least a portion of the field of view of the first device and a portion of the field of view of the second device correspond to the same portion of the physical environment surrounding the first device and the second device. The method also includes displaying: during a shared annotation session, one or more first virtual annotations corresponding to annotation input directed to a corresponding location in the physical environment made through a first device are displayed via a first display generating component, and one or more second virtual annotations corresponding to annotation input directed to a corresponding location in the physical environment made through a second device are displayed via the first display generating component, provided that the corresponding location is included in the field of view of the first group of cameras.

[0011] According to some embodiments, the electronic device includes a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting contact intensity with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, optionally one or more posture sensors for detecting posture changes, one or more processors, and a memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, a computer-readable storage medium has instructions stored therein, which, when executed by an electronic device having a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting contact intensity with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors, causes the device to perform the operations of any of the methods described herein or causes the operations of any of the methods described herein to be performed. According to some embodiments, a graphical user interface on an electronic device having a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting contact intensity with a touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors, a memory, and one or more processors for executing one or more programs stored in the memory includes one or more elements displayed in any of the methods described herein, and the one or more elements are updated in response to input, as described in any of the methods described herein. According to some embodiments, the electronic device includes: a display generation component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting contact intensity with a touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors for detecting posture changes; and a device for performing or causing the performance of operations of any of the methods described herein.According to some embodiments, an information processing device used in an electronic device having a display generating component, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors for detecting posture changes includes a device for performing the operations of any method described herein or causing the operations of any method described herein to be performed.

[0012] Thus, improved methods and interfaces for displaying virtual objects in various scenarios are provided for electronic devices having display generation components, optionally one or more input devices, optionally one or more touch-sensitive surfaces, optionally one or more cameras, optionally one or more sensors for detecting contact intensity with the touch-sensitive surface, optionally one or more audio output generators, optionally one or more device orientation sensors, optionally one or more tactile output generators, and optionally one or more posture sensors, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces can supplement or replace conventional methods for displaying virtual objects in various scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0014] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0015] Figure 1B is a block diagram illustrating example components for event handling according to some embodiments.

[0016] Figure 1C is a block diagram illustrating a tactile output module according to some embodiments.

[0017] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0018] Figure 3 is a block diagram of an example multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0019] Figure 4A An example user interface is shown for an application menu on a portable multifunction device according to some embodiments.

[0020] Figure 4B An example user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0021] FIG. 4C to FIG. 4E Examples of dynamic intensity thresholds are shown according to some embodiments.

[0022] 5A to 5AF An exemplary user interface for relocalizing annotations according to some embodiments is shown.

[0023] FIG. 6A to FIG. 6N An exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause location of a video is shown according to some embodiments.

[0024] 7A to 7BF An example user interface for adding a virtual object to a previously captured media object is shown according to some embodiments.

[0025] FIG. 8A to FIG. 8W An exemplary user interface is shown for illustrating an exemplary user interface for initiating a shared annotation session according to some embodiments.

[0026] 9A to 9F is a flowchart of a process for relocalizing annotations according to some embodiments.

[0027] FIG. 10A to FIG. 10B is a flowchart of a process for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause location of a video according to some embodiments.

[0028] FIG. 11A to FIG. 11F is a flowchart of a process for adding a virtual object to a previously captured media object according to some embodiments.

[0029] FIG. 12A to FIG. 12D is a flowchart of a process for initiating a shared annotation session according to some embodiments. DETAILED DESCRIPTION

[0030] Conventional methods of augmenting media typically require multiple separate inputs (e.g., separate annotations of multiple frames and / or placement of augmentations relative to objects in the media) to achieve a desired result (e.g., annotating a portion of a stored video or a live video feed and / or displaying a virtual object at a location corresponding to a surface of a physical object in the stored media). Embodiments herein provide a user with an intuitive way to augment media, such as stored content, still images, and / or live video captured by one or more cameras of a device (e.g., by placing augmentations using depth data stored and / or captured with the image data and maintaining a fixed spatial relationship between the augmentations and portions of the physical environment in the camera's field of view).

[0031] The systems, methods, and GUIs described herein improve user interface interactions for augmenting media in a variety of ways. For example, they make it easier to: relocalize annotations, annotate videos, add virtual objects to previously captured media, and initiate a shared annotation session.

[0032] under, Figures 1A to 1C , Figure 2 and Figure 3 A description of an exemplary apparatus is provided. FIG. 4A to FIG. 4B , 5A to 5AF , FIG. 6A to FIG. 6N , 7A to 7BF as well as FIG. 8A to FIG. 8W Exemplary user interfaces for displaying virtual objects in various scenarios are shown. 9A to 9F A process for relocalizing annotations is shown. FIG. 10A to FIG. 10B A process for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause location of a video is shown. FIG. 11A to FIG. 11F A process for adding a virtual object to a previously captured media object is shown. FIG. 12A to FIG. 12D A process for initiating a shared annotation session is shown. 5A to 5AF , FIG. 6A to FIG. 6N , 7A to 7BF as well as FIG. 8A to FIG. 8W The user interface in 9A to 9F , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11F as well as FIG. 12A to FIG. 12D process.

[0033] Exemplary Devices

[0034] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0035] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0036] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0037] As used herein, the term "if" is optionally interpreted to mean "when" followed by "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0038] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple iPod from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices are optionally used, such as laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or trackpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or trackpads).

[0039] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0040] The device typically supports a variety of applications, such as one or more of the following applications: a note-taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital camcorder application, a web browsing application, a digital music player application, and / or a digital video player application.

[0041] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0042] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 167 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0043] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0044] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. The access of other components of the device 100 (such as the CPU 120 and the peripheral device interface 118) to the memory 102 is optionally controlled by the memory controller 122.

[0045] Peripheral interface 118 may be used to couple the device's input and output peripherals to CPU 120 and memory 102. One or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0046] In some embodiments, peripherals interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0047] RF (radio frequency) circuit 108 receives and sends RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / from electromagnetic signals and communicates with a communication network and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec chipset, a subscriber identity module (SIM) card, memory, and the like. RF circuit 108 optionally communicates with networks and other devices via wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), an intranet, and / or a wireless network (such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN)). The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data-only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11d), IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11d, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0048] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single or dual-ear headset) and input (e.g., a microphone).

[0049] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from / to other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, the one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0050] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some visual outputs or all of the visual outputs correspond to user interface objects. As used herein, the term "indicator" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0051] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In some embodiments, the point of contact between the touch-sensitive display system 112 and the user corresponds to the user's finger or stylus.

[0052] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In some embodiments, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0053] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0054] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0055] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0056] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some embodiments, an optical sensor is located on the rear portion of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user is watching other video conference participants on the touch screen, etc.).

[0057] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to a force sensor controller 159 in the I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 located on the front of the device 100.

[0058] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0059] Device 100 optionally also includes one or more tactile output generators 167 . Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. In some embodiments, the tactile output generator 167 includes one or more electroacoustic devices such as a speaker or other audio component; and / or an electromechanical device for converting energy into linear motion such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component for converting an electrical signal into a tactile output on the device). The tactile output generator 167 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front of the device 100.

[0060] Device 100 optionally also includes one or more accelerometers 168 . Figure 1A An accelerometer 168 is shown coupled to the peripheral device interface 118. Alternatively, the accelerometer 168 is optionally coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on an analysis of data received from the one or more accelerometers. The device 100 optionally includes a magnetometer (not shown) and a GPS (or GLONASS or other global navigation system) receiver (not shown) in addition to the accelerometer 168 for obtaining information about the position and orientation (e.g., portrait or landscape) of the device 100.

[0061] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in FIG. 1A and FIG. Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0062] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0063] The communication module 128 facilitates communication with other devices through one or more external ports 124, and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices, or indirectly through a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device that is compatible with some of Apple Inc. (Cupertino, California). iPod and In some embodiments, the external port is a multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in Apple Inc. (Cupertino, California). iPod and A Lightning connector that is the same as or similar to and / or compatible with the Lightning connector used in the device.

[0064] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or stylus contact) or multiple points of simultaneous contact (e.g., "multi-touch" / multi-finger contact). In some embodiments, contact / motion module 130 and display controller 156 detect contact on a touch pad.

[0065] The contact / motion module 130 optionally detects gesture input by the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0066] In some embodiments, detecting a finger tap gesture depends on detecting the length of time between a finger press event and a finger lift event, but is independent of the strength of the finger contact between the finger press event and the finger lift event. In some embodiments, a tap gesture is detected based on determining that the length of time between the finger press event and the finger lift event is less than a predetermined value (e.g., less than 0.1, 0.2, 0.3, 0.4, or 0.5 seconds), regardless of whether the strength of the finger contact during the tap reaches a given strength threshold (greater than a nominal contact detection strength threshold), such as a light press or deep press strength threshold. Therefore, a finger tap gesture can meet a specific input criterion that does not require the characteristic strength of the contact to meet a given strength threshold to meet the specific input criterion. For clarity, the finger contact in a tap gesture is generally required to meet a nominal contact detection strength threshold to detect a finger press event, and the contact will not be detected below the nominal contact detection strength threshold. A similar analysis applies to detecting a tap gesture by a stylus or other contact. In the case where the device is capable of detecting contact with a finger or stylus hovering over the touch-sensitive surface, the nominal contact detection intensity threshold optionally does not correspond to physical contact between the finger or stylus and the touch-sensitive surface.

[0067] The same concepts apply in a similar manner to other types of gestures. For example, a swipe gesture, a pinch gesture, an expand gesture, and / or a long press gesture may be optionally detected based on satisfying criteria that are independent of the strength of the contacts included in the gesture or that do not require one or more contacts performing the gesture to reach a strength threshold in order to be recognized. For example, a swipe gesture is detected based on the amount of movement of one or more contacts; a zoom gesture is detected based on the movement of two or more contacts toward each other; a zoom gesture is detected based on the movement of two or more contacts away from each other; and a long press gesture is detected based on the duration of a contact on a touch-sensitive surface having less than a threshold amount of movement. Thus, the statement that a particular gesture recognition criterion does not require the strength of the contacts to satisfy a corresponding strength threshold in order to satisfy the particular gesture recognition criterion means that the particular gesture recognition criterion can be satisfied when the contacts in the gesture do not reach the corresponding strength threshold, and can also be satisfied when one or more contacts in the gesture reach or exceed the corresponding strength threshold. In some embodiments, a tap gesture is detected based on determining that a finger down event and a finger up event are detected within a predefined time period, regardless of whether the contact is above or below a corresponding strength threshold during the predefined time period, and a swipe gesture is detected based on determining that the contact moves greater than a predefined amount, even if the contact is above the corresponding strength threshold at the end of the contact movement. Even in specific implementations where detection of gestures is affected by the strength of the contact performing the gesture (e.g., the device detects a long press more quickly when the strength of the contact is above the strength threshold, or the device delays detection of a tap input when the strength of the contact is higher), detection of these gestures does not require the contact to reach a particular strength threshold (e.g., even if the amount of time required to recognize the gesture varies), as long as the criteria for recognizing the gesture can be met without the contact reaching the particular strength threshold.

[0068] In some cases, the contact intensity threshold, duration threshold, and movement threshold are combined in various combinations to create a heuristic algorithm to distinguish between two or more different gestures for the same input element or area, so that multiple different interactions with the same input element can provide a richer set of user interactions and responses. The statement that a particular set of gesture recognition criteria does not require the intensity of one or more contacts to meet the corresponding intensity threshold in order to meet the particular gesture recognition criteria does not exclude the simultaneous evaluation of other intensity-related gesture recognition criteria to identify other gestures with criteria that are met when the gesture includes a contact with an intensity above the corresponding intensity threshold. For example, in some cases, a first gesture recognition criterion for a first gesture (which does not require the intensity of the contact to meet the corresponding intensity threshold to meet the first gesture recognition criterion) competes with a second gesture recognition criterion for a second gesture (which depends on the contact reaching the corresponding intensity threshold). In such a competition, if the second gesture recognition criterion for the second gesture is met first, the gesture is optionally not recognized as satisfying the first gesture recognition criterion for the first gesture. For example, if the contact reaches the corresponding intensity threshold before the contact moves a predefined amount of movement, a deep press gesture is detected instead of a swipe gesture. Conversely, if the contact moves the predefined amount of movement before the contact reaches the corresponding intensity threshold, a swipe gesture is detected instead of a deep press gesture. Even in such cases, the first gesture recognition criteria for the first gesture still does not require the intensity of the contact to satisfy the corresponding intensity threshold to satisfy the first gesture recognition criteria because if the contact remains below the corresponding intensity threshold until the gesture ends (e.g., a swipe gesture with an intensity of the contact that does not increase above the corresponding intensity threshold), the gesture will be recognized as a swipe gesture by the first gesture recognition criteria. Thus, a particular gesture recognition criterion that does not require the intensity of the contact to satisfy the corresponding intensity threshold to satisfy the particular gesture recognition criterion will (A) in some cases ignore the intensity of the contact relative to the intensity threshold (e.g., for a tap gesture) and / or (B) in some cases fail to satisfy the particular gesture recognition criterion (e.g., for a long press gesture) if a set of competing intensity-related gesture recognition criteria (e.g., for a deep press gesture) recognizes the input as corresponding to an intensity-related gesture before the particular gesture recognition criterion recognizes the gesture corresponding to the input, and in this sense still depends on the intensity of the contact relative to the intensity threshold (e.g., for a long press gesture competing with the deep press gesture for recognition).

[0069] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0070] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0071] Haptic feedback module 133 includes various software components for generating instructions (e.g., instructions used by haptic feedback controller 161) to generate tactile output at one or more locations on device 100 using tactile output generator 167 in response to user interaction with device 100.

[0072] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0073] The GPS module 135 determines the location of the device and provides this information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as the Weather widget, the Local Pages widget, and the Maps / Navigation widget).

[0074] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0075] Contacts module 137 (sometimes called address book or contact list);

[0076] Telephone module 138;

[0077] Video conferencing module 139;

[0078] Email client module 140;

[0079] Instant messaging (IM) module 141;

[0080] Fitness support module 142;

[0081] A camera module 143 for still images and / or video images;

[0082] Image management module 144;

[0083] Browser module 147;

[0084] Calendar module 148;

[0085] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets acquired by the user, and a user-created widget 149-6;

[0086] A widget creator module 150 for forming a user-created widget 149-6;

[0087] Search module 151;

[0088] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0089] Memo module 153;

[0090] Map module 154; and / or

[0091] Online video module 155.

[0092] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0093] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and the like.

[0094] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0095] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0096] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0097] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, transmitting the corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs or IMPS).

[0098] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and video and music player module 152, fitness support module 142 includes executable instructions for creating workouts (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0099] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0100] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0101] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0102] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0103] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0104] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0105] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0106] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0107] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notepads, to-do lists, etc. according to user instructions.

[0108] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0109] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0110] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present disclosure (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented as independent software programs, processes, or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores subsets of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0111] In some embodiments, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0112] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using a trackpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a trackpad.

[0113] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0114] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event distributor module 174. In some embodiments, the application 136-1 includes an application internal state 192, which indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0115] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0116] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 168, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0117] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0118] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0119] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0120] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0121] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0122] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0123] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0124] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0125] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring in a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit (not shown) or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in application view 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of data updater 176, object updater 177, and GUI updater 178 are included in the corresponding application view 191.

[0126] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0127] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (for example, from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0128] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and others. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0129] In some embodiments, event definition 187 includes definitions of events for corresponding user interface objects. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with the sub-event and the object that triggered the hit test.

[0130] In some embodiments, the definition of the corresponding event 187 also includes delay actions that delay the delivery of the event information until it has been determined that the sub-event sequence does or does not correspond to the event type of the event identifier.

[0131] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0132] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0133] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0134] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.

[0135] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video or music player module 152. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0136] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0137] It should be understood that the above discussion of event processing of user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or hold; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining the event to be recognized.

[0138] Figure 1C 1 is a block diagram showing a tactile output module according to some embodiments. In some embodiments, the I / O subsystem 106 (e.g., the tactile feedback controller 161 ( Figure 1A ) and / or one or more other input controllers 160 ( Figure 1A ))include Figure 1C At least some of the example components shown. In some embodiments, the peripheral device interface 118 includes Figure 1C At least some of the example components shown.

[0139] In some embodiments, the tactile output module includes a tactile feedback module 133. In some embodiments, the tactile feedback module 133 aggregates and combines tactile outputs from software applications on the electronic device for user interface feedback (e.g., feedback in response to user input corresponding to the displayed user interface and prompts and other notifications indicating the execution of operations or the occurrence of events in the user interface of the electronic device). The tactile feedback module 133 includes one or more of a waveform module 123 (for providing a waveform for generating tactile output), a mixer 125 (for mixing waveforms, such as waveforms in different channels), a compressor 127 (for reducing or compressing the dynamic range of the waveform), a low-pass filter 129 (for filtering out high-frequency signal components in the waveform), and a thermal controller 131 (for adjusting the waveform according to thermal conditions). In some embodiments, the tactile feedback module 133 is included in a tactile feedback controller 161 ( Figure 1A In some embodiments, a separate unit of the tactile feedback module 133 (or a separate specific implementation of the tactile feedback module 133) is also included in the audio controller (e.g., the audio circuit 110, Figure 1A ) and used to generate an audio signal. In some embodiments, a single tactile feedback module 133 is used to generate an audio signal and to generate a waveform for a tactile output.

[0140] In some embodiments, the tactile feedback module 133 also includes a trigger module 121 (e.g., a software application, operating system, or other software module that determines that a tactile output is to be generated and triggers a process for generating the corresponding tactile output). In some embodiments, the trigger module 121 generates a trigger signal for triggering (e.g., by the waveform module 123) the generation of a waveform. For example, the trigger module 121 generates the trigger signal based on a preset timing standard. In some embodiments, the trigger module 121 receives a trigger signal from outside the tactile feedback module 133 (e.g., in some embodiments, the tactile feedback module 133 receives the trigger signal from a hardware input processing module 146 located outside the tactile feedback module 133) and relays the trigger signal to other components within the tactile feedback module 133 (e.g., the waveform module 123) or a software application that triggers an operation (e.g., using the trigger module 121) based on the activation of a user interface element (e.g., an application icon or an enable representation within an application) or a hardware input device (e.g., a main button or an intensity-sensitive input interface, such as an intensity-sensitive touch screen). In some embodiments, the trigger module 121 also (e.g., from the tactile feedback module 133, Figure 1A and Figure 3 ) receives a tactile feedback generation instruction. In some embodiments, the trigger module 121 responds to the tactile feedback module 133 (or the trigger module 121 in the tactile feedback module 133) (e.g., from the tactile feedback module 133, Figure 1Aand Figure 3 ) receives tactile feedback instructions and generates a trigger signal.

[0141] Waveform module 123 receives a trigger signal as input (e.g., from trigger module 121) and provides a waveform for generating one or more tactile outputs (e.g., a waveform selected from a predefined set of waveforms assigned for use by waveform module 123) in response to receiving the trigger signal.

[0142] The mixer 125 receives waveforms as input (e.g., from the waveform module 123) and mixes the waveforms together. For example, when the mixer 125 receives two or more waveforms (e.g., a first waveform in a first channel and a second waveform in a second channel that at least partially overlaps the first waveform), the mixer 125 outputs a combined waveform corresponding to the sum of the two or more waveforms. In some embodiments, the mixer 125 also modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the rest of the two or more waveforms (e.g., by increasing the scale of the particular waveform and / or reducing the scale of the other waveforms in the waveforms). In some cases, the mixer 125 selects one or more waveforms to remove from the combined waveform (e.g., when waveforms from more than three sources have been requested to be output simultaneously by the tactile output generator 167, the waveform from the oldest source is discarded).

[0143] Mixer 127 receives waveforms (e.g., combined waveforms from mixer 125) as input and modifies these waveforms. In some embodiments, compressor 127 reduces these waveforms (e.g., based on tactile output generator 167 ( Figure 1A ) or 357( Figure 3 ) such that the tactile output corresponding to these waveforms is reduced. In some embodiments, compressor 127 limits the waveforms, such as by imposing a predefined maximum amplitude on the waveforms. For example, compressor 127 reduces the amplitude of the waveform portion that exceeds the predefined amplitude threshold, while maintaining the amplitude of the waveform portion that does not exceed the predefined amplitude threshold. In some embodiments, compressor 127 reduces the dynamic range of the waveform. In some embodiments, compressor 127 dynamically reduces the dynamic range of the waveform so that the combined waveform remains within the performance specifications of tactile output generator 167 (e.g., force and / or movable mass displacement limits).

[0144] Low-pass filter 129 receives a waveform (e.g., a compressed waveform from compressor 127) as input and filters (e.g., smoothes) the waveform (e.g., removes or reduces high-frequency signal components in the waveform). For example, in some cases, compressor 127 includes extraneous signals (e.g., high-frequency signal components) in the compressed waveform that interfere with tactile output generation and / or exceed the performance specifications of tactile output generator 167 when generating tactile output based on the compressed waveform. Low-pass filter 129 reduces or removes such extraneous signals in the waveform.

[0145] Thermal controller 131 receives a waveform (e.g., a filtered waveform from low pass filter 129) as input and adjusts the waveform based on a thermal condition of device 100 (e.g., based on an internal temperature detected within device 100, such as the temperature of haptic feedback controller 161, and / or an external temperature detected by device 100). For example, in some cases, the output of haptic feedback controller 161 varies based on temperature (e.g., in response to receiving the same waveform, haptic feedback controller 161 generates a first tactile output when haptic feedback controller 161 is at a first temperature and generates a second tactile output when haptic feedback controller 161 is at a second temperature different from the first temperature). For example, the magnitude (or amplitude) of the tactile output may vary based on temperature. To reduce the effects of temperature changes, the waveform is modified (e.g., the amplitude of the waveform is increased or decreased based on temperature).

[0146] In some embodiments, the tactile feedback module 133 (e.g., trigger module 121) is coupled to the hardware input processing module 146. In some embodiments, Figure 1A The one or more other input controllers 160 in the embodiment include a hardware input processing module 146. In some embodiments, the hardware input processing module 146 receives input from a hardware input device 145 (e.g. Figure 1A In some embodiments, the hardware input device 145 is any input device described herein, such as a touch-sensitive display system 112 ( Figure 1A )、Keyboard / Mouse 350( Figure 3 )、Touchpad 355( Figure 3 ), one of the other input or control devices 116 ( Figure 1A ) or intensity-sensitive primary buttons. In some embodiments, the hardware input device 145 is composed of intensity-sensitive primary buttons rather than touch-sensitive display system 112 ( Figure 1A )、Keyboard / Mouse 350( Figure 3 ) or touchpad 355( Figure 3). In some embodiments, in response to input from a hardware input device 145 (e.g., an intensity-sensitive home button or a touch screen), the hardware input processing module 146 provides one or more trigger signals to the tactile feedback module 133 to indicate that a user input that meets predefined input criteria has been detected, such as an input corresponding to a home button "click" (e.g., a "press click" or a "release click"). In some embodiments, the tactile feedback module 133 provides a waveform corresponding to a home button "click" in response to an input corresponding to a home button "click", thereby simulating the tactile feedback of pressing a physical home button.

[0147] In some embodiments, the tactile output module includes a tactile feedback controller 161 (eg, Figure 1A 145 ) which controls the generation of tactile output. In some embodiments, the tactile feedback controller 161 is coupled to a plurality of tactile output generators, and selects one or more tactile output generators from the plurality of tactile output generators and sends a waveform to the selected one or more tactile output generators for generating a tactile output. In some embodiments, the tactile feedback controller 161 coordinates a tactile output request corresponding to activation of the hardware input device 145 and a tactile output request corresponding to a software event (e.g., a tactile output request from the tactile feedback module 133), and modifies one or more of the two or more waveforms to emphasize a particular waveform relative to the remaining waveforms of the two or more waveforms (e.g., by increasing the scale of a particular waveform and / or reducing the scale of the remaining waveforms of the waveforms to prioritize the tactile output corresponding to the activation of the hardware input device 145 over the tactile output corresponding to the software event).

[0148] In some embodiments, such as Figure 1C As shown, the output of the tactile feedback controller 161 is coupled to the audio circuit of the device 100 (eg, the audio circuit 110, Figure 1A ), and provides the audio signal to the audio circuit of device 100. In some embodiments, tactile feedback controller 161 provides both a waveform for generating a tactile output and an audio signal for providing an audio output together with generating the tactile output. In some embodiments, tactile feedback controller 161 modifies the audio signal and / or the waveform (for generating the tactile output) so that the audio output and the tactile output are synchronized (e.g., by delaying the audio signal and / or the waveform). In some embodiments, tactile feedback controller 161 includes a digital-to-analog converter for converting a digital waveform into an analog signal, which is received by amplifier 163 and / or tactile output generator 167.

[0149] In some embodiments, the tactile output module includes an amplifier 163. In some embodiments, the amplifier 163 receives a waveform (e.g., from the tactile feedback controller 161), amplifies the waveform, and then sends the amplified waveform to the tactile output generator 167 (e.g., the tactile output generator 167 ( Figure 1A ) or 357( Figure 3 )). For example, amplifier 163 amplifies the received waveform to a signal level that meets the physical specifications of tactile output generator 167 (e.g., amplifies to the voltage and / or current required by tactile output generator 167 to generate a tactile output so that the signal sent to tactile output generator 167 generates a tactile output corresponding to the waveform received from tactile feedback controller 161) and sends the amplified waveform to tactile output generator 167. In response, tactile output generator 167 generates a tactile output (e.g., by shifting the movable mass back and forth in one or more dimensions relative to the neutral position of the movable mass).

[0150] In some embodiments, the tactile output module includes a sensor 169 coupled to the tactile output generator 167. The sensor 169 detects the state or state change (e.g., mechanical position, physical displacement, and / or movement) of the tactile output generator 167 or one or more components of the tactile output generator 167 (e.g., one or more moving components, such as a membrane, for generating a tactile output). In some embodiments, the sensor 169 is a magnetic field sensor (e.g., a Hall effect sensor) or other displacement and / or motion sensor. In some embodiments, the sensor 169 provides information (e.g., the position, displacement, and / or movement of one or more components in the tactile output generator 167) to the tactile feedback controller 161, and based on the information about the state of the tactile output generator 167 provided by the sensor 169, the tactile feedback controller 161 adjusts the waveform output from the tactile feedback controller 161 (e.g., the waveform sent to the tactile output generator 167, optionally via the amplifier 163).

[0151] Figure 2 A device having a touch screen (e.g., Figure 1AThe portable multifunction device 100 of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In these embodiments and in other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, when the user interrupts contact with the one or more graphics, selection of the one or more graphics will occur. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0152] The device 100 optionally also includes one or more physical buttons, such as a "home button" or menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0153] In some embodiments, the device 100 includes a touch screen display, a menu button 204 (sometimes referred to as a home button 204), a push button 206 for powering the device on / off and for locking the device, a volume adjustment button 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button pressed for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the strength of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 167 for generating tactile output for a user of the device 100.

[0154] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 167 described above), sensors 359 (e.g., optical sensors, acceleration sensors, proximity sensors, touch sensors, and / or sensors similar to those described above) Figure 1A The memory 370 may include one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0155] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to the instruction set for performing the above-mentioned functions. The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subset of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0156] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0157] Figure 4A An example user interface 400 of an application menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0158] One or more signal strength indicators of one or more wireless communications, such as cellular signals and Wi-Fi signals;

[0159] ·time;

[0160] Bluetooth indicator;

[0161] Battery status indicator;

[0162] A tray 408 with icons for commonly used applications, such as:

[0163] o An icon 416 of the phone module 138 labeled "Phone," the icon 416 optionally including an indicator 414 of the number of missed calls or voice messages;

[0164] o an icon 418 of the email client module 140 labeled “Mail”, the icon 418 optionally including an indicator 410 of the number of unread emails;

[0165] o An icon 420 labeled "Browser" of the browser module 147; and

[0166] o An icon 422 labeled “Music” of the video and music player module 152;

[0167] as well as

[0168] Icons for other apps, such as:

[0169] o Icon 424 labeled "Message" of IM module 141;

[0170] o An icon 426 labeled “Calendar” of the calendar module 148;

[0171] o Icon 428 labeled “Photos” of the image management module 144;

[0172] o An icon 430 labeled “Camera” of the camera module 143;

[0173] ○ Icon 432 labeled “Online Video” of the online video module 155;

[0174] ○ Icon 434 labeled “Stock Market” of the Stock Market Desktop Applet 149-2;

[0175] o An icon 436 labeled “Map” of the map module 154;

[0176] ○ Icon 438 labeled “Weather” of the Weather widget 149-1;

[0177] ○ Icon 440 labeled as “Clock” of the alarm clock desktop applet 149-4;

[0178] o An icon 442 labeled “Fitness Support” of the fitness support module 142;

[0179] o An icon 444 labeled "Notepad" of the Notepad module 153; and

[0180] ○ An icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136 .

[0181] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0182] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touchpad 355) Figure 3 Although many of the examples that follow will be given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects contact with touch-sensitive surface 451 at a location corresponding to a corresponding location on the display (e.g., Figure 4B 460 and 462) (e.g., in Figure 4B 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B 450 in ) is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0183] In addition, although the following examples are given primarily with reference to finger inputs (e.g., finger contacts, single-finger tap gestures, finger swipe gestures, etc.), it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0184] As used herein, the term "focus selector" refers to an input element used to indicate the current portion of a user interface with which a user is interacting. In some implementations that include a cursor or other position marker, the cursor acts as a "focus selector" such that when the cursor is over a particular user interface element (e.g., a button, window, slider, or other user interface element), a focus selector is displayed on a touch-sensitive surface (e.g., Figure 3 Touchpad 355 or Figure 4B In the event that an input (e.g., a press input) is detected on the touch-sensitive surface 451 in the display, the particular user interface element is adjusted according to the detected input. In the case that the touch-sensitive surface 451 in the display includes a touch-screen display (e.g., Figure 1AA touch-sensitive display system 112 or Figure 4A In some implementations of a touch screen in a touch screen, a contact detected on the touch screen acts as a "focus selector" such that when an input (e.g., a press input by a contact) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touch screen display, the particular user interface element is adjusted according to the detected input. In some implementations, the focus moves from one area of ​​the user interface to another area of ​​the user interface without corresponding movement of a cursor or movement of a contact on the touch screen display (e.g., by using a tab key or arrow keys to move the focus from one button to another); in these implementations, the focus selector moves according to the movement of the focus between different areas of the user interface. Regardless of the specific form taken by the focus selector, the focus selector is typically a user interface element (or contact on the touch screen display) that is controlled by the user to convey the user's desired interaction with the user interface (e.g., by indicating to the device the element of the user interface with which the user desires to interact). For example, when a press input is detected on a touch-sensitive surface (e.g., a trackpad or touch screen), the position of a focus selector (e.g., a cursor, contact, or selection box) over a corresponding button will indicate that the user desires to activate the corresponding button (rather than other user interface elements shown on the device display).

[0185] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). The intensity of the contact is used as an attribute of the user input, thereby allowing the user to access additional device functions that the user cannot easily access on a smaller device with limited real estate for displaying indications (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical control / mechanical control such as a knob or button).

[0186] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of the device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, the user of the device is provided with a software setting for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0187] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact begins to move, before contact ends, before or after contact is detected to increase in intensity, and / or before or after contact is detected to decrease in intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, the value generated by low-pass filtering the contact intensity over a predefined time period or starting from a predefined time, etc. In some embodiments, the duration of the contact is used in determining the feature strength (e.g., when the feature strength is the average of the strength of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this embodiment, a contact whose feature strength does not exceed the first strength threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second strength threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more strength thresholds is used to determine whether to perform one or more operations (e.g., whether to execute the corresponding option or give up performing the corresponding operation), rather than to determine whether to perform the first operation or the second operation.

[0188] In some embodiments, a portion of a gesture is identified for determining a characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position and reaches an end position (e.g., a drag gesture) where the strength of the contact increases. In this example, the characteristic strength of the contact at the end position may be based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm may be applied to the strength of the swipe gesture before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the strength of the swipe contact to achieve the purpose of determining the characteristic strength.

[0189] The user interface diagrams described herein optionally include various intensity diagrams that illustrate the intensity of contacts on the touch-sensitive surface relative to one or more intensity thresholds (e.g., contact detection intensity threshold IT0, light press intensity threshold IT1, and light touch intensity threshold IT2). L , Deep compression intensity threshold IT D (For example, at least initially above IT L ) and / or one or more other intensity thresholds (e.g., L Low intensity threshold IT H )). This intensity graph is not typically part of the displayed user interface, but is provided to help explain the graph. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold IT0, contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector according to the movement of the contact on the touch-sensitive surface without performing operations associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0190] In some embodiments, the response of the device to the input detected by the device depends on a criterion based on the contact strength during the input. For example, for some "light press" inputs, the strength of the contact exceeding the first strength threshold during the input triggers a first response. In some embodiments, the response of the device to the input detected by the device depends on a criterion including both the contact strength during the input and a time-based criterion. For example, for some "deep press" inputs, as long as a delay time passes between satisfying the first strength threshold and satisfying the second strength threshold, the strength of the contact exceeding the second strength threshold greater than the first strength threshold of the light press during the input triggers a second response. The duration of the delay time is typically less than 200ms (milliseconds) (e.g., 40ms, 100ms, or 120ms, depending on the magnitude of the second strength threshold, where the delay time increases as the second strength threshold increases). The delay time helps avoid accidentally identifying a deep press input. As another example, for some "deep press" inputs, a period of reduced sensitivity will occur after reaching the first strength threshold. During this period of reduced sensitivity, the second strength threshold increases. This temporary increase in the second intensity threshold also helps avoid accidental deep press inputs.For other deep press inputs, the response to detecting the deep press input does not depend on time-based criteria.

[0191] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Example factors are described in U.S. patent applications 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0192] For example, Figure 4C A dynamic intensity threshold 480 is shown that changes over time based in part on the intensity of touch input 476 over time. Dynamic intensity threshold 480 is the sum of two components: a first component 474 that decays over time after a predefined delay time p1 from when touch input 476 is initially detected, and a second component 478 that tracks the intensity of touch input 476 over time. The initial high intensity threshold of first component 474 reduces accidental triggering of a “deep press” response while still allowing an immediate “deep press” response if touch input 476 provides sufficient intensity. Second component 478 reduces unintentional triggering of a “deep press” response through gradual intensity fluctuations of the touch input. In some embodiments, when touch input 476 satisfies dynamic intensity threshold 480 (e.g., at Figure 4C ), triggering a "deep press" response.

[0193] Figure 4D Another dynamic intensity threshold 486 (eg, intensity threshold I D ). Figure 4D Two other intensity thresholds are also shown: a first intensity threshold I H and the second intensity threshold I L .exist Figure 4D Although the touch input 484 satisfies the first intensity threshold I before time p2, H and the second intensity threshold I L , but no response is provided until a delay time p2 has elapsed at time 482. Similarly, Figure 4D , the dynamic intensity threshold 486 decays over time, wherein the decay is from time 482 (triggering the second intensity threshold I L This type of dynamic intensity threshold reduction begins at time 488 after a predefined delay time p1 has elapsed when the trigger is associated with a lower threshold intensity (such as the first intensity threshold I H Or the second intensity threshold I L ) is triggered after or simultaneously with the response associated with the dynamic intensity threshold I D The associated response.

[0194] Figure 4E Another dynamic intensity threshold 492 (eg, intensity threshold I D ).exist Figure 4E In the example, after a delay time p2 has passed since the touch input 490 was initially detected, the trigger is triggered with the intensity threshold I L At the same time, the dynamic intensity threshold 492 decays after a predefined delay time p1 has passed since the touch input 490 was initially detected. L The associated response then decreases the intensity of the touch input 490, and then increases the intensity of the touch input 490 without releasing the touch input 490, which may trigger the touch input 490 to be in accordance with the intensity threshold I. D The associated response (e.g., at time 494) may be performed even when the intensity of touch input 490 is below another intensity threshold (e.g., intensity threshold I L ) is also the case.

[0195] The contact characteristic strength is below the light press strength threshold IT L The intensity increases to between the light pressure intensity threshold IT L and deep compression intensity threshold IT D The intensity between IT and the deep press intensity threshold is sometimes referred to as a "light press" input. D The intensity of the deep compression increases to above the deep compression intensity threshold IT DThe intensity of the contact feature increases from an intensity below the contact detection intensity threshold IT0 to an intensity between the contact detection intensity threshold IT0 and the light press intensity threshold IT0. L A decrease in the characteristic intensity of a contact from an intensity above the contact detection intensity threshold IT0 to an intensity below the contact detection intensity threshold IT0 is sometimes referred to as detecting a contact lifted off the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some examples, shaded circles or ellipses are used to represent the intensity of a contact on the touch-sensitive surface. In some examples, unshaded circles or ellipses are used to represent corresponding contacts on the touch-sensitive surface without specifying the intensity of the corresponding contacts.

[0196] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the strength of the contact (or multiple contacts) increases to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting that the strength of the corresponding contact increases to above the press input strength threshold (e.g., the corresponding operation is performed on a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and the strength of the contact subsequently decreases to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the press input threshold (e.g., the corresponding operation is performed on an "up stroke" of the corresponding press input).

[0197] In some embodiments, the device employs intensity hysteresis to avoid unexpected input, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., performing the corresponding operation on an "up stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact strength increases from an intensity equal to or below a hysteresis strength threshold to an intensity equal to or above a press input strength threshold and, optionally, the contact strength subsequently decreases to an intensity equal to or below the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact strength or a decrease in contact strength, depending on the circumstances).

[0198] For ease of explanation, descriptions of operations performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input are optionally triggered in response to detecting that the intensity of the contact increases above the press input intensity threshold, the intensity of the contact increases from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, the intensity of the contact decreases below the press input intensity threshold, or the intensity of the contact decreases below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where operations are described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operations are optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold. As described above, in some embodiments, triggering of these operations is also dependent on satisfying a time-based criterion (e.g., a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold).

[0199] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when the smoothness of the touch-sensitive surface does not change, the movement of the touch-sensitive surface may be optionally interpreted or sensed by the user as the "roughness" of the touch-sensitive surface. Although such interpretation of touch by the user will be limited by the user's individualized sensory perception, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a specific sensory perception of a user (e.g., "press click", "release click", "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or its components that will generate the sensory perception of a typical (or ordinary) user. Providing tactile feedback to the user using tactile output enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), thereby further reducing power usage and extending the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0200] User interface and associated processes

[0201] Attention now turns to embodiments of a user interface ("UI") and associated processes that may be implemented on an electronic device such as portable multifunction device 100 or device 300 having a display, a touch-sensitive surface, (optionally) one or more tactile output generators for generating tactile output, and (optionally) one or more sensors for detecting intensity of contact with the touch-sensitive surface.

[0202] 5A to 5AF An exemplary user interface for relocalizing annotations according to some embodiments is shown. The user interface in these figures is used to illustrate the process described below, including 9A to 9F , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11F as well as FIG. 12A to FIG. 12D For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device with touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, in response to detecting a contact on touch-sensitive surface 451 when the user interface shown in the figure is displayed on display 450 together with the focus selector, similar operations are optionally performed on a device with display 450 and a separate touch-sensitive surface 451.

[0203] Figure 5A An annotation user interface 5003 is shown displayed on the touch screen display 112 of the device 100. The user interface displays the field of view of the camera of the device 100 as it captures the physical environment 5000 of the device 100. A table 5001a and a mug 5002a are located in the physical environment 5000. The camera view displayed in the user interface 5003 includes a visual representation 5001b of the physical table 5001a. The user interface 5003 includes a control 5004 for switching between still image mode and video mode, a control 5006 for controlling the camera flash setting, a control 5008 for accessing the camera settings, and a mode indicator 5010 indicating the current mode.

[0204] from FIG. 5A to FIG. 5B , device 100 has moved relative to the physical environment, such that the portion of table 5001b visible in the camera's field of view has changed, and such that visual representation 5002b of physical mug 5002a is now visible in the camera's field of view.

[0205] exist Figure 5C , stylus 5012 has touch-pressed touch screen display 112 at a point indicated by contact 5014 in annotation user interface 5003. In response to detecting contact 5014, a still image of the camera's field of view is captured and displayed in user interface 5003. The state of mode control 5004 and mode indicator 5010 has changed to indicate that the active mode of the user interface has changed from video mode to still image mode. The transition to still mode that occurs in response to the touch-press of the stylus (and / or in response to other types of input, as further described below) allows annotation input to be received relative to the view of the physical environment captured by camera 143 without being affected by changes in the spatial relationship between device 100 and physical environment 5000 caused by movement of device 100.

[0206] from FIG. 5C to FIG. 5D, the device 100 has moved relative to the physical environment 5000. Because the active mode of the user interface has changed from video mode to still image mode, the image displayed by the touch screen display 112 has changed from FIG. 5C to FIG. 5D No change.

[0207] from Figure 5E to Figure 5G , contact 5014 moves along the path indicated by arrow 5016, creating a first annotation 5018 at a portion of the still image that includes the visual representation 5002b of the physical mug 5002a.

[0208] exist Figure 5H , stylus 5012 provides input at a location corresponding to mode control 5004, as indicated by contact 5019. Fig.5I , in response to input from stylus 5012, the state of mode control 5004 and mode indicator 5010 has changed to indicate that the active mode of annotation user interface 5003 has changed from still image mode to video mode.

[0209] from FIG. 5I to FIG. 5L , Fig.5I The still image displayed in the full-size view of the still image (such as Fig.5I 5003 in the user interface) gradually changes to a micro view 5020 (as shown in FIG. Figure 5J to Figure 5K ) and changes to indicator point 5022 (as shown in user interface 5003 in FIG. Figure 5L 5003 in the user interface). Figure 5J middle, Fig.5I The micro view 5020 of the still image shown is shown to cover the current field of view of the camera. The size of the micro view 5020 is from Figure 5J Reduce to Figure 5K (For example, to provide Fig.5I The still image displayed by the device 100 in Figure 5L An indication of the correspondence between indicator points 5022 displayed on the video corresponding to the field of view of the device camera).

[0210] from Figure 5L to Figure 5M , the position of the device 100 changes relative to the physical environment 5000. As the device 100 moves, the field of view of the camera of the device 100 changes and the position of the indicator 5022 in the user interface 5003 changes. The movement of the indicator 5022 within the user interface 5003 provides an indication of the virtual space position of the annotation 5018 relative to the current position of the device 100. In this way, visual feedback is provided to the user indicating that the device 100 needs to be moved in the direction indicated by the indicator 5022 to redisplay the annotation 5018.

[0211] from Figure 5M to Figure 5N, the position of device 100 continues to change relative to physical environment 5000. Due to the movement of device 100, the field of view of the camera of device 100 is updated so that Fig.5I A portion of the physical environment 5000 captured in the annotated still image of the physical mug 5002a is visible in the camera's field of view. Annotations 5018 are displayed in the video (e.g., in each image frame that includes the visual representation of the physical mug 5002a) at locations corresponding to where the annotations 5018 were received in the still image (e.g., at locations corresponding to where the visual representation 5002b of the physical mug 5002a was received (e.g., as shown in FIG. 5 ). Fig.5I )). In some embodiments, as the device 100 moves closer to the physical mug 5002a, the visual representation 5002b of the physical mug 5002a will appear larger in the video image, and the annotation 5018 will also appear larger in accordance with the change in size of the visual representation 5002b. In some embodiments, as the device 100 moves around the physical mug 5002a, the visual representation 5002b of the physical mug 5002a will be updated to reflect the different perspectives of the physical mug 5002a, and the appearance of the annotation 5018 in the video image will also be updated in accordance with the change in perspective of the physical mug 5002a (e.g., viewed from a different angle).

[0212] exist Fig.5O 5003, stylus 5012 has touched down touch screen display 112 at a point indicated by contact 5024 in annotation user interface 5003. In response to detecting contact 5024, a second still image of the camera's field of view is captured and displayed in user interface 5003.

[0213] from Figure 5P to Figure 5R , while the second still image is displayed, input from stylus 5012 is received at the location indicated by contact 5026. Movement of contact 5026 creates a second annotation 5028 at a portion of the still image that includes visual representation 5002b of physical mug 5002a. Figure 5S , the stylus 5012 has been lifted off the touch screen display 112.

[0214] exist Figure 5T , an input is detected at a location on touch screen display 112 corresponding to control 5004 for switching between still image mode and video mode, as indicated by contact 5030. Figure 5U In response to the input, the state of the mode control 5004 and the mode indicator 5010 has changed to indicate that the active mode of the annotation user interface 5003 has changed from still image mode to video mode. Figure 5T The portion of the physical environment 5000 captured in the annotated still image of is already visible in the camera's field of view (e.g. Figure 5U3 in the video mode), so annotations 5018 and 5028 are displayed at locations in the video that correspond to the corresponding locations in the still image where the annotations were received (e.g., at locations corresponding to the still image). Figure 5G and Figure 5R The corresponding position of the visual representation 5002b of the physical mug 5002a is shown at a position corresponding to the corresponding position of the visual representation 5002b of the physical mug 5002a).

[0215] from Figure 5U to Figure 5V , the position of device 100 changes relative to physical environment 5000 so that the field of view of the device camera displayed in user interface 5003 does not include the portion of physical environment 5000 that includes mug 5002a. Indicator point 5022 corresponding to annotation 5018 and indicator point 5032 corresponding to annotation 5028 are displayed (e.g., indicator points 5022 and 5032 are displayed at off-screen virtual space locations relative to physical environment 5000 that indicate annotations 5018 and 5028, respectively (e.g., Figure 5G and Figure 5R 5002a ) as shown in the figure.

[0216] from Figure 5V to Figure 5W , the position of device 100 continues to change relative to physical environment 5000. As device 100 moves downward, the field of view of the camera of device 100 changes and indicators 5022 and 5032 move upward in user interface 5003 (e.g., to indicate the virtual space positions of annotations 5018 and 5028, respectively, relative to the current position of device 100). Figure 5W 5003, stylus 5012 has touched down touch screen display 112 at a point indicated by contact 5034 in annotation user interface 5003. In response to detecting contact 5034, a third still image of the camera's field of view is captured and displayed in user interface 5003.

[0217] from Figure 5W to Figure 5X , while the third still image is displayed, contact 5034 moves along the touch screen display 112 to create a third annotation 5036 at a portion of the lower right surface of the third still image that includes the visual representation 5001b of the physical table 5001a.

[0218] exist Figure 5Y , stylus 5012 provides input at a location corresponding to mode control 5004, as indicated by contact 5038. Figure 5Z 5012, the state of mode control 5004 and mode indicator 5010 has changed to indicate that the active mode of annotation user interface 5003 has changed from still image mode to video mode. Because the lower right surface of table 5001a is visible in the camera's field of view (e.g., Figure 5Z3 in video mode), so annotation 5036 is displayed in the video (e.g., in the video including Figure 5X 5028) in the image frame of the portion of the desktop shown in the still image at a location corresponding to the location where annotation 5036 was received in the still image (e.g., at a location corresponding to the lower right surface of visual representation 5001b of physical table 5001a). Because the portion of physical environment 5000 as displayed in the video mode of user interface 5003 and visible in the camera's field of view does not include the portion of the physical environment that corresponds to the spatial locations of annotations 5018 and 5028 (e.g., mug 5002a is not visible in the camera's field of view), indicator dots 5022 and 5032 are displayed at locations indicating the off-screen virtual spatial locations of annotations 5018 and 5028, respectively, relative to physical environment 5000.

[0219] from Figure 5Z to Figure 5AA , the position of device 100 changes relative to physical environment 5000. As device 100 moves upward, the field of view of the camera of device 100 changes and indicators 5022 and 5032 move downward in user interface 5003. Because the lower right surface of table 5001a is no longer visible in the camera's field of view (as displayed by the user interface in video mode), indicator point 5040 is displayed in the video at a location corresponding to the location in the still image where annotation 5036 was received.

[0220] from Figure 5AA to Figure 5AB , the position of device 100 changes relative to physical environment 5000. As device 100 continues to move upward, the field of view of the camera of device 100 changes and indicators 5022, 5032, and 5040 move downward in user interface 5003 to indicate the off-screen virtual space positions of annotations 5018, 5028, and 5036 relative to physical environment 5000, respectively.

[0221] from Figures 5AB to 5AC , the position of device 100 changes, resulting in a change in the field of view of the camera of device 100, as displayed by user interface 5003. The positions of indicators 5022, 5032, and 5040 are updated based on the off-screen virtual space positions of annotations 5018, 5028, and 5036, respectively.

[0222] from Figures 5AC to 5AD , the position of device 100 changes relative to physical environment 5000 so that the field of view of the device camera displayed in user interface 5003 includes the portion of physical environment 5000 that includes mug 5002a. Annotations 5018 and 5028 are displayed in user interface 5003, and indicator dots 5022 and 5032 cease to be displayed.

[0223] from Figures 5AD to 5AE, the position of device 100 changes relative to physical environment 5000 so that the field of view of the device camera displayed in user interface 5003 includes a portion of physical environment 5000 that includes the lower right surface of table 5001a. Annotation 5036 is displayed in user interface 5003, and indicator dot 5040 ceases to be displayed.

[0224] from Figure 5AE to Figure 5AF , device 100 has moved around the perimeter and over table 5001a, causing the position and perspective of annotations 5018, 5028, and 5036 to change.

[0225] FIG. 6A to FIG. 6N An exemplary user interface for receiving annotations on a portion of a physical environment captured in a still image corresponding to a paused position of a video is shown according to some embodiments. The user interface in these figures is used to illustrate the process described below, including 9A to 9F , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11F as well as FIG. 12A to FIG. 12D For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device with touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, in response to detecting a contact on touch-sensitive surface 451 when the user interface shown in the figure is displayed on display 450 together with the focus selector, similar operations are optionally performed on a device with display 450 and a separate touch-sensitive surface 451.

[0226] Fig. 6AA user interface 6000 is shown that includes a video playback area 6002. In some embodiments, the user interface 6000 is accessed via a list of media content objects (e.g., in an image and / or video viewing application). In some embodiments, the user interface 6000 also includes a timeline 6004 (e.g., corresponding to a set of sample frames 6006 of a continuous segment of a video). The timeline 6004 includes a current position indicator 6008 that indicates a position on the timeline 6004 corresponding to a frame displayed in the video playback area 6002 and throughout the video displayed in the video playback area 6002. In some embodiments, the user interface includes a mark control 6010 (e.g., for starting a mark mode for marking a video displayed in the video playback area 6002), a rotation control 6012 (e.g., for rotating a video displayed in the video playback area 6002), an edit control 6014 (e.g., for editing a video displayed in the video playback area 6002), a cancel control 6020 (e.g., for canceling a current operation), a rotation control 6022 (e.g., for rotating a video displayed in the video playback area 6002), and a play / pause switch control 6016 (e.g., for playing back and pausing a video displayed in the video playback area 6002). Contact 6018 (e.g., input by a user's finger) with the touch screen display 112 is detected at a location corresponding to the play / pause switch control 6016.

[0227] exist Figure 6B In response to input detected at a position corresponding to the play / pause switch control 6016, playback of the video is initiated in the video playback area 6002.

[0228] exist Figure 6C 6024. In the example embodiment, while the video continues to play back, input is detected at a location corresponding to the tag control 6010, as indicated by contact 6024. In response to the input made by contact 6024 at the location corresponding to the tag control 6010, playback of the video is paused, a still image corresponding to the paused location of the video is displayed, a tagging mode is initiated, wherein input received at a location corresponding to the video playback area 6002 will tag the video, and the state of the tag control 6010 is changed so that the text "Done" is displayed (e.g., to indicate that input selecting control 6010 will terminate the tagging mode).

[0229] exist FIG. 6D to FIG. 6F , annotation input is detected at a location within video playback area 6002, as indicated by contact 6026. FIG. 6D to FIG. 6F As the path shown moves, an annotation 6030 is received. In some embodiments, the annotation is received at a location corresponding to an object in the video (e.g., kite 6028). Figure 6G, while the marking mode is active and the text “Done” is displayed, input via contact 6031 is detected at a location corresponding to marking control 6010. In response to input via contact 6031 to terminate the marking session (and / or after the time elapsed since the last input was received increases above a threshold amount of time), the play / pause toggle control 6016 is redisplayed, as shown. Figure 6H shown.

[0230] exist Fig.6I , input is detected at a location corresponding to the play / pause switch control 6016, as indicated by contact 6032.

[0231] exist FIG. 6I to FIG. 6K In response to an input detected at a location corresponding to the play / pause switch control 6016, the video resumes playback in the video playback area 6002. Figure 6J As shown, the sample frames of timeline 6004 include marks at positions corresponding to kite 6028 in the corresponding image. For example, sample frame 6006 includes mark 6034 at a position corresponding to kite 6036. Annotation 6030 applied to kite object 6028 in video playback area 6002 has been applied to sample frames in timeline 6004, including video frames that occur before the point in time when annotation 6030 is received in the video (e.g., frames to the left of swipe control 6008 in timeline 6004, such as sample frame 6006) and video frames that occur after the point in time when annotation 6030 is received in the video (e.g., frames to the right of swipe control 6008 in timeline 6004). In each of the sample frames of the timeline 6004 in which a kite object 6036 (corresponding to the kite object 6028 to which the annotation 6030 is applied in the video playback area 6002) is visible, a marker (e.g., marker 6034, corresponding to the annotation 6030) is displayed at a position corresponding to the changed position of the kite object 6036 within the sample frame. The marker is displayed with a size and orientation scaled corresponding to the changed size and orientation of the kite object 6036 within the sample frame. When the video displayed in the video playback area 6002 is played forward to a frame in the video after the frame in which the video stopped playing to receive the annotation input, the annotation 6030 is displayed with a size and orientation scaled corresponding to the changed size and orientation of the kite object 6026, as shown in FIG. Figure 6K Thus, as the object moves and changes size (e.g., due to a change in its distance from the camera) and orientation (e.g., changes in any direction in three-dimensional space) during video playback, the annotations received in the video playback area 6002 are applied to the object (e.g., so that the annotations track the object).

[0232] exist Figure 6L6004, an input is detected at a location corresponding to timeline 6004, as indicated by contact 6038. As contact 6038 moves along the path indicated by arrows 6040 and 6044, the video displayed in video playback area 6002 rewinds, as shown in FIG. Figures 6L to 6N For example, as contact 6038 moves across timeline 6004, time indication 6042 associated with the frame currently displayed in video playback area 6002 decreases. When the video displayed in video playback area 6002 is rewound to a frame prior to the frame at which the video was stopped to receive annotation input, annotation 6030 is displayed with a size and orientation that scales correspondingly to the changed size and orientation of kite object 6026.

[0233] 7A to 7BF An example user interface for adding a virtual object to a previously captured media object is shown according to some embodiments. The user interface in these figures is used to illustrate the process described below, including 9A to 9F , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11F as well as FIG. 12A to FIG. 12D For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device with touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, in response to detecting a contact on touch-sensitive surface 451 when the user interface shown in the figure is displayed on display 450 together with the focus selector, similar operations are optionally performed on a device with display 450 and a separate touch-sensitive surface 451.

[0234] Fig. 7AA user interface 7000 is shown displayed by the touch screen display 112 of the device 100, the user interface including a media object display area 7002 and a navigation area 7004. In some embodiments, the user interface 7000 is accessed via a list of media content objects (e.g., in an image and / or video viewing application). A previously captured image is displayed in the media object display area. Information corresponding to the previously captured image (e.g., the location where the image was captured, "Cupertino") is displayed in the information area 7003. The navigation area 7004 includes a previous media object control 7006 (e.g., for navigating to a previous media object), a current media object indicator 7008 (e.g., which indicates the location of the previously captured image relative to other media objects stored by the device 100 (which are represented by non-magnified dots) (which are represented by magnified dots), and a subsequent media object control 7010 (e.g., for navigating to subsequent media objects). The user interface 7000 includes controls 7012 to 7024 for adding various virtual objects to the previously captured image displayed in the media object display area 7002, as described below with reference to FIG. 7B to FIG. 7B G is further described. As described according to various embodiments herein, the manner in which the virtual object is displayed relative to the physical object in the previously captured image provides the following indication to the user: the depth data of the previously captured image is stored and the virtual object is able to interact with various surfaces in the various images. The interaction of the virtual object with the physical object in the physical environment captured in the image provides the user with an indication of the presence of a detectable surface in the image.

[0235] FIG. 7B to FIG. 7L 7002. The virtual ball object is shown interacting with the surface of a physical object depicted in a previously captured image displayed in the media object display area 7002. For example, the captured image stores depth data that is used to determine the location of the surface (e.g., horizontal plane and / or vertical plane) corresponding to the physical object in the physical environment captured in the image.

[0236] exist Figure 7B In the embodiment, an input (e.g., a tap input) is received through contact 7026 at a position corresponding to ball control 7012 to add a virtual ball to the previously captured image displayed in media object display area 7002. Figure 7C7012, the mode of user interface 7000 changes to a ball spawning mode, as indicated by a "Ball" label in information area 7003 and a changed visual state of ball control 7012. An input (e.g., a tap input) is received at a location indicated by contact 7028. In response to detecting the contact, a virtual ball 7030 is added to the previously captured image displayed in media object display area 7002. For example, adding virtual ball 7030 to the previously captured image includes launching virtual ball 7030 in a trajectory upward from the point at which contact 7028 was detected and along a path indicated by dashed line 7032 (e.g., virtual ball 7030 falls under the influence of simulated gravity, bounces off a surface of chair object 7036 in the previously captured image, bounces off a surface of table object 7038 in the previously captured image, falls to floor surface 7040 in the previously captured image, and rolls along floor surface 7040).

[0237] exist Fig.7D 7042, an input (e.g., a tap input) is received at the location indicated by contact 7042. In response to detecting the contact, a virtual ball 7044 is added to the previously captured image displayed in media object display area 7002, and virtual ball 7044 moves along path 7046 (e.g., bounces off and lands on floor 7040).

[0238] exist Fig. 7E 7048. In the example of FIG. 7049, several additional virtual balls have been added to the previously captured image displayed in the media object display area 7002, some of which have landed and settled on the surfaces of the chair object 7036 and the table object 7038. An input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the subsequent media object control 7010, as indicated by contact 7048. In response to the input, the touch screen display 112 is moved to the touch screen display 112. Fig. 7E The display of the first previously captured image shown is replaced by Figure 7F The virtual ball added to the previously captured image displayed in the media object display area 7002 is added to the second previously captured image, such as Figure 7F as shown (e.g., animated so that the virtual ball appears to move from Fig. 7E The floor surface 7040 in the first previously captured image "falls" onto the top of the physical environment in the second previously captured image, as shown in FIG. FIG. 7F to FIG. 7IAs the virtual ball falls, it settles on surfaces in the physical environment captured in the second previously captured image, such as the surfaces of lamp 7052 and table 7054 and in the arms of persons 7056 and 7058. In some embodiments, the virtual object has a depth that is equivalent to the depth data corresponding to the physical object in the previously captured image to determine the placement of the virtual object relative to the physical object (e.g., to determine whether the physical object will occlude the virtual object or vice versa). For example, in Figure 7G , the virtual ball 7045 is partially obscured by the physical table 7054.

[0239] exist Figure 7J , an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the subsequent media object control 7010, as indicated by contact 7050. In response to the input, Figure 7J The display of the second previously captured image shown is replaced by Figure 7K The virtual ball added to the second previously captured image displayed in the media object display area 7002 is added to the third previously captured image, as shown. Figure 7K as shown (e.g., animated so that the virtual ball appears to move from Figure 7J The surface in the second previously captured image "falls" onto the top of the physical environment in the third previously captured image, as Figures 7K to 7L As the virtual ball falls, it settles on a surface in the physical environment captured in the third previously captured image, such as the surface of the sofa 7060.

[0240] exist Figure 7L , an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the text insertion control 7014, as indicated by contact 7062. In response to the input, the virtual balls (e.g., balls 7034 and 7044) cease to be displayed, and a text object 7064 is added to the third previously captured image, as shown. Figure 7M shown.

[0241] FIG. 7M to FIG. 7T It shows how the virtual text object 7064 interacts with the surface of the physical object depicted in the previously captured image displayed in the media object display area 7002.

[0242] exist Figure 7M , input by contact 7066 is detected at a location on touch screen display 112 corresponding to text object 7064. Figure 7M to Figure 7P, contact 7066 moves along the path indicated by arrows 7068, 7070, and 7072. As contact 7066 moves, text object 7064 is "dragged" by contact 7066, so that the movement of text object 7064 corresponds to the movement of contact 7066. When text object 7064 is dragged to a position corresponding to sofa 7060, text object 7064 interacts with the detected surface of sofa 7060 by "traveling" over the armrests of sofa 7060, as shown in FIG. FIG. 7N to FIG. 7O For example, in Fig.7O , when the text object 7064 is dragged to the area of ​​the third previously captured image corresponding to the sofa 7060, a first portion of the text object 7064 is adjacent to a first surface of the sofa 7060 (above the armrests of the sofa), and a second portion of the text object 7064 is adjacent to a second surface of the sofa 7060 (e.g., above the seat of the sofa 7060).

[0243] exist Figure 7Q In the embodiment, an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the text object 7064, as indicated by contact 7062. In response to the input, a text editing mode for the text object 7064 is started, as shown in FIG. Figure 7R As indicated by the display of cursor 7080 and keyboard 7078. Figure 7S , the input provided via keyboard 7078 has changed the text of text object 7064 from the word "text" to the word "cold."

[0244] exist Figure 7T , an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the decal insertion control 7016, as indicated by contact 7064. In response to the input, the text object 7064 is stopped from being displayed, and the decal object 7084 is added to the third previously captured image, as shown. Figure 7U shown.

[0245] Figure 7U to Figure 7Y It shows how the virtual decal object 7084 interacts with the surface of the physical object depicted in the previously captured image displayed in the media object display area 7002.

[0246] exist Figure 7U , input via contact 7086 is detected at a location on touch screen display 112 corresponding to decal object 7084. Figure 7U to Figure 7X, contact 7086 moves along the path indicated by arrows 7088, 7090, and 7092. As contact 7086 moves, decal object 7084 is "dragged" by contact 7086, such that the movement of decal object 7084 corresponds to the movement of contact 7086. When decal object 7084 is dragged over the surface of sofa 7060, decal object 7064 conforms to the detected horizontal and vertical surfaces of sofa 7060 and floor 7094, as shown in FIG. Figure 7U to Figure 7X For example, in Figure 7V , when decal object 7084 is dragged over sofa 7060, a first portion of decal object 7084 is adjacent to a first surface of sofa 7060 (lying flat on the seat of the sofa), and a second portion of decal object 7084 is adjacent to a second surface of sofa 7060 (e.g., covering over a front portion of sofa 7060). Figure 7X , when the decal object 7084 has been dragged onto the floor 7094 , the decal object 7084 is partially obscured by the sofa 7060 .

[0247] exist Figure 7Y In the embodiment of the present invention, an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the emoji insertion control 7018, as indicated by contact 7096. In response to the input, the decal object 7084 is stopped from being displayed, and the emoji object 7098 is added to the third previously captured image, as shown in FIG. Figure 7Z shown.

[0248] Figure 7Z to Figure 7AE It shows how the virtual emoji object 7098 interacts with the surface of the physical object depicted in the previously captured image displayed in the media object display area 7002.

[0249] exist Figure 7AA , input by contact 7100 is detected at a location on touch screen display 112 corresponding to emoticon object 7098. Figures 7AA to 7AB , contact 7100 moves along the path indicated by arrow 7102. As contact 7100 moves, emoticon object 7098 is "dragged" by contact 7100 so that the movement of emoticon object 7098 corresponds to the movement of contact 7100. Figure 7AC , when emoji object 7098 is suspended in the space above the surface (e.g., sofa 7060 and floor 7094) in the third previously captured image, contact 7100 has lifted off touch screen display 112. In response to the lifting of contact 7100, emoji object 7098 falls under the influence of simulated gravity, as shown in FIG. Figure 7AC to Figure 7AE As shown. Figure 7AC, emoji object 7098 encounters physical object sofa 7060, causing the orientation of emoji object 7098 to change as it rolls over the armrests of sofa 7060 (e.g., Figure 7AD as shown), and continues its fall until it reaches floor 7094.

[0250] exist Figure 7AF , an input (e.g., a tap input) is detected at a location on touch screen display 112 corresponding to confetti insertion control 7020, as indicated by contact 7104. In response to the input, display of emoji object 7098 stops, and a confetti object (e.g., confetti object 7106) is added to the third previously captured image, as shown. Figure 7AG shown.

[0251] Figure 7AG to Figure 7AT 7002 shows how virtual confetti objects interact with the surfaces of physical objects depicted in previously captured images and videos displayed in media object display area 7002. Figure 7AG middle,

[0252] exist Figures 7AG to 7AJ In , virtual confetti objects are added continuously (e.g. Figure 7AH , Figure 7AI and Figure 7AJ 5060 and floor 7094) and falls under the influence of simulated gravity, converging on a detected surface in a third previously captured image (e.g., a substantially horizontal surface such as the horizontal surface of sofa 5060 and floor 7094).

[0253] exist Figure 7AJ , an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the previous media object control 5006, as indicated by contact 7104. In response to the input, Figure 7AJ The display of the third previously captured image shown is replaced by Figure 7AK The confetti added to the third previously captured image displayed in the media object display area 7002 is added to the second previously captured image, as shown. Figure 7AK as shown (for example, in Figure 7AJ confetti is shown in the same position).

[0254] exist Figures 7AK to 7AL In the present invention, virtual confetti objects are continuously added and fall under the influence of simulated gravity, gathering on the detected surface in the second previously captured image.

[0255] exist Figure 7AJ, an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the previous media object control 5006, as indicated by contact 7104. In response to the input, Figure 7AJ The display of the third previously captured image shown is replaced by Figure 7AK The confetti added to the third previously captured image displayed in the media object display area 7002 is added to the second previously captured image, as shown. Figure 7AK as shown (for example, in Figure 7AJ confetti is shown in the same position).

[0256] exist Figures 7AK to 7AL In the present invention, virtual confetti objects are continuously added and fall under the influence of simulated gravity, gathering on the detected surface in the second previously captured image.

[0257] exist Figure 7AL In the embodiment of the present invention, an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the subsequent media object control 7010, as indicated by contact 7110 (e.g., receiving multiple taps to advance the currently displayed media object twice). In response to the input, Figure 7AL The display of the second previously captured image shown is replaced by Figure 7AM Confetti added to the second previously captured image displayed in the media object display area 7002 is added to the video, as shown Figure 7AM as shown (for example, in Figure 7AL confetti is shown in the same position).

[0258] exist Figure 7AM to Figure 7AT , virtual confetti objects are continuously added and fall under the influence of simulated gravity, gathering on detected surfaces in the video (e.g., the edges of the kite object 7112). Figure 7AN , Figure 7AO and Figure 7AP As video playback continues in , virtual confetti objects are continuously added and fall under the influence of simulated gravity, gathering on the edge of the kite object 7112 and the bottom edge of the video frame.

[0259] exist Figure 7AP In , an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the playback control 7114, as indicated by contact 7116. In response to the input, the video is repeatedly played back. Figure 7AQ When replay is initiated, it will be added to Figure 7AM to Figure 7ATFor example, in FIG. AQ, when playback of the video is initiated and the kite object 7112 is shown at a different position in the video, the confetti accumulated on the kite object 7112 is removed from the video. Figure 7AP Drop to the location shown.

[0260] exist Figure 7AQ to Figure 7AT , virtual confetti objects are continuously added and fall under the influence of simulated gravity, gathering on detected surfaces in the video (e.g., the edges of the kite object 7112). Figure 7AR , Figure 7AS and Figure 7AT As video playback continues in , virtual confetti objects are continuously added and fall under the influence of simulated gravity, gathering on the edge of the kite object 7112 and the bottom edge of the video frame.

[0261] In some embodiments, a displayed virtual confetti object fades from the display and / or ceases to be displayed (e.g., when the amount of time that has elapsed since the virtual confetti object was displayed increases above a threshold time).

[0262] Figure 7AU to Figure 7AX It shows how the virtual spotlight object 7118 interacts with the physical objects depicted in the previously captured image displayed in the media object display area 7002.

[0263] exist Figure 7AU , a second previously captured image is displayed in the media object display area 7002, and the spotlight mode has been activated (e.g., in response to input received at the spotlight control 7022). In spotlight mode, the spotlight virtual object 7118 is shown as illuminating a portion of the image (e.g., person 7058), and areas of the image outside the spotlight virtual object 7118 are shown as dimmed. In this way, the spotlight virtual object 7118 allows attention to be drawn to a specific portion of the image. In some embodiments, the initial physical object to be illuminated is automatically selected (e.g., based on a determination of the physical object closest to the foreground). The spotlight virtual object 7118 includes a simulated light beam 7122 and a simulated lighting point 7124 that illuminates a portion of the floor in the previously captured image. In some embodiments, the light beam 7122 illuminates at least a portion of the representation of the physical object in the previously captured image. In some embodiments, the lighting point 7124 illuminates a portion of the image corresponding to a horizontal surface detected in the image, such as the floor 7124.

[0264] exist Figure 7AV , input by contact 7128 is detected at a location on the touch screen display 112 corresponding to spotlight object 7118. Figure 7AV to Figure 7AW, contact 7128 moves along the path indicated by arrow 7102. As contact 7128 moves, spotlight object 7118 is "dragged" by contact 7128 so that the movement of spotlight object 7118 corresponds to the movement of contact 7128. Figure 7AW , the position of spotlight object 7118 has shifted so that person 7056 is illuminated by spotlight object 7118. Because spotlight object 7118 has moved, the size of illumination point 7124 has changed.

[0265] exist Figure 7AX , an input (e.g., a tap input) is detected at a location on the touch screen display 112 corresponding to the measurement control 7020, as indicated by contact 7026. In response to the input, the spotlight object 7118 stops being displayed.

[0266] exist Figure 7AY , input is detected at the locations indicated by contacts 7132 and 7134 on touch screen display 112. In 7AZ, in response to detecting contacts 7132 and 7134, a virtual measurement object 7136 is displayed at a location that spans the distance between contacts 7132 and 7134 (e.g., corresponding to the height of person 7058). Measurement indicator 7138 indicates that the distance between the points corresponding to contacts 7132 and 7134 in the physical environment captured in the image is 1.8 m (e.g., determined using depth data stored with a previously captured image).

[0267] from Figure 7AZ to Figure 7BA , contact 7132 moves along the path indicated by arrow 7140. As contact 7132 moves, the size of virtual measurement object 7136 is adjusted to span the adjusted distance between contacts 7132 and 7134, and the measurement value indicated by measurement value indicator 7138 is updated.

[0268] exist Figure 7BB , contacts 7132 and 7134 have been lifted off the touch screen display 112. Virtual measurement object 7136 and measurement value indicator 7138 remain displayed. Input is detected at the location indicated by contact 7142 on the touch screen display 112. Figure 7BB to Figure 7BC , contact 7142 moves along the path indicated by arrows 7144 and 7146. In response to movement of contact 7142 (e.g., movement exceeding a threshold amount), a virtual measurement object 7148 and a measurement value indicator 7150 are displayed. Figure 7BC to Figure 7BD As the contact 7142 continues to move, the size of the virtual measurement object 7148 is adjusted, and the measurement value indicated by the measurement value indicator 7150 is updated. The dashed portion of the virtual measurement object 7148 indicates a portion of the virtual measurement object 7148 that passes through the physical object (eg, person 7056).

[0269] In 7BE, contact 7142 has been lifted off the touch screen display 112. Virtual measurement object 7148 and measurement value indicator 7148 remain displayed. Fig.7BF , an input is detected at a location indicated by contact 7152 on the touch screen display 112. In response to the input, an end of the virtual measurement object 7148 (eg, the end closest to the received input) moves to a location corresponding to the location of contact 7152.

[0270] FIG. 8A to FIG. 8W An exemplary user interface for initiating a shared annotation session according to some embodiments is shown. The user interface in these figures is used to illustrate the process described below, including 9A to 9F , FIG. 10A to FIG. 10B , FIG. 11A to FIG. 11F as well as FIG. 12A to FIG. 12D For ease of explanation, some embodiments in the implementation scheme will be discussed with reference to operations performed on a device with touch-sensitive display system 112. In such embodiments, the focus selector is optionally: a corresponding finger or stylus contact, a representative point corresponding to the finger or stylus contact (e.g., the center of gravity of the corresponding contact or a point associated with the corresponding contact), or the center of gravity of two or more contacts detected on touch-sensitive display system 112. However, in response to detecting a contact on touch-sensitive surface 451 when the user interface shown in the figure is displayed on display 450 together with the focus selector, similar operations are optionally performed on a device with display 450 and a separate touch-sensitive surface 451.

[0271] FIG. 8A to FIG. 8G The establishment of a shared annotation session between two devices is shown.

[0272] Fig. 8A A physical environment 8000 is shown in which a first user operates a first device 100-1 (e.g., device 100) and a second user operates a second device 100-2 (e.g., device 100). A collaborative user interface 8002 displayed by device 100-1 is shown in an inset 8004 corresponding to device 100-1. Inset 8006 shows a web browser user interface currently displayed by device 100-2. A prompt 8008 displayed in collaborative user interface 8002 includes instructions for initiating a shared annotation session.

[0273] exist Figure 8B8, an input is received via contact 8012 at a location corresponding to control 8010 displayed by device 100-1 for initiating a shared annotation session. In response to the input, a request to initiate a shared annotation session is sent from device 100-1 to a remote device (e.g., device 100-2). When the request has been sent and a response indicating acceptance of the request has not yet been received, a notification 8014 is displayed by device 100-1, such as Figure 8C shown.

[0274] exist Figure 8C 8, in response to receiving a request to initiate a shared annotation session, device 100-2 displays prompt 8016 including instructions for accepting the request for the shared annotation session. Input via contact 8020 is received at a location corresponding to control 8018 displayed by device 100-2 for accepting the request for the shared annotation session. In response to the input, an acceptance of the request is sent from device 100-2 to a remote device (e.g., device 100-1).

[0275] exist Fig.8D , device 100-1 has received an indication of an acceptance of the request to initiate a shared annotation session. Device 100-1 displays prompt 8022, which includes an instruction to move device 100-1 toward device 100-2. Prompt 8022 includes a representation 8026 of device 100-1 and a representation 8028 of device 100-2. Device 100-2 displays prompt 8024, which includes an instruction to move device 100-2 toward device 100-1. Prompt 8024 includes a representation 8030 of device 100-1 and a representation 8032 of device 100-2.

[0276] FIG. 8D to FIG. 8E Animations are shown displayed in prompts 8022 and 8024. In prompt 8022, representation 8026 of device 100-1 is animated so that it moves toward representation 8028 of device 100-2. In prompt 8024, representation 8032 of device 100-2 is animated so that it moves toward representation 8030 of device 100-1.

[0277] exist Figure 8F, connection criteria have been met (e.g., the first device 100-1 and the second device 100-2 have moved toward each other and / or at least a portion of the physical space 8000 captured in the field of view of one or more cameras of the first device 100-1 corresponds to at least a portion of the physical space 8000 captured in the field of view of one or more cameras of the device 100-2). A notification 8034 displayed by the first device 100-1 and a notification 8036 displayed by the second device 100-2 each include an indication that a shared annotation session has been initiated. The first device 100-1 displays a representation of the field of view of the one or more cameras of the first device 100-1 (e.g., overlaid by the notification 8034). A poster 8038a in the physical environment 8000 is visible in the field of view of the one or more cameras of the first device 100-1, as indicated by a representation 8038b of the poster 8038a displayed by the first device 100-1. The second device 100-2 displays a representation of the field of view of one or more cameras of the second device 100-2 (e.g., overlaid by notification 8036). Devices 100-1 and 100-2 display a shared field of view (e.g., at least a portion of the physical space 8000 captured in the field of view of the one or more cameras of the first device 100-1 corresponds to at least a portion of the physical space 8000 captured in the field of view of the one or more cameras of device 100-2). For example, a poster 8038a in the physical environment 8000 is visible in the field of view of the one or more cameras of the second device 100-2, as indicated by representation 8038c of poster 8038a displayed by the second device 100-2. Figure 8G , the respective fields of view of the cameras are displayed by the first device 100-1 and the second device 100-2 without notifications 8034 and 8036 (eg, notifications 8034 and 8036 have stopped being displayed).

[0278] Figures 8H to 8M Annotation input received during a shared annotation session is shown. FIG. 8H to FIG. 8J An annotation input provided at the second device 100 - 2 is shown. Figures 8K to 8M An annotation input provided at the first device 100 - 1 is shown.

[0279] exist Figure 8H 8044, the second device 100-2 detects input made through contact 8044 (e.g., input received at the touch screen display of the second device 100-2). When the second device 100-2 detects the input, the first device 100-1 displays an avatar 8048 at a location in the shared field of view that corresponds to the location in the shared field of view where the input was received at the second device 100-2. As contact 8044 moves along the path indicated by arrow 8046, as shown in FIG. FIG. 8H to FIG. 8IAs shown, the annotation corresponding to the movement of the contact 8044 is displayed by the first device 100-1 (as the annotation 8050-1) and the second device 100-2 (as the annotation 8050-2). Figure 8J , further annotation input has been provided by further movement of contact 8044.

[0280] exist Figure 8K , first device 100-1 detects input through contact 8052 (e.g., input received at a touch screen display of first device 100-1). When first device 100-1 detects the input, second device 100-2 displays avatar 8054 at a location in the shared field of view that corresponds to the location in the shared field of view where the input was received at first device 100-1. When contact 8052 moves, as shown in FIG. Figures 8K to 8M As shown, an annotation corresponding to the movement of the contact 8052 is displayed by the first device 100-1 (as annotation 8056-1) and the second device 100-2 (as annotation 8056-2).

[0281] exist Figure 8M to Figure 8P In FIG. 1 , the movement of the first device 100 - 1 increases the distance between the first device 100 - 1 and the second device 100 - 2. Figure 8N , when the first device 100-1 moves away from the second device 100-2, the representation of the field of view of the camera of the first device 100-1 displayed by the first device 100-1 is adjusted (e.g., the portion of the representation 8038b of the physical poster 8038a displayed by the first device 100-1 is reduced).

[0282] In some embodiments, one or more annotations (e.g., 8050-1, 8050-2, 8056-1, and / or 8056-2) have a fixed spatial relationship relative to a portion of the physical environment 8000 (e.g., such that movement of the device camera relative to the physical environment 8000 changes the display location of the annotation). Fig.8O , when the first device 100-1 continues to move away from the second device 100-2 so that the first device 100-1 no longer displays the annotation 8056-1, the first device 100-1 displays a visual indication 8058 corresponding to the annotation 8056-1 (e.g., to indicate the movement direction required for the first device 100-1 to redisplay the annotation 8056-1).

[0283] exist Figure 8P, movement of first device 100-1 away from second device 100-2 increases the distance between first device 100-1 and second device 100-2 to a degree (e.g., above a threshold distance) that causes the shared annotation session to be disconnected. Device 100-1 displays prompt 8060, which includes instructions to move device 100-1 toward device 100-2. Device 100-2 displays prompt 8062, which includes instructions to move device 100-2 toward device 100-1. In some embodiments, prompt 8060 includes an animation element (e.g., as shown in FIG. 1 ). FIG. 8D to FIG. 8E described above).

[0284] from Figure 8P to Figure 8Q , the movement of the first device 100-1 reduces the distance between the first device 100-1 and the second device 100-2. Figure 8Q , the distance between the first device 100-1 and the second device 100-2 has been sufficiently reduced to resume the shared annotation session. Devices 100-1 and 100-2 have stopped displaying respective prompts 8060 and 8062, and redisplay the respective fields of view of the respective device cameras.

[0285] Figure 8R to Figure 8W A gaming application using a shared session between a first device 100-1 and a second device 100-2 is shown (eg, as described above with reference to FIG. 1 ). FIG. 8A to FIG. 8G The established shared session).

[0286] Figure 8R A physical environment 8068 is shown in which a first user 8064a operates a first device 100-1 (e.g., device 100) and a second user 8066a operates a second device 100-2 (e.g., device 100). A gaming application user interface 8063-1 displayed by the first device 100-1 is shown in an illustration 8070 corresponding to the first device 100-1. A gaming application user interface 8063-2 displayed by the second device 100-2 is shown in an illustration 8072 corresponding to the second device 100-2. User 8064a faces user 8066a so that a representation 8066b of user 8066a is visible in the field of view of one or more cameras (e.g., a rear-facing camera) of device 100-1, as shown in user interface 8063-1. A representation 8064b of user 8064a is also visible in the field of view of one or more cameras (e.g., a rear-facing camera) of device 100-2, as shown in user interface 8063-2.

[0287] Game application user interfaces 8063-1 and 8063-2 display basketball hoops 8074 and 8076, respectively. By providing input in the respective game application user interfaces, the user launches a virtual basketball object into the displayed representation of the field of view of the respective device camera in an attempt to shoot a ball into the respective basketball hoop. In some embodiments, the respective basketball hoop is fixed to the spatial location of the respective device, so that the respective user can move the device in an attempt to set a challenge for the opponent. Game application user interfaces 8063-1 and 8063-2 also display game data areas 8078 and 8080, respectively. The game data displayed in the game data area includes, for example, a score of points earned from successful shots and a distance between device 100-1 and device 100-2 (e.g., to serve as a basis for assigning a score to a given shot).

[0288] exist Figure 8S In the example of FIG. 1 , device 100-2 detects an input (e.g., a tap input) made through contact 8082 to launch a virtual basketball object. In response to detecting the input made through contact 8082, a virtual basketball object 8084 is added to the field of view of the camera of device 100-2 displayed in game application user interface 8063-2, as shown in FIG. Figure 8T shown. Figure 8T Device 100-2 is also shown detecting an input (e.g., a tap input) made through contact 8086 to launch a virtual basketball object. In response to detecting the input made through contact 8086, a virtual basketball object 8088 is added to the field of view of the camera of device 100-1 displayed in game application user interface 8063-1, as shown in FIG. Figure 8V As shown. Figure 8V In the example of FIG. 1 , device 100-2 detects input (e.g., a tap input) made through contact 8090 to launch a virtual basketball object. In response to detecting the input made through contact 8090, a virtual basketball object 8092 is added to the field of view of the camera of device 100-2 displayed in game application user interface 8063-2, as shown in FIG. Figure 8W As shown. Figure 8V to Figure 8W , user 8064a has lowered device 100-1, so that the display position of box 8076 and representation 8064b of user 8064 have changed in user interface 8063-2.

[0289] 9A to 9Fis a flow chart showing method 900 of localizing annotations according to some embodiments. Method 900 is performed on an electronic device (e.g., a display generating component (e.g., a display, a projector, a heads-up display, etc.), one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control or a touch screen display that also serves as a display generating component, a mouse, a joystick, a wand controller, and / or a camera that tracks one or more features of a user such as the position of the user's hands), and one or more cameras (e.g., one or more rear-facing cameras on a side of the device opposite the display and the touch-sensitive surface). Figure 3 Device 300 or Figure 1A The method 900 is performed at a portable multifunction device 100 of the present invention. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display and the touch-sensitive surface are separate. Some operations in method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0290] The device displays (902) via the display generation component a first user interface area (e.g., user interface 5003), the first user interface area including a representation of the field of view of the one or more cameras, the representation being updated as the field of view of the one or more cameras changes over time (e.g., continuously updating (e.g., continuously updating at a preset frame rate such as 24fps, 48fps, or 60fps, etc.) the representation of the field of view in accordance with changes occurring in a physical environment surrounding the cameras and in accordance with movement of the cameras relative to the physical environment). For example, FIG. 5A to FIG. 5B As shown, when the device 100 moves, the view of the physical environment 5000 in the field of view of one or more cameras is updated according to the change in position of the camera of the device 100.

[0291] While displaying a first user interface area that includes a representation of the field of view of the one or more cameras, the device receives (904) via the one or more input devices a first request to add an annotation (e.g., text or numbers generated and / or positioned by movement of a contact (e.g., contact by a finger or stylus) on the touch-sensitive surface) to the displayed representation of the field of view of the one or more cameras (e.g., the first request is input made by a contact detected in the first user interface area on the touch screen display (e.g., at a location corresponding to a control for initiating annotation or at a location in the first user interface area (e.g., at a location where the annotation is to be initiated)). For example, the request to add the annotation is input made by stylus 5012 (e.g., input to initiate the annotation, as described with reference to FIG. 5 ). Figure 5C and / or input by contact at a position corresponding to a control (e.g., control 5004 for switching between still image mode and video mode), as described with reference to Figure 5H described above).

[0292] In response to a first request to add an annotation to the displayed representation of the field of view of the one or more cameras (e.g., including in response to detecting a touch depression or movement of a finger or stylus on a touch-sensitive surface at a location corresponding to a portion of the physical environment captured within the representation of the field of view of the camera or a user interface object (e.g., a button for activating an AR annotation mode)), the device replaces (906) display of the representation of the field of view of the one or more cameras in a first user interface area with a still image of the field of view of the one or more cameras (e.g., pausing a real-time feed of the field of view of the one or more cameras (e.g., displaying a still image of the current field of view while the field of view continues to change as the device moves), the still image being captured at a time corresponding to receipt of the first request to add an annotation; and displaying the still image corresponding to the paused real-time feed of the field of view of the one or more cameras in the first user interface area. For example, in response to input made through stylus 5012 (e.g., as described with reference to Figure 5C ), a representation of the field of view of the device camera (e.g., as referenced FIG. 5A to FIG. 5B The ) is replaced by a display of a still image of the field of view of the one or more cameras captured at a time corresponding to receipt of the first request to add an annotation (e.g., as described with reference to FIG. 5C to FIG. 5D described above).

[0293] While the still image is displayed in the first user interface area, the device receives (908) via the one or more input devices a first annotation (e.g., drawing input) on a first portion of the still image, wherein the first portion of the still image corresponds to a first portion of the physical environment captured in the still image. For example, annotation 5018 is received on a portion of the still image (e.g., a portion including representation 5002b of physical mug 5002a), the portion of the still image corresponding to a portion of the physical environment captured in the still image (e.g., a portion of physical environment 5000 that includes physical mug 5002a), as described with reference to FIG. Figure 5D to Figure 5G In some embodiments, while displaying a still image and receiving annotation input on the still image, the device continues to track the position of the camera relative to the surrounding physical environment (e.g., based on changes in the camera's field of view and input from other sensors (e.g., motion sensors, gravity sensors, gyroscopes, etc.)). In some embodiments, the device determines whether the physical location or object corresponding to the location of the annotation in the still image has been moved out of the camera's field of view; and if so, the device also determines the spatial relationship between the portion of the physical environment currently in the camera's field of view and the physical location or object that is the target of the annotation. For example, mug 5002 is the target of annotation 5018.

[0294] While displaying the first annotation on the first portion of the still image in the first user interface area (e.g., after the first annotation on the first portion of the still image has been received), the device receives (910) via the one or more input devices a first request to redisplay the representation of the field of view of the one or more cameras in the first user interface area. For example, the request to redisplay the representation of the field of view of the one or more cameras in the first user interface area is input made by stylus 5012 at control 5004 for switching between still image mode and video mode, as described with reference to Figure 5H described.

[0295] In response to receiving a first request to redisplay a representation of the field of view of the one or more cameras in the first user interface area (e.g., including in response to not detecting contact on the touch-sensitive surface for a threshold amount of time (e.g., deeming the drawing session terminated), or detecting a tap on a user interface object (e.g., a button for deactivating the AR annotation mode), the device replaces (910) display of the still image with a representation of the field of view of the one or more cameras in the first user interface area (e.g., continuously updating the representation of the field of view at a preset frame rate (e.g., 24fps, 48fps, or 60fps, etc.) based on changes occurring in the physical environment surrounding the cameras and based on movement of the cameras relative to the physical environment). For example, in response to input made by stylus 5012 at control 5004, as described with reference to Figure 5H As described above, in the first user interface area 5003, FIG. 5C to FIG. 5H The still image shown in is replaced with a representation of the field of view of the one or more cameras, as shown in reference FIG. 5I to FIG. 5N Based on determining that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment that includes an object relative to which the first annotation is received) is currently outside the field of view of the one or more cameras (e.g., this is due to device movement that occurs after the real-time feed of the camera view is paused), the device simultaneously displays an indication of the current spatial relationship of the one or more cameras (e.g., as part of a computing system) relative to the first portion of the physical environment captured in the still image and a representation of the field of view of the one or more cameras (e.g., displaying a visual indication such as a dot or other shape at an edge of the displayed camera field of view and at a location on the edge closest to the first portion of the physical environment on which the annotation is made, or displaying a simplified diagram of the physical environment simultaneously with the representation of the camera's field of view and marking the location of the first portion of the physical environment relative to the device on the diagram). For example, as described with reference to Figure 5L The portion of the physical environment captured in the still image (e.g., including the mug 5002 relative to which the annotation 5018 is received) is currently outside the field of view of the one or more cameras (e.g., within Figure 5J to Figure 5L5002a is outside the field of view of the camera, as displayed by user interface 5003), while displaying an indication of the current spatial relationship of the one or more cameras relative to the first portion of the physical environment captured in the still image (e.g., indicator point 5022) and a representation of the field of view of the one or more cameras (e.g., as displayed by user interface 5003). Figure 5L In response to determining that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including an object relative to which the first annotation is received) is currently within the field of view of the one or more cameras, the device abandons displaying the indication. For example, in response to input to redisplay the field of view of the camera (e.g., input by stylus 5012 at control 5004 for switching between still image mode and video mode, as described with reference to Figure 5T ), based on determining that the portion of the physical environment captured in the still image (e.g., including the mug 5002 relative to which the annotations 5018 and 5028 are received) is currently within the field of view of the one or more cameras (e.g., within Figure 5T 5002a is visible in the field of view of the camera displayed in the user interface 5003 (as visible representation 5002b), and no indication of the current spatial relationship of the one or more cameras relative to the first portion of the physical environment captured in the still image is displayed (e.g., as shown in FIG. 5002b). Figure 5T ). Displaying an indication of the current spatial relationship of the one or more cameras relative to the portion of the physical environment captured in the still image in response to determining that the portion of the physical environment captured in the still image is currently out of view of the one or more cameras provides visual feedback to the user (e.g., indicating that the camera needs to be moved to view the portion of the physical environment captured in the still image). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment that corresponds to the annotated portion of the still image), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0296] In some embodiments, displaying (912) an indication of the current spatial relationship of the one or more cameras relative to the first portion of the physical environment captured in the still image includes: displaying an indicator near an edge of the representation of the field of view of the one or more cameras, and moving the indicator along the edge in accordance with movement of the one or more cameras relative to the physical environment. Figure 5L As described, indicator 5022 is displayed near the left edge of the camera's field of view, as displayed in user interface 5003, and as shown in reference Figure 5L to Figure 5MAs described, the indicator moves along the edge according to the movement of the camera of the device 100. In some embodiments, for the rectangular representation of the field of view of the one or more cameras, the indicator is a visual indication such as a dot or other shape that moves along the edge of the rectangular representation of the field of view, and the visual indication may move along a straight edge according to the first movement of the one or more cameras, and the visual indication may also jump from one straight edge to another straight edge according to the second movement of the one or more cameras. Moving the indicator along the edge of the camera view provides visual feedback to the user (e.g., indicating the direction of movement of the camera required to view the portion of the physical environment captured in the still image). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment corresponding to the annotated portion of the still image), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0297] In some embodiments, while displaying an indication of a current spatial relationship of the one or more cameras relative to a first portion of a physical environment captured in a still image, the device detects (914) a first movement of the one or more cameras (e.g., as part of a computing system). In response to detecting the first movement of the one or more cameras, the device updates a representation of the field of view of the one or more cameras in accordance with a change in the field of view of the one or more cameras caused by the first movement. Based on determining that the first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including an object relative to which the first annotation is received) is currently outside the field of view of the one or more cameras, the device updates an indication of a current spatial relationship of the one or more cameras relative to the first portion of the physical environment captured in the still image in accordance with the first movement of the one or more cameras (e.g., moving a visual indication such as a dot or other shape along an edge of the representation of the field of view in accordance with the first movement of the cameras). Based on determining that the first portion of the physical environment captured in the still image (e.g., a portion of the physical environment including an object relative to which the first annotation is received) is currently within the field of view of the one or more cameras, the device stops displaying the indication. For example, when the camera of device 100 moves, when a portion of the physical environment captured in the still image is outside the field of view of the camera (e.g., mug 5002a is outside the field of view of the camera, as shown in FIG. Figure 5L to Figure 5M ), updating indication 5022 (e.g., moving upward in user interface 5003), and no longer displaying indication 5022 (e.g., as shown in reference to FIG. 5 ) when the portion of the physical environment captured in the still image is within the field of view of the camera. Figure 5L to Figure 5NIn some embodiments, a zoomed-out representation of the still image with the first annotation (e.g., zoomed-out representation 5020) is displayed at an edge of the representation of the field of view near the location of the last displayed visual indicator, with the still image view zooming out and moving toward the location of the first annotation shown in the representation of the camera's field of view. In some embodiments, when the first portion of the physical environment moves out of the camera's field of view due to additional movement of the camera relative to the physical environment, the indicator is redisplayed (e.g., as shown in reference to Figure 5U to Figure 5V The present invention also provides a method for displaying an image of a physical environment captured in a still image in a manner that is consistent with the present invention. The method further provides a method for displaying an image of a physical environment captured in a still image in a manner that is consistent with the present invention. The method further provides a method for displaying an image of a physical environment captured in a still image in a manner that is consistent with the present invention. The method further provides a method for displaying an image of a physical environment captured in a still image in a manner that is consistent with the present invention. The method further provides a method for displaying an image of a physical environment captured in a still image in a manner that is consistent with the present invention. The method further provides a method for displaying an image of a physical environment captured in a still image in a manner that is consistent with the present invention.

[0298] In some embodiments, in response to receiving a first request to redisplay a representation of the field of view of the one or more cameras in a first user interface area (916): based on determining that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment that includes an object relative to which the first annotation is received) is currently within the field of view of the one or more cameras, the device displays the first annotation at the first portion of the physical environment captured within the representation of the field of view of the one or more cameras. Figure 5N As described, based on determining that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment 5000 that includes the physical mug 5002a) is currently within the field of view of the one or more cameras, the device displays an annotation 5018 at a portion of the physical environment captured within the representation of the field of view of the one or more cameras (e.g., the annotation 5018 is displayed at a location corresponding to the visual representation 5002b of the physical mug 5002a). Displaying the annotation of the portion of the physical environment captured in the annotated still image provides visual feedback to the user (e.g., indicating that the portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment that corresponds to the annotated portion of the still image), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0299] In some embodiments, a first annotation is displayed (918) as a two-dimensional object (e.g., annotation 5018) on a first depth plane in a first portion of the physical environment captured within the representation of the field of view of the one or more cameras. In some embodiments, the first depth plane is detected based on detection of a physical object (e.g., physical mug 5002a) or object feature at the first depth plane in the first portion of the physical environment. Displaying the annotation on the depth plane in the portion of the physical environment in the field of view of the one or more cameras provides improved visual feedback to the user (e.g., indicating that the annotation has a fixed spatial relationship with the physical environment). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to cause objects in the physical world to have additional information contained in the annotation), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0300] In some embodiments, a first annotation (e.g., annotation 5018) is displayed (920) at a location in space in a first portion of the physical environment captured within the representation of the field of view of the one or more cameras. In some embodiments, the first annotation floats in space separate from any physical objects detected in the first portion of the physical environment. Displaying the annotation at a location in space in a portion of the physical environment captured in the static image provides improved visual feedback to the user (e.g., indicating that the annotation has a fixed spatial relationship to the physical environment). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to cause objects in the physical world to have additional information contained in the annotation), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0301] In some embodiments, a first annotation (e.g., annotation 5018) is displayed (922) at a location on a physical object (e.g., physical mug 5002a) detected in a first portion of the physical environment captured within the representation of the field of view of the one or more cameras. In some embodiments, the first annotation is attached to the physical object (or feature of the physical object) detected in the first portion of the physical environment. Displaying the annotation of the portion of the physical environment captured in the still image at a location on the physical object detected in the portion provides improved visual feedback to the user (e.g., indicating that the annotation has a fixed spatial relationship to the physical environment). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to cause objects in the physical world to have additional information contained in the annotation), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0302] In some embodiments, in response to receiving a first request to redisplay a representation of the field of view of the one or more cameras in a first user interface area (924): based on determining that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment that includes an object relative to which the first annotation is received) is not currently within the field of view of the one or more cameras, the device simultaneously displays in the first user interface area a visual representation of the first annotation (e.g., a drawing input) drawn on the first portion of the still image and a representation of the field of view of the one or more cameras (e.g., a reduced scale representation of the still image with the first annotation displayed on an edge of the representation of the field of view proximate a location closest to the first portion of the physical space currently represented in the field of view of the camera). For example, in response to a request to redisplay a representation of the field of view of the one or more cameras in the first user interface area (e.g., input made by stylus 5012 at control 5004 for switching between still image mode and video mode, as described with reference to Figure 5H ), based on determining that a first portion of the physical environment captured in the still image is currently within the field of view of the one or more cameras (the portion of the physical environment captured in the still image (e.g., including the mug 5002 relative to which the annotation 5018 is received) is currently within the field of view of the one or more cameras), the device simultaneously displays the annotation 5018 drawn on the first portion of the still image and a representation of the field of view of the one or more cameras in a first user interface area (e.g., as shown in FIG. 1 ). Figure 5N In some embodiments, in response to receiving a first request to redisplay a representation of the field of view of the one or more cameras in a first user interface area: based on determining that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment that includes an object relative to which the first annotation is received) is currently outside the field of view of the one or more cameras, the computing system abandons displaying a visual representation of the first annotation drawn on the first portion of the still image (e.g., Figure 5L and Figure 5M 5018 is not displayed). Based on determining that the first portion of the physical environment is not within the field of view of the camera while simultaneously displaying the visual representation of the annotation of the still image and the representation of the field of view of the camera, improved visual feedback is provided to the user (e.g., indicating that the indication of the current spatial relationship (e.g., point) corresponds to the received annotation input). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment that corresponds to the annotated portion of the still image), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0303] In some embodiments, a first annotation shown in the representation of the field of view has (926) a first viewing perspective, the first viewing perspective being based on a current spatial relationship of the one or more cameras (e.g., as part of a computing system) relative to a first portion of a physical environment captured in the still image and being different from a second viewing perspective of the first annotation shown on the still image (e.g., a perspective of the first annotation shown in a reduced scale representation of the still image displayed adjacent to the representation of the field of view). In some embodiments, an animated transition is displayed showing the reduced scale representation of the still image being converted to a representation of the current field of view. Displaying the annotation having a viewing perspective different from that shown in the still image provides improved visual feedback to the user (e.g., indicating that the annotation is fixed to a portion of the physical environment captured in the still image). Figure 5AE and Figure 5AF Examples of annotations 5018 with different viewing perspectives shown in a representation of a field of view are provided. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment that corresponds to the annotated portion of the still image), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0304] In some embodiments, in response to receiving a first request to redisplay a representation of the field of view of the one or more cameras in a first user interface area (928): based on determining that a first portion of the physical environment captured in the still image (e.g., a portion of the physical environment that includes an object relative to which the first annotation is received) is currently outside the field of view of the one or more cameras, simultaneously displaying in the first user interface area a visual representation of the first annotation (e.g., a drawing input) drawn on the first portion of the still image and a representation of the field of view of the one or more cameras (e.g., a reduced scale representation of the still image with the first annotation displayed near a position on an edge of the representation of the field of view closest to the first portion of the physical space currently represented in the field of view of the cameras), wherein the visual representation of the first annotation drawn on the first portion of the still image (e.g., the reduced scale representation 5020) is converted to an indication of a current spatial relationship of the one or more cameras (e.g., as part of a computing system) relative to the first portion of the physical environment captured in the still image (e.g., indication 5022) (e.g., as described with reference to FIG. 5 ). FIG. 5I to FIG. 5LAs described). For example, the indication is a visual indication such as a dot or other shape displayed at an edge of the displayed camera field of view and at a location on the edge closest to the first portion of the physical environment on which the annotation was made, and before displaying the visual indication, a miniature view of the still image with the first annotation is displayed at the location and converted to the visual indication. Converting the visual representation of the annotation (e.g., a reduced scale representation of the still image) to an indication (e.g., a dot) of the current spatial relationship of the camera relative to the portion of the physical environment captured in the still image provides improved visual feedback to the user (e.g., the indication (e.g., the dot) and the visual representation (e.g., the reduced scale image) of the annotation are different representations of the same annotation). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment corresponding to the annotated portion of the still image), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0305] In some embodiments, while displaying a first user interface area including a representation of the field of view of the one or more cameras, and before detecting a first request to add an annotation to the displayed representation of the field of view, the device displays (930) an indication of the current spatial relationship of the one or more cameras (e.g., as part of a computing system) relative to a second portion of the physical environment to which a previously received second annotation was previously added (e.g., indication 5022) (e.g., displaying a visual indication such as a dot or other shape at an edge of the displayed camera field of view and at a location on the edge closest to the second portion of the physical environment on which the second annotation was made, or displaying a simplified diagram of the physical environment and the representation of the camera's field of view simultaneously, with the location of the second portion of the physical environment relative to the device marked on the diagram). In some embodiments, the second annotation is added to the second portion of the physical environment shown in the representation of the camera's field of view in the same manner as the first annotation was added to the first portion of the physical environment shown in the representation of the camera's field of view. Displaying the indication of the current spatial relationship of the camera relative to the second portion of the physical environment to which the previously received annotation was added provides improved visual feedback to the user (e.g., indicating that the previously received annotation needs to be moved relative to the portion of the physical environment captured in the still image to be viewed). For example, before detecting a request to add annotation 5028, indication 5022 is displayed, as shown in FIG. Figure 5P to Figure 5RProviding improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment that corresponds to the previously annotated portion of the still image), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0306] In some embodiments, after receiving a first request to redisplay a representation of the field of view of the one or more cameras in the first user interface area: based on determining that both the first portion of the physical environment and the second portion of the physical environment are outside the field of view of the one or more cameras, the device simultaneously displays (932) an indication of a current spatial relationship of the one or more cameras relative to the first portion of the physical environment and an indication of a current spatial relationship of the one or more cameras relative to the second portion of the physical environment. For example, when multiple annotations (e.g., annotations 5018 and 5036) have been added to different portions of the physical environment, indicators corresponding to different annotations (e.g., indicators 5018 and 5040, Figure 5AA ) are simultaneously displayed around an edge of a representation of the camera's field of view at a corresponding location closest to a portion of the physical environment corresponding to the annotation. Simultaneously displaying an indication of the current spatial relationship of the camera relative to a first portion of the physical environment and an indication of the current spatial relationship of the camera relative to a second portion of the physical environment provides improved visual feedback to the user (e.g., indicating a direction of movement of the camera required to view one or more of a plurality of received annotations). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment corresponding to the previously annotated portion of the still image), which in turn reduces power usage and extends battery life of the device by enabling the user to use the device more quickly and efficiently.

[0307] In some embodiments, while simultaneously displaying an indication of a current spatial relationship of the one or more cameras relative to a first portion of the physical environment and an indication of a current spatial relationship of the one or more cameras relative to a second portion of the physical environment, the device detects (934) a second movement of the one or more cameras relative to the physical environment; and in response to detecting the second movement of the one or more cameras relative to the physical environment: based on determining that both the first portion of the physical environment and the second portion of the physical environment are outside the field of view of the one or more cameras, the device updates the one or more cameras relative to the first portion and the second portion of the physical environment, respectively, in accordance with the second movement of the one or more cameras relative to the physical environment (e.g., moving visual indicators in different directions and / or at different speeds along an edge of the representation of the field of view). For example, respective visual indicators corresponding to different annotations pointing to different portions of the physical environment are displayed at different locations on the edge of the representation of the field of view of the camera, and as the device moves, the respective visual indicators move in different directions and at different speeds in accordance with changes in the current spatial relationship of their respective annotations relative to the device (e.g., indicators 5018 and 5040 move in accordance with movement of device 100, as described with reference to 5AA to 5AD). The visual indicators may move together or separately and / or at different speeds depending on the actual spatial relationship between different portions of the physical environment marked with different annotations. Updating the indication of the current spatial relationship of the camera relative to the first portion and the second portion of the physical environment based on the movement of the camera provides improved visual feedback to the user (e.g., indicating whether the movement of the camera brings the camera closer to or further away from the portion of the physical environment corresponding to the annotated portion of the still image). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment corresponding to the previously annotated portion of the still image), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0308] In some embodiments, based on determining that the first portion and the second portion of the physical environment are within a predefined range of the one or more cameras, an indication of a current spatial relationship of the one or more cameras relative to the first portion of the physical environment and an indication of a current spatial relationship of the one or more cameras relative to the second portion of the physical environment are displayed (936). In some embodiments, the user interface provides a method for selecting a subset of annotations from all annotations that have been added to various portions of the physical environment, and only indications corresponding to the selected subset of annotations are displayed together with a representation of the field of view of the camera. Based on determining that the first portion and the second portion of the physical environment are within a predefined range of the one or more cameras, the indication relative to the first portion of the physical environment and relative to the second portion of the physical environment are displayed, providing improved visual feedback to the user (e.g., by reducing clutter of the user interface with indicators when the first portion and the second portion are outside the predefined range). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately locate the portion of the physical environment that corresponds to the previously annotated portion of the still image), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0309] It should be understood that 9A to 9F The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 1000, 1100, and 1200) also apply in a similar manner to the details of the processes described above with respect to 9A to 9F Method 9000 is described. For example, the contacts, inputs, annotations, physical objects, user interface areas, fields of view, movements, and / or animations described above with reference to method 9000 optionally have one or more of the features of the contacts, inputs, annotations, physical objects, user interface areas, fields of view, movements, and / or animations described herein with reference to other methods described herein (e.g., methods 1000, 1100, and 1200). For the sake of brevity, these details are not repeated here.

[0310] FIG. 10A to FIG. 10B 1 is a flow chart illustrating a method 1000 for receiving annotations on a portion of a physical environment captured in a still image corresponding to a pause position of a video according to some embodiments. The method 1000 is performed on an electronic device (e.g., a display, a projector, a heads-up display, etc.) having a display generating component (e.g., a display, a projector, a heads-up display, etc.) and one or more input devices (e.g., a touch screen display that also serves as a display generating component). Figure 3 Device 300 or Figure 1A The method 1000 is performed at a portable multifunction device 100 of the present invention. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display and the touch-sensitive surface are separate. Some operations in method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0311] The device displays (1002) a user interface including a video playback area via a display generation component. For example, the device 100 displays a user interface 6000 including a video playback area 6002 via a touch screen display 112, as shown in FIG. Fig. 6A described.

[0312] While displaying playback of the first portion of the video in the video playback area, the device receives (1004) via the one or more input devices a request to add an annotation to the video playback (e.g., the request is input made by a contact detected in the video playback user interface on the touch screen display (e.g., at a location corresponding to a control for initiating annotation or at a location in the video playback area (e.g., at the location where the annotation is to be initiated)). For example, the request to add the annotation to the video playback is input made by contact 6024 detected at a location corresponding to mark control 6010.

[0313] In response to receiving the request to add an annotation, the device (1006): pauses playback of the video at a first position in the video (eg, identifies a current frame of the video (eg, the pause position) and stops playback of the video at the current frame).

[0314] The device displays (1008) a still image (e.g., a frame of the video) corresponding to the first pause position of the video (e.g., displays the current frame of the video as shown when the request is received). FIG. 6C to FIG. 6D As described, in response to input made through contact 6024 at a position corresponding to mark control 6010, playback of the video is paused and a still image corresponding to the paused position of the video is displayed.

[0315] While the still image is displayed (e.g., in a video playback area), the device receives (1008) via the one or more input devices an annotation (e.g., drawing input by contact) on a first portion of the physical environment captured in the still image. FIG. 6E to FIG. 6FIn the embodiment, an annotation 6030 is received on a portion of the physical environment corresponding to the kite object 6028 in the displayed still image. It should be understood that in the case of reference to a "physical environment" herein, a non-physical environment (e.g., a computer-generated environment) may be included in the still image. For example, an annotation is received on a portion of the image corresponding to the pause position of the video (e.g., as part of a computer-generated video). In some embodiments, the computer-generated image includes depth data and / or objects, and the annotations and / or virtual objects are positioned relative to these depth data and / or objects.

[0316] After receiving the annotation, the device displays (1010) in the video playback area a second portion of the video corresponding to a second location in the video (e.g., during continued playback of the video or upon receiving an input to swipe forward or backward through the video (e.g., on a timeline)), the second location being different from the first location in the video (e.g., before or after a pause location in the video), wherein the first portion of the physical environment is captured in the second portion of the video and the annotation is displayed in the second portion of the video. For example, after receiving the annotation 6030 (as described in reference to FIG. 6E to FIG. 6F ), the annotation 6030 is displayed in the second portion of the video (e.g., as shown in FIG. Figure 6J As described). In some embodiments, when a second portion of the video is displayed, the annotation is displayed at a second location in the video playback area, the second location being different from the first location of the annotation received when the still image was displayed (e.g., the annotation is "attached" to a location (e.g., a physical object) in the physical environment captured in the video clip, such that the annotation moves as the location (e.g., the physical object) moves as the video progresses. Displaying the annotation in a portion of the video (different from the portion of the video to which the annotation is applied) without requiring further input (e.g., to identify the surface to which the annotation is to be applied) enhances the operability of the device. Performing operations without requiring further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing a user to add information to a previously captured video by annotating the video directly without the need to re-record the video or provide annotations in multiple portions of the video), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0317] In some embodiments, during relative movement of the camera and the physical environment, a video is captured (1012) by the camera (e.g., during video capture, movement data of the camera and depth data of the physical environment are simultaneously captured and stored with simultaneously captured image data), and during the relative movement of the camera and the physical environment, a third portion of the video is captured between the first portion and the second portion of the video and does not include the first portion of the physical environment. In some embodiments, an annotation (e.g., annotation 6030) received on the still image (e.g., as FIG. 6E to FIG. 6F 6028) is pointed to a first object (e.g., kite 6028) located in a first portion of the physical environment and is displayed at a location corresponding to the first object in a second portion of the video (e.g., without being annotated directly on a still image of any frame of the second portion of the video). Figure 6J , annotation 6030 is displayed in the second portion of the video at a location corresponding to kite 6028. In some embodiments, the annotation is not displayed in a third portion of the video that does not include the first object (e.g., the annotation is not persistently displayed and is only shown when the current frame includes the first object). In some embodiments, the annotation is rotated and scaled according to the distance and viewing angle of the first object so that the annotation appears to be displayed at the location of the first object. For example, in Figures 6J to 6N, rotate and scale the annotation 6030 so that the annotation appears to be displayed at the location of the kite 5026 in the video. In some embodiments, the first part, the third part, and the second part of the video are continuously captured parts of the video, or the second part, the third part, and the first part of the video are continuously captured parts of the video. Since the captured material is not continuous in the field of view of the camera (e.g., during the capture of the third part of the video), the first part of the physical environment captured in the first part of the video cannot be identified as the same as the first part of the physical environment captured in the third part of the video based only on the image data of the separate video (e.g., by identifying tracking points on consecutive frames by frame-to-frame comparison). Use the movement and / or depth data of the camera (optionally, in combination with the image data) to create a three-dimensional or quasi-three-dimensional model of the physical environment captured in the video, so that the specific location in the physical environment can be identified in each frame of the video regardless of its appearance in the frame or viewing angle. Displaying the annotation at a location corresponding to the object to which the annotation points and not displaying the annotation in a portion of the video that does not include the object provides improved visual feedback to the user (e.g., by providing an indication that the annotation is fixed at a location corresponding to the object). Performing operations without requiring further user input enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing a user to add information to a video without requiring the video to be re-recorded or providing annotations in multiple portions of the video), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0318] In some embodiments, the device displays (1014) a timeline of the video (e.g., displayed simultaneously with the video (e.g., during playback of the video, and / or when the video is paused)) (e.g., a swipe bar having a position indicator indicating the position of a currently displayed frame, a scrollable sequence of scaled-down images of sample frames from a continuous segment of the video with a still mark for the currently displayed frame), wherein in response to user input swiping the timeline of the video to a second position in the video (e.g., user input dragging the position indicator along the swipe bar, or user input scrolling the sequence of scaled-down images of sample frames past the still mark for the currently displayed frame), displaying a second portion of the video is performed. For example, input is received via contact 6038 at a location corresponding to timeline 6004, the timeline including a sequence of sample frames (e.g., sample frame 6006, as shown in reference to Fig. 6A ), and in response to the input (e.g., as described with reference to Figures 6L to 6NThe device 6002 may be used to rewind a video displayed in the video playback area 6002 by performing an operation without requiring further user input. ...

[0319] In some embodiments, displaying the second portion of the video is performed as a result of (1016) rewinding the video from the first position to the second position (e.g., the second position precedes the first position on the timeline of the video). For example, receiving input via contact 6038 to rewind the video displayed in video playback area 6002, as described with reference to FIG. Figures 6L to 6N In response to a rewind input, displaying annotations in a portion of a video (different from the portion of the video to which the annotations are applied) without requiring further user input enhances the operability of the device (e.g., by allowing the user to access the annotations displayed in the second portion of the video using an existing swipe control without requiring a separate control or input). Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0320] In some embodiments, displaying the second portion of the video is performed as a result of (1018) fast forwarding the video from a first position to a second position (e.g., the second portion is after the first portion on a timeline of the video and the playback speed is faster than normal playback speed). In response to the fast forward input, the annotation is displayed in a portion of the video (different from the portion of the video to which the annotation is applied) without requiring further user input, enhancing the operability of the device (e.g., by allowing the user to access the annotation displayed in the second portion of the video using an existing swipe control without requiring a separate control or input). Performing the operation without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0321] In some embodiments, displaying the second portion of the video is performed because (1020) the video is playing back normally from the first position to the second position (e.g., the second portion is after the first portion on the timeline of the video, and the playback speed is faster than the normal playback speed). In some embodiments, when the user finishes providing annotations on the still image, the user exits the annotation mode by selecting a "Done" button displayed with the still image (e.g., as described in reference to Figure 6G as described); and thus, the device continues to play back the video from the first location, wherein the annotation is displayed in each subsequent frame including the first portion of the physical environment, and at the same physical location (e.g., the physical object), even when the first portion of the physical environment was captured at a different distance and / or from a different perspective compared to the still image. In response to normal playback of the video, displaying the annotation in a portion of the video (different from the portion of the video to which the annotation is applied) without requiring further user input enhances the operability of the device (e.g., by allowing a user to access the annotation displayed in a second portion of the video using an existing swipe control without requiring a separate control or input). Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0322] In some embodiments, the device displays (1022) a list of media content objects including a video (e.g., displaying a representation of the video in a media library) via a display generation component. The device receives input selecting a video from the list of media content objects, and in response to receiving input selecting the video, displays a user interface object (e.g., a "mark" button for adding annotations) in the video playback area along with the display of the video, wherein the user interface object is configured to receive a request to add an annotation to the video during playback of the video (e.g., a tap input for activating a button to add an annotation). Displaying a user interface object configured to receive a request to add an annotation to the video during playback of the video provides improved feedback (e.g., indicating an option to add an annotation to a portion of the video during playback of the video). In some embodiments, when a touch input is detected on the video playback area during video playback, displaying a mark button (e.g., displayed with other playback controls) provides improved feedback, and when the mark button is activated, the currently displayed frame is shown as being in a mark-enabled state, i.e., ready for drawing annotations to be received directly on the image of the currently displayed frame. Providing enhanced feedback enhances the operability of the device (e.g., by allowing the user to access annotations displayed in the second portion of the video using existing swipe controls without requiring a separate control or input). Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0323] It should be understood that FIG. 10A to FIG. 10BThe specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 900, 1100, and 1200) are also applicable in a similar manner to the details of the above with respect to FIG. 10A to FIG. 10B The method 1000 described herein. For example, the contacts, inputs, annotations, physical objects, user interface areas, fields of view, movements, and / or animations described above with reference to the method 1000 optionally have one or more of the features of the contacts, inputs, annotations, physical objects, user interface areas, fields of view, movements, and / or animations described herein with reference to other methods described herein (e.g., methods 900, 1100, and 1200). For the sake of brevity, these details are not repeated here.

[0324] FIG. 11A to FIG. 11F 1 is a flow chart showing a method 1100 of adding a virtual object to a previously captured media object. The method 1100 is performed on an electronic device (e.g., a display generating component (e.g., a display, a projector, a heads-up display, etc.) and one or more input devices (e.g., a touch-sensitive surface such as a touch-sensitive remote control or a touch screen display that also serves as a display generating component, a mouse, a joystick, a wand controller, and / or a camera that tracks one or more features of a user such as the position of a user's hands). Figure 3 Device 300 or Figure 1A The method 1100 is performed at a portable multifunction device 100 of the present invention. In some embodiments, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display and the touch-sensitive surface are separate. Some operations in method 1100 are optionally combined, and / or the order of some operations is optionally changed.

[0325] The device displays (1102) via a display generation component a first previously captured media object including one or more first images (e.g., a still photo, a real-time photo, or a video including a sequence of image frames), wherein the first previously captured media object and first depth data corresponding to a first physical environment captured in each of the one or more first images (e.g., the first depth data generated by one or more depth sensors of the device (e.g., emitter / detector systems such as infrared, sonar and / or lidar, and / or image analysis systems (e.g., video clip analysis and / or stereo image / video analysis)), for example, at a time corresponding to the time when the first media object was captured by one or more cameras) are recorded and stored together. For example, as described with reference to Fig. 7A As described, the device 100 displays the previously captured image in the media object display area 7002 via the touch screen display 112 .

[0326] While displaying a first previously captured media object (e.g., displaying a still image, displaying a representative image of a real-time photo, displaying frames of a video during playback of a video, or displaying frames of a video when the video is in a paused or stopped state), the device receives (1104) via the one or more input devices a first user request to add a first virtual object (e.g., a falling ball, confetti, text, a spotlight, an emoticon, paint, a measurement graphic) to the first previously captured media object. For example, the request to add the virtual object to the previously captured media object is a tap input received on the touch screen display 112 for adding the virtual object to the previously captured image, as described with reference to FIG. Figure 7C described.

[0327] In response to a first user request to add a first virtual object to a first previously captured media object, the device displays (1106) the first virtual object over at least a portion of a corresponding image in the first previously captured media object, wherein the first virtual object is displayed to have at least a first position or orientation (or movement path), the first position or orientation being determined using first depth data corresponding to the corresponding image in the first previously captured media object. For example, as described with reference to Figure 7C As described, in response to a tap input received on the touch screen display 112, a virtual object (e.g., a virtual ball object 7030) is added to a previously captured image. Adding a virtual object to a previously captured media object using depth data from the previously captured media object without requiring user input (e.g., to indicate the location of a plane in the media object) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0328] In some embodiments, displaying a first virtual object over at least a portion of a corresponding image in a first previously captured media object includes (1108): after placing the first virtual object over a corresponding image in the one or more first images, displaying a first movement of the first virtual object relative to a first physical surface captured in the first previously captured media object, wherein the first movement of the first virtual object is constrained by a first simulated surface corresponding to the first physical surface and determined based on the depth data and a simulated gravity direction (e.g., the simulated gravity direction is optionally determined based on the gravity direction recorded with the depth data when the first image was captured, or is the same as the actual gravity direction relative to the current orientation of a device displaying the corresponding image). For example, as referenced Figure 7CAs described, the virtual ball object 7030 is subject to the constraints of the simulated surface and simulated gravity direction corresponding to the physical surface (e.g., floor surface 7040) and determined based on the depth data. In some embodiments, different types of virtual objects have different simulated physical features (e.g., shape, size, weight, elasticity, etc.) that interact with the simulated surface in different ways. In one example, if the still image captures a sofa with curved armrests and a flat seating area, the virtual rubber ball is shown as falling from the top of the image, landing on the curved surface of the armrest, rebounding from the curved surface of the armrest, landing on the flat seating area, and then rolling down to the floor. In contrast, a piece of confetti is shown as floating down from the top of the image, landing on the curved surface of the armrest, sliding off the curved surface of the armrest, landing on the flat seating area, and remaining on the flat seating area. In another example, the 3D letter "A" is placed on the top of the curved surface of the armrest by the user's finger, and when the user's finger is lifted off the touch screen, the 3D letter "A" falls sideways and lands on the flat surface of the seating area. In some embodiments, a surface mesh corresponding to the physical environment captured in the still image is generated based on the depth data, and the virtual object inserted into the still image is animated during insertion and / or after initial placement to show movement and final position / orientation consistent with basic physics (such as laws related to gravity, forces, and physical interactions between objects). The virtual object is moved to the physical surface of the previously captured media object (e.g., based on simulated gravity) to indicate the position of the plane in the media object), thereby allowing the movement of the virtual object to occur automatically without further input (e.g., without requiring the user to provide input to guide the movement path of the virtual object), enhancing the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0329] In some embodiments, displaying a first virtual object over at least a portion of a corresponding image in a first previously captured media object includes (1110): after placing the first virtual object over a corresponding image in the one or more first images, displaying the first virtual object according to a shape change of a first physical surface captured in the first previously captured media object, wherein the shape change of the first virtual object is constrained by a first simulated surface corresponding to the first physical surface and determined based on the first depth data. For example, as described with reference to Figure 7U to Figure 7XAs described, the shape of the virtual decal object 7084 changes as the object moves over the surface of the sofa 7060 and onto the floor 7004 depicted in the previously captured image. In some embodiments, if the still image captures a sofa with a curved armrest and a flat seat area, the virtual paint ball is shown as being shot into the image and landing on the curved surface of the armrest, then the virtual paint is splashed over the curved surface of the armrest. In contrast, if the virtual paint ball is shown as being shot into the image and landing on the flat surface of the seat area, then the virtual paint is splashed over the flat surface of the seat area. In another example, the virtual long ribbon hangs over the curved surface of the armrest when it falls over the curved surface of the armrest, and the virtual long ribbon lies flat on the flat surface of the seat area when it falls over the flat seat area. Displaying the shape change of the virtual object according to the surface in the previously captured media object allows the shape change of the virtual object to occur automatically without further input (e.g., without requiring the user to provide input indicating the shape change of the virtual object), thereby enhancing the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0330] In some embodiments, when a first virtual object is displayed over at least a portion of a corresponding image in a first previously captured media object, the device detects (1112) a second user input (e.g., a horizontal swipe input on the first virtual object to show a previous or next item in a horizontally arranged list of media objects, a vertical swipe on the first virtual object to show a previous or next item in a vertically arranged list of media objects, or a tap on a forward or back button for switching to a previous or next media object) to switch from displaying the first previously captured media object to displaying a second previously captured media object, wherein the second previously captured media object includes one or more second images, and the second previously captured media object and second depth data corresponding to a second physical environment captured in each of the one or more second images (e.g., second depth data generated by one or more depth sensors of the device (e.g., emitter / detector systems such as infrared, sonar and / or lidar, and / or image analysis systems (e.g., video clip analysis and / or stereo image / video analysis)), for example, at a time corresponding to the time when the second media object was captured by one or more cameras) are recorded and stored together. For example, as described with reference to FIG. 7E to FIG. 7F As described above, when the virtual ball object is displayed on top of the first previously captured image, as Fig. 7EAs shown, detecting a request to switch from displaying a first previously captured image to displaying a second previously captured image (eg, an input at a subsequent media object control 7010) (eg, Figure 7F In response to receiving a second user request to switch from displaying a first previously captured media object to displaying a second previously captured media object: the device replaces display of the first previously captured media object with display of the second previously captured media object (e.g., sliding out the first previously captured media object and sliding in the second previously captured media object in the direction of the swipe input (e.g., a horizontal swipe or a vertical swipe input)). For example, in response to receiving an input (e.g., referring to Fig. 7E ), the device displays the first previously captured media object (such as Fig. 7E ) is switched to displaying a second previously captured image (as shown) in the media object display area 7002. Figure 7F The device displays a first virtual object over at least a portion of a corresponding image in a second previously captured media object, wherein the first virtual object is displayed as having at least a second position or orientation (or movement path), the second position or orientation being determined based on the first position or orientation (or movement path) of the first virtual object in the corresponding image in the first previously captured media object and based on second depth data corresponding to the corresponding image in the second previously captured media object. For example, adding Fig. 7E The virtual ball objects 7034 and 7044 of the first previously captured image displayed in FIG. Figure 7F. In some embodiments, if the first virtual object is a piece of virtual confetti or a virtual ball that has fallen downward in the image of the first previously captured media object and landed on a first surface in the image (e.g., a flat surface of a sofa seating area), when the user switches to displaying the second image by swiping horizontally on the first image, the second image slides horizontally, and the virtual confetti ball or the virtual ball begins to move from its position in the first image (e.g., falling downward from a position corresponding to the surface of the sofa seating area) and lands on a second surface (e.g., the surface of the floor or the surface of a cushion on the floor, etc.). In other words, the virtual object remains unchanged when switching between media objects, and the position, orientation, and movement path of the virtual object in the next image are affected by the position, orientation, and movement path of the virtual object in the previous image. Switching from displaying the virtual object over a first previously captured media object to displaying the virtual object over a second previously captured media object at a position determined based on the position or orientation of the virtual object in the first previously captured media object, without requiring user input (e.g., to indicate the position of the virtual object in the second previously captured media object), enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0331] In some embodiments, the first user request is (1114) to move a first type of virtual object (e.g., as referenced in FIG. 7C to FIG. 7EIn some embodiments, the first user request is a request to add multiple instances of the virtual ball objects 7034 and 7044) to a previously captured media object (e.g., adding falling virtual confetti or virtual balls to an image over time), and the first virtual object is one of the multiple instances of a first type of virtual object added to the first previously captured media object. In response to receiving a second user request to switch from displaying the first previously captured media object to displaying a second previously captured media object: the device displays a second virtual object above at least a portion of a corresponding image in the second previously captured media object, wherein: the second virtual object is an instance of the first type of virtual object that is different from the first virtual object and is not added to the first previously captured media object, and the second virtual object is displayed as having at least a third position or orientation (or movement path), and the third position or orientation is determined using second depth data corresponding to the corresponding image in the second previously captured media object. For example, in some embodiments, the first user request is a request to add a series of virtual objects of the same type over time (e.g., in a continuous manner) to create an effect on the image (such as falling confetti, raindrops, or fireworks, etc.). When the effect is applied to a first image or video (e.g., when multiple instances of virtual confetti, raindrops, or fireworks are added to the first image), if the user switches to the next image or video (e.g., by swiping horizontally or vertically on the first image or video), the effect is also automatically applied to the next image or video (e.g., new instances of virtual confetti, raindrops, or fireworks are also added to the next image) without the user explicitly invoking the effect for the next image or video (e.g., activating a control for the effect). Switching from displaying a first virtual object over a first previously captured media object to displaying a second virtual object over a second previously captured media object at a position determined based on the position or orientation of the virtual object in the first previously captured media object without requiring user input (e.g., to indicate the position of the virtual object in the second previously captured media object) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0332] In some embodiments, the first previously captured media object and the second previously captured media object are two different still images (e.g., as shown in FIG. 116 ) that were previously recorded and stored with different depth data corresponding to different physical environments and / or different views of the same physical environment. Fig. 7E The first previously captured image shown in the media object display area 7002 and the Figure 7FThe second previously captured image shown in the media object display area 7002 of the image). For example, the second still image need not have any connection to the first still image in terms of the material captured in the image so that the same effect (e.g., falling confetti, virtual balls, fireworks, virtual block letters, etc.) continues to be applied to the second still image. Switching from displaying the virtual object over the first previously captured media object to displaying the second virtual object over the second previously captured media object having different depth data than the first previously captured media object without requiring further user input (e.g., to indicate the location of the virtual object in the second previously captured media object) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0333] In some embodiments, the first previously captured media object is a video comprising a sequence of consecutive image frames (e.g., as shown in FIG. Figure 7AM to Figure 7ATThe method of claim 1114 wherein the first virtual object is displayed over at least a portion of a corresponding image in a first previously captured media object and wherein the first virtual object is displayed over at least a portion of a corresponding image in a first previously captured media object. The method of claim 1115 wherein the first virtual object is displayed over at least a portion of a corresponding image in a first previously captured media object and ... In normal or fast forward playback, the second image frame immediately follows the first image frame in the media object; in reverse playback of the media object, the second image frame immediately precedes the first image frame in the media object; or in looped playback of the media object, the second image is the initial frame of the media object and the first image is the last frame of the media object), displaying a first virtual object over a second portion of the second image frame, wherein the first virtual object is displayed as having a position or orientation (or movement path) determined based on the position or orientation (or movement path) of the first virtual object in the first image frame and based on a portion of the first depth data corresponding to the second image frame of the first previously captured media object. For example, if the first virtual object is a piece of virtual confetti or a virtual ball that has fallen down onto a surface (e.g., a surface of a moving or stationary object) in a first image frame of the video, then when the video continues to play and the surface is shown in a next image frame, the position and / or orientation and / or movement path of the virtual confetti or virtual ball will change based on the position and orientation of the surface in the new image frame. For example, if the surface is a stationary table surface, the virtual confetti will appear to be located at the same location on the stationary table surface, and the virtual ball will appear to roll along the stationary table surface, even though the table surface is now viewed from a different perspective and occupies a different area on the second image frame compared to the first image frame. Similarly, if the surface is the top of a trapdoor that suddenly opens in the video, the virtual confetti will begin to gradually fall from its stationary position on the top of the trapdoor, and the virtual ball will appear to accelerate from its position on the top of the trapdoor due to simulated gravity. In some embodiments, the first user request is a request to add a series of virtual objects of the same type over time (e.g., in a continuous manner) to create an effect on the image (such as falling confetti or fireworks, etc.). For example, in Figure 7AM to Figure 7ATAs video playback occurs in the video, the virtual confetti object resting on the edge of the physical kite object 7112 is displayed as having a changed position, orientation, and movement path. When the effect is applied to the first image frame (e.g., when multiple instances of virtual confetti or fireworks are added to the first image frame), the effect is also automatically applied to the next image frame as video playback continues (e.g., new instances of virtual confetti or fireworks are also added to the next image frame). In some embodiments, at the end of the video, the virtual objects added to the last image frame include virtual objects that were added to multiple earlier image frames and have been stabilized to their final position and orientation in the final image frame based on the following previous interactions: previous interactions with simulated surfaces corresponding to the physical environment depicted in the earlier image frames and simulated surfaces corresponding to the physical environment depicted in the last image frame. Displaying a virtual object over a second frame of video displayed immediately after displaying a first frame of the video, wherein the position or orientation of the virtual object is determined in the second image frame using depth data from the second image frame without requiring further input (e.g., without requiring the user to provide input indicating the position of the virtual object in each frame of the video), enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0334] In some embodiments, displaying the first previously captured media object includes (1120) playing the video according to a first timeline, the first timeline including at least one of looping, fast forwarding, or reversing a sequence of consecutive image frames; and displaying the first virtual object over at least a portion of a corresponding image in the first previously captured media object includes: during playback of the video according to the first timeline, displaying changes in the position or orientation (or movement path) of the first virtual object according to a forward timeline (e.g., the virtual object's previous position and orientation in each previously displayed image frame affects the position and orientation in the currently displayed image frame), the forward timeline being associated with an actual order of the sequence of image frames displayed during playback of the video (e.g., including: looping from the end to the beginning of the video, switching frames at an uneven rate during video playback, playing the video backwards from a later frame to an earlier frame, etc.). In other words, the timeline of the virtual object's movement in the displayed image frames is independent of the timeline according to which the media object is played back. Displaying changes in the position or orientation of a virtual object according to a timeline associated with the order of the sequence of image frames without requiring further input (e.g., without requiring the user to provide input indicating the change in the position of the virtual object in each frame of the video) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0335] In some embodiments, displaying a first virtual object over at least a portion of a corresponding image in a first previously captured media object includes (1122): when the first virtual object is placed over a corresponding image in the one or more first images, displaying the first virtual object according to a shadow of a first physical surface captured in the first previously captured media object, wherein the shadow of the first virtual object is constrained by a first simulated surface corresponding to the first physical surface and determined based on the first depth data. For example, if the still image captures a sofa having a curved armrest and a flat seating area, a virtual letter A placed on the curved surface of the armrest and then dropped sideways onto the flat surface of the seating area will have a shadow, the shape of which changes based on the surface on which the virtual letter A is currently located and the current orientation of the virtual letter A relative to the surface. In some embodiments, a three-dimensional or quasi-three-dimensional mesh is generated based on depth data associated with an image or video, the mesh surface exhibiting shape features of the physical environment captured in the image or video, and the shadow is projected on the mesh surface based on a simulated light source and the position and orientation of the virtual object relative to the mesh surface. Displaying virtual objects with shadows constrained by simulated surfaces corresponding to physical surfaces and based on depth data of previously captured media objects without requiring further input (e.g., to identify surfaces in previously captured media objects) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0336] In some embodiments, the first user request is (1124) a user request to place a virtual first text object at a first position in a corresponding image in a first previously captured media object. For example, the user request is an input provided at the virtual text object 7064 to initiate an editing mode for the virtual text object 7064, as described with reference to Figure 7Q to Figure 7SThe device receives (1126) user input to update a virtual first text object, including adding a first virtual letter to the virtual first text object (e.g., editing a text input area by typing a letter at the end of an existing text input); and in response to receiving the user input, the device displays the first virtual letter at a second position in a corresponding image in a first previously captured media object adjacent to a previous virtual letter in the virtual first text object and according to a portion of the first depth data corresponding to the second position in the corresponding image. In some embodiments, the text has lighting and shadows generated based on a surface mesh of an environment captured in the corresponding image. Displaying a text object at a position in a previously captured media object and placing the letters of the text object according to the depth data in the media object without requiring further input (e.g., to identify the depth data in the previously captured media object) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0337] In some embodiments, displaying the first virtual object above at least a portion of the corresponding image in the first previously captured media object includes (1128): based on determining that a simulated surface adjacent to a current position of the first virtual object in the corresponding image is a horizontal surface, displaying the first virtual object above the horizontal surface (e.g., as opposed to below the horizontal surface); and based on determining that the simulated surface adjacent to the current position of the first virtual object in the corresponding image is a vertical surface, displaying the first virtual object in front of the vertical surface. Displaying the virtual object on top of or in front of a surface adjacent to the virtual object based on whether the surface is a horizontal surface or a vertical surface without requiring further input (e.g., to indicate whether the surface is a horizontal surface or a vertical surface) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0338] In some embodiments, displaying a first virtual object above at least a portion of a corresponding image in a first previously captured media object includes (1130): based on determining that the corresponding image includes a first simulated surface (e.g., a foreground object) and a second simulated surface (e.g., a background object) that are adjacent to a current position of the first virtual object in the corresponding image and have different depths, displaying the first virtual object at a depth between the first simulated surface and the second simulated surface (e.g., where at least a first portion of the first virtual object is obscured by the first simulated surface, and where at least a second portion of the first virtual object obscures at least a portion of the second simulated surface, or is below an object represented by the first simulated surface). For example, as described with reference to Figure 7G As described, a virtual ball object 7045 is displayed at a depth between a first simulated surface (e.g., the back wall of a room depicted in a previously captured image) and a second simulated surface (e.g., table 7054). In some embodiments, a complete three-dimensional model of a physical environment cannot be established based solely on the depth data of the image. There is no spatial information about the space between the first simulated surface and the second simulated surface. The first virtual object is placed between the first simulated surface and the second simulated surface, regardless of the absence of spatial information within the depth range between the first simulated surface and the second simulated surface. Displaying a virtual object having a depth between the first simulated surface and the second simulated surface without further input (e.g., to indicate the depth of the virtual object, the depth of the first simulated surface, and / or the depth of the second simulated surface) enhances the operability of the device. Performing operations without further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0339] In some embodiments, the device detects (1132) an object positioning input (e.g., a drag input or a tap input) directed to the first virtual object through contact on the touch-sensitive surface, wherein the object positioning input specifies a placement position of the first virtual object on the corresponding image, and the placement position corresponds to the final position (and optionally, the final movement speed and direction) of the contact on the touch-sensitive surface. For example, the object positioning input is an input for positioning the virtual text object 7064 through contact 7066, as shown in reference Figure 7MIn response to detecting the object positioning input, the device places the first virtual object at the placement location on the corresponding image according to the object positioning input. When the first virtual object is displayed at the placement location on the corresponding image, the device detects termination of the object positioning input, including detecting the lifting of the contact from the touch-sensitive surface. In response to detecting termination of the object positioning input, the device moves the first virtual object from the placement location to a final position (e.g., using rotation and / or translation movement) based on depth data corresponding to a portion of the physical environment surrounding the placement location (e.g., a surface mesh of the physical environment created based on depth data near the placement location) and based on the portion of the physical environment and one or more simulated physical properties of the first virtual object (e.g., weight, gravity, elasticity, plasticity, rate, speed, etc.). Moving the virtual object according to the depth data without the need for further input (e.g., to indicate how the movement of the virtual object will occur and / or to indicate depth data of a previously captured media image) enhances the operability of the device. For example, when the contact 5066 is lifted off, the virtual text object is placed at the final position on the horizontal surface of the sofa 7060 in the previously captured image, as shown in reference Figure 7M to Figure 7P Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0340] In some embodiments, the first virtual object includes (1134) a simulated spotlight (e.g., virtual spotlight object 7118, as shown in FIG. Figure 7AU Displaying the first virtual object over at least a portion of the corresponding image in the first previously captured media object includes displaying a simulated light beam (e.g., light beam 7120) having a simulated three-dimensional shape (e.g., a conical or cylindrical light beam) in the corresponding image, wherein the simulated three-dimensional shape remains constant as the simulated light beam moves in the corresponding image (e.g., the appearance of the shape may be different to a user depending on the viewing angle of the light beam from the user's vantage point); and displaying a simulated lighting point (e.g., lighting point 7122) having a two-dimensional shape in the corresponding image, wherein the two-dimensional shape changes according to the movement of the simulated light beam in the corresponding image and according to a relative spatial relationship between the simulated light beam and a simulated intersecting surface corresponding to a physical surface in the physical environment (e.g., a surface of a physical object (e.g., a curved or flat surface) and determined based on the first depth data. For example, as described with reference to Figure 7AV to Figure 7AWAs described, the shape of the simulated lighting point 7122 changes as the virtual spotlight object 7118 moves in response to user input. Displaying a simulated lighting point having a two-dimensional shape that changes according to the following without further input (e.g., to indicate depth data of a previously captured media image) enhances the operability of the device: the movement of the simulated light beam, and the relative spatial relationship between the simulated light beam and the simulated surface corresponding to the surface determined based on the depth data. Performing operations without further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0341] In some embodiments, the first virtual object includes (1136) a graphical object (e.g., a two-dimensional or three-dimensional emoticon), and displaying the first virtual object over at least a portion of the corresponding image in the first previously captured media object includes: displaying the graphical object at a location in the corresponding image that corresponds to free space in the physical environment (e.g., the emoticon floats in the air at a location in the physical environment selected by contact of a user's finger). For example, the graphical object is as described in reference to Figure 7AB The virtual emoticon object 7098. Displaying the graphical object at a location corresponding to free space in the physical environment (e.g., captured in a media image) without requiring further input (e.g., to indicate the location of the free space in the previously captured media image) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0342] In some embodiments, the device detects (1138) two concurrent contacts on the touch-sensitive surface, and in response to detecting the two concurrent contacts, the device displays a first measurement object having two respective ends, the two respective ends being located at respective positions on the respective image corresponding to the respective positions of the two concurrent contacts on the touch-sensitive surface. For example, in response to detecting concurrent contacts 7132 and 7134, a virtual measurement object 7136 is displayed having ends located at positions corresponding to the positions of the contacts, as shown in FIG. Figure 7AY to Figure 7AZ Displaying a measurement object having an end located at a position on the image corresponding to the position of the contact provides visual feedback to the user (e.g., indicating a portion of the image corresponding to the measurement). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately determine and / or adjust the portion of the image corresponding to the measurement), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0343] In some embodiments, displaying the first measurement object includes (1140): based on determining that two corresponding ends of the first measurement object are located on both sides of a simulated foreground surface corresponding to a physical surface captured in a corresponding image (e.g., the first measurement object spans the simulated foreground surface), displaying a first portion of the first measurement object located on a first side of the simulated foreground surface and having a first appearance, and displaying a second portion (different from the first portion) of the first measurement object located on a second side (different from the first side) of the simulated foreground surface and having a second appearance (different from the first appearance) (e.g., the first side of the measurement object is foreground and fully visible, and the second side of the measurement object is obscured by the foreground surface and has a shadow appearance). For example, in Figure 7BD , a first portion of the measurement object 7148 (e.g., the dashed portion) is located on a first side of the simulated foreground surface, and a second portion of the measurement object 7148 (e.g., the solid portion) is located on a second side of the simulated foreground surface. Displaying the first portion of the first measurement object located on the first side of the simulated foreground surface and having a first appearance, and displaying the second portion of the first measurement object located on the second side of the simulated foreground surface and having a second appearance, provides visual feedback to the user (e.g., indicating that the measurement object spans different sides of the simulated surface corresponding to the physical surface in the captured image). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by allowing the user to quickly and accurately determine and / or adjust the portion of the image corresponding to the measurement), which in addition reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0344] In some embodiments, displaying the first measurement object includes (1142): when displaying the first measurement object having a first end and a second end in a corresponding image, detecting a positioning input performed by contact on the touch-sensitive surface, wherein the positioning input specifies a new end position of the first measurement object in the corresponding image; and in response to detecting the positioning input performed by the contact, moving one of the first end and the second end of the first measurement object that is closer to the new end position to the new end position in the corresponding image specified by the positioning input. For example, as referenced Fig.7BF As described above, the positioning input performed by contact 7152 specifies the new end position of the previously displayed measurement object 7148, as shown in reference Figure 7BB to Figure 7BEMoving one of the first end and the second end of the first measurement object to a new end position without requiring further input (e.g., canceling a previous measurement input to provide a new measurement input) enhances the operability of the device. Performing operations without requiring further user input makes the user-device interface more efficient, which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0345] It should be understood that FIG. 11A to FIG. 11F The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 900, 1000, and 1200) also apply in a similar manner to the details of the above with respect to FIG. 11A to FIG. 11F The method 1200 described. For example, the contacts, inputs, annotations, virtual objects, physical objects, user interface areas, fields of view, movements, and / or animations described above with reference to the method 1100 optionally have one or more of the features of the contacts, inputs, annotations, virtual objects, physical objects, user interface areas, fields of view, movements, and / or animations described herein with reference to other methods described herein (e.g., methods 900, 1000, and 1200). For the sake of brevity, these details are not repeated here.

[0346] FIG. 12A to FIG. 12D 1 is a flow chart illustrating a method 1200 for initiating a shared annotation session. The method 1200 is performed on an electronic device (e.g., a display, a projector, a heads-up display, etc.) having a first display generating component (e.g., a display, a projector, a heads-up display, etc.), a first set of one or more in...

Claims

1. A method comprising: At a computer system in communication with a display generating component and one or more sensors for detecting user input, the following operations are performed: receiving a request to display a first virtual effect with the view of the physical environment while the view of the physical environment is visible via the display generation component; as well as In response to detecting the request to display the first virtual effect with the view of the physical environment, displaying one or more virtual objects overlaid on the view of the physical environment, comprising: displaying respective animated movements of the one or more virtual objects on at least a portion of the view of the physical environment, wherein the respective animated movements are constrained according to a direction of simulated gravity associated with the view of the physical environment; constraining the corresponding animated movement of a first virtual object among the one or more virtual objects according to the first surface detected in the view of the physical environment based on determining that a current position of the first virtual object during the corresponding animated movement of the first virtual object corresponds to a first surface detected in the view of the physical environment: and Based on determining that a current position of a second virtual object among the one or more virtual objects during the corresponding animated movement of the second virtual object corresponds to a second surface detected in the view of the physical environment, wherein the second surface is lower than the first surface in the direction of the simulated gravity, the corresponding animated movement of the second virtual object is constrained according to the second surface detected in the view of the physical environment.

2. The method of claim 1 , wherein displaying one or more virtual objects overlaid on the view of the physical environment comprises: Continue to add additional instances of the first type of virtual objects overlaid on the view of the physical environment, wherein when a first plurality of virtual objects of the first type have been gathered near the first surface in the view of the physical environment and a second plurality of virtual objects of the first type have been gathered near the second surface in the view of the physical environment, the additional instances of the first type of virtual objects move in the direction of simulated gravity associated with the view of the physical environment.

3. The method according to claim 1, comprising: while displaying the respective animated movements of the one or more virtual objects on at least a portion of the view of the physical environment, detecting a first event that changes the view of the physical environment from a first view of a first physical environment to a second view of a second physical environment that is different from the first view of the first physical environment; as well as In response to detecting the first event: ceasing to display the first view of the first physical environment; and The respective animated movements of the one or more virtual objects are displayed on at least a portion of the second view of the second physical environment.

4. The method according to claim 1, comprising: concurrently displaying, while the view of the physical environment is visible via the display generation component, a first control corresponding to the first virtual effect and a second control corresponding to a second virtual effect different from the first virtual effect, wherein receiving the request to display the first virtual effect with the view of the physical environment comprises detecting a first user input selecting the first control; as well as Display of the second control corresponding to the second virtual effect is maintained while the corresponding animated movement of the one or more virtual objects is displayed on at least the portion of the view of the physical environment.

5. The method according to claim 4, comprising: While displaying the respective animated movements of the one or more virtual objects on at least the portion of the view of the physical environment and maintaining display of the second control corresponding to the second virtual effect, detecting a second user input selecting the second control; and In response to detecting the second user input selecting the second control: ceasing to display the one or more virtual objects on at least the portion of the view of the physical environment; and A corresponding virtual object of the second virtual effect is displayed on at least the portion of the view of the physical environment.

6. The method of claim 1 , wherein displaying the corresponding animated movement of the one or more virtual objects comprises: Animated movement of two or more virtual objects at different depths relative to the view of the physical environment is displayed. 7 . The method of claim 1 , wherein the view of the physical environment comprises a view of one or more physical objects moving in the physical environment.

8. The method of claim 7, wherein displaying the corresponding animated movement of the one or more virtual objects comprises: At least one of the one or more virtual objects is moved according to the movement of the one or more physical objects moving in the physical environment.

9. A computer system comprising: Display generated components; one or more sensors for detecting user input; one or more processors; as well as A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for: receiving a request to display a first virtual effect with the view of the physical environment while the view of the physical environment is visible via the display generation component; as well as In response to detecting the request to display the first virtual effect with the view of the physical environment, displaying one or more virtual objects overlaid on the view of the physical environment, comprising: displaying respective animated movements of the one or more virtual objects on at least a portion of the view of the physical environment, wherein the respective animated movements are constrained according to a direction of simulated gravity associated with the view of the physical environment; constraining the corresponding animated movement of a first virtual object among the one or more virtual objects according to the first surface detected in the view of the physical environment based on determining that a current position of the first virtual object during the corresponding animated movement of the first virtual object corresponds to a first surface detected in the view of the physical environment; and Based on determining that a current position of a second virtual object among the one or more virtual objects during the corresponding animated movement of the second virtual object corresponds to a second surface detected in the view of the physical environment, wherein the second surface is lower than the first surface in the direction of the simulated gravity, the corresponding animated movement of the second virtual object is constrained according to the second surface detected in the view of the physical environment.

10. The computer system of claim 9, wherein the one or more programs include instructions for executing the method of any one of claims 2 to 8.

11. A computer program comprising instructions which, when executed by a computer system in communication with a display generating component and one or more sensors for detecting user input, cause the computer system to: receiving a request to display a first virtual effect with the view of the physical environment while the view of the physical environment is visible via the display generation component; as well as In response to detecting the request to display the first virtual effect with the view of the physical environment, displaying one or more virtual objects overlaid on the view of the physical environment, comprising: displaying respective animated movements of the one or more virtual objects on at least a portion of the view of the physical environment, wherein the respective animated movements are constrained according to a direction of simulated gravity associated with the view of the physical environment; constraining the corresponding animated movement of a first virtual object among the one or more virtual objects according to the first surface detected in the view of the physical environment based on determining that a current position of the first virtual object during the corresponding animated movement of the first virtual object corresponds to a first surface detected in the view of the physical environment; and Based on determining that a current position of a second virtual object among the one or more virtual objects during the corresponding animated movement of the second virtual object corresponds to a second surface detected in the view of the physical environment, wherein the second surface is lower than the first surface in the direction of the simulated gravity, the corresponding animated movement of the second virtual object is constrained according to the second surface detected in the view of the physical environment.

12. Computer program according to claim 11, wherein the one or more programs comprise instructions which, when executed by the computer system, cause the computer system to perform the method according to any one of claims 2 to 8.

Images