Device, method and graphical user interface for unified annotation layer

Abstract

Embodiments of the present disclosure relate to devices, methods, and graphical user interfaces for unifying annotation layers. The present invention discloses an electronic device that displays a first user interface of a first application. The first application is a corresponding application among a plurality of applications, and the first user interface includes first application content. While displaying the first user interface, and in response to detecting a first input, the device continues to display the first application content and displays an annotation layer above at least a first portion of the first user interface. The annotation layer is associated with a user application that is different from the plurality of applications. While displaying the annotation layer above at least the first portion, and in response to detecting a second input, the device adds a representation of at least a second portion of the first user interface to the annotation layer. While displaying the representation, the device detects a set of one or more additional inputs, and in response, adds annotations to the annotation layer.

Description

[0001] This application is a divisional application of an invention patent application with an application date of August 31, 2017, which entered the Chinese national phase on February 28, 2019, with Chinese national application number 201780053635.X, and invention name "Device, method and graphical user interface for annotating a unified annotation layer to content displayed on a device". Technical Field

[0002]

[0013] This document generally relates to electronic devices with touch-sensitive displays, including but not limited to electronic devices with touch-sensitive displays for manipulating a user interface by adding annotations to a unified annotation layer. Background Art

[0003] The use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has grown significantly in recent years. Examples of touch-sensitive surfaces include touchpads and touch screen displays. Such surfaces are widely used to manipulate user interface objects on displays.

[0004] The user interface can be manipulated by adding annotations. The conventional method of adding annotations to the user interface is cumbersome and inefficient. Summary of the invention

[0005] Therefore, electronic devices need to have faster, more efficient methods and interfaces for adding annotations to user interfaces. Such methods and interfaces alternatively supplement or replace conventional methods for adding annotations to user interfaces. Such methods and interfaces reduce the amount, degree, and / or nature of input from the user and produce a more efficient human-computer interface. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0006] The above-mentioned defects and other problems associated with the user interface of an electronic device with a touch-sensitive surface can be reduced or eliminated by means of the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the device is a personal electronic device (e.g., a wearable electronic device, such as a watch). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI mainly through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, presentation, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, note-taking and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0007] According to some embodiments, a method is performed at an electronic device having a touch-sensitive display and one or more sensors for detecting signals from a stylus associated with the device. The method includes: displaying a first user interface of a first application. The first application is a corresponding application in a plurality of applications, and the first user interface includes first application content. The method also includes, while displaying the first user interface of the first application, detecting a first input; and in response to detecting the first input, continuing to display the first application content, and in addition, displaying an annotation layer above at least a first portion of the first user interface. The annotation layer is associated with a user application different from the plurality of applications. The method also includes, while displaying the annotation layer above at least a first portion of the first user interface, detecting a second input; in response to detecting the second input, adding a representation of at least a second portion of the first user interface to the annotation layer; while displaying the representation in the annotation layer, detecting a set of one or more additional inputs; and in response to detecting the set of one or more additional inputs, adding annotations to the annotation layer.

[0008] According to some embodiments, an electronic device includes a touch-sensitive display unit configured to display a first user interface of a first application and receive user input; and a processing unit coupled to the touch-sensitive display unit. In some embodiments, the electronic device includes one or more sensor units (e.g., coupled to the processing unit) for detecting signals from a stylus associated with the device. The processing unit is configured to enable the display of a first user interface of a first application. The first application is a corresponding application among multiple applications, and the first user interface includes first application content. The processing unit is also configured to, when displaying the first user interface of the first application, detect a first input; and in response to detecting the first input, enable continued display of the first application content, and in addition, enable display of an annotation layer above at least a first portion of the first user interface. The annotation layer is associated with a user application different from the multiple applications. The processing unit is further configured to, when displaying the annotation layer above at least a first portion of the first user interface, detect a second input; in response to detecting the second input, add a representation of at least a second portion of the first user interface to the annotation layer; when displaying the representation in the annotation layer, detect a set of one or more additional inputs; and in response to detecting a set of one or more additional inputs, add annotations to the annotation layer.

[0009] According to some embodiments, an electronic device includes: a touch-sensitive display, one or more alternative sensors for detecting signals from a stylus associated with the device, one or more processors, a memory, and one or more programs; the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, a computer-readable storage medium stores therein instructions that, when executed by an electronic device having a touch-sensitive display and one or more alternative sensors for detecting signals from a stylus associated with the device, cause the device to perform the operations of any of the methods described herein or cause the operations of any of the methods described herein to be performed. According to some embodiments, a graphical user interface on an electronic device having a touch-sensitive display, one or more alternative sensors for detecting signals from a stylus associated with the device, a memory, and one or more processors executing one or more programs stored in the memory includes one or more elements displayed in any of the methods described herein, and the one or more elements are updated in response to input, as described in any of the methods described herein. According to some embodiments, an electronic device includes: a touch-sensitive display; one or more alternative sensors for detecting signals from a stylus associated with the device; and means for performing the operations of any of the methods described herein or causing the operations of any of the methods described herein to be performed. According to some embodiments, an information processing device for use in an electronic device having a touch-sensitive display and one or more alternative sensors for detecting signals from a stylus associated with the device includes means for performing the operations of any of the methods described herein or causing the operations of any of the methods described herein to be performed.

[0010] Thus, a faster, more efficient method and interface for adding annotations to a user interface using a unified annotation layer is provided for an electronic device having a touch-sensitive display and an alternative one or more sensors for detecting signals from a stylus associated with the device, thereby improving the effectiveness, efficiency, and user satisfaction of such devices. Such methods and interfaces may supplement or replace conventional methods for adding annotations to a user interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0012] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0013] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0014] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0015] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0016] Figure 4 is a block diagram of an exemplary electronic stylus according to some embodiments.

[0017] FIG. 5A to FIG. 5B The position state of the stylus relative to the touch-sensitive surface according to some embodiments is shown.

[0018] Fig. 6A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0019] Figure 6B An example user interface is shown for a multifunction device having a touch-sensitive surface that is separate from the display according to some embodiments.

[0020] Figure 7A-7RRR An exemplary user interface for adding annotations to an annotation layer is shown according to some embodiments.

[0021] Fig. 8A is a block diagram of an annotation application according to some embodiments.

[0022] Figure 8B is a block diagram illustrating exemplary components for a preserved annotation layer according to some embodiments.

[0023] Figure 9A-9E is a flow chart illustrating a method of adding annotations to an annotation layer according to some embodiments.

[0024] Fig.10 is a functional block diagram of an electronic device according to some embodiments. DETAILED DESCRIPTION

[0025] Many electronic devices have multiple applications for viewing and editing documents. Some of these applications allow users to add annotations to documents. However, existing methods for annotating documents are slow and inefficient. For example, different applications use different methods to annotate documents and / or save annotated documents and annotations. In addition, for documents associated with applications that do not allow annotations to be added, a separate application is required to annotate. In addition, annotations added to documents in one application may not be saved for display or editing in other applications. The following embodiments address these problems by providing a method that allows annotations to be added to a unified annotation layer that operates in conjunction with multiple applications and is controlled by the annotation application independently of the multiple applications.

[0026] under, Figure 1A to Figure 1B , Figure 2 , Figure 3 , Figure 4 , Figure 5A-Figure 5B as well as FIG. 6A to FIG. 6B A description of an exemplary apparatus is provided. Figure 7A-7RRR An exemplary user interface for adding annotations to an annotation layer is shown. Figure 8A-8B is an exemplary block diagram of an annotation application and associated annotation layer. Figure 9A-9E is a flow chart illustrating a method of adding annotations to an annotation layer. Figure 7A-7RRR The user interface in Figure 9A-9E process.

[0027] Exemplary Devices

[0028] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0029] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact, unless the context clearly indicates otherwise.

[0030] The terms used in the description of various described embodiments in this article are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" and / or "comprising" are used in this specification to specify the presence of stated features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0031] As used herein, the term "if" is optionally interpreted to mean "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0032] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple iPod from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0033] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0034] The device typically supports a variety of applications, such as one or more of the following applications: a note-taking application, a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital camcorder application, a web browsing application, a digital music player application, and / or a digital video player application.

[0035] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0036] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A 1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0037] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface may optionally be interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0038] It should be understood that device 100 is merely one example of a portable multifunction device, and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0039] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory 102 by other components of the device 100 (such as one or more CPUs 120 and peripheral device interface 118) is optionally controlled by a memory controller 122.

[0040] Peripheral interface 118 may be used to couple the device's input and output peripherals to memory 102 and one or more CPUs 120. The one or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0041] In some embodiments, peripherals interface 118, one or more CPUs 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0042] RF (radio frequency) circuit 108 receives and transmits RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, user identity modules (SIM) cards, memory, and the like. RF circuit 108 optionally communicates with networks and other devices through wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communications protocol including communications protocols that have not yet been developed on the date of filing of this document.

[0043] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0044] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0045] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some visual outputs or all of the visual outputs correspond to user interface objects. As used herein, the term "indicator" refers to a user-interactive graphical user interface object (e.g., a graphical user interface object configured to respond to input directed to a graphical user interface object). Examples of user-interactive graphical user interface objects include, but are not limited to, buttons, sliders, icons, selectable menu items, switches, hyperlinks, or other user interface controls.

[0046] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or sensor group that accepts input from a user based on tactile and / or haptic contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary embodiment, the point of contact between the touch-sensitive display system 112 and the user corresponds to a finger or stylus of the user.

[0047] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0048] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0049] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0050] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0051] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some embodiments, an optical sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 on the front of the device, so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device to obtain an image of the user (e.g., for self-portraits, for video conferencing when the user is watching other video conference participants on the touch screen, etc.).

[0052] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to a force sensor controller 159 in the I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 located on the front of the device 100.

[0053] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0054] Device 100 optionally also includes one or more tactile output generators 163. Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. One or more tactile output generators 163 optionally include one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components for converting electrical signals into tactile outputs on the device). The tactile output generator 163 receives tactile feedback generation instructions from the tactile feedback module 133 and generates tactile outputs on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates tactile outputs by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front portion of the device 100 .

[0055] The device 100 optionally also includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (eg, as part of an inertial measurement unit (IMU)) for obtaining information about the position (eg, attitude) of the device. Figure 1A Sensors such as accelerometer 167, gyroscope 168, and magnetometer 169 are shown coupled to peripheral device interface 118. Alternatively, sensors such as accelerometer 167, gyroscope 168, and magnetometer 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from the one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location and orientation (e.g., portrait or landscape) of device 100.

[0056] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a position module (or instruction set) 131, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, such as in Figure 1A and Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0057] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0058] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a device compatible with some of Apple Inc. (Cupertino, California). iPod A multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in iPod devices. In some embodiments, the external port is compatible with some of Apple Inc. (Cupertino, California) iPod A Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in iPod devices.

[0059] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes various software components for performing various operations related to contact detection (e.g., by a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement across the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has stopped (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or a stylus contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0060] The contact / motion module 130 optionally detects gesture input from a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0061] In conjunction with accelerometer 167, gyroscope 168, and / or magnetometer 169, position module 131 optionally detects position information about the device, such as the attitude (roll, pitch, and / or yaw) of the device in a particular reference frame. Position module 130 includes software components for performing various operations related to detecting device position and detecting changes in device position. In some embodiments, position module 131 uses information received from a stylus used with the device to detect position information about the stylus, such as detecting a position state of the stylus relative to the device and detecting changes in the position state of the stylus.

[0062] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0063] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0064] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 163 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0065] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts module 137, email client module 140, IM 141, browser module 147, and any other application requiring text input).

[0066] The GPS module 135 determines the location of the device and provides such information for use in various applications (e.g., to the phone module 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and maps / navigation widgets).

[0067] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0068] Contacts module 137 (sometimes called address book or contact list);

[0069] Telephone module 138;

[0070] Video conferencing module 139;

[0071] Email client module 140;

[0072] Instant messaging (IM) module 141;

[0073] Fitness support module 142;

[0074] A camera module 143 for still images and / or video images;

[0075] Image management module 144;

[0076] Browser module 147;

[0077] Calendar module 148;

[0078] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, and a user-created widget 149-6;

[0079] A widget creator module 150 for forming a user-created widget 149-6;

[0080] Search module 151;

[0081] A video and music player module 152, optionally consisting of a video player module and a music player module;

[0082] Notepad module 153;

[0083] Map module 154;

[0084] Online video module 155; and / or

[0085] An annotation application 195 for providing annotations to the user interface and optionally storing and / or accessing saved annotations 196 in the memory 102 .

[0086] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0087] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating telephone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing telephone numbers and / or email addresses to initiate and / or facilitate communications via the telephone module 138, video conferencing module 139, email client module 140, or instant messaging module 141; and the like.

[0088] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a sequence of characters corresponding to a phone number, one or more phone numbers in contact module 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0089] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0090] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0091] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs or IMPS).

[0092] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, fitness support module 142 includes executable instructions for creating a workout (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0093] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0094] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0095] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0096] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0097] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0098] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0099] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0100] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0101] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notepads, to-do lists, etc. according to user instructions.

[0102] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0103] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch-sensitive display system 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0104] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the above modules and data structures. In addition, memory 102 optionally stores other modules and data structures not described above.

[0105] In some embodiments, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0106] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0107] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0108] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event distributor module 174. In some embodiments, the application 136-1 includes an application internal state 192, which indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0109] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0110] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0111] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0112] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0113] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0114] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0115] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it is identified as the hit view.

[0116] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with one particular view, higher views in the hierarchy will still remain as actively participating views.

[0117] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receivers 182.

[0118] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0119] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring within a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit (not shown) or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in the application views 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0120] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0121] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0122] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), and other events. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0123] In some embodiments, event definition 186 includes a definition of an event for a corresponding user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and the object that triggered the hit test.

[0124] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0125] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0126] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0127] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0128] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.

[0129] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 177 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0130] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0131] It should be understood that the above discussion of event processing for user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining an event to be recognized.

[0132] Figure 2 A device having a touch screen (e.g., Figure 1AThe portable multifunction device 100 of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In this embodiment and other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, when the user breaks contact with the one or more graphics, selection of the one or more graphics will occur. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with the graphics will not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0133] The device 100 optionally also includes one or more physical buttons, such as a "home" button, or a menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0134] In some embodiments, the device 100 includes a touch screen display, a menu button 204, a push button 206 for powering the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output for a user of the device 100.

[0135] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 163), sensors 359 (e.g., similar to those described above) Figure 1A The touch-sensitive display system 112, optical sensor 164, contact force sensor 165, proximity sensor 166, accelerometer 167, gyroscope 168 and magnetometer 169 are touch-sensitive sensors, optical sensors, contact force sensors, proximity sensors, acceleration sensors, gesture and / or magnetic sensors). The memory 370 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) stored in memory 102 of portable multifunction device 100, or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0136] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device. Each module in the above-mentioned identified module corresponds to a group of instructions for performing the above-mentioned functions. The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 370 optionally stores the subgroup of the above-mentioned modules and data structures. In addition, memory 370 optionally stores additional modules and data structures not described above.

[0137] Figure 4 is a block diagram of an exemplary electronic stylus 203 according to some embodiments. Electronic stylus 203 is sometimes referred to simply as a stylus. Stylus 203 includes memory 402 (which optionally includes one or more computer-readable storage media), memory controller 422, one or more processing units (CPUs) 420, peripheral device interface 418, RF circuit 408, input / output (I / O) subsystem 406, and other input or control devices 416. Stylus 203 optionally includes external port 424 and one or more optical sensors 464. Stylus 203 optionally includes one or more intensity sensors 465 for detecting the intensity of contact of stylus 203 on device 100 (e.g., when stylus 203 is used with a touch-sensitive surface such as touch-sensitive display system 112 of device 100) or on other surfaces (e.g., a table surface). Stylus 203 optionally includes one or more tactile output generators 463 for generating tactile output on stylus 203. These components optionally communicate via one or more communication buses or signal lines 403 .

[0138] In some embodiments, the term "tactile output" discussed above refers to a physical displacement of an accessory (e.g., stylus 203) of a device (e.g., device 100) relative to a previous position of the accessory, a physical displacement of a component of the accessory relative to another component of the accessory, or a displacement of a component relative to the center of mass of the accessory that will be detected by a user by virtue of the user's sense of touch. For example, in the case where an accessory or a component of the accessory is in contact with a touch-sensitive surface of a user (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation that corresponds to a perceived change in a physical characteristic of the accessory or component of the accessory. For example, movement of a component (e.g., the housing of stylus 203) is optionally interpreted by the user as a "click" on a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "click", even when a physical actuation button associated with the stylus that is physically pressed (e.g., displaced) by the user's movement does not move. Although such user interpretations of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Thus, when a tactile output is described as corresponding to a particular perception of a user (e.g., a “click”), unless otherwise stated, the tactile output generated corresponds to a physical displacement of the device or a component thereof that would generate that perception for a typical (or average) user.

[0139] It should be understood that stylus 203 is merely one example of an electronic stylus, and that stylus 203 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 4 The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0140] Memory 402 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more flash memory devices or other non-volatile solid-state memory devices. Access to memory 402 by other components of stylus 203 (such as one or more CPUs 420 and peripheral device interface 418) is optionally controlled by memory controller 422.

[0141] Peripheral device interface 418 can be used to couple the input peripherals and output peripherals of the stylus to one or more CPUs 420 and memory 402. The one or more processors 420 run or execute various software programs and / or instruction sets stored in memory 402 to perform various functions of stylus 203 and process data.

[0142] In some embodiments, peripherals interface 418, one or more CPUs 420, and memory controller 422 are optionally implemented on a single chip, such as chip 404. In some other embodiments, they are optionally implemented on separate chips.

[0143] RF (radio frequency) circuit 408 receives and sends RF signals, also known as electromagnetic signals. RF circuit 408 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with device 100 or device 300, a communication network, and / or other communication devices via electromagnetic signals. RF circuit 408 optionally includes well-known circuits for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, user identity modules (SIM) cards, memory, and the like. RF circuit 408 optionally communicates with networks and other devices through wireless communications, such as the Internet (also known as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). The wireless communication optionally uses any of a variety of communication standards, protocols and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence Utilizing Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS)), and / or Short Message Service (SMS), or any other suitable communications protocol including communications protocols that have not yet been developed on the date of filing of this document.

[0144] The I / O subsystem 406 couples input / output peripherals on the stylus 203, such as other input or control devices 416, to a peripheral device interface 418. The I / O subsystem 406 optionally includes an optical sensor controller 458, an intensity sensor controller 459, a tactile feedback controller 461, and one or more input controllers 460 for other input or control devices. The one or more input controllers 460 receive / send electrical signals from / to other input or control devices 416. Other input or control devices 416 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, click wheels, etc. In some alternative embodiments, the one or more input controllers 460 are optionally coupled to any of the following (or none of the following): an infrared port and / or a USB port.

[0145] Stylus 203 also includes a power system 462 for powering various components. Power system 462 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device and / or portable accessory.

[0146] Stylus 203 optionally also includes one or more optical sensors 464 . Figure 4 An optical sensor is shown coupled to an optical sensor controller 458 in the I / O subsystem 406. One or more optical sensors 464 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 464 receive light projected through one or more lenses from the environment and convert the light into data representing an image.

[0147] Stylus 203 optionally also includes one or more contact intensity sensors 465 . Figure 4 A contact force sensor is shown coupled to force sensor controller 459 in I / O subsystem 406. One or more contact force sensors 465 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of a contact on a surface). One or more contact force sensors 465 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to the tip of stylus 203.

[0148] Stylus 203 optionally also includes one or more proximity sensors 466 . Figure 4 A proximity sensor 466 is shown coupled to the peripherals interface 418. Alternatively, the proximity sensor 466 is coupled to the input controller 460 in the I / O subsystem 406. In some embodiments, the proximity sensor determines the proximity of the stylus 203 to the electronic device (eg, device 100).

[0149] Stylus 203 optionally also includes one or more tactile output generators 463 . Figure 4 A tactile output generator coupled to a tactile feedback controller 461 in the I / O subsystem 406 is shown. One or more tactile output generators 463 optionally include one or more electroacoustic devices such as speakers or other audio components; and / or electromechanical devices for converting energy into linear motion such as motors, solenoids, electroactive polymers, piezoelectric actuators, electrostatic actuators, or other tactile output generating components (e.g., components for converting electrical signals into tactile outputs on a device). One or more tactile output generators 463 receive tactile feedback generation instructions from the tactile feedback module 433 and generate tactile outputs on the stylus 203 that can be felt by the user of the stylus 203. In some embodiments, at least one tactile output generator is juxtaposed or adjacent to the length of the stylus 203 (e.g., the body or housing), and optionally generates tactile outputs by moving the stylus 203 vertically (e.g., in a direction parallel to the length of the stylus 203) or laterally (e.g., in a direction perpendicular to the length of the stylus 203).

[0150] Stylus 203 optionally also includes one or more accelerometers 467 , gyroscopes 468 , and / or magnetometers 469 for obtaining information about the positioning and positional state of stylus 203 (eg, as part of an inertial measurement unit (IMU)). Figure 4 Sensors such as accelerometer 467, gyroscope 468, and magnetometer 469 are shown coupled to peripherals interface 418. Alternatively, sensors such as accelerometer 467, gyroscope 468, and magnetometer 469 are optionally coupled to input controller 460 in I / O subsystem 406. Stylus 203 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location of stylus 203.

[0151] In some embodiments, the software components stored in memory 402 include an operating system 426, a communication module (or instruction set) 428, a contact / motion module (or instruction set) 430, a location module (or instruction set) 431, and a global positioning system (GPS) module (or instruction set) 435. In addition, in some embodiments, memory 402 stores device / global internal state 457, such as Figure 4 Device / global internal state 457 includes one or more of the following: sensor state, including information obtained from various sensors and other input or control devices 416 of the stylus; position state, including information about the position of the stylus relative to the device (e.g., device 100) (e.g., position, orientation, tilt, rotation, and / or distance, such as Figure 5A and Figure 5B and location information about the stylus location (eg, determined by GPS module 435).

[0152] The operating system 426 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, power management, etc.), and facilitates communication between various hardware components and software components.

[0153] The communication module 428 optionally facilitates communication with other devices through one or more external ports 424, and also includes various software components for processing data received by the RF circuit 408 and / or the external port 424. The external port 424 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, wireless LAN, etc.). In some embodiments, the external port is a device that is compatible with some of Apple Inc. (Cupertino, California). iPod A Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in iPod devices.

[0154] The contact / motion module 430 optionally detects contact with the stylus 203 and other touch-sensitive devices of the stylus 203 (e.g., buttons or other touch-sensitive components of the stylus 203). The contact / motion module 430 includes software components for performing various operations related to the detection of contact (e.g., detection of contact between the tip of the stylus and a touch-sensitive display such as the touch-sensitive display system 112 of the device 100 or with another surface such as a table surface), such as determining whether contact has occurred (e.g., detecting a touch down event), determining the strength of the contact (e.g., the force or pressure of the contact, or a surrogate for the force or pressure of the contact), determining whether there is movement of the contact and tracking across that movement (e.g., across the touch-sensitive display system 112 of the device 100), and determining whether the contact has ceased (e.g., detecting a lift-off event or contact disconnection). In some embodiments, the contact / motion module 430 receives contact data from the I / O subsystem 406. Determining movement of a point of contact optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the point of contact, the movement of the point of contact being represented by a series of contact data. As described above, in some embodiments, one or more of these operations related to detection of contact are performed by the device using contact / motion module 130 (in addition to or in lieu of the stylus using contact / motion module 430).

[0155] The contact / motion module 430 optionally detects gesture input from the stylus 203. Different gestures performed with the stylus 203 have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single tap gesture includes detecting a touch down event, followed by detecting a lift-off event at the same location (or substantially the same location) as the touch down event (e.g., at the location of an icon). As another example, detecting a swipe gesture includes detecting a touch down event, followed by detecting one or more stylus drag events, and then followed by detecting a lift-off event. As described above, in some embodiments, gesture detection is performed by the device using the contact / motion module 130 (in addition to or in lieu of the stylus using the contact / motion module 430).

[0156] In conjunction with accelerometer 467, gyroscope 468 and / or magnetometer 469, position module 431 optionally detects position information about the stylus, such as the attitude (roll, pitch and / or yaw) of the stylus in a particular reference frame. In conjunction with accelerometer 467, gyroscope 468 and / or magnetometer 469, position module 431 optionally detects stylus movement gestures, such as flicks, taps, and rotations of the stylus. Position module 431 includes software components for performing various operations related to detecting the position of the stylus and detecting changes in the position of the stylus in a particular reference frame. In some embodiments, position module 431 detects a position state of the stylus relative to the device and detects changes in the position state of the stylus relative to the device. As described above, in some embodiments, device 100 or device 300 uses position module 131 to determine the position state of the stylus relative to the device and changes in the position state of the stylus (in addition to or instead of using position module 431 for the stylus).

[0157] Haptic feedback module 433 includes various software components for generating instructions used by one or more tactile output generators 463 to produce tactile output at one or more locations on stylus 203 in response to user interaction with stylus 203 .

[0158] GPS module 435 determines the location of the stylus and provides this information for use in various applications (eg, to applications that provide location-based services, such as applications for finding lost devices and / or accessories).

[0159] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 402 optionally stores a subset of the above modules and data structures. In addition, memory 402 optionally stores additional modules and data structures not described above.

[0160] FIG. 5A to FIG. 5B The position state of the stylus 203 relative to the touch-sensitive surface (e.g., the touch-sensitive display system 112 of the device 100) according to some embodiments is shown. In some embodiments, the position state of the stylus 203 corresponds to (or indicates): the projected position of the tip (or other representative portion) of the stylus on the touch-sensitive surface (e.g., Figure 5A 504), the orientation of the stylus relative to the touch-sensitive surface (e.g., Figure 5A 506), the tilt of the stylus relative to the touch-sensitive surface (e.g., Figure 5B 512) and / or the distance of the stylus relative to the touch-sensitive surface (e.g., Figure 5B In some embodiments, the position state of stylus 203 corresponds to (or indicates) the pitch, yaw, and / or roll of the stylus (e.g., the attitude of the stylus relative to a particular reference frame such as a touch-sensitive surface (e.g., touch-sensitive display system 112) or the ground). In some embodiments, the position state includes a set of position parameters (e.g., one or more position parameters). In some embodiments, the position state is detected based on one or more measurement results from stylus 203 sent to an electronic device (e.g., device 100). For example, the stylus measures the tilt of the stylus (e.g., Figure 5B of tilt 512) and / or orientation (e.g., Figure 5A The device 100 may further include a touch-sensitive surface (e.g., a touch-sensitive display system 112 of the device 100) that receives the position state of the stylus 203 and detects the position state of the stylus 203. The touch-sensitive surface may also ... surface) that receives the position state of the stylus 203 and detects the position state of the stylus 203 based on the sensor measurements generated by the stylus.

[0161] Figure 5A 2 shows stylus 203 relative to a touch-sensitive surface (e.g., touch-sensitive display system 112 of device 100) from a perspective directly above the touch-sensitive surface according to some embodiments. Figure 5A , the z-axis 594 points out of the page (i.e., in a direction perpendicular to the plane of the touch-sensitive display system 112), the x-axis 590 is parallel to a first edge of the touch-sensitive display system 112 (e.g., length), the y-axis 592 is parallel to a second edge of the touch-sensitive display system 112 (e.g., width), and the y-axis 592 is perpendicular to the x-axis 590.

[0162] Figure 5AThe tip of stylus 203 is shown at (x, y) position 504. In some embodiments, the tip of stylus 203 is the end of the stylus that is configured to determine the proximity of the stylus to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, the projection of the tip of the stylus onto the touch-sensitive surface is an orthogonal projection. In other words, the projection of the tip of the stylus onto the touch-sensitive surface is a point at the end of a line from the tip of the stylus to the surface of the touch-sensitive surface that is perpendicular to the touch-sensitive surface (e.g., (x, y) position 504, where the tip of the stylus would touch the touch-sensitive surface if the stylus was moved directly along a path perpendicular to the touch-sensitive surface). In some embodiments, the (x, y) position at the lower left corner of touch-sensitive display system 112 is position (0, 0) (e.g., (0, 0) position 502) and other (x, y) positions on touch-sensitive display system 112 are relative to the lower left corner of touch-sensitive display system 112. Alternatively, in some embodiments, the (0,0) position is located at another location of the touch-sensitive display system 112 (e.g., at the center of the touch-sensitive display system 112) and the other (x,y) positions are relative to the (0,0) position of the touch-sensitive display system 112.

[0163] in addition, Figure 5A Stylus 203 is shown having orientation 506. In some embodiments, orientation 506 is the orientation of a projection of stylus 203 on touch-sensitive display system 112 (e.g., an orthogonal projection of the length of stylus 203 or a line corresponding to a line between projections of two different points of stylus 203 on touch-sensitive display system 112). In some embodiments, orientation 506 is relative to at least one axis in a plane parallel to touch-sensitive display system 112. In some embodiments, orientation 506 is relative to a single axis in a plane parallel to touch-sensitive display system 112 (e.g., axis 508, with a clockwise rotation angle starting from axis 508 ranging from 0 degrees to 360 degrees, as shown in FIG. 1 ). Figure 5A Alternatively, in some embodiments, orientation 506 is relative to a pair of axes in a plane parallel to touch-sensitive display system 112 (e.g., x-axis 590 and y-axis 592, as shown). Figure 5A As shown, or a pair of axes associated with an application displayed on the touch-sensitive display system 112).

[0164] In some embodiments, an indication (e.g., indication 516) is displayed on a touch-sensitive display (e.g., touch-sensitive display system 112 of device 100). In some embodiments, indication 516 shows the location where the stylus will touch (or mark) the touch-sensitive display before the stylus touches the touch-sensitive display. In some embodiments, indication 516 is part of the mark drawn on the touch-sensitive display. In some embodiments, indication 516 is separate from the mark drawn on the touch-sensitive display and corresponds to a virtual "pen tip" or other element indicating the location where the mark will be drawn on the touch-sensitive display.

[0165] In some embodiments, indication 516 is displayed based on the position state of stylus 203. For example, in some cases, indication 516 is displaced from (x, y) position 504 (e.g., Figure 5A and Figure 5B ), and in other cases, indication 516 is not displaced from (x,y) position 504 (for example, when tilt 512 is zero degrees, indication 516 is displayed at or near (x,y) position 504). In some embodiments, indication 516 is displayed with varying colors, sizes (or radius or area), opacity, and / or other characteristics depending on the position state of the stylus. In some embodiments, the displayed indication takes into account the thickness of the glass layer on the touch-sensitive display so as to implement the indication into the "pixels" of the touch-sensitive display rather than displaying the indication on the "glass" covering the pixels.

[0166] Figure 5B 2 shows stylus 203 relative to a touch-sensitive surface (e.g., touch-sensitive display system 112 of device 100) viewed from a side view of the touch-sensitive surface according to some embodiments. Figure 5B , the z-axis 594 points in a direction perpendicular to the plane of the touch-sensitive display system 112, the x-axis 590 is parallel to a first edge (e.g., length) of the touch-sensitive display system 112, the y-axis 592 is parallel to a second edge (e.g., width) of the touch-sensitive display system 112, and the y-axis 592 is perpendicular to the x-axis 590.

[0167] Figure 5B Stylus 203 is shown having tilt 512. In some embodiments, tilt 512 is an angle relative to a normal to the surface of the touch-sensitive surface (also referred to simply as the normal to the touch-sensitive surface) (e.g., normal 510). Figure 5B As shown, when the stylus is perpendicular / perpendicular to the touch-sensitive surface (eg, when stylus 203 is parallel to normal 510), tilt 512 is zero, and increases as the stylus tilts closer to being parallel to the touch-sensitive surface.

[0168] in addition, Figure 5B514 is shown relative to the touch-sensitive surface of the stylus 203. In some embodiments, the distance 514 is the distance from the tip of the stylus 203 to the touch-sensitive surface in a direction perpendicular to the touch-sensitive surface. Figure 5B , distance 514 is the distance from the tip of stylus 203 to (x,y) position 504 .

[0169] Although the terms "x-axis", "y-axis", and "z-axis" are used herein to illustrate certain directions in certain drawings, it should be understood that these terms do not refer to absolute directions. In other words, the "x-axis" can be any corresponding axis, and the "y-axis" can be a specific axis different from the x-axis. Typically, the x-axis is perpendicular to the y-axis. Similarly, the "z-axis" is different from the "x-axis" and the "y-axis", and is typically perpendicular to both the "x-axis" and the "y-axis".

[0170] in addition, Figure 5B A rotation 518 is shown - a rotation about the length (major axis) of stylus 203 .

[0171] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0172] Fig. 6A An exemplary user interface for a menu of applications on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 600 includes the following elements, or a subset or superset thereof:

[0173] One or more signal strength indicators 602 of one or more wireless communications such as cellular signals and Wi-Fi signals;

[0174] Time 604;

[0175] Bluetooth indicator 605;

[0176] Battery status indicator 606;

[0177] A tray 608 with icons for commonly used applications, such as:

[0178] o An icon 616 of the phone module 138 labeled "Phone," which optionally includes an indicator 614 of the number of missed calls or voice messages;

[0179] o an icon 618 of the email client module 140 labeled “Mail”, the icon 618 optionally including an indicator 610 of the number of unread emails;

[0180] o An icon 620 labeled "Browser" of the browser module 147; and

[0181] ○ Video and music player module 152 (also known as (Trademark of Apple Inc.)

[0182] module 152) and an icon 622 labeled "iPod"; and

[0183] Icons for other apps, such as:

[0184] o Icon 624 labeled "Messages" of the IM module 141;

[0185] o An icon 626 labeled “Calendar” of the calendar module 148;

[0186] o Icon 628 labeled “Photos” of the image management module 144;

[0187] o An icon 630 labeled “Camera” of the camera module 143;

[0188] o Icon 632 labeled "Video Editing" of the online video module 155;

[0189] ○ Icon 634 labeled “Stock Market” of the Stock Market Desktop Applet 149-2;

[0190] o An icon 636 labeled “Map” of the map module 154;

[0191] ○ Icon 638 labeled “Weather” of the weather widget 149-1;

[0192] ○ Icon 640 labeled as “clock” of the alarm clock desktop applet 149-4;

[0193] o An icon 642 labeled “Fitness Support” of the fitness support module 142;

[0194] o An icon 644 labeled "Notepad" of the Notepad module 153; and

[0195] ○ An icon 646 for a settings application or module that provides access to settings for the device 100 and its various applications 136 .

[0196] It should be noted that Fig. 6AThe icon labels shown in are exemplary only. For example, in some embodiments, icon 622 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0197] Figure 6B A touch-sensitive surface 651 (eg, Figure 3 A device (e.g., a tablet or touchpad 355) Figure 3 Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contact on touch-sensitive surface 651, and / or one or more tactile output generators 357 for generating tactile output for a user of device 300.

[0198] Figure 6B A touch-sensitive surface 651 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3 Although many of the examples that follow will be given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 6B In some embodiments, the touch-sensitive surface (e.g., Figure 6B 651) has a main axis (e.g., Figure 6B 653) corresponding to the main axis (for example, Figure 6B According to these embodiments, the device detects contact with the touch-sensitive surface 651 at a location corresponding to a corresponding location on the display (e.g., Figure 6B 660 and 662 in (e.g., in Figure 6B , 660 corresponds to 668 and 662 corresponds to 670). Thus, on a touch-sensitive surface (e.g., Figure 6B 651) and a display of a multi-function device (e.g., Figure 6B 650 in ) is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 660 and 662 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0199] In addition, although the following examples are given primarily with reference to finger input (e.g., finger contact, single-finger tap gesture, finger swipe gesture, etc.) and / or stylus input, it should be understood that in some embodiments, one or more of these finger inputs are replaced by input from another input device (e.g., mouse-based input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact (or stylus contact) are optionally used simultaneously.

[0200] As used in this specification and claims, the term "intensity" of a contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of a contact (e.g., a finger contact or a stylus contact) on the touch-sensitive surface, or to a surrogate (surrogate) of the force or pressure of a contact on the touch-sensitive surface. The intensity of a contact has a range of values ​​that includes at least four different values ​​and more typically includes hundreds of different values ​​(e.g., at least 256). The intensity of a contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at different points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged or summed) to determine an estimated contact force. Similarly, the pressure-sensitive tip of the stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size of the contact area detected on the touch-sensitive surface and / or its change, the capacitance of the touch-sensitive surface near the contact and / or its change, and / or the resistance of the touch-sensitive surface near the contact and / or its change are optionally used as a substitute for the force or pressure of the contact on the touch-sensitive surface. In some embodiments, the substitute measurement of the contact force or pressure is used directly to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some embodiments, the substitute measurement of the contact force or pressure is converted to an estimated force or pressure, and the estimated force or pressure is used to determine whether the intensity threshold has been exceeded (e.g., the intensity threshold is a pressure threshold measured in pressure units). The intensity of the contact is used as an attribute of the user input, thereby allowing the user to access additional device functions that the user cannot easily access on a smaller device with limited real estate for displaying an indication (e.g., on a touch-sensitive display) and / or receiving user input (e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical control / mechanical control such as a knob or button).

[0201] In some embodiments, contact / motion module 130 and / or contact / motion module 430 uses a set of one or more intensity thresholds to determine whether an operation has been performed by a user (e.g., to determine whether a user has "clicked" an icon). In some embodiments, at least a subset of the intensity thresholds are determined based on software parameters (e.g., the intensity thresholds are not determined by the activation threshold of a particular physical actuator and can be adjusted without changing the physical hardware of device 100). For example, without changing the touchpad or touchscreen display hardware, the mouse "click" threshold of a touchpad or touchscreen display can be set to any one of a large range of predefined thresholds. In addition, in some specific implementations, a software setting is provided to the user of the device for adjusting one or more intensity thresholds in a set of intensity thresholds (e.g., by adjusting individual intensity thresholds and / or by utilizing a system-level click on an "intensity" parameter to adjust multiple intensity thresholds at once).

[0202] As used in the specification and claims, the term "characteristic intensity" of a contact refers to a characteristic of a contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a predefined number of intensity samples or a set of intensity samples collected during a predetermined time period (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds) relative to a predefined event (e.g., after contact is detected, before contact is detected to be lifted off, before or after contact starts to move, before contact ends, before or after contact intensity is detected to increase, and / or before or after contact intensity is detected to decrease). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the value at the top 10% of the contact intensity, the half-maximum value of the contact intensity, the 90% maximum value of the contact intensity, etc. In some embodiments, the duration of the contact is used when determining the characteristic intensity (e.g., when the characteristic intensity is the average value of the intensity of the contact over time). In some embodiments, the feature strength is compared to a set of one or more strength thresholds to determine whether the user has performed an operation. For example, the set of one or more strength thresholds may include a first strength threshold and a second strength threshold. In this example, a contact whose feature strength does not exceed the first threshold results in a first operation, a contact whose feature strength exceeds the first strength threshold but does not exceed the second strength threshold results in a second operation, and a contact whose feature strength exceeds the second threshold results in a third operation. In some embodiments, a comparison between the feature strength and one or more strength thresholds is used to determine whether to perform one or more operations (e.g., whether to perform the corresponding option or give up performing the corresponding operation), rather than to determine whether to perform the first operation or the second operation.

[0203] In some embodiments, a portion of a gesture is identified for determining a characteristic strength. For example, a touch-sensitive surface may receive a continuous swipe contact that transitions from a starting position and reaches an end position (e.g., a drag gesture) where the strength of the contact increases. In this example, the characteristic strength of the contact at the end position may be based only on a portion of the continuous swipe contact, rather than the entire swipe contact (e.g., only a portion of the swipe contact at the end position). In some embodiments, a smoothing algorithm may be applied to the strength of the swipe gesture before determining the characteristic strength of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted sliding average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some cases, these smoothing algorithms eliminate narrow peaks or dips in the strength of the swipe contact to achieve the purpose of determining the characteristic strength.

[0204] The user interface diagrams described herein optionally include various intensity diagrams that illustrate the intensity of contacts on the touch-sensitive surface relative to one or more intensity thresholds (e.g., contact detection intensity threshold IT0, light press intensity threshold IT1, and light touch intensity threshold IT2). L , Deep compression intensity threshold IT D (For example, it is at least partially higher than I L ) and / or one or more other intensity thresholds (e.g., ratio I L Lower intensity threshold I H )). This intensity graph is not typically part of the displayed user interface, but is provided to help explain the graph. In some embodiments, the light press intensity threshold corresponds to an intensity at which the device will perform an operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, the deep press intensity threshold corresponds to an intensity at which the device will perform an operation different from the operation typically associated with clicking a button of a physical mouse or touchpad. In some embodiments, when a contact with a characteristic intensity below the light press intensity threshold (e.g., and above a nominal contact detection intensity threshold IT0, contacts lower than the nominal contact detection intensity threshold are no longer detected) is detected, the device will move the focus selector according to the movement of the contact on the touch-sensitive surface without performing operations associated with the light press intensity threshold or the deep press intensity threshold. Generally speaking, unless otherwise stated, these intensity thresholds are consistent between different groups of user interface illustrations.

[0205] In some embodiments, the response of the device to the input detected by the device depends on a criterion based on the contact strength during the input. For example, for some "light press" inputs, the strength of the contact exceeding the first strength threshold during the input triggers a first response. In some embodiments, the response of the device to the input detected by the device depends on a criterion including both the contact strength during the input and a time-based criterion. For example, for some "deep press" inputs, as long as a delay time passes between satisfying the first strength threshold and satisfying the second strength threshold, the strength of the contact exceeding the second strength threshold greater than the first strength threshold of the light press during the input triggers a second response. The duration of the delay time is typically less than 200ms (e.g., 40ms, 100ms, or 120ms, depending on the magnitude of the second strength threshold, where the delay time increases as the second strength threshold increases). The delay time helps avoid accidental deep press inputs. As another example, for some "deep press" inputs, a period of reduced sensitivity will occur after reaching the first strength threshold. During this period of reduced sensitivity, the second strength threshold increases. This temporary increase in the second strength threshold also helps avoid accidental deep press inputs. For other deep press inputs, the response to detecting the deep press input is not dependent on time-based criteria.

[0206] In some embodiments, one or more of the input intensity thresholds and / or corresponding outputs vary based on one or more factors such as user settings, contact motion, input timing, application running, rate at which intensity is applied, number of simultaneous inputs, user history, environmental factors (e.g., ambient noise), focus selector position, etc. Example factors are described in U.S. patent applications 14 / 399,606 and 14 / 624,296, which are incorporated herein by reference in their entirety.

[0207] The contact characteristic strength is below the light press strength threshold IT L The intensity increases to between the light pressure intensity threshold IT L and deep compression intensity threshold IT D The intensity between IT and the deep press intensity threshold is sometimes referred to as a "light press" input. D The intensity of the deep compression increases to above the deep compression intensity threshold IT D The intensity of the contact feature increases from an intensity below the contact detection intensity threshold IT0 to an intensity between the contact detection intensity threshold IT0 and the light press intensity threshold IT0. LA decrease in the characteristic intensity of a contact from an intensity above the contact detection intensity threshold IT0 to an intensity below the contact detection intensity threshold IT0 is sometimes referred to as detecting a contact lifted off the touch surface. In some embodiments, IT0 is zero. In some embodiments, IT0 is greater than zero. In some examples, shaded circles or ellipses are used to represent the intensity of a contact on the touch-sensitive surface. In some examples, unshaded circles or ellipses are used to represent corresponding contacts on the touch-sensitive surface without specifying the intensity of the corresponding contacts.

[0208] In some embodiments described herein, one or more operations are performed in response to detecting a gesture that includes a corresponding press input or in response to detecting a corresponding press input performed using a corresponding contact (or multiple contacts), wherein the corresponding press input is detected at least in part based on detecting that the strength of the contact (or multiple contacts) increases to above a press input strength threshold. In some embodiments, the corresponding operation is performed in response to detecting that the strength of the corresponding contact increases to above the press input strength threshold (e.g., the corresponding operation is performed on a "down stroke" of the corresponding press input). In some embodiments, the press input includes an increase in the strength of the corresponding contact to above the press input strength threshold and the strength of the contact subsequently decreases to below the press input strength threshold, and the corresponding operation is performed in response to detecting that the strength of the corresponding contact subsequently decreases to below the press input threshold (e.g., the corresponding operation is performed on an "up stroke" of the corresponding press input).

[0209] In some embodiments, the device employs intensity hysteresis to avoid unexpected input, sometimes referred to as "jitter," where the device defines or selects a hysteresis intensity threshold that has a predefined relationship to a press input intensity threshold (e.g., the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable proportion of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of the corresponding contact to above the press input intensity threshold and a subsequent decrease in the intensity of the contact to below a hysteresis intensity threshold corresponding to the press input intensity threshold, and a corresponding operation is performed in response to detecting that the intensity of the corresponding contact subsequently decreases to below the hysteresis intensity threshold (e.g., performing the corresponding operation on an "up stroke" of the corresponding press input). Similarly, in some embodiments, a press input is detected only when the device detects that the contact strength increases from an intensity equal to or below a hysteresis strength threshold to an intensity equal to or above a press input strength threshold and, optionally, the contact strength subsequently decreases to an intensity equal to or below the hysteresis strength, and a corresponding operation is performed in response to detecting the press input (e.g., an increase in contact strength or a decrease in contact strength, depending on the circumstances).

[0210] For ease of explanation, descriptions of operations performed in response to a press input associated with a press input intensity threshold or in response to a gesture including a press input are optionally triggered in response to detecting that the intensity of the contact increases above the press input intensity threshold, the intensity of the contact increases from an intensity below a hysteresis intensity threshold to an intensity above the press input intensity threshold, the intensity of the contact decreases below the press input intensity threshold, or the intensity of the contact decreases below a hysteresis intensity threshold corresponding to the press input intensity threshold. Additionally, in examples where operations are described as being performed in response to detecting that the intensity of the contact decreases below the press input intensity threshold, the operations are optionally performed in response to detecting that the intensity of the contact decreases below a hysteresis intensity threshold that corresponds to and is less than the press input intensity threshold. As described above, in some embodiments, triggering of these operations is also dependent on satisfying a time-based criterion (e.g., a delay time has elapsed between satisfying the first intensity threshold and satisfying the second intensity threshold).

[0211] User interface and associated processes

[0212] Attention now turns to embodiments of a user interface ("UI") and associated processes that may be implemented on an electronic device such as portable multifunction device 100 or device 300 having a touch-sensitive display and an optional sensor or sensors for detecting signals from a stylus associated with the device.

[0213] Figure 7A-7RRR An exemplary user interface for adding annotations to an annotation layer according to some embodiments is shown. The user interface in these figures is used to illustrate the process described below, including Figure 9A-9E Although some of the examples that follow will be given with reference to input on a touch screen display (where a touch-sensitive surface and a display are combined, such as on touch-sensitive display system 112), in some embodiments, the device detects input on a touch-sensitive surface 651 that is separate from the display 650, such as Figure 6B as shown in .

[0214] Fig. 7A 702 on a display 750 according to some embodiments. In some embodiments, the display 750 may be a touch-sensitive display (e.g., a touch-sensitive display 112, Figure 1A ).exist Figures 7A-7SS In the example shown in FIG. 7 , user interface 702 includes a user interface of a web browser application (e.g., browser module 147, Figure 1A ). The user interface of the web browser application includes displaying a portion of a web page including an image 704 and a map 706.

[0215] Figure 7B-7C The user interface 702 is shown Fig. 7A In particular, Figure 7B-7C shows user input 708 detected while displaying the user interface 702 of the web browser application, and Figure 7C Also shown is an annotation layer 712 that is displayed in response to detecting user input 708. In some embodiments, the annotation layer 712 is displayed in response to an input gesture, such as a drag gesture (e.g., a drag gesture that starts at or near an edge of the display 750, such as a downward drag gesture that starts at or near the top edge of the display 750). For example, Figure 7B An initial location (eg, initial contact location) of user input 708 by stylus 710 detected at or near an upper edge of display 750 is shown. Figure 7C User input 708 by stylus 710 is shown from Figure 7B 714. As the user input 708 of the stylus 710 moves along the path indicated by the arrow 714 while the user interface 702 of the web browser application continues to be displayed, the annotation layer 712 is gradually displayed over at least a portion of the user interface 702 as the user input 708 of the stylus 710 moves along the path indicated by the arrow 714.

[0216] Figure 7D-Figure 7E The user interface 702 is shown Fig. 7A In particular, Figure 7D-Figure 7E Shown in the display Fig. 7A The web browser application's user interface 702 is shown with alternative user input 716 detected. Fig.7D 7. An initial position of a user input 716 (eg, a contact) of a finger 718 detected at or near an upper edge of the display 750 is shown. Fig.7D The initial position of the image is shown moving along the path indicated by arrow 720, without showing the annotation layer (e.g., Figure 7C 712 shown in FIG. 702 ). Conversely, in some embodiments, as user input 716 of finger 718 moves along the path indicated by arrow 720, a different user interface, such as notification user interface 722, is gradually displayed above user interface 702. In some embodiments, detecting user input (e.g., an input gesture, such as a drag gesture from an edge of the display, or input corresponding to a location of a control affordance for displaying an annotation layer) includes determining whether the user input is provided using a stylus (e.g., Figure 7B-7C ), rather than, for example, fingers (e.g. Figure 7D-Figure 7E shown).

[0217] Figure 7F-7G The user interface 702 is shown Fig. 7AAnother transition of user interface 702 in . Figure 7F 7 shows an initial position of input 724 (eg, contact) detected at or near a lower edge of display 750. In response to input 724 being Figure 7F As shown, the initial position moves along the path indicated by arrow 726, and a control user interface 728 including multiple control affordances is displayed, such as Figure 7G As shown. In some embodiments, the multiple control enable representations include a control enable representation for displaying an annotation layer such as annotation icon 730. In some embodiments, the annotation layer is displayed based on detecting an input (e.g., a tap gesture) at a location corresponding to annotation icon 730. In some embodiments, after the control user interface 728 is displayed in response to movement of the input, a subsequent input 732 is detected (e.g., a subsequent contact detected after the contact of input 724 is lifted off), and in response to movement of the subsequent input 732 along the path indicated by arrow 734, the annotation layer is displayed.

[0218] Figure 7H A user interface 702 (e.g., Fig. 7A In some embodiments, the annotation layer 712 is displayed above the annotation layer 712 (as shown). In some embodiments, the user interface 702 continues to be displayed and is at least partially visible through the annotation layer 712 (e.g., the annotation layer 712 is translucent). In some embodiments, the annotation layer 712 includes one or more control enable indications. For example, in some embodiments, the annotation layer 712 includes an annotation tool icon 740, which indicates the currently selected annotation tool (e.g., a virtual knife tool, a marking or highlighting tool, a felt tip pen tool, a pencil tool, an eraser tool, etc.), and optionally, the currently selected color (e.g., if the currently selected annotation tool has an associated color option). In some embodiments, the annotation layer 712 includes an annotation layer undo enable indication 742 displayed at or near the edge of the annotation layer 712 and the display 750. In some embodiments, when input is detected at a location corresponding to the annotation layer undo enable indication 742 (e.g., a tap gesture is detected at a location corresponding to the annotation layer undo enable indication 742), the annotation layer 712 stops being displayed.

[0219] Figure 7I-7J In response to detecting user input 744, annotation layer 712 is shown Figure 7H In particular, Fig.7I The location of user input 744 (e.g., contact by a finger) is shown at a location in the annotation layer 712 that does not correspond to an annotation tool icon 740 or an annotation layer de-enable indication 742. Figure 7J Subsequent lift-off of contact for user input 744 , or more generally, cessation of detecting user input 744 , is shown. Figure 7J Also shown is a set of annotation controls (e.g., in addition to and including) of the annotation layer 712 displayed in response to detecting the user input 744 (or a subsequent liftoff of the contact of the user input 744). Figure 7H For example, in some embodiments, the set of annotation controls also includes a camera icon 746 (e.g., for capturing application content to be added to the annotation layer, such as a full-screen snapshot of the application content), a texture selection icon 748 (e.g., for selecting a texture for the annotation layer), an opacity slider 749 (e.g., for changing the opacity level of the annotation layer), and a share icon 752 (e.g., for sharing and / or storing the annotation layer). Figure 7J , the opacity slider 749 is set to an opacity level corresponding to the slide position 753.

[0220] In some embodiments, input for displaying a set of annotation controls is distinguished from input for generating annotations on the annotation layer. For example, in some embodiments, the set of annotation controls is displayed based on a determination that user input 744 (e.g., a tap gesture) is provided using a finger, while a similar user input in another manner provided using a stylus generates annotations on the annotation layer (e.g., a stippling). In some embodiments, the set of annotation controls is displayed in response to input that meets the intensity criteria (e.g., in response to a deep press input), whereas, for example, a similar user input in another manner that does not meet the intensity criteria (e.g., a light press input) may generate annotations on the annotation layer.

[0221] Figure 7K Similar to Figure 7J , the difference is Figure 7K An annotation layer 712 is shown with an opacity slider set to an opacity level corresponding to slider position 754, which corresponds to a ratio of Figure 7J The opacity of the annotation layer 712 with a low opacity level (e.g., a higher opacity) indicated by the slider position 753 in FIG. Figure 7K In the example shown in FIG, the visibility of the application content displayed below the annotation layer is increased so that the user interface 702 is more visible through the annotation layer 712 than the user interface 702. Figure 7J User interface 702 is shown in FIG.

[0222] Figure 7L Similar to Figure 7J-7K , the difference is Figure 7L An annotation layer 712 is shown with an opacity slider set to an opacity level corresponding to slider position 756, which corresponds to a ratio of 0. Figure 7K The opacity of the annotation layer 712 is lower (e.g., more transparent) than the opacity level indicated by the slider position 754 in . Figure 7LIn the example shown, slider position 756 corresponds to the lowest opacity (e.g., the highest transparency). In some embodiments, slider position 756 corresponds to zero opacity (e.g., completely transparent), and therefore, the application content displayed below the annotation layer is fully visible, making the user interface 702 Fig. 7A It should be noted that, according to the zero opacity setting, even though none of the textures of annotation layer 712 are visible, annotation layer 712 remains above user interface 702, as indicated by annotation controls (e.g., control affordances 740, 742, 746, 748, 749, 752, and 756).

[0223] Figure 7M Similar to Figure 7J-7L , the difference is Figure 7M shows that the ratio is set to Figure 7J-7L The annotation layer 712 of the opacity slider of the high opacity (e.g., low opacity) indicated by any of the slider positions 753, 754, 756 in FIG. Figure 7M In the example shown, slider position 758 corresponds to the highest opacity (e.g., the lowest transparency). In some embodiments, slider position 758 corresponds to full opacity (e.g., zero transparency), and therefore, the application content displayed below the annotation layer is not visible, so that only the annotation layer 712 and its annotation controls (e.g., control affordances 740, 742, 746, 748, 749, 752, and 758) are visible.

[0224] Figure 7N-Figure 7O In response to detecting user input 760, annotation layer 712 is shown Figure 7M In particular, Figure 7N The location of user input 760 (eg, contact of a finger) is shown at a location in annotation layer 712 corresponding to camera icon 746 . Fig.7O Ceasing to detect user input 760 (eg, based on subsequent lift-off of the contact) is shown. Fig.7O Also shown is that, in response to detecting user input 760, a representation of user interface 702 (e.g., a screenshot of the web browser application content below) is added to annotation layer 712. It should be noted that the opacity slider is still set to full opacity, as indicated by slider position 758, and therefore, Fig.7OThe application content displayed in annotation layer 712 in is not itself user interface 702, because when the annotation layer is set to full opacity, user interface 702 will not be visible through annotation layer 712. Instead, the application content displayed in annotation layer 712 is a representation of user interface 702, which, in some embodiments, is created in response to user input 760 (e.g., activating camera icon 746).

[0225] Figure 7P-Figure 7Q The annotation layer 712 is shown from Figure 7K In particular, Figure 7P The annotation layer 712 is shown to have an adjustable texture that can be selected from a plurality of texture options (e.g., a currently selected grid texture icon 761, a horizontal grid texture icon 762, a solid or semi-transparent texture icon 764, a dotted texture icon 766, etc.). In some embodiments, in response to detecting a Figure 7K The texture selection control area 768 is displayed based on user input at the location of the texture selection icon 748 in the annotation layer 712. In some embodiments, texture icons 761, 762, 764, 766 are displayed within the texture selection control area 768.

[0226] Figure 7P Also shown is detection of user input 763 (eg, a tap gesture) at a location corresponding to horizontal checkered texture icon 762 .

[0227] Figure 7Q The annotation layer 712 is shown as Figure 7P In particular, Figure 7Q The texture of the annotation layer 712 has been changed from Figure 7P The grid texture shown in FIG. 1 is changed to a horizontal grid texture (e.g., in response to Figure 7P In some embodiments, the update Figure 7Q The texture selection icon 748 is moved to a horizontal checkered texture icon 762 (eg, to match the horizontal checkered texture icon 762) to indicate that the currently selected texture is a horizontal checkered texture.

[0228] Figure 7R The annotation layer 712 is shown as Figure 7K In particular, Figure 7RMultiple available tools or tool modes for the annotation layer 712 (e.g., virtual knife tool icon 770, felt tip pen tool icon 771, pencil tool icon 772, eraser tool icon 773, (currently selected) marker or highlight tool icon 740) and multiple color options for the currently selected tool (e.g., color option icons 774, 775, 776, 777) are shown. In some embodiments, in response to detecting a color in the annotation layer 712 corresponding to a Figure 7K The tool selection control area 778 is displayed based on user input at the location of the annotation tool icon 740. In some embodiments, tool option icons and / or color option icons are displayed within the tool selection control area 778. In some embodiments, the currently selected tool is displayed in the center of the tool selection control area 778, and / or includes an indication (e.g., a black, thick ring displayed with the marking tool icon 740) that this is the currently selected tool. In some embodiments, the currently selected color option of the currently selected tool includes a visual indication (e.g., a white ring displayed with the color option icon 777) indicating that it is the currently selected color. In some embodiments, at least a portion of the currently selected tool icon (e.g., the marker pen tip icon 740) reflects the currently selected color option (e.g., the color of the currently selected tool). Those of ordinary skill in the art will recognize that many other tool options may be represented in the tool selection control affordance representation, such as line thickness (e.g., for a felt tip pen or marker pen tool) or selection mode (e.g., shape-based and freeform selection modes for a virtual knife tool).

[0229] Figure 7S-Figure 7T The annotation layer 712 is shown as Figure 7R In particular, Figure 7S The location of a user input 780 (eg, contact of a finger) is shown at a location in the annotation layer 712 corresponding to a virtual knife tool icon 770 . Figure 7T The user input 780 is shown to cease being detected (e.g., upon lift-off of the contact), and further shown to be a virtual indication (e.g., a virtual indication having a marking tool icon 740) in response to detecting the user input 780. Figure 7R ) is removed and the virtual indication is displayed with a virtual knife tool icon 770 to indicate that the currently selected tool has changed from a marking tool to a virtual knife tool. For example, the virtual knife tool icon 770 is displayed in the center of the tool selection control area 778 with a black ring. In addition, in some embodiments, the virtual knife tool represented by the virtual knife tool icon 770 does not have or require an associated color selection. Therefore, in Figure 7T In the example shown, the color options icon 777 no longer displays an indication of the current selection (e.g., stops at Figure 7T Display Figure 7R) as shown in the white ring.

[0230] Figure 7U-7W The annotation layer 712 is shown as follows: Figure 7T In particular, Figure 7U The initial position of user input 782 by stylus 710 is shown detected near the upper left corner of image 704. Figure 7U , the virtual knife tool icon 770 and the annotation layer undo enable indication 742 are only displayed when no additional controls (e.g., Figure 7K The camera icon, texture selection icon, opacity slider, and share icon, or control submenus such as Figure 7P The texture selection control affordance indicates or Figure 7R The control affordances that appear when you select a tool (for example, similar to Figure 7H ). In some embodiments, display of the additional controls is stopped in response to detecting user input for creating an annotation on the annotation layer (e.g., user input provided using a stylus and / or at a location on the annotation layer that does not correspond to a control affordance).

[0231] Figure 7V The user input 782 of the stylus 710 is shown from Figure 7U The initial position is shown to move along the path indicated by arrow 783 to a position near the lower right corner of the caption of image 704 to select the first portion of user interface 702 to be added to annotation layer 712. Figure 7V In the example shown in , the input of the stylus 710 corresponds to a selection mode having a predefined shape (e.g., a rectangular selection tool) for a virtual knife tool corresponding to the virtual knife tool icon 770. Thus, in response to the movement of the user input 782 of the stylus 710, a selection indication 784 is displayed to indicate the first portion of the user interface 702 selected by the user input.

[0232] Figure 7W The annotation layer 712 is shown in response to ceasing to detect user input 782 by the stylus 710 (e.g., detecting lift-off of the contact). Figure 7V In particular, Figure 7W A representation 785 of the first portion selected by the user interface 702 is shown (eg, Figure 7V 785) has been added to the annotation layer 712. Therefore, the texture of the annotation layer 712 is not visible above the representation 785. Figure 7W In the example shown in , representation 785 includes image 704 and its associated title, and has Figure 7VThe selection indication 784 in FIG. 7A coincides with the boundaries of the selected portion of the user interface 702 (eg, giving the impression of cutting away the annotation layer using a virtual knife).

[0233] Figure 7X-Figure 7Y The annotation layer 712 is shown by adding other following application content from Figure 7W In particular, Figure 7X User input 786 by stylus 710 is shown in a free-form or freehand selection mode for virtual knife tool icon 770. A selection indication 787 is displayed in response to movement of user input 786 by stylus 710 along a free-form selection path.

[0234] Figure 7Y The annotation layer 712 is shown in response to ceasing to detect user input 786 by the stylus 710 (e.g., detecting lift-off of the contact). Figure 7X In particular, Figure 7Y A representation 788 of the selected portion of the user interface 702 is shown added to the annotation layer 712. In some embodiments, such as in Figure 7X-Figure 7Y In the example shown in , even if the selection path of the user input deviates from the predefined content boundaries (e.g., based on the automatically determined alpha channel), the boundaries of the user interface portion added to the annotation layer 712 are automatically and / or intelligently determined based on the structure of the underlying application content. Figure 7X ), Figure 7Y The representation 788 in FIG. 78 has a rectangular border and includes a complete paragraph of text from the underlying web browser application content (rather than just Figure 7X The portion surrounded by the selection indicator 787).

[0235] Figure 7Z and Figure 7AA-Figure 7EE The annotation layer 712 is shown as Figure 7Y A series of transitions of annotation layer 712 in . Figure 7Z Detection of user input 790 (eg, contact) at a location corresponding to annotation tool icon 770 is shown. Figure 7AA The user input 790 is shown to cease being detected (e.g., upon lift-off of the contact), and additional annotation tools are also shown displayed in response to detecting the user input 790, including a tool selection control area 778 (e.g., as described above with reference to Figure 7R-Figure 7T described above).

[0236] Next, Figure 7BB Detection of user input 791 (eg, contact) at a location corresponding to felt tip pen tool icon 771 is shown. In response to user input 791, Figure 7CC As shown, the felt tip pen tool represented by the felt tip pen tool icon 771 becomes the currently selected tool (e.g., displayed in the center position of the tool selection control area 778 with a black ring) and the previously selected tool (e.g., the virtual knife tool represented by the virtual knife tool icon 770) is redisplayed in the left half of the tool selection control area 778. In addition, the color option icon 777 is displayed with a current selection indication (e.g., a white ring) to indicate that it is the currently selected color option of the felt tip pen tool represented by the felt tip pen tool icon 771.

[0237] Next, Figure 7DD The following diagram shows a time interval corresponding to a slider position 754 (eg, Figure 7CC ) at a location 792 (eg, a touch). Figure 7EE As user input 792 is shown moving, the opacity of annotation layer 712 increases, and the visibility of web browser application content displayed beneath annotation layer 712 decreases.

[0238] Figure 7FF-Figure 7NN The annotation layer 712 is shown as Figure 7EE A series of transitions in the annotation layer 712 in FIG. In particular, Figure 7FF-Figure 7HH A first annotation is shown being added to the annotation layer 712 in response to detecting a first additional user input 793 . Figure 7FF The initial location of the user input 793 (eg, contact) by the stylus 710 on the annotation layer 712 is shown. Figure 7U In some embodiments, in response to detecting that Figure 7EE 793 provided using stylus 710 at a location on annotation layer 712 represented by any control affordances displayed in the diagram, additional controls (e.g., Figure 7EE The icons 770, 740, 777, 746, 748 and 752 shown in the figure stop being displayed. Such input (for example, when not corresponding to Figure 7EE User input 793) provided using stylus 710 at a location on annotation layer 712 represented by any control affordances represented in is considered intended to be used to create annotations on the annotation layer. Figure 7GG-Figure 7HH A handwritten annotation (eg, the phrase “Gettysburg Address”) is shown added to annotation layer 712 in response to movement of user input 793 provided using stylus 710 .

[0239] Figure 7II-7MM A second annotation is shown added to the annotation layer 712 in response to detecting additional user input. In particular, Fig.7II The annotation layer 712 is shown as Figure 7HH712 and illustrates the initial location of user input 794 by stylus 710 on annotation layer 712. Figure 7JJ It is shown that in response to movement of user input 794 using stylus 710 , a first portion of a handwritten annotation 795 (eg, partial text “—Nove”) having a line color set to the color option represented by color option icon 777 is added to annotation layer 712 .

[0240] In some embodiments, when detecting movement of stylus 710 to generate handwritten annotation 795, finger input is detected on a corresponding one of the annotation controls. For example, Figure 7KK A color option icon 776 (e.g., Figure 7JJ In response to detecting user input 796 on the color option represented by color option icon 776, the line color of stylus 710 is changed from the color option represented by color option icon 777 to the color option represented by color option icon 776. Figure 7KK It is shown that, according to user input 796, color option icon 776 is displayed with a current selection indication (e.g., a white ring) to indicate that the corresponding color (e.g., the color represented by color option icon 776) is the currently selected color. Therefore, the current selection indication is no longer displayed on color option icon 777. Figure 7LL It is shown that in response to continued movement of user input 794 using stylus 710 and based on finger input 796 on color option icon 776, the line color of at least a portion of handwritten annotation 795 is changed to the color represented by color option icon 776.

[0241] Figure 7MM A handwritten annotation 795 (e.g., the phrase "November 19, 1963") is shown completed in response to stopping detecting user input 794 (e.g., when touch lift-off of stylus 710). A first portion of handwritten annotation 795 (e.g., partial text "-Novem") has a line color represented by color option icon 777 (the color option selected for the first portion of handwritten annotation 795), while a second portion of handwritten annotation 795 (e.g., partial text "November 19, 1963") has a line color based on color option 776 (the color option selected by user input 796 for the second portion of handwritten annotation 795).

[0242] Figure 7NN The annotation layer 712 is shown as Figure 7MMIn some embodiments, the handwritten annotation is processed to generate searchable text associated with the handwritten annotation. In some embodiments, the searchable text is stored as metadata of the handwritten annotation or, more generally, the annotation layer. In some embodiments, optical character recognition (OCR) is performed on the handwritten annotation to recognize the handwritten characters. In some embodiments, the display of the handwritten annotation is replaced with digital text in the annotation layer (e.g., corresponding to the recognized characters). Figure 7NN In the example shown in Figure 7MM The handwritten annotations shown in (eg, the phrases “Gettysburg Address” and “-November 19, 1963”) are replaced with digital text annotations including the recognized characters while maintaining other features of the handwritten annotations, such as the line color and approximate location in the annotation layer 712.

[0243] Figure 7OO-Figure 7RR Sharing and collaborative editing of annotation layer 712 is shown. Figure 7OO The display 750 of device A is shown. Figure 7NN In response to user input from the first user of device A to share the annotation layer 712 (eg, user input is detected at a location corresponding to the share icon 752, such as a tap gesture), the annotation layer 712 is shared with the second user using device B. Figure 7PP The annotation layer 712 is shown to be transferred to the second user via device B (e.g., by sending or transmitting an object or other data including at least the annotation layer 712 to device B) and displayed on the display 751 of device B. In addition, a user icon 7100 marked with a "J" (representing the first user of device A) and a user icon 7101 marked with an "M" (representing the second user of device B) are displayed (e.g., in the upper left corner of the respective devices) to indicate which users are participating in the sharing session. In some embodiments, as Figure 7PP As shown, user icons representing corresponding users of corresponding devices are displayed in the leftmost user icon position on the corresponding devices (for example, user icon 7100 marked with "J" representing the first user at device A is displayed in the leftmost user icon position on the display 750 of device A, and user icon 7101 marked with "M" representing the second user at device B is displayed in the leftmost user icon position on the display 751 of device B).

[0244] In some embodiments, the annotation layer is configured to accept input from both a first user using device A and a second user using device B. For example, Figure 7QQ 7 shows a hand-drawn annotation 797 added to the shared annotation layer during a sharing session of a second user at device B using stylus 711. Figure 7RRAs indicated in , the hand-drawn annotation 797 is transmitted and displayed during the sharing session in the shared annotation layer on device A.

[0245] Figure 7SS The user interface 702 is shown Fig. 7A In particular, Figure 7SS A user input 798 corresponding to a cancel pinch gesture detected on the user interface 702 at a location corresponding to the map 706 is shown (eg, a Figure 7SS ) and the two contacts moving away from each other as indicated by the arrows.

[0246] Figure 7TT 7 shows a user interface 703 on a display 750 according to some embodiments. Figure 7TT In the example shown in , user interface 703 includes a map user interface. In some embodiments, in response to Figure 7SS The map interface is displayed by canceling the pinch gesture on the user interface 702 of the web browser application shown in FIG. In other embodiments, the map interface is associated with a map application (e.g., a map module 154, Figure 1A ) is associated with. Figure 7TT Also shown is an initial location (eg, initial contact location) of user input 7102 by stylus 710 detected at or near an upper edge of display 750 .

[0247] Figure 7UU The user interface 703 is shown as being inputted from a stylus 710 by a user 7102 Figure 7TT The movement of its initial position from Figure 7TT When the user interface 703 continues to be displayed, the user input 7102 of the stylus 710 is moved along Figure 7UU As the indicated path moves, an annotation layer 713 is gradually displayed over at least a portion of the user interface 703 .

[0248] Figure 7VVAn annotation layer 713 is shown displayed above the user interface 703 on the display 750 according to some embodiments. In some embodiments, the user interface 703 continues to be displayed and is at least partially visible through the annotation layer 713. In some embodiments, the annotation layer 713 includes one or more control enable indications (e.g., an annotation tool icon 771 indicating that the felt tip pen tool is the currently selected tool, a camera icon 746, a texture selection icon 748, an opaque slider 749 with a sliding position 754, a share icon 752, and an annotation layer undo enable indication 742). In some embodiments, the annotation layer 713 includes an annotation recording control area 7104, which includes, for example, a saved annotation icon 7106, an annotation recording button 7108, and a reset button 7110.

[0249] Figure 7WW-Figure 7GGG Shows from Figure 7VV A recorded annotation sequence is added to the annotation layer 713 over a series of transitions in the annotation layer 713. In some embodiments, the recorded annotation sequence can be saved and / or shared with another user. Figure 7WW-Figure 7GGG In the example shown, annotations are made and recorded on annotation layer 713 to provide map directions for recipients of the recorded annotation sequence. Figure 7WW , annotation recording button 7108 indicates that annotation recording has been initiated (e.g., in response to user input at a location corresponding to annotation recording button 7108, such as a tap gesture). Figure 7WW Also shown is user input 7111 detected on annotation layer 713 by stylus 710 .

[0250] Figure 7XX It is shown that, in response to user input 7111 (e.g., a tap gesture by stylus 710), a location annotation 7112 (e.g., to indicate a location or landmark on a map) is added to annotation layer 713. Figure 7XX Also shown is detection of user input 7113 by stylus 710 .

[0251] Figure 7YY It is shown that in response to movement of user input 7113 by stylus 710 , arrow annotation 7114 is added to annotation layer 713 .

[0252] Figure 7ZZ Detection of user input 7115 by stylus 710 is shown. Figure 7AAA 710, a handwritten annotation 7116 is added to the annotation layer 713 in response to movement of the user input 7115 by the stylus 710. Figure 7AAA In the example shown in , the handwritten annotation 7116 includes the phrase "You are here," to indicate the starting location (eg, of the recipient of the recorded annotation sequence).

[0253] Figure 7BBB-Figure 7CCC Additional annotations added to the annotation layer 713 are shown. In particular, Figure 7BBB Annotation layer 713 is shown after adding annotations such as route or path annotations 7117 (e.g., to indicate a route from one location to the next), location annotations 7118, route annotations 7119, and route information annotations 7120 (e.g., to provide information about the route, such as distance traveled or duration). Figure 7CCC The annotation layer 713 is shown after annotations such as arrow annotations 7121 and handwritten annotations 7122 are added to provide information about positioning annotations 7123.

[0254] Figure 7DDD-Figure 7FFF The annotation layer 713 is shown as Figure 7CCC In particular, Figure 7DDD Positioning annotation 7124 is shown near the bottom edge of annotation layer 713. Figure 7DDD Also shown is the initial position of user input 7125 by stylus 710 near positioning annotation 7123.

[0255] Figure 7EEE The movement of user input 7125 by stylus 710 along the path indicated by dashed line 7126 is shown. In some embodiments, the path indicated by dashed line 7126 is displayed on display 750. In some embodiments, the path indicated by dashed line 7126 is not displayed on display 750. Figure 7EEE As shown, dashed line 7126 is drawn approximately between location annotation 7123 and location annotation 7124 (e.g., dashed line 7126 begins at a location adjacent to location annotation 7123 and ends at a location adjacent to location annotation 7124), and generally follows a portion of the road on user interface 703 between those location annotations (e.g., dashed line 7126 remains within a predefined distance from a particular road on the map, such as "Washington Avenue").

[0256] Figure 7FFF It is shown that in response to user input 7125 (e.g., upon lift-off of contact of stylus 710), annotations 7127 and 7128 are added to annotation layer 713. In particular, annotation 7127 is a route annotation indicating a route from location annotation 7123 to location annotation 7124. Annotation 7128 is a route information annotation providing information about the route corresponding to route annotation 7127. In some embodiments, the route information annotation is automatically created based on route information determined from data or metadata of the underlying map content. In some embodiments, the route information annotation is manually added by the user.

[0257] Figure 7GGGThe annotation layer 713 is shown when the sequence of annotations added to the annotation layer 713 is complete. Figure 7GGG , the annotation recording button 7108 indicates that annotation recording has been stopped. Annotation layer 713 shows a complete set of annotations added to this layer, which includes the above relative Figure 7WW-Figure 7FFF As described in the comments. Figure 7GGG As shown, the complete set of annotations added to the annotation layer 713 also includes arrow annotations 7130 and image annotations 7131, which correspond to and alternatively provide information about the location annotation 7124 and the route annotation 7132 and the route information annotation 7133 (e.g., a photo of), which indicates the route to the location annotation 7134 and provides information about the route.

[0258] Figure 7HHH-Figure 7III A sequence of annotations added to a record of the annotation layer is shown. Figure 7HHH The display 750 of device A shows Figure 7GGG The annotation layer 713 is configured as described above. Figure 7WW-Figure 7GGG The complete set of added annotations has a minimum opacity of 756. In addition, Figure 7HHH Additional annotation controls are shown, including a share icon 752 .

[0259] Fig.7III The annotation layer 713 and its recorded annotation sequence are shown displayed on the display 751 of device B. In some embodiments, in response to user input from a first user of device A to share the recorded annotation layer 713 (e.g., user input at a location corresponding to the share icon 752), the recorded annotation layer 713 is shared with a second user on device B. The recorded annotation layer 713 is displayed on device B with a play icon 7136, which, when activated (e.g., by a tap on the icon), initiates playback of the recorded annotation sequence (e.g., as a video) in the annotation layer 713 on the display 751.

[0260] Figure 7JJJ-Figure 7RRR The playback of the recorded annotation sequence added to the annotation layer is shown. In some embodiments, the recorded annotation sequence is played back (e.g., the annotation video is added to the annotation layer or the animation of the annotation is displayed on the annotation layer in the same order in which the annotations are added). Figure 7JJJ User input 799 (eg, finger contact) by a second user on device B at a location corresponding to play icon 7136 is shown.

[0261] Figure 7KKK It is shown that, in response to user input 799 (or in response to ceasing to detect user input 799, e.g., in response to a detected lift-off of a finger contact), playback of the recorded annotation sequence is initiated on display 751. Figure 7KKK , the positioning annotation 7112 is first displayed on the display 751.

[0262] Next, in Figure 7LLL In the figure, corresponding to the positioning annotation 7112, an arrow annotation 7114 and a handwriting annotation 7116 are displayed.

[0263] Next, in Figure 7MMM In the figure, route annotation 7117 is displayed, showing the route from location annotation 7112 to location annotation 7118.

[0264] Next, in Figure 7NNN In the figure, route annotation 7119 is displayed with route information annotation 7120, showing the route from location annotation 7118 to location annotation 7123.

[0265] Next, in Figure 7OO In the figure, corresponding to the positioning annotation 7123, an arrow annotation 7121 and a handwriting annotation 7122 are displayed.

[0266] Next, in Figure 7 PPP In the figure, route annotation 7127 is displayed with route information annotation 7128, showing the route from location annotation 7123 to location annotation 7124.

[0267] Next, in Figure 7QQQ In the figure, corresponding to the positioning annotation 7124, an arrow annotation 7130 and an image annotation 7131 are displayed.

[0268] Finally, in Figure 7RRR In the figure, route annotation 7132 is displayed with route information annotation 7133, showing the route from location annotation 7124 to location annotation 7134.

[0269] Fig. 8A 802 (e.g., application content underlying the annotation layer) from a user interface application (e.g., a web browser application) that is independent of the annotation application, and in some embodiments, in response to user input (e.g., user input corresponding to the camera icon 746, Figure 7J Alternatively, the content 802 is pre-processed (e.g., using a pre-processing module 806) before being added to the annotation layer 818. For example, in some embodiments, the content 802 to be added to the annotation layer 818 is determined based on a user selection 804 of a portion of the application content (e.g., as described above with reference to Figure 7U-Figure 7Y described above).

[0270] In some embodiments, the annotation application receives handwritten user input 808 (e.g., including input provided using a stylus, as described above with reference to Figure 7FF-Figure 7MM ) and thus add annotations to annotation layer 818. Alternatively, before adding to annotation layer 818 (e.g., as described above with reference to Figure 7MM-Figure 7NN Before the handwriting input 808 is processed (described above), the handwriting input 808 is converted to text (eg, using a text conversion module 810).

[0271] In some embodiments, the annotation application 195 receives input corresponding to the tool and / or color selection 812 and updates the annotation layer 818 accordingly. Similarly, in some embodiments, the annotation application 195 receives input corresponding to the texture control selection 814 and / or the transparency control selection 816 and updates the annotation 818 accordingly.

[0272] In some embodiments, the annotation layer 818 is shared to or with other devices. For example, the annotation layer 818 and its corresponding annotations can be shared as a shared annotation layer 822 via a second device (e.g., to a second user using the second device). In some embodiments, for the shared annotation layer 822 (e.g., as described above with reference to Figure 7OO-Figure 7RR The invention relates to establishing a live and / or collaborative sharing session.

[0273] In some embodiments, the annotation layer 818 is stored in a memory (e.g., memory 102, Figure 1A ) in the saved annotations 196. In some embodiments, in conjunction with saving to the saved annotations 196, the annotation layer 818 is recorded (e.g., recorded using the recording module 820). For example, in some embodiments, the saved annotation layer includes a recorded annotation sequence (e.g., as described above with reference to Figure 7WW-Figure 7GGG In some embodiments, a saved annotation layer (e.g., from saved annotations 196) is called into annotation application 195 and redisplayed. In some embodiments, a saved annotation layer including annotations with recorded sequence annotations can be shared from annotation application 195 to another device (e.g., as described above with reference to Figure 7HHH - Figure 7RRR described above).

[0274] Figure 8B 1 is a block diagram illustrating exemplary components for storing annotation layers according to some embodiments. In some embodiments, one or more annotation layers are stored in a memory (e.g., memory 102, Figure 1A ) in the saved annotations 196. In some embodiments, the one or more saved annotation layers are saved in the saved annotations 196 as annotation objects (e.g., annotation objects 850-1 to 850-N).

[0275] In some embodiments, the corresponding annotation object 850-1 includes annotation metadata 852-1 and annotation data / state 854-1. Annotation data / state 854-1 represents the data and / or state of the saved annotation layer, and in some embodiments, includes data of captured content 856-1 (e.g., content captured from an application user interface, such as representations 785 and 788, Figure 7Y , or content 802, Fig. 8A ) and Note 858-1 (e.g., Notes 7112, 7114, 7116, 7118-7124, etc., Figure 7XX-Figure 7GGG , or handwriting input 808, Fig. 8A ). In some embodiments, the annotation metadata 852-1 for the saved annotation layer includes metadata for the underlying application user interface content, even if the content was not added to the annotation layer as captured content.

[0276] In some embodiments, the corresponding annotation object 850-1 includes a recorded sequence of annotations added to the annotation layer represented by the annotation object 850-1. In some embodiments, the corresponding annotation object 850-1 includes data for a sequence 860-1 of recorded annotations 858-1.

[0277] Figure 9A-9E is a flowchart illustrating a method 900 for adding annotations to an annotation layer according to some embodiments. The method 900 is performed on an electronic device (e.g., device 300, Figure 3 ; or a portable multifunction device 100, Figure 1A Some operations in method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0278] As described below, method 900 provides an intuitive way to add annotations to a unified annotation layer controlled by a standalone annotation application, allowing annotations across multiple applications and application user interfaces.

[0279] In response to user input, a representation of at least a portion of a user interface is added to the annotation layer while being displayed with the annotation layer, providing a quick transfer of a portion of the user interface to the annotation layer and providing visual feedback to the user indicating that the portion of the user interface is being transferred to the annotation layer. Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently. In addition, when adding annotations to application content, the method reduces the amount, degree and / or nature of input from the user, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, users are enabled to input faster and more efficiently save power and increase the interval between battery charges.

[0280] The device displays (902) a first user interface of a first application (eg, user interface 702 of a web browser application, Fig. 7A ). A first application (eg, browser module 147, Figure 1A ) is a corresponding application among the plurality of applications (e.g., application 136 other than annotation application 195, Figure 1A ), and the first user interface includes first application content.

[0281] In some embodiments, the plurality of applications includes (904) at least two applications selected from the group consisting of: a browser application, a photo application, a communication application, and a content editing application. In some embodiments, the plurality of applications also includes a map application.

[0282] While displaying a first user interface of a first application, the device detects (906) a first input (eg, user input 708, Figure 7B Or the user enters 732, Figure 7G ). In some embodiments, the first input is a gesture. In some embodiments, the gesture is an application-independent predefined gesture, independent of any of a plurality of different software applications. In some embodiments, the gesture is a swipe or pan gesture. In some embodiments, the first input is a tap gesture (e.g., by finger contact).

[0283] In some embodiments, the device also includes (908) one or more sensors to detect signals from a stylus associated with the device, and the first input is a first gesture performed using the stylus and selected from the group consisting of: an edge gesture (e.g., user input 708, Figure 7B-7C ), including a pencil flick gesture of a flicking motion toward the touch-sensitive display, and a gesture on a predefined affordance displayed on the touch-sensitive display. In some embodiments, the predefined affordance is not responsive to touch input from a finger contact. In some embodiments, the predefined affordance is smaller than the standard size of a touch target contacted by a finger (e.g., because a stylus is more precise and can be used to reliably select smaller touch targets).

[0284] In some embodiments, detecting the first input includes (910) determining that the first gesture is performed using a stylus. In some embodiments, in response to detecting the non-stylus input as an edge gesture, the device performs a different operation other than displaying the annotation layer. For example, in some embodiments, when an edge gesture performed with the aid of a finger is detected, a different user interface other than the annotation user interface is displayed, such as a control center user interface (including controls for controlling device functions), a notification center user interface (including notifications of communications received by the device), or a previously displayed page user interface (e.g., due to performing a navigation operation such as navigating back through one or more pages). In some embodiments, if the edge gesture is being performed with the stylus, the device determines whether the edge gesture is being performed by a finger or a stylus when the edge gesture is detected, and displays the annotation user interface (e.g., as described above with reference to FIG. 1 ). Figure 7B-7C ), and displaying a different user interface if the edge gesture is being performed with a finger (e.g., as described above with reference to Figure 7D-Figure 7E described above).

[0285] In some embodiments, the first input is detected (912) independently of the plurality of applications. In some embodiments, the first input is detected while any of the plurality of applications is executed or displayed.

[0286] In response to detecting a first input (eg, user input 708, Figure 7C ), the device continues (914, Fig. 9B ) displays the first application content, and in addition, at least a first portion of a first user interface (e.g., a user interface 702 of a web browser application, Figure 7C ) above the annotation layer (e.g., annotation layer 712, Figure 7C ). The annotation layer is associated with a user application (e.g., annotation application 195, Figure 1A) is associated with.

[0287] In some embodiments, the annotation layer has (916) an adjustable opacity (e.g., as described above with reference to Figure 7J-7M ), and the device changes the opacity of the annotation layer in response to a fifth input (e.g., by slider position 753, 754, 756, or 758, Figure 7J-7M ), where increasing the opacity of the annotation layer reduces the visibility of application content displayed beneath the annotation layer.

[0288] In some embodiments, the annotation layer has (918) an adjustable texture (e.g., as described above with reference to Figure 7P-Figure 7Q In some embodiments, the adjustable texture is solid, lattice, grid, dotted or translucent. In some embodiments, the texture includes fading, blurring and / or translucency.

[0289] While the annotation layer is displayed over at least the first portion of the first user interface, the device detects (920) a second input (e.g., user input 782, Figure 7U ).

[0290] In some embodiments, in response to detecting the second input (922), the device creates (924) a representation of at least a second portion of the user interface added to the annotation layer.

[0291] In response to detecting the second input (922), the device displays a representation of at least a second portion of the first user interface (e.g., representation 785, Figure 7W ) is added (926) to the annotation layer (e.g., a screen shot or screen capture of the application content displayed side by side with the annotation layer for an overlay annotation layer configuration below the annotation layer, or a side by side configuration in which the annotation layer is displayed side by side with the first application content).

[0292] In some embodiments, the second input is (928) a gesture having a path, and the boundary of the second portion of the user interface to be represented in the annotation layer is determined based on the path of the second input gesture (e.g., as described above with reference to Figure 7U-7W In some embodiments, the second input corresponds to a virtual knife tool (eg, virtual knife tool icon 770, Figure 7U ) is used to select content to be added to the annotation layer. In some embodiments, the virtual knife tool is one of the available tools or tool modes for the annotation layer (e.g., as described above with reference to Figure 7R In some embodiments, the virtual knife tool is one of the available tools or tool modes for the stylus. In some embodiments, the border of the second portion of the user interface (e.g., indicated by selection indicator 784, Figure 7V) is the content area to be added to the annotation layer (eg, corresponding to the content of representation 785, Figure 7W ). In some embodiments, the boundary is based on the path of the gesture and, alternatively, the structure of the content (e.g., the underlying application content). In some embodiments, even if the gesture path deviates from the content boundary (e.g., as described above with reference to Figure 7X-Figure 7Y The boundary of the content area is automatically or intelligently determined based on the content structure.

[0293] When displaying the representation in the annotation layer, the device detects (930, Fig. 9C ) a set of one or more additional inputs (e.g. Figure 7FF User input in 793 or Fig.7II User input in 794).

[0294] In response to detecting the set of one or more additional inputs, the device adds (932) an annotation to the annotation layer (e.g., annotation 795, Figure 7MM ).

[0295] In some embodiments, in different applications (e.g., annotation application layer program 195, Figure 1A ) to add (934) a representation to the annotation layer under the control of a user application of the plurality of applications, and to add annotations to the annotation layer under the control of a user application different from the plurality of applications.

[0296] In some embodiments, annotations are added to the annotation layer (936) in response to both a set of one or more additional inputs and application content displayed below the annotation layer. In some embodiments, this is done using metadata of the underlying application content. For example, when drawing on the annotation layer above a map, the annotation lines can snap to objects on the map, such as roads and landmarks (e.g., as described above with reference to FIG. Figure 7DDD-Figure 7FFF described above).

[0297] In some embodiments, the device repeats the method described above for a second application (e.g., different from the first application) using the same annotation layer previously used with the first application. More specifically, in some embodiments, the representation is (938) a first representation, the group of one or more additional inputs is a first group of one or more additional inputs, and the annotations are a first group of annotations. After adding the first group of annotations to the annotation layer, the device displays a second user interface (e.g., different from the first application) of a second application (e.g., a map application, a photo application, a communication application, a content editing application, etc.). The second user interface includes second application content (e.g., different from the first application content). While displaying the second user interface of the second application, the device detects a third input. In some embodiments, the third input includes multiple user inputs to select an annotation layer of a representation of at least a second portion of the first user interface previously added (e.g., the same annotation layer previously used with the first application). In response to detecting the third input, the device continues to display the second application content, and in addition, displays the annotation layer above at least a first portion of the second user interface. While displaying the annotation layer above at least the first portion of the second user interface, the device detects a fourth input. In response to detecting the fourth input, the device adds a second representation of at least a second portion of the second user interface to the annotation layer. While displaying the second representation in the annotation layer, the device detects a second set of one or more additional inputs, and in response to detecting the second set of one or more additional inputs, the device adds a second set of annotations to the annotation layer.

[0298] In some embodiments, the device repeats the above method for a second application (e.g., different from the first application) using a different annotation layer than the annotation layer previously used with the first application (e.g., a blank canvas). More specifically, in some embodiments, the annotation layer is (940, Fig.9D) a first annotation layer, the representation being a first representation, the set of one or more additional inputs being a first set of one or more additional inputs, and the annotations being a first set of annotations. After adding the first set of annotations to the annotation layer, the device displays a second user interface of a second application (e.g., different from the first application) of the plurality of applications. The second user interface includes second application content (different from the first application content). While displaying the second user interface of the second application, the device detects a third input. In response to detecting the third input, the device continues to display the second application content, and in addition, displays a second annotation layer different from the first annotation layer (e.g., a blank canvas) over at least a first portion of the second user interface. While displaying the second annotation layer over at least the first portion of the second user interface, the device detects a fourth input. In response to detecting the fourth input, the device adds a second representation of at least a second portion of the second user interface to the second annotation layer. While displaying the second representation in the second annotation layer, the device detects a second set of one or more additional inputs, and in response to detecting the second set of one or more additional inputs, the device adds a second set of annotations to the annotation layer.

[0299] In some embodiments, the electronic device further includes (942, Fig.9E ) one or more sensors to detect signals from a stylus associated with the device. In some embodiments, the one or more additional inputs include a first additional input, and the first additional input is input to the electronic device using the stylus.

[0300] In some embodiments, the one or more additional inputs include (944) a second additional input, and the second additional input includes input from both the stylus and one or more finger contacts (e.g., user input 794 from stylus 710 and user input 796 from finger contacts, Figure 7KK ). For example, in some embodiments, the device detects finger input on the annotation layer and, in response, displays a set of drawing controls for controlling lines output by the stylus (e.g., as described above with reference to Figure 7II-7MM In some embodiments, the device detects movement of the stylus (e.g., user input 794, Figure 7KK ) to draw an annotation (e.g., a line), and in conjunction with drawing the annotation based on the movement of the stylus, detecting a finger input (e.g., user input 796, Figure 7KK ), and as the stylus continues to move on the touch-sensitive display, in response to detecting finger input on the corresponding drawing, changing the annotation output (e.g., the color, style, opacity, or width of the line). Figure 7KK-Figure 7LLAs described, the color of annotation 795 changes in response to both continued drawing input 794 by stylus 710 and finger contact 796 on color option 776 .

[0301] In some embodiments, at least one of the annotations (946) is a handwritten annotation input using a stylus (e.g., handwritten annotation 795, Figure 7JJ-Figure 7MM ).

[0302] In some embodiments, the device processes (948) the handwritten annotation to generate searchable text associated with the handwritten annotation (e.g., as described above with reference to Figure 7NN described above).

[0303] In some embodiments, the one or more additional inputs include a third additional input, and based on determining that the third additional input is provided using a finger, the device displays (950) a set of annotation controls for the annotation layer (e.g., as described above with reference to Figure 7I-7J In some embodiments, based on determining that the third additional input is provided using a stylus, the annotation control is not displayed (e.g., as described above with reference to Figure 7U-7FF described above).

[0304] In some embodiments, the device detects (952) a sixth input and, in response to the sixth input, sends or transmits an object including at least the annotation layer (e.g., as described above with reference to Figure 7OO and Figure 7HHH In some embodiments, the object is a file, such as a note, image file, or other document. In some embodiments, the object includes displayed application content. In some embodiments, the object includes metadata for the displayed application content below the annotation layer. In some embodiments, each performance of the methods described herein includes storing the object in a memory, or sending or transmitting the object to another electronic device. In some embodiments, sending or transmitting the object to another electronic device includes transmitting the annotation layer to a second user of the second electronic device (e.g., a second user using device B, Figure 7OO or Figure 7HHH ) for presentation to the second user. In some embodiments, in response to input from the first user at the first device when the annotation layer is shared, the first device transmits the annotations added to the annotation layer to the second device. In some embodiments, the annotation layer is configured to receive input and / or annotations from both the first user and the second user. In some embodiments, in response to input from the second user at the second device when the annotation is shared, the device displays the annotations added to the annotation layer at the first device (e.g., as a live share or virtual whiteboard session, see above with reference to Figure 7OO-Figure 7RR described above).

[0305] In some embodiments, the device stores (954) an object including at least one annotation layer; after storing the object including at least the annotation layer, detects a subsequent input; and in response to detecting the subsequent input, displays at least the annotation layer (e.g., the subsequent input initiates a call to the annotation layer from the stored object, alternatively by activating Figure 7VV , and the saved annotation icon 7106 shown in , and the annotation layer showing the call).

[0306] In some embodiments, the object including at least the annotation layer further includes (956): a method for preparing an annotation sequence for the annotation layer, and in response to detecting a subsequent input (eg, user input 799, Figure 7JJJ ) displaying at least the annotation layer comprises: replaying the annotation sequence used to prepare the annotation layer (e.g., as described above with reference to Figure 7KKK-Figure 7RRR and after playing back the annotation sequence, displaying the annotation layer (e.g., as described above with reference to Figure 7RRR In some embodiments, after playback of the annotation sequence, all annotations used to prepare the annotation layer are displayed simultaneously.

[0307] It should be understood that 9A to 9E The specific order in which the operations are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations can be performed. A person of ordinary skill in the art will recognize many ways to reorder the operations described herein.

[0308] According to some embodiments, Fig.10 1 shows a functional block diagram of an electronic device 1000 configured according to the principles of the various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, or a combination of hardware and software that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Fig.10 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0309] like Fig.10 As shown, the electronic device 1000 includes a touch-sensitive display unit 1002 and a processing unit 1004 coupled to the touch-sensitive display unit 1002, the touch-sensitive display unit being configured to display a first user interface of a first application and receive user input. In some embodiments, the processing unit 1004 includes a display enabling unit 1006, a detection unit 1008, an adding unit 1010, a creating unit 1012, a text processing unit 1014, a changing unit 1016, a communication unit 1018, and a storage unit 1020.

[0310] Processing unit 1004 is configured to: enable display of a first user interface of the first application (e.g., with the aid of display enabling unit 1006), wherein the first application is a corresponding application among multiple applications, and wherein the first user interface includes first application content; while displaying the first user interface of the first application, detect a first input (e.g., with the aid of detection unit 1008); in response to detecting the first input, enable continued display of the first application content, and in addition, enable display of an annotation layer above at least a first portion of the first user interface (e.g., with the aid of display enabling unit 1006), wherein the annotation layer is associated with a user application different from the multiple applications; while displaying the annotation layer above at least the first portion of the first user interface, detect a second input (e.g., with the aid of detection unit 1008); in response to detecting the second input, add a representation of at least a second portion of the first user interface to the annotation layer (e.g., with the aid of adding unit 1010); while displaying the representation in the annotation layer, detect a set of one or more additional inputs (e.g., with the aid of detection unit 1008); and in response to detecting the set of one or more additional inputs, add annotations to the annotation layer (e.g., with the aid of adding unit 1010).

[0311] In some embodiments, processing unit 1004 is configured to, in response to detecting the second input, create a representation of at least a second portion of the user interface added to the annotation layer (e.g., with the help of creating unit 1012).

[0312] In some embodiments, representations are added to the annotation layer under the control of a user application that is distinct from the plurality of applications, and annotations are added to the annotation layer under the control of a user application that is distinct from the plurality of applications.

[0313] In some embodiments, the second input is a gesture having a path, and the boundaries of the second portion of the user interface to be represented in the annotation layer are determined based on the path of the second input gesture.

[0314] In some embodiments, the representation is a first representation, the one or more additional inputs are a first group of one or more additional inputs, the annotations are a first group of annotations, and the processing unit 1004 is configured to, after adding the first group of annotations to the annotation layer, enable display of a second user interface of a second application among the multiple applications (e.g., with the aid of a display enabling unit 1006), wherein the second user interface includes second application content; while displaying the second user interface of the second application, detect a third input (e.g., with the aid of a detection unit 1008); in response to detecting the third input, enable continued display of the second application content, and in addition, enable the annotation layer in the display over at least a first portion of a second user interface (e.g., with the aid of display enabling unit 1006); detecting a fourth input (e.g., with the aid of detecting unit 1008) while the annotation layer is displayed over at least the first portion of the second user interface; in response to detecting the fourth input, adding a second representation of at least a second portion of the second user interface to the annotation layer (e.g., with the aid of adding unit 1010); while displaying the second representation in the annotation layer, detecting a second group of one or more additional inputs (e.g., with the aid of detecting unit 1008); and in response to detecting the second group of one or more additional inputs, adding a second group of annotations to the annotation layer (e.g., with the aid of adding unit 1010).

[0315] In some embodiments, the annotation layer is a first annotation layer, the representation is a first representation, the set of one or more additional inputs is a first set of one or more additional inputs, the annotations are a first set of annotations, and the processing unit 1004 is configured to, after adding the first set of annotations to the annotation layer, enable display of a second user interface of a second application among the multiple applications (for example, with the aid of a display enabling unit 1006), wherein the second user interface includes second application content; while displaying the second user interface of the second application, detect a third input (for example, with the aid of a detection unit 1008); in response to detecting the third input, enable continued display of the second application content, and in addition, enable a display different from the first annotation layer. The invention also provides for the step of displaying a second annotation layer over at least a first portion of the second user interface (e.g., with the aid of display enabling unit 1006); detecting a fourth input (e.g., with the aid of detecting unit 1008) while the second annotation layer is displayed over at least the first portion of the second user interface; in response to detecting the fourth input, adding a second representation of at least a second portion of the second user interface to the second annotation layer (e.g., with the aid of adding unit 1010); detecting a second set of one or more additional inputs (e.g., with the aid of detecting unit 1008) while the second representation is displayed in the second annotation layer; and in response to detecting the second set of one or more additional inputs, adding a second set of annotations to the annotation layer (e.g., with the aid of adding unit 1010).

[0316] In some embodiments, the electronic device 1000 also includes one or more sensor units (e.g., sensor unit 1022) configured to detect signals from a stylus associated with the device, wherein the first input is a first gesture performed using the stylus and selected from the group consisting of: an edge gesture moving from an edge of the touch-sensitive display unit 1002 to an internal position of the touch-sensitive display unit 1002, a pencil flick gesture including a flick motion toward the touch-sensitive display unit 1002, and a gesture on a predefined affordance displayed on the touch-sensitive display unit 1002.

[0317] In some embodiments, detecting the first input includes determining that the first gesture is performed using a stylus.

[0318] In some embodiments, the electronic device 1000 also includes one or more sensor units (e.g., sensor unit 1022) configured to detect signals from a stylus associated with the device; one or more additional inputs include a first additional input; and the first additional input is input into the electronic device using the stylus.

[0319] In some embodiments, the one or more additional inputs include a second additional input; and the second additional input includes input from both the stylus and one or more finger contacts.

[0320] In some embodiments, at least one of the annotations is a handwritten annotation entered using a stylus.

[0321] In some embodiments, processing unit 1004 is configured to process handwritten annotations to generate searchable text associated with the handwritten annotations (e.g., with the aid of text processing unit 1014).

[0322] In some embodiments, the one or more additional inputs include a third additional input, and the processing unit 1004 is configured to enable display of a set of annotation controls of the annotation layer (e.g., with the aid of the display enabling unit 1006) based on determining that the third additional input is performed using a finger.

[0323] In some embodiments, the first input is detected independent of the multiple applications.

[0324] In some embodiments, the plurality of applications includes at least two applications selected from the group consisting of: a browser application, a photo application, a communication application, and a content editing application.

[0325] In some embodiments, the annotation layer has an adjustable opacity, and the processing unit 1004 is configured to change the opacity of the annotation layer in response to a fifth input (e.g., via a changing unit 1016), wherein increasing the opacity of the annotation layer reduces visibility of application content displayed below the annotation layer.

[0326] In some embodiments, the annotation layer has an adjustable texture, and the adjustable texture is solid, lattice, grid, dotted, or semi-transparent.

[0327] In some embodiments, the processing unit 1004 is configured to: detect a sixth input (eg, with the detection unit 1008 ); and in response to the sixth input, send or transmit an object including at least the annotation layer (eg, with the communication unit 1018 ).

[0328] In some embodiments, the processing unit 1004 is configured to: store an object including at least the annotation layer (e.g., with the aid of the storage unit 1020); after storing the object including at least the annotation layer, detect subsequent input (e.g., with the aid of the detection unit 1008); and in response to detecting the subsequent input, enable display of at least the annotation layer (e.g., with the aid of the display enabling unit 1006).

[0329] In some embodiments, the object including at least the annotation layer also includes an annotation sequence for generating the annotation layer; and displaying at least the annotation layer in response to detecting subsequent input includes: playing back the annotation sequence for generating the annotation layer; and after playing back the annotation sequence, displaying the annotation layer.

[0330] In some embodiments, adding an annotation to the annotation layer is responsive to both a set of one or more additional inputs and application content displayed below the annotation layer.

[0331] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (for example, as described above with respect to Figure 1A and Figure 3 described) or application-specific chips.

[0332] References 9A to 9E The operation is optionally performed by Figure 1A to Figure 1B or Fig.10 . For example, display operation 902, detection operation 906, and response operation 916 are optionally implemented by event classifier 170, event identifier 180, and event handler 190. Event monitor 171 in event classifier 170 detects contact (or proximity) on touch-sensitive display 112, and event distributor module 174 transmits event information to application 136-1. The corresponding event identifier 180 of application 136-1 compares the event information with the corresponding event definition 186, and determines whether the first contact (or proximity) at the first position on the touch-sensitive surface (or whether the rotation of the device) corresponds to a predefined event or sub-event, such as the selection of an object on a user interface, or the rotation of the device from one orientation to another orientation. When a corresponding predefined event or sub-event is detected, event identifier 180 activates an event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art will clearly know that based on Figure 1A to Figure 1B How the components depicted in the figure can implement other processes.

[0333] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best illustrate the principles of the present invention and its practical application, so that other technical personnel in the field can best use the present invention and the various described embodiments with various modifications suitable for the specific purpose contemplated.

Claims

1. A method for annotation processing, comprising: At an electronic device communicating with a display device: displaying, via the display device, a user interface including application content generated by the electronic device; While displaying the user interface including the application content generated by the electronic device, detecting a first input moving over a portion of the application content; In response to detecting the first input, displaying an annotation user interface simultaneously with a representation of the application content over which the first input moves; and When the annotation user interface is displayed: displaying a markup tool set including a plurality of tools for adding marks to the application content over which the first input is moved, wherein the markup tool set is displayed simultaneously with the representation of the application content over which the first input is moved; detecting a second input selecting a first marking tool in the marking tool set; adding one or more marks on an annotation layer overlying the application content over which the first input moves when the first markup tool is selected, wherein the one or more marks are generated by the electronic device based on one or more additional inputs using the selected first markup tool; as well as An opacity of the annotation layer is changed, wherein increasing the opacity of the annotation layer reduces visibility of the application content over which the first input moves relative to the one or more markers on the annotation layer.

2. The method of claim 1, wherein the annotation layer comprises an adjustable texture, and the method comprises: When the annotation layer is displayed, the texture of the annotation layer is changed according to the selection of a corresponding tool in the marking tool set.

3. The method according to claim 1, further comprising: while displaying the annotation layer, detecting a third input selecting a portion of the application content; as well as In response to detecting the third input, a representation of the selected portion of the application content is added to the annotation layer.

4. The method according to claim 1, further comprising: displaying in the user interface a first affordance for capturing a snapshot of at least a portion of the application content visible through the annotation layer; detecting a fourth input selected by the first affordance; as well as In response to detecting the fourth input selecting the first affordance, capturing a snapshot of the at least a portion of the application content visible through the annotation layer.

5. The method according to any one of claims 1 to 4, further comprising: The annotation layer including the one or more tags and metadata for the at least a portion of the application content visible through the annotation layer is saved.

6. The method according to any one of claims 1 to 4, further comprising: displaying in the user interface a second affordance for sharing a view of the annotation layer and a portion of the application content visible through the annotation layer; detecting a fifth input selected by said second affordance; as well as In response to detecting the fifth input, the annotation layer and the view of the portion of the application content visible through the annotation layer are shared.

7. The method according to any one of claims 1 to 4, further comprising: detecting the one or more additional inputs, the one or more additional inputs comprising one or more gestures; as well as In response to detecting the one or more additional inputs, the one or more tags are added on the annotation layer based at least in part on the metadata of the at least a portion of the application content visible through the annotation layer and in accordance with the one or more gestures.

8. The method of any one of claims 1 to 4, further comprising, based on the structure of the at least a portion of the application content visible through the annotation layer: determining boundaries of a content area within the application content; and The content region is selected, wherein the selected content region is added to the annotation layer based on the one or more additional inputs.

9. The method according to any one of claims 1 to 4, wherein: The at least a portion of the application content includes one or more objects within the application content; and The method further comprises: detecting the one or more additional inputs; and In response to detecting the one or more additional inputs, a line is added to the annotation layer, wherein the line added to the annotation layer is snapped to an object of the one or more objects within the application content.

10. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by an electronic device having a display device, cause the electronic device to: displaying, via the display device, a user interface including application content generated by the electronic device; While displaying the user interface including the application content generated by the electronic device, detecting a first input moving over a portion of the application content; as well as in response to detecting the first input, displaying an annotation user interface simultaneously with a representation of the application content over which the first input is moved; When the annotation user interface is displayed: displaying a markup tool set including a plurality of tools for adding marks to the application content over which the first input is moved, wherein the markup tool set is displayed simultaneously with the representation of the application content over which the first input is moved; detecting a second input selecting a first marking tool in the marking tool set; adding one or more marks on an annotation layer overlying the application content over which the first input moves when the first markup tool is selected, wherein the one or more marks are generated by the electronic device based on one or more additional inputs using the selected first markup tool; as well as An opacity of the annotation layer is changed, wherein increasing the opacity of the annotation layer reduces visibility of the application content over which the first input moves relative to the one or more markers on the annotation layer.

11. The computer-readable storage medium of claim 10, wherein the one or more programs include instructions that, when executed by the electronic device, cause the electronic device to perform the method of any one of claims 2 to 9.

12. An electronic device comprising: Display devices; one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for: displaying, via the display device, a user interface including application content generated by the electronic device; While displaying the user interface including the application content generated by the electronic device, detecting a first input moving over a portion of the application content; In response to detecting the first input, displaying an annotation user interface simultaneously with a representation of the application content over which the first input moves; and When the annotation user interface is displayed: displaying a markup tool set including a plurality of tools for adding marks to the application content over which the first input is moved, wherein the markup tool set is displayed simultaneously with the representation of the application content over which the first input is moved; detecting a second input selecting a first marking tool in the marking tool set; adding one or more marks on an annotation layer overlying the application content over which the first input moves when the first markup tool is selected, wherein the one or more marks are generated by the electronic device based on one or more additional inputs using the selected first markup tool; as well as An opacity of the annotation layer is changed, wherein increasing the opacity of the annotation layer reduces visibility of the application content over which the first input moves relative to the one or more markers on the annotation layer.

13. The electronic device according to claim 11, wherein the one or more programs include instructions for executing the method according to any one of claims 2 to 9.

Images