Device, method and graphical user interface for annotating text

The implementation of methods for converting non-text areas into drawing panels and rearranging text and annotations on touch-sensitive displays addresses inefficiencies in existing text annotation methods, enhancing user satisfaction and energy conservation.

CN114675774BActive Publication Date: 2024-12-06APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210384766.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2016-09-23
Filing Date
2017-09-25
Publication Date
2024-12-06
Estimated Expiration
2037-09-25

AI Technical Summary

Technical Problem

Existing methods for annotating text on electronic devices with touch-sensitive surfaces are cumbersome and inefficient, imposing a cognitive burden on users and wasting energy, particularly in battery-powered devices.

Method used

Implementing methods and interfaces that allow for faster and more efficient text annotation by converting a non-text area into a drawing panel for input and enabling rearrangement of text and annotations on a touch-sensitive display.

Benefits of technology

Enhances user satisfaction and reduces cognitive burden while conserving energy by providing intuitive and efficient text annotation methods on devices with touch-sensitive surfaces.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114675774B_ABST
    Figure CN114675774B_ABST
Patent Text Reader

Abstract

The present disclosure relates to devices, methods, and graphical user interfaces for annotating text. A method is performed at a device having a touch-sensitive display, the method comprising displaying a text area, the text area comprising a first portion of text and a second portion of text, detecting input within the text area, in response to the detected input, moving the first portion of text relative to the second portion of text to display a non-text area between the first portion of text and the second portion of text, and converting a portion of the non-text area into a drawing panel, the drawing panel being configured to receive drawing input via the touch-sensitive display and display corresponding graphics in response to receiving the drawing input. Another method is performed at a device having a touch-sensitive display, the method comprising displaying text, detecting a set of annotation inputs via the touch-sensitive display and displaying annotations associated with a portion of the text, receiving a request to rearrange the text, displaying a second arrangement of the text, and displaying a second representation of the annotation adjacent to a corresponding rearranged portion of the text.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of PCT international application No. 201780058411.8, which entered the Chinese national phase and whose international application date is September 25, 2017 and whose invention name is “Device, method and graphical user interface for annotating text”. Technical Field

[0002]

[0013] This document generally relates to electronic devices with touch-sensitive surfaces, including but not limited to electronic devices with touch-sensitive surfaces that allow annotation of displayed text. Background Art

[0003] The use of touch-sensitive surfaces as input devices for computers and other electronic computing devices has grown significantly in recent years. Exemplary touch-sensitive surfaces include touch pads and touch screen displays. Such surfaces are widely used to manipulate user interface objects on displays.

[0004] Exemplary manipulations include adjusting the position and / or size of one or more user interface objects, or activating a button or opening a file / application represented by a user interface object, as well as associating metadata with one or more user interface objects or otherwise manipulating the user interface. Exemplary user interface objects include digital images, videos, text, icons, control elements (such as buttons), and other graphics. In some cases, a user will need to perform such manipulations of user interface objects in: a file management program (e.g., Finder from Apple Inc. (Cupertino, California)); an image management application (e.g., Aperture, iPhoto, Photos from Apple Inc. (Cupertino, California)); a digital content (e.g., video and music) management application (e.g., iTunes from Apple Inc. (Cupertino, California)); a drawing application; a presentation application (e.g., Keynote from Apple Inc. (Cupertino, California)); a word processing application (e.g., Pages from Apple Inc. (Cupertino, California)); a website creation application (e.g., iWeb from Apple Inc. (Cupertino, California)); a disk editing application (e.g., iDVD from Apple Inc. (Cupertino, California)); or a spreadsheet application (e.g., Numbers from Apple Inc. (Cupertino, California)).

[0005] Some user interfaces display text and allow users to annotate the text. However, methods for adding annotations to text are cumbersome and inefficient. For example, using a sequence of mouse-based inputs to select one or more user interface objects and perform one or more actions on the selected user interface objects is tedious and imposes a significant cognitive burden on the user. In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the invention

[0006] Therefore, electronic devices need to have faster and more efficient methods and interfaces for annotating text. Such methods and interfaces optionally supplement or replace conventional methods for annotating text. Such methods and interfaces reduce the cognitive burden on users and produce more efficient human-computer interfaces. For battery-powered devices, such methods and interfaces can save power and increase the time between battery charges.

[0007] The above-mentioned defects and other problems associated with the user interface of an electronic device with a touch-sensitive surface can be reduced or eliminated by means of the disclosed device. In some embodiments, the device is a desktop computer. In some embodiments, the device is portable (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the device has a touchpad. In some embodiments, the device has a touch-sensitive display (also referred to as a "touch screen" or a "touch screen display"). In some embodiments, the device has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI mainly through stylus and / or finger contact and gestures on the touch-sensitive surface. In some embodiments, these functions optionally include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet production, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0008] According to some embodiments, the present invention discloses a method performed at a device having one or more processors, a non-volatile memory, and a touch-sensitive display. The method includes displaying a text area on a touch-sensitive display, the text area including a first portion of text and a second portion of text. The method includes detecting input within the text area on the touch-sensitive display. The method includes, in response to detecting input within the text area, moving the first portion of the text relative to the second portion of the text to display a non-text area between the first portion of the text and the second portion of the text, the non-text area not including any text, and converting at least a portion of the non-text area into a drawing panel, the drawing panel being configured to receive drawing input via the touch-sensitive display and display corresponding graphics in response to receiving the drawing input.

[0009] According to some embodiments, the present invention discloses a method performed at a device having one or more processors, a non-volatile memory, and a touch-sensitive display. The method includes displaying a first arrangement of text on a touch-sensitive display, wherein a first portion of the text has a first position relative to a second portion of the text. The method includes, when the text is displayed in a first arrangement, detecting a set of one or more annotation inputs via the touch-sensitive display. The method includes, in response to detecting a set of one or more annotation inputs, displaying a first representation of an annotation associated with both the first portion of the text and the second portion of the text on the touch-sensitive display, wherein at least a portion of the first representation is displayed adjacent to the first portion of the text in a first arrangement, and at least a portion of the first representation is displayed adjacent to the second portion of the text in a first arrangement. The method includes receiving a request to rearrange the text after displaying the first representation of the annotation. The method includes, in response to receiving a request to rearrange the text, concurrently displaying a second arrangement of the text and a second representation of the annotation on the display, in which the first portion of the text has a second position relative to the second portion of the text, the second representation of the annotation is associated with both the first portion of the text and the second portion of the text, wherein at least a portion of the second representation is displayed adjacent to the first portion of the text in a second arrangement, and at least a portion of the second representation is displayed adjacent to the second portion of the text in a second arrangement.

[0010] According to some embodiments, an electronic device includes a display unit configured to display a user interface, one or more input units configured to receive user input, and a processing unit coupled to the display unit and the one or more input units. The processing unit is configured to display a text area on the display unit, the text area including a first part of the text and a second part of the text. The processing unit is configured to detect input in the text area via one or more input units. The processing unit is configured to, in response to detecting input in the text area, move the first part of the text relative to the second part of the text to display a non-text area between the first part of the text and the second part of the text, the non-text area does not include any text, and convert at least a portion of the non-text area into a drawing panel, the drawing panel being configured to receive drawing input via a touch-sensitive display and display corresponding graphics in response to receiving the drawing input.

[0011] According to some embodiments, an electronic device includes a display unit configured to display a user interface, one or more input units configured to receive user input, and a processing unit coupled to the display unit and the one or more input units. The processing unit is configured to display a first arrangement of text on the display unit, wherein a first portion of the text has a first position relative to a second portion of the text. The processing unit is configured to detect a set of one or more annotation inputs via one or more input units when the text is displayed in the first arrangement. The processing unit is configured to display a first representation of an annotation associated with both the first portion of the text and the second portion of the text on the display unit in response to detecting a set of one or more annotation inputs. At least a portion of the first representation is displayed adjacent to the first portion of the text in the first arrangement, and at least a portion of the first representation is displayed adjacent to the second portion of the text in the first arrangement. The processing unit is configured to receive a request to rearrange the text after displaying the first representation of the annotation. The processing unit is configured to concurrently display a second arrangement of the text and a second representation of the annotation on the display unit in response to receiving a request to rearrange the text, in which the first portion of the text has a second position relative to the second portion of the text in the second arrangement herein; the second representation of the annotation is associated with both the first portion of the text and the second portion of the text. The second position is different from the first position. The second representation is different from the first representation. At least a portion of the second representation is displayed adjacent to the first portion of the text in a second arrangement, and at least a portion of the second representation is displayed adjacent to the second portion of the text in a second arrangement.

[0012] According to some embodiments, an electronic device includes a display, an input device, one or more processors, a non-volatile memory, and one or more programs; one or more programs are stored in the non-volatile memory and configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the operations of any method described herein. According to some embodiments, a non-volatile computer-readable storage medium stores instructions therein, and when these instructions are executed by one or more processors of an electronic device having a display and an input device, the device performs or causes the execution of the operations of any of the methods described herein. According to some embodiments, a graphical user interface on an electronic device includes one or more elements displayed in any of the methods described above, wherein the electronic device has a display, an input device, a memory, and one or more processors for executing one or more programs stored in the non-volatile memory, and the one or more elements are updated in response to input, as described in any of the methods described herein. According to some embodiments, an electronic device includes: a display, an input device; and a device for performing or causing the execution of the operations of any of the methods described herein. According to some embodiments, an information processing device used in an electronic device having a display and an input device includes a device for performing or causing the execution of the operations of any of the methods described herein.

[0013] Thus, electronic devices having a display, a touch-sensitive surface, and optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface have faster, more efficient methods and interfaces for annotating text, thereby increasing effectiveness, efficiency, and user satisfaction with such devices. Such methods and interfaces can supplement or replace conventional methods for annotating text. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0015] Figure 1A is a block diagram illustrating a portable multifunction device with a touch-sensitive display according to some embodiments.

[0016] Figure 1B is a block diagram illustrating example components for event processing according to some embodiments.

[0017] Figure 2 A portable multifunction device with a touch screen according to some embodiments is shown.

[0018] Figure 3 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments.

[0019] Figure 4A An exemplary user interface for an application menu on a portable multifunction device according to some embodiments is shown.

[0020] Figure 4B An exemplary user interface for a multifunction device having a touch-sensitive surface separate from the display is shown according to some embodiments.

[0021] Figures 5A-5AF An exemplary user interface for adding annotations to text is shown according to some embodiments.

[0022] Figures 6A-6D is a flow chart illustrating a method of adding drawing annotations to text according to some embodiments.

[0023] Figures 7A-7D is a flow chart illustrating a method of adding hand-drawn annotations to reflowable text according to some embodiments.

[0024] Figures 8 to 9 is a functional block diagram of an electronic device according to some embodiments. DETAILED DESCRIPTION

[0025] Many electronic devices have graphical user interfaces for displaying text, such as e-books or word processing documents. Some applications allow users to add annotations to text, such as notes or highlights. In the embodiments described below, the application allows users to add hand-drawn annotations to text. For example, in some embodiments, a drawing panel is provided, in which a user can provide input for drawing annotations associated with a part of the text. For example, in some embodiments, the input for hand-drawn annotations can be directly applied to reflowable text, and the annotations reflow with the text.

[0026] under, Figure 1A-1B , Figure 2 and Figure 3 A description of an exemplary apparatus is provided. Figure 4A-4B and Figures 5A-5AF An exemplary user interface for adding annotations to text is shown. Figures 6A-6D A flow chart showing a method of adding drawing annotations to text. Figures 7A-7D A flow chart of a method of adding hand-drawn annotations to reflowable text is shown. 5A to 5AF The user interface in FIG. 6A to FIG. 6D and FIG. 7A to FIG. 7D process.

[0027] Exemplary Devices

[0028] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description in order to provide a full understanding of the various described embodiments. However, it will be apparent to one of ordinary skill in the art that the various described embodiments can be practiced without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, so as not to unnecessarily obscure various aspects of the embodiments.

[0029] It will also be understood that, although in some cases, the terms "first", "second", etc. are used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first contact can be named as the second contact, and similarly, the second contact can be named as the first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact unless the context clearly indicates otherwise.

[0030] The terms used in the description of various described embodiments herein are only for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of various described embodiments and in the appended claims, the singular forms "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "includes", "including", "comprises" and / or "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their grouping.

[0031] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that ..." or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0032] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a PDA and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple iPod from Apple Inc. (Cupertino, California). iPod and Device. Other portable electronic devices, such as laptops or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads) are optionally used. It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0033] In the following discussion, an electronic device including a display and a touch-sensitive surface is described. However, it should be understood that the electronic device optionally includes one or more other physical user interface devices, such as a physical keyboard, mouse and / or joystick.

[0034] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk editing application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a fitness support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music player application, and / or a digital video player application.

[0035] Various applications executed on the device optionally use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the device are optionally adjusted and / or varied for different applications, and / or adjusted and / or varied within the respective applications. In this way, the common physical architecture of the device (such as a touch-sensitive surface) optionally supports a variety of applications with a user interface that is intuitive and clear to the user.

[0036] Attention is now turned to embodiments of portable devices having touch-sensitive displays. Figure 1A1 is a block diagram illustrating a portable multifunction device 100 with a touch-sensitive display system 112 according to some embodiments. The touch-sensitive display system 112 is sometimes referred to as a "touch screen" for convenience, and is sometimes simply referred to as a touch-sensitive display. The device 100 includes a memory 102 (which optionally includes one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral device interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input, or control devices 116, and an external port 124. The device 100 optionally includes one or more optical sensors 164. The device 100 optionally includes one or more intensity sensors 165 for detecting the intensity of contact on the device 100 (e.g., a touch-sensitive surface, such as the touch-sensitive display system 112 of the device 100). Device 100 optionally includes one or more tactile output generators 163 for generating tactile output on device 100 (e.g., generating tactile output on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touch pad 355 of device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0037] As used in this specification and claims, the term "tactile output" refers to a physical displacement of a device relative to a previous position of the device, a physical displacement of a component of the device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or a displacement of a component relative to the center of mass of the device, which will be detected by a user using the user's sense of touch. For example, in the case where a device or a component of the device is in contact with a user's touch-sensitive surface (e.g., a finger, palm, or other part of the user's hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in a physical characteristic of the device or component of the device. For example, movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) is optionally interpreted by the user as a "press click" or "release click" to a physical actuation button. In some cases, the user will feel a tactile sensation, such as a "press click" or "release click", even when the physical actuation button associated with the touch-sensitive surface that is physically pressed (e.g., displaced) by the user's movement does not move. As another example, even when there is no change in the smoothness of the touch-sensitive surface, movement of the touch-sensitive surface is optionally interpreted or sensed by the user as "roughness" of the touch-sensitive surface. Although such a user's interpretation of touch will be limited by the user's individualized sensory perceptions, many sensory perceptions of touch are common to most users. Therefore, when a tactile output is described as corresponding to a particular sensory perception of a user (e.g., "press click," "release click," "roughness"), unless otherwise stated, the generated tactile output corresponds to a physical displacement of the device or a component thereof that would generate that sensory perception for a typical (or average) user.

[0038] It should be understood that device 100 is merely one example of a portable multifunction device and that device 100 optionally has more or fewer components than shown, optionally combines two or more components, or optionally has a different configuration or arrangement of the components. Figure 1A The various components shown in the drawings are implemented in hardware, software, firmware, or any combination thereof (including one or more signal processing circuits and / or application specific integrated circuits).

[0039] The memory 102 optionally includes high-speed random access memory, and optionally also includes non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory 102 by other components of the device 100 (such as one or more CPUs 120 and peripheral device interface 118) is optionally controlled by a memory controller 122.

[0040] Peripheral interface 118 may be used to couple the device's input and output peripherals to memory 102 and one or more CPUs 120. The one or more processors 120 run or execute various software programs and / or instruction sets stored in memory 102 to perform various functions of device 100 and process data.

[0041] In some embodiments, peripherals interface 118, one or more CPUs 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0042] RF (radio frequency) circuit 108 receives and transmits RF signals, also referred to as electromagnetic signals. RF circuit 108 converts electrical signals into / converts electromagnetic signals into electrical signals, and communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 108 optionally includes well-known circuits for performing these functions, including but not limited to antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, user identity modules (SIM) cards, memory, and the like. RF circuit 108 optionally communicates with networks and other devices through wireless communications, such as the Internet (also referred to as the World Wide Web (WWW)), intranets, and / or wireless networks (such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs)). The wireless communication optionally uses any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolution-Data Only (EV-DO), HSPA, HSPA+, Dual Cell HSPA (DC-HSPDA), Long Term Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), instant messaging (e.g., Extensible Messaging and Presence Protocol (XMPP), Session Initiation Protocol for Instant Messaging and Presence using Extensions (SIMPLE), Instant Messaging and Presence Service (IMPS), and / or Short Message Service (SMS)), or any other suitable communication protocol including communication protocols that have not been developed on the date of filing of this document.

[0043] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between a user and the device 100. The audio circuit 110 receives audio data from the peripheral device interface 118, converts the audio data into electrical signals, and transmits the electrical signals to the speaker 111. The speaker 111 converts the electrical signals into sound waves audible to humans. The audio circuit 110 also receives electrical signals converted from sound waves by the microphone 113. The audio circuit 110 converts the electrical signals into audio data and transmits the audio data to the peripheral device interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or the RF circuit 108 by the peripheral device interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., Figure 2 The headset jack provides an interface between the audio circuit 110 and a removable audio input / output peripheral device, such as an output-only headset or a headset having both output (e.g., a single-ear headset or a dual-ear headset) and input (e.g., a microphone).

[0044] The I / O subsystem 106 couples input / output peripherals on the device 100, such as a touch-sensitive display system 112 and other input or control devices 116, to a peripheral device interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a tactile feedback controller 161, and one or more input controllers 160 for other input or control devices. The one or more input controllers 160 receive / send electrical signals from other input or control devices 116. Other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons, etc.), dials, slide switches, joysticks, click wheels, etc. In some alternative embodiments, one or more input controllers 160 are optionally coupled to any (or none) of the following: a keyboard, an infrared port, a USB port, a stylus, and / or a pointer device such as a mouse. One or more buttons (e.g., Figure 2 208) optionally includes an up / down button for volume control of the speaker 111 and / or the microphone 113. The one or more buttons optionally include a push button (e.g., Figure 2 206).

[0045] The touch-sensitive display system 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from the touch-sensitive display system 112 and / or sends electrical signals to the touch-sensitive display system 112. The touch-sensitive display system 112 displays visual output to the user. The visual output optionally includes graphics, text, icons, videos, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output corresponds to a user interface object.

[0046] The touch-sensitive display system 112 has a touch-sensitive surface, sensor, or set of sensors that accepts input from a user based on haptic / tactile contact. The touch-sensitive display system 112 and display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of that contact) on the touch-sensitive display system 112 and convert the detected contact into interaction with a user interface object (e.g., one or more soft keys, icons, web pages, or images) displayed on the touch-sensitive display system 112. In an exemplary embodiment, the point of contact between the touch-sensitive display system 112 and the user corresponds to a finger or stylus of the user.

[0047] The touch-sensitive display system 112 optionally uses LCD (liquid crystal display) technology, LPD (light emitting polymer display) technology, or LED (light emitting diode) technology, although other display technologies are used in other embodiments. The touch-sensitive display system 112 and display controller 156 optionally use any of a variety of touch sensing technologies now known or later developed, including but not limited to capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive display system 112 to detect contact and any movement or interruption thereof. In an exemplary embodiment, projected mutual capacitance sensing technology is used, such as the . iPod and Technology found in.

[0048] The touch-sensitive display system 112 optionally has a video resolution of more than 100 dpi. In some embodiments, the touch screen video resolution exceeds 400 dpi (e.g., 500 dpi, 800 dpi or greater). The user optionally uses any suitable object or attachment such as a stylus, finger, etc. to contact the touch-sensitive display system 112. In some embodiments, the user interface is designed to work with finger-based contacts and gestures, which may not be as accurate as stylus-based input due to the larger contact area of ​​the finger on the touch screen. In some embodiments, the device converts rough finger-based input into precise pointer / cursor positions or commands for performing the actions desired by the user.

[0049] In some embodiments, in addition to the touch screen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touch screen, does not display visual output. The touchpad is optionally a touch-sensitive surface that is separate from the touch-sensitive display system 112, or an extension of the touch-sensitive surface formed by the touch screen.

[0050] The device 100 also includes a power system 162 for powering the various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator (e.g., a light emitting diode (LED)), and any other components associated with the generation, management, and distribution of power in a portable device.

[0051] Device 100 optionally also includes one or more optical sensors 164 . Figure 1A An optical sensor coupled to an optical sensor controller 158 in the I / O subsystem 106 is shown. One or more optical sensors 164 optionally include a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. One or more optical sensors 164 receive light projected through one or more lenses from the environment and convert the light into data representing an image. In conjunction with an imaging module 143 (also called a camera module), one or more optical sensors 164 optionally capture static images and / or videos. In some embodiments, an optical sensor is located on the rear portion of the device 100 opposite to the touch-sensitive display system 112 on the front of the device so that the touch screen can be used as a viewfinder for static image and / or video image acquisition. In some embodiments, another optical sensor is located on the front of the device so that the image of the user is captured (e.g., for self-portraits, for video conferencing when the user is watching other video conference participants on the touch screen, etc.).

[0052] Device 100 optionally also includes one or more contact intensity sensors 165 . Figure 1A A contact force sensor coupled to a force sensor controller 159 in the I / O subsystem 106 is shown. One or more contact force sensors 165 optionally include one or more piezoresistive strain gauges, capacitive force sensors, electrical force sensors, piezoelectric force sensors, optical force sensors, capacitive touch-sensitive surfaces, or other force sensors (e.g., sensors for measuring the force (or pressure) of contact on a touch-sensitive surface). One or more contact force sensors 165 receive contact force information (e.g., pressure information or a surrogate for pressure information) from the environment. In some embodiments, at least one contact force sensor is juxtaposed or adjacent to a touch-sensitive surface (e.g., touch-sensitive display system 112). In some embodiments, at least one contact force sensor is located on the rear of the device 100 opposite to the touch-sensitive display system 112 located on the front of the device 100.

[0053] Device 100 optionally also includes one or more proximity sensors 166 . Figure 1A A proximity sensor 166 is shown coupled to the peripherals interface 118. Alternatively, the proximity sensor 166 is coupled to the input controller 160 in the I / O subsystem 106. In some embodiments, when the multifunction device is placed near the user's ear (e.g., when the user is on a phone call), the proximity sensor turns off and disables the touch-sensitive display system 112.

[0054] Device 100 optionally also includes one or more tactile output generators 163. Figure 1A A tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106 is shown. The tactile output generator 163 optionally includes: one or more electroacoustic devices, such as a speaker or other audio component; and / or an electromechanical device that converts energy into linear motion, such as a motor, a solenoid, an electroactive polymer, a piezoelectric actuator, an electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The tactile output generator 163 receives tactile feedback generation instructions from the tactile feedback module 133 and generates a tactile output on the device 100 that can be felt by a user of the device 100. In some embodiments, at least one tactile output generator is arranged in juxtaposition or proximity to a touch-sensitive surface (e.g., a touch-sensitive display system 112), and optionally generates a tactile output by moving the touch-sensitive surface vertically (e.g., inward / outward of the surface of the device 100) or laterally (e.g., backward and forward in the same plane as the surface of the device 100). In some embodiments, at least one tactile output generator sensor is located on a rear portion of the device 100 opposite the touch-sensitive display system 112 located on the front of the device 100.

[0055] The device 100 optionally also includes one or more accelerometers 167, gyroscopes 168, and / or magnetometers 169 (eg, as part of an inertial measurement unit (IMU)) for obtaining information about the position (eg, attitude) of the device. Figure 1A Sensors 167, 168, and 169 are shown coupled to peripheral device interface 118. Alternatively, sensors 167, 168, and 169 are optionally coupled to input controller 160 in I / O subsystem 106. In some embodiments, information is displayed in a portrait view or a landscape view on the touch screen display based on analysis of data received from the one or more accelerometers. Device 100 optionally includes a GPS (or GLONASS or other global navigation system) receiver (not shown) for obtaining information about the location of device 100.

[0056] In some embodiments, the software components stored in memory 102 include an operating system 126, a communication module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a tactile feedback module (or instruction set) 133, a text input module (or instruction set) 134, a global positioning system (GPS) module (or instruction set) 135, and an application (or instruction set) 136. In addition, in some embodiments, memory 102 stores device / global internal state 157, as shown in FIG. 1A and FIG. Figure 3 . The device / global internal state 157 includes one or more of the following: an active application state, which indicates which applications (if any) are currently active; a display state, which indicates what applications, views, or other information occupy various areas of the touch-sensitive display system 112; a sensor state, including information obtained from the device's various sensors and other input or control devices 116; and position and / or orientation information about the device's position and / or posture.

[0057] The operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded operating system such as VxWorks) includes various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitates communication between various hardware and software components.

[0058] The communication module 128 facilitates communication with other devices through one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or the external port 124. The external port 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) is suitable for coupling directly to other devices or indirectly via a network (e.g., the Internet, a wireless LAN, etc.). In some embodiments, the external port is a device compatible with some of Apple Inc. (Cupertino, California). iPod A multi-pin (e.g., 30-pin) connector that is the same as or similar to and / or compatible with the 30-pin connector used in iPod devices. In some embodiments, the external port is compatible with some of Apple Inc. (Cupertino, California) iPod A Lightning connector that is the same as, similar to, and / or compatible with the Lightning connector used in iPod devices.

[0059] The contact / motion module 130 optionally detects contact with the touch-sensitive display system 112 (in conjunction with the display controller 156) and other touch-sensitive devices (e.g., a touchpad or physical click wheel). The contact / motion module 130 includes software components for performing various operations related to contact detection (e.g., a finger or stylus), such as determining whether contact has occurred (e.g., detecting a finger press event), determining the strength of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of the contact and tracking the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events), and determining whether the contact has ceased (e.g., detecting a finger lift event or contact disconnection). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of a contact point optionally includes determining a rate (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point, the movement of which is represented by a series of contact data. These operations are optionally applied to a single point of contact (e.g., a single finger contact or a stylus contact) or multiple points of contact simultaneously (e.g., "multi-touch" / multi-finger contact and / or stylus contact). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touch pad.

[0060] The contact / motion module 130 optionally detects gesture input from a user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motions, timings, and / or intensities of the detected contacts). Therefore, gestures are optionally detected by detecting specific contact patterns. For example, detecting a single-finger tap gesture includes detecting a finger press event, and then detecting a finger lift (lift-off) event at the same location (or substantially the same location) as the finger press event (e.g., at an icon location). As another example, detecting a finger swipe gesture on the touch-sensitive surface includes detecting a finger press event, then detecting one or more finger drag events, and then detecting a finger lift (lift-off) event. Similarly, taps, swipes, drags, and other gestures of the stylus are optionally detected by detecting a specific contact pattern of the stylus.

[0061] Graphics module 132 includes various known software components for rendering and displaying graphics on touch-sensitive display system 112 or other displays, including components for changing the visual impact (e.g., brightness, transparency, saturation, contrast, or other visual attributes) of displayed graphics. As used herein, the term "graphics" includes any object that can be displayed to a user, including, without limitation, text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0062] In some embodiments, the graphics module 132 stores data representing graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes for specifying the graphics to be displayed from an application program, etc., and also receives coordinate data and other graphic attribute data together if necessary, and then generates screen image data to be output to the display controller 156.

[0063] Haptic feedback module 133 includes various software components for generating instructions used by tactile output generator 163 to produce tactile output at one or more locations on device 100 in response to user interaction with device 100 .

[0064] Text input module 134, which is optionally a component of graphics module 132, provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other application requiring text input).

[0065] The GPS module 135 determines the location of the device and provides such information for use in various applications (e.g., to the phone 138 for location-based dialing; to the camera 143 as picture / video metadata; and to applications that provide location-based services such as weather widgets, local yellow pages widgets, and maps / navigation widgets).

[0066] The application 136 optionally includes the following modules (or instruction sets) or a subset or superset thereof:

[0067] Contacts module 137 (sometimes called address book or contact list);

[0068] Telephone module 138;

[0069] Video conferencing module 139;

[0070] Email client module 140;

[0071] Instant messaging (IM) module 141;

[0072] Fitness support module 142;

[0073] Camera module 143 for still images and / or video images;

[0074] Image management module 144;

[0075] Browser module 147;

[0076] Calendar module 148;

[0077] A widget module 149, which optionally includes one or more of the following: a weather widget 149-1, a stock widget 149-2, a calculator widget 149-3, an alarm widget 149-4, a dictionary widget 149-5, and other widgets obtained by the user, and a user-created widget 149-6;

[0078] A widget creator module 150 for forming a user-created widget 149-6;

[0079] Search module 151;

[0080] Video and music player module 152, which optionally consists of a video player module and a music player module;

[0081] Notepad module 153;

[0082] Map module 154; and / or

[0083] Online video module 155.

[0084] Examples of other applications 136 optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, rendering applications, JAVA-enabled applications, encryption, digital rights management, voice recognition, and voice replication.

[0085] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the contacts module 137 includes executable instructions for managing an address book or contact list (e.g., stored in the application internal state 192 of the contacts module 137 in memory 102 or memory 370), including: adding names to the address book; deleting names from the address book; associating phone numbers, email addresses, physical addresses, or other information with names; associating images with names; categorizing and classifying names; providing phone numbers and / or email addresses to initiate and / or facilitate communications via telephone 138, video conferencing 139, email 140, or instant messaging 141; and the like.

[0086] In conjunction with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, phone module 138 includes executable instructions for entering a character sequence corresponding to a phone number, accessing one or more phone numbers in address book 137, modifying an entered phone number, dialing a corresponding phone number, conducting a conversation, and disconnecting or hanging up when the conversation is complete. As described above, wireless communications optionally use any of a variety of communication standards, protocols, and technologies.

[0087] In combination with RF circuitry 108, audio circuitry 110, speaker 111, microphone 113, touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, text input module 134, contact list 137, and telephone module 138, video conferencing module 139 includes executable instructions for initiating, conducting, and terminating a video conference between a user and one or more other participants in accordance with user instructions.

[0088] In conjunction with RF circuit 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, email client module 140 includes executable instructions for creating, sending, receiving, and managing emails in response to user instructions. In conjunction with image management module 144, email client module 140 makes it very easy to create and send emails with still images or video images captured by camera module 143.

[0089] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display controller 156, the contact module 130, the graphics module 132, and the text input module 134, the instant messaging module 141 includes executable instructions for performing the following operations: inputting a character sequence corresponding to an instant message, modifying previously input characters, sending a corresponding instant message (e.g., using a short message service (SMS) or multimedia message service (MMS) protocol for phone-based instant messaging or using XMPP, SIMPLE, Apple Push Notification Service (APNs) or IMPS for Internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photos, audio files, video files, and / or other attachments supported in MMS and / or enhanced messaging services (EMS). As used herein, "instant messaging" refers to both phone-based messages (e.g., messages sent using SMS or MMS) and Internet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs or IMPS).

[0090] In combination with RF circuitry 108, touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module 146, fitness support module 142 includes executable instructions for creating a workout (e.g., with time, distance, and / or calorie burn goals); communicating with fitness sensors (in sports equipment and smart watches); receiving fitness sensor data; calibrating sensors for monitoring fitness; selecting and playing music for a workout; and displaying, storing, and transmitting fitness data.

[0091] In conjunction with touch-sensitive display system 112, display controller 156, one or more optical sensors 164, optical sensor controller 158, contact module 130, graphics module 132, and image management module 144, camera module 143 includes executable instructions for capturing still images or videos (including video streams) and storing them in memory 102, modifying characteristics of still images or videos, and / or deleting still images or videos from memory 102.

[0092] In conjunction with touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, text input module 134, and camera module 143, image management module 144 includes executable instructions for arranging, modifying (e.g., editing), or otherwise manipulating, labeling, deleting, displaying (e.g., in a digital slide show or album), and storing still images and / or video images.

[0093] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132 and the text input module 134, the browser module 147 includes executable instructions for browsing the Internet (including searching, linking to, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages) in accordance with user instructions.

[0094] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, the email client module 140 and the browser module 147, the calendar module 148 includes executable instructions for creating, displaying, modifying and storing calendars and data associated with the calendar (e.g., calendar entries, to-do items, etc.) in accordance with user instructions.

[0095] In conjunction with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget module 149 is a mini-application that is optionally downloaded and used by the user (e.g., the weather desktop widget 149-1, the stock desktop widget 149-2, the calculator desktop widget 149-3, the alarm desktop widget 149-4, and the dictionary desktop widget 149-5), or a mini-application created by the user (e.g., the user-created desktop widget 149-6). In some embodiments, the desktop widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the desktop widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., the Yahoo! desktop widget).

[0096] In combination with the RF circuit 108, the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, the text input module 134, and the browser module 147, the desktop widget creator module 150 includes executable instructions for creating a desktop widget (e.g., transferring a user-specified portion of a web page to a desktop widget).

[0097] In conjunction with the touch-sensitive display system 112, the display system controller 156, the contact module 130, the graphics module 132, and the text input module 134, the search module 151 includes executable instructions for searching the memory 102 for text, music, sound, images, videos, and / or other files that match one or more search criteria (e.g., one or more user-specified search terms) in accordance with user instructions.

[0098] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuitry 110, speaker 111, RF circuitry 108, and browser module 147, video and music player module 152 includes executable instructions that allow a user to download and play back recorded music and other sound files stored in one or more file formats (such as MP3 or AAC files), as well as executable instructions for displaying, presenting, or otherwise playing back video (e.g., on touch-sensitive display system 112 or on an external display wirelessly connected via external port 124). In some embodiments, device 100 optionally includes the functionality of an MP3 player such as an iPod (trademark of Apple Inc.).

[0099] In conjunction with the touch-sensitive display system 112, display controller 156, contact module 130, graphics module 132, and text input module 134, the notepad module 153 includes executable instructions for creating and managing notepads, to-do lists, etc. according to user instructions.

[0100] In combination with RF circuitry 108, touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147, map module 154 includes executable instructions for receiving, displaying, modifying, and storing maps and data associated with the maps (e.g., driving routes; data about stores and other points of interest at or near a particular location; and other location-based data) in accordance with user instructions.

[0101] In conjunction with touch-sensitive display system 112, display system controller 156, contact module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, online video module 155 includes executable instructions that allow a user to access, browse, receive (e.g., by streaming and / or downloading), play back (e.g., on touch screen 112 or on an external display connected wirelessly or via external port 124), send an email with a link to a particular online video, and otherwise manage online videos in one or more file formats such as H.264. In some embodiments, instant messaging module 141 is used instead of email client module 140 to send a link to a particular online video.

[0102] Each module and application identified above corresponds to a set of executable instructions for performing one or more functions and methods described in the present application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) do not have to be implemented with independent software programs, processes or modules, so various subsets of these modules are optionally combined or otherwise rearranged in various embodiments. In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. In addition, memory 102 optionally stores other modules and data structures not described above.

[0103] In some embodiments, the device 100 is a device where the operation of a predefined set of functions on the device is performed exclusively through a touch screen and / or a touch pad. By using a touch screen and / or a touch pad as the primary input control device for operating the device 100, the number of physical input control devices (e.g., push buttons, dials, etc.) on the device 100 is optionally reduced.

[0104] A predefined set of functions that are performed exclusively through the touch screen and / or trackpad optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 to a main menu, a home menu, or a root menu from any user interface displayed on the device 100. In such embodiments, a "menu button" is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0105] Figure 1B 1 is a block diagram illustrating exemplary components for event processing according to some embodiments. In some embodiments, memory 102 ( Figure 1A ) or memory 370 ( Figure 3 ) includes an event classifier 170 (e.g., in the operating system 126) and a corresponding application 136-1 (e.g., any one of the aforementioned applications 136, 137 to 155, 380 to 390).

[0106] The event classifier 170 receives event information and determines the application 136-1 and the application view 191 of the application 136-1 to which the event information is to be delivered. The event classifier 170 includes an event monitor 171 and an event distributor module 174. In some embodiments, the application 136-1 includes an application internal state 192, which indicates one or more current application views displayed on the touch-sensitive display system 112 when the application is active or executing. In some embodiments, the device / global internal state 157 is used by the event classifier 170 to determine which application(s) is currently active, and the application internal state 192 is used by the event classifier 170 to determine the application view 191 to which the event information is to be delivered.

[0107] In some embodiments, the application internal state 192 includes additional information, such as one or more of the following: resumption information to be used when the application 136-1 resumes execution, user interface state information indicating that information is being displayed or is ready to be displayed by the application 136-1, a state queue for enabling a user to return to a previous state or view of the application 136-1, and a repeat / undo queue of previous actions taken by the user.

[0108] Event monitor 171 receives event information from peripherals interface 118. Event information includes information about sub-events (e.g., a user touch on touch-sensitive display system 112 as part of a multi-touch gesture). Peripherals interface 118 transmits information it receives from I / O subsystem 106 or sensors such as proximity sensor 166, one or more accelerometers 167, one or more gyroscopes 168, one or more magnetometers 169, and / or microphone 113 (through audio circuit 110). The information received by peripherals interface 118 from I / O subsystem 106 includes information from touch-sensitive display system 112 or a touch-sensitive surface.

[0109] In some embodiments, event monitor 171 sends requests to peripheral device interface 118 at predetermined intervals. In response, peripheral device interface 118 transmits event information. In other embodiments, peripheral device interface 118 transmits event information only when there is a significant event (e.g., receiving an input above a predetermined noise threshold and / or receiving an input for more than a predetermined duration).

[0110] In some embodiments, the event classifier 170 also includes a hit view determination module 172 and / or an active event identifier determination module 173.

[0111] When the touch-sensitive display system 112 displays more than one view, the hit view determination module 172 provides software procedures for determining where within one or more views a sub-event has occurred. A view consists of controls and other elements that a user can see on the display.

[0112] Another aspect of the user interface associated with an application is a set of views, sometimes also referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application views (of the respective application) in which a touch is detected optionally correspond to a programmatic level within the programmatic or view hierarchy of the application. For example, the lowest level view in which a touch is detected is optionally referred to as a hit view, and the set of events recognized as correct input is optionally determined at least in part based on the hit view of the initial touch that started the touch-based gesture.

[0113] Hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views organized in a hierarchy, hit view determination module 172 identifies the hit view as the lowest view in the hierarchy where the sub-events should be processed. In most cases, the hit view is the lowest level view in which the initiating sub-event (i.e., the first sub-event in a sequence of sub-events that form an event or potential event) occurs. Once a hit view is identified by the hit view determination module, the hit view typically receives all sub-events related to the same touch or input source for which it was identified as the hit view.

[0114] Active event recognizer determination module 173 determines which view or views within the view hierarchy should receive a particular sequence of sub-events. In some embodiments, active event recognizer determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, active event recognizer determination module 173 determines that all views that include the physical location of the sub-event are actively participating views, and therefore determines that all actively participating views should receive a particular sequence of sub-events. In other embodiments, even if a touch sub-event is completely confined to an area associated with a particular view, higher views in the hierarchy should still remain actively participating views.

[0115] Event distributor module 174 distributes event information to event recognizers (e.g., event recognizers 180). In embodiments including active event recognizer determination module 173, event distributor module 174 delivers the event information to the event recognizers determined by active event recognizer determination module 173. In some embodiments, event distributor module 174 stores the event information in an event queue, which is retrieved by corresponding event receiver module 182.

[0116] In some embodiments, operating system 126 includes event classifier 170. Alternatively, application 136-1 includes event classifier 170. In yet another embodiment, event classifier 170 is a standalone module or is part of another module stored in memory 102, such as contact / motion module 130.

[0117] In some embodiments, application 136-1 includes multiple event handlers 190 and one or more application views 191, each of which includes instructions for handling touch events occurring in a corresponding view of the user interface of the application. Each application view 191 of application 136-1 includes one or more event recognizers 180. Typically, the corresponding application view 191 includes multiple event recognizers 180. In other embodiments, one or more event recognizers in event recognizers 180 are part of an independent module, which is a higher-level object such as a user interface toolkit (not shown) or application 136-1 from which methods and other properties are inherited. In some embodiments, the corresponding event handler 190 includes one or more of the following: data updater 176, object updater 177, GUI updater 178, and / or event data 179 received from event classifier 170. Event handler 190 optionally utilizes or calls data updater 176, object updater 177 or GUI updater 178 to update application internal state 192. Alternatively, one or more of the application views in the application views 191 include one or more corresponding event handlers 190. In addition, in some embodiments, one or more of the data updater 176, the object updater 177, and the GUI updater 178 are included in the corresponding application view 191.

[0118] A corresponding event identifier 180 receives event information (e.g., event data 179) from event classifier 170 and identifies an event from the event information. Event identifier 180 includes an event receiver 182 and an event comparator 184. In some embodiments, event identifier 180 also includes metadata 183 and at least a subset of event delivery instructions 188 (which optionally include sub-event delivery instructions).

[0119] Event receiver 182 receives event information from event classifier 170. Event information includes information about sub-events such as touch or touch movement. Depending on the sub-event, the event information also includes additional information, such as the location of the sub-event. When the sub-event involves the movement of the touch, the event information optionally also includes the speed and direction of the sub-event. In some embodiments, the event includes the device rotating from one orientation to another orientation (e.g., from a longitudinal orientation to a transverse orientation, or vice versa), and the event information includes corresponding information about the current orientation of the device (also referred to as the device posture).

[0120] Event comparator 184 compares event information with predefined event or sub-event definitions, and determines an event or sub-event based on the comparison, or determines or updates the state of an event or sub-event. In some embodiments, event comparator 184 includes event definition 186. Event definition 186 includes the definition of an event (e.g., a predefined sequence of sub-events), such as event 1 (187-1), event 2 (187-2), etc. In some embodiments, sub-events in event 187 include, for example, touch start, touch end, touch move, touch cancel, and multi-touch. In one example, the definition of event 1 (187-1) is a double-click on a displayed object. For example, a double-click includes a first touch (touch start) of a predetermined duration on a displayed object, a first lift (touch end) of a predetermined duration, a second touch (touch start) of a predetermined duration on a displayed object, and a second lift (touch end) of a predetermined duration. In another example, the definition of event 2 (187-2) is a drag on a displayed object. For example, dragging includes a touch (or contact) of a predetermined duration on a displayed object, movement of the touch on the touch-sensitive display system 112, and lifting of the touch (touch end). In some embodiments, the event also includes information for one or more associated event handlers 190.

[0121] In some embodiments, event definition 187 includes a definition of an event for a corresponding user interface object. In some embodiments, event comparator 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view displaying three user interface objects on touch-sensitive display system 112, when a touch is detected on touch-sensitive display system 112, event comparator 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with a corresponding event handler 190, the event comparator uses the result of the hit test to determine which event handler 190 should be activated. For example, event comparator 184 selects an event handler associated with a sub-event and an object that triggers the hit test.

[0122] In some embodiments, the definition of the corresponding event 187 also includes a delay action that delays the delivery of the event information until it has been determined whether the sub-event sequence does or does not correspond to the event type of the event identifier.

[0123] When a corresponding event recognizer 180 determines that a sequence of sub-events does not match any event in event definition 186, the corresponding event recognizer 180 enters an event impossible, event failed, or event ended state, after which subsequent sub-events of the touch-based gesture are ignored. In this case, other event recognizers (if any) that remain active for the hit view continue to track and process sub-events of the ongoing touch-based gesture.

[0124] In some embodiments, the corresponding event recognizers 180 include metadata 183 with configurable properties, flags, and / or lists that indicate how the event delivery system should perform sub-event delivery to actively participating event recognizers. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate how event recognizers interact or can interact with each other. In some embodiments, metadata 183 includes configurable properties, flags, and / or lists that indicate whether sub-events are delivered to different levels in a view or programmatic hierarchy.

[0125] In some embodiments, when one or more specific sub-events of an event are identified, the corresponding event recognizer 180 activates an event handler 190 associated with the event. In some embodiments, the corresponding event recognizer 180 delivers event information associated with the event to the event handler 190. Activating the event handler 190 is different from sending (and deferred sending) the sub-events to the corresponding hit view. In some embodiments, the event recognizer 180 throws a tag associated with the identified event, and the event handler 190 associated with the tag obtains the tag and performs a predefined process.

[0126] In some embodiments, the event delivery instructions 188 include a sub-event delivery instruction that delivers event information about a sub-event without activating an event handler. Instead, the sub-event delivery instruction delivers the event information to an event handler associated with a sub-event sequence or to an actively participating view. The event handler associated with the sub-event sequence or with an actively participating view receives the event information and performs a predetermined process.

[0127] In some embodiments, data updater 176 creates and updates data used in application 136-1. For example, data updater 176 updates phone numbers used in contact module 137 or stores video files used in video player module 145. In some embodiments, object updater 177 creates and updates objects used in application 136-1. For example, object updater 176 creates new user interface objects or updates the position of user interface objects. GUI updater 178 updates the GUI. For example, GUI updater 178 prepares display information and sends the display information to graphics module 132 for display on a touch-sensitive display.

[0128] In some embodiments, event handler 190 includes or has access to data updater 176, object updater 177, and GUI updater 178. In some embodiments, data updater 176, object updater 177, and GUI updater 178 are included in a single module of the corresponding application 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0129] It should be understood that the above discussion of event processing for user touches on a touch-sensitive display also applies to other forms of user input that utilize input devices to operate the multifunction device 100, and not all user input is initiated on the touch screen. For example, mouse movement and mouse button presses, optionally in conjunction with single or multiple keyboard presses or holddowns; contact movement on a touch pad, such as tapping, dragging, scrolling, etc.; stylus input; movement of the device; verbal commands; detected eye movement; biometric input; and / or any combination thereof are optionally used as input corresponding to sub-events defining an event to be recognized.

[0130] Figure 2 A device having a touch screen (e.g., Figure 1AThe portable multifunction device 100 of the present invention is a touch-sensitive display system 112 of the present invention. The touch screen optionally displays one or more graphics within the user interface (UI) 200. In this embodiment and other embodiments described below, the user can select one or more of the graphics by, for example, making gestures on the graphics using one or more fingers 202 (not drawn to scale in the figure) or one or more styluses 203 (not drawn to scale in the figure). In some embodiments, the selection of one or more graphics will occur when the user breaks contact with the one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of the fingers that have been in contact with the device 100 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with the graphics does not select the graphics. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application.

[0131] The device 100 optionally also includes one or more physical buttons, such as a "home" button, or a menu button 204. As previously described, the menu button 204 is optionally used to navigate to any application 136 in a set of applications that are optionally executed on the device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touch screen display.

[0132] In some embodiments, the device 100 includes a touch screen display, a menu button 204, a push button 206 for powering the device on / off and for locking the device, one or more volume adjustment buttons 208, a user identity module (SIM) card slot 210, an earphone jack 212, and a docking / charging external port 124. The push button 206 is optionally used to turn the device on / off by pressing the button and keeping the button in a pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In some embodiments, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 also optionally includes one or more contact strength sensors 165 for detecting the intensity of contact on the touch-sensitive display system 112, and / or one or more tactile output generators 163 for generating tactile output for a user of the device 100.

[0133] Figure 3300 is a block diagram of an exemplary multifunction device with a display and a touch-sensitive surface according to some embodiments. Device 300 does not have to be portable. In some embodiments, device 300 is a laptop, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home controller or an industrial controller). Device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, a memory 370, and one or more communication buses 320 for interconnecting these components. The communication bus 320 optionally includes circuits (sometimes referred to as a chipset) that interconnect system components and control communications between system components. Device 300 includes an input / output (I / O) interface 330 with a display 340, which is typically a touch screen display. The I / O interface 330 also optionally includes a keyboard and / or a mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 for generating tactile output on the device 300 (e.g., similar to the above reference). Figure 1A The one or more tactile output generators 163), sensors 359 (e.g., similar to those described above) Figure 1A The memory 370 may include touch sensors, optical sensors, contact intensity sensors, proximity sensors, acceleration sensors, gesture and / or magnetic sensors of the sensors 112, 164, 165, 166, 167, 168 and 169). The memory 370 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located away from the CPU 310. In some embodiments, the memory 370 stores information related to the portable multifunction device 100 ( Figure 1A ) stored in memory 102 of portable multifunction device 100, or a subset thereof. In addition, memory 370 optionally stores additional programs, modules, and data structures not present in memory 102 of portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk editing module 388, and / or a spreadsheet module 390, while portable multifunction device 100 ( Figure 1A )'s memory 102 optionally does not store these modules.

[0134] Figure 3Each element in the above-mentioned identified element is optionally stored in one or more memory devices in the previously mentioned memory device.Each module in the above-mentioned identified module corresponds to a group of instructions for performing the above-mentioned functions.The above-mentioned identified module or program (that is, instruction set) need not be implemented as a separate software program, process or module, so the various subsets of these modules are optionally combined or otherwise rearranged in various embodiments.In some embodiments, memory 370 optionally stores the module identified above and the subgroup of data structure.In addition, memory 370 optionally stores other modules and data structure not described above.

[0135] Attention is now turned to an embodiment of a user interface (“UI”) that is optionally implemented on portable multifunction device 100 .

[0136] Figure 4A An exemplary user interface of an application menu on portable multifunction device 100 is shown according to some embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, user interface 400 includes the following elements, or a subset or superset thereof:

[0137] Signal strength indicators 402 for wireless communications, such as cellular signals and Wi-Fi signals;

[0138] Time 404;

[0139] Bluetooth indicator 405;

[0140] Battery status indicator 406;

[0141] Tray 408, which has icons for commonly used applications, such as:

[0142] o an icon 416 of the phone module 138 labeled “Phone”, the icon 416 optionally including an indicator 414 of the number of missed calls or voice messages;

[0143] o an icon 418 of the email client module 140 labeled “Mail,” which optionally includes an indicator 410 of the number of unread emails;

[0144] o An icon 420 labeled “Browser” of the browser module 147; and

[0145] o an icon 422 labeled “iPod” of a video and music player module 152 (also referred to as an iPod (trademark of Apple Inc.) module 152); and

[0146] Icons for other apps, such as:

[0147] o Icon 424 of the IM module 141 labeled “Text”;

[0148] o Icon 426 labeled “Calendar” of calendar module 148;

[0149] o Icon 428 labeled “Photos” of the image management module 144;

[0150] o An icon 430 labeled “Camera” of the camera module 143;

[0151] o Icon 432 labeled “Online Video” of the online video module 155;

[0152] o Icon 434 labeled "Stock Market" of the Stock Market widget 149-2;

[0153] o an icon 436 labeled “MAP” of the map module 154;

[0154] o Icon 438 labeled "Weather" of the Weather widget 149-1;

[0155] o Icon 440 labeled “Clock” of the alarm clock widget 169-6;

[0156] o an icon 442 labeled “Fitness Support” of the fitness support module 142;

[0157] o An icon 444 labeled "Notepad" of the notepad module 153; and

[0158] o An icon 446 for a settings application or module that provides access to settings for the device 100 and its various applications 136.

[0159] It should be noted that Figure 4A The icon labels shown in are exemplary only. For example, in some embodiments, icon 422 of video and music player module 152 is labeled "Music" or "Music Player". Other labels are optionally used for various application icons. In some embodiments, the label of the corresponding application icon includes the name of the application corresponding to the corresponding application icon. In some embodiments, the label of a particular application icon is different from the name of the application corresponding to the particular application icon.

[0160] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting intensity of contact on touch-sensitive surface 451 and / or one or more tactile output generators 359 for generating tactile output for a user of device 300.

[0161] Figure 4B A touch-sensitive surface 451 (eg, Figure 3 A device (e.g., a tablet or touch pad 355) Figure 3 Although many of the examples that follow will be given with reference to input on a touch screen display 112 (where a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface that is separate from the display, such as Figure 4B In some embodiments, the touch-sensitive surface (e.g., Figure 4B 451) has a main axis (e.g., Figure 4B 453) corresponding to the main axis (for example, Figure 4B According to these embodiments, the device detects contact with touch-sensitive surface 451 at a location corresponding to a corresponding location on the display (e.g., Figure 4B 460 and 462) (e.g., in Figure 4B 460 corresponds to 468 and 462 corresponds to 470). Thus, on a touch-sensitive surface (e.g., Figure 4B 451) and a display of a multi-function device (e.g., Figure 4B 450 in ) is separated, the user input detected by the device on the touch-sensitive surface (e.g., contacts 460 and 462 and their movement) is used by the device to manipulate the user interface on the display. It should be understood that similar methods are optionally used for other user interfaces described herein.

[0162] In addition, although the following examples are given primarily with reference to finger input (e.g., finger contact, single-finger tap gesture, finger swipe gesture, etc.) or stylus input, it should be understood that in some embodiments, one or more of these finger inputs or stylus inputs are replaced by input from another input device (e.g., mouse-based input). For example, a swipe gesture is optionally replaced by a mouse click (e.g., instead of contact), followed by movement of the cursor along the path of the swipe (e.g., instead of movement of the contact). As another example, a tap gesture is optionally replaced by a mouse click when the cursor is over the location of the tap gesture (e.g., instead of detecting contact, followed by ceasing to detect contact). Similarly, when multiple user inputs are detected simultaneously, it should be understood that multiple computer mice are optionally used simultaneously, or a mouse and finger contact are optionally used simultaneously.

[0163] User interface and associated processes

[0164] Attention now turns to embodiments of a user interface ("UI") and associated processes that may be implemented on an electronic device such as portable multifunction device (PMD) 100 or device 300 having a display, a touch-sensitive surface, and one or more sensors for detecting intensity of contact with the touch-sensitive surface.

[0165] Figures 5A-5AF An exemplary user interface for annotating text according to some embodiments is shown. The user interface in these figures is used to illustrate the following process, including Figures 6A to 6D and Figures 7A to 7D Although some of the examples that follow will be given with reference to input on a touch screen display (in which a touch-sensitive surface and a display are combined), in some embodiments, the device detects input on a touch-sensitive surface 451 that is separate from the display 450, such as Figure 4B as shown in .

[0166] Figure 5A An e-book application is shown as part of user interface 500 displayed by portable multifunction device 100. Within the application area, in addition to the e-book application, user interface 500 includes a device bar 570 at the top of the display that includes an identifier of portable multifunction device 100 (e.g., "iPad"), a wireless connection indicator, the current time, and a battery indicator indicating the charge level of portable multifunction device 100.

[0167] Below the device bar 570 and across the rest of the display, the user interface 500 includes an application area for an e-book application. Although initially described herein in the context of an e-book application (see Figures 5A-5AF), but the aspects described herein can be applied to other applications that display text, including word processing applications, PDF (Portable Document Format) reader applications, and Web browser applications.

[0168] The application area includes an application bar 510 located at the top of the application area. In some embodiments, the application bar 510 is displayed at the bottom of the application area or at another location within the application area. The application bar 510 includes a library affordance 511 for displaying a library view within the application. The library view includes affordances for selecting e-books that can be displayed by the application. The application bar 510 includes a directory affordance 512 for displaying a catalog view within the application. As further described below, the catalog view includes a table of contents for the e-book, which has affordances for displaying corresponding portions of the text of the e-book. As further described below, the notes view can be accessed from the catalog view.

[0169] Application bar 510 includes author text 513 indicating the author of the electronic book and title text 516 indicating the title of the electronic book. Application bar 510 includes text display option affordance 517, which is used to change the display of the text of the electronic book, such as the font or font size. Application bar 510 includes search affordance 518, which is used to search the text of the electronic book. Application bar 510 includes bookmark affordance 519, which is used to add a bookmark associated with the displayed electronic book text. Application bar 510 includes hand-drawn annotation affordance 590, which is described in detail below.

[0170] The application area includes a text display area 520 in which at least a portion of the text of the electronic book is displayed. Figure 5A As shown, the text display area 520 includes a first column 521 and a second column 522, each column including a portion of the text of the electronic book. In various specific implementations, the text display area 520 includes a single column or more than two columns.

[0171] Figure 5B Shows Figure 5A 5A and 5B are diagrams of a user interface 500 in which finger contact 581A is detected at a location of the text. Specifically, finger contact 581A is located at the location of the word "apple" in first column 521 of text display area 520. Although various finger contacts (illustrated as dashed ovals) and stylus contacts (illustrated as dashed circles with stylus 203) are shown throughout the figures and described herein, it should be understood that in various implementations, finger contacts may be replaced with stylus contacts, and similarly, stylus contacts may be replaced with finger contacts. In some embodiments, finger contacts are processed in the same manner as stylus contacts. In some embodiments, finger contacts are processed differently than stylus contacts.

[0172] Figure 5C A user interface 500 is shown, which responds to detecting Figure 5B In response to detecting finger contact 581A, the word "apple" in text display area 520 is selected, and the selected text menu 530 is displayed in association with the selected text. In various specific implementations, if finger contact 581A is a tap (e.g., the contact time is less than a threshold), the selected text menu 530 is not displayed. In some embodiments, the user interface 500 does not change in response to detecting a finger tap. In some embodiments, the user interface 500 changes in a manner other than displaying the selected text menu 530, such as scrolling text or displaying the next page. In various specific implementations, if finger contact 581A is a long press (e.g., the contact time is greater than a threshold), the selected text menu 530 is displayed.

[0173] The selected text menu 530 includes a copy affordance 531 for copying the selected text to the clipboard. The selected text menu 530 includes a define affordance 532 for displaying the definition of the selected text. The selected text menu 530 includes a highlight affordance 533 for adding a highlight annotation to the selected text. The selected text menu 530 includes a note affordance 534 for adding a note annotation to the selected text. The selected text menu 530 includes a draw affordance 535 for adding a drawing annotation to the selected text.

[0174] Figure 5C Finger contact 581B detected at the location of drawing indication 535 is shown.

[0175] Figure 5D A user interface 500 is shown, which responds to detecting Figure 5C In response to detecting the finger contact 581B, the first portion 551A of the text displayed in the first column 521 moves upward, and the second portion 551B of the text displayed in the first column 521 moves downward, thereby revealing the non-text area. Figure 5D , at least a portion of the non-text area is converted into a drawing panel 540.

[0176] like Figure 5D As shown, moving the first portion 551A of the displayed text relative to the second portion 551B of the displayed text to display the non-text area between the first portion 551A and the second portion 551B does not change the display of the second column 522. In various implementations, some of the second portion 551B of the displayed text is reflowed onto the second column 522.

[0177] A drawing panel 540 is provided to receive drawing input and display corresponding graphics in response to receiving the drawing input. The drawing panel 540 includes a plurality of drawing enable representations 541-544, which are used to change the color or modify the virtual drawing tools of the drawing input and the corresponding graphics. Specifically, the drawing panel 540 includes a drawing mode enable representation 541, which is used to enter a drawing mode (e.g., select a virtual pencil tool) to add lines to the drawing within the drawing panel 540. The drawing panel 540 includes a brush enable representation 540, which is used to select a virtual brush for adding strokes to the drawing within the drawing panel 542. The drawing panel 540 includes an eraser enable representation 540, which is used to erase some parts of the drawing within the drawing panel 543. The drawing panel 540 includes a color enable representation 540, which is used to select the color to be added to the drawing within the drawing panel 544.

[0178] Figure 5E Shows Figure 5D The user interface 500 is displayed in response to detecting a drawing input within the drawing panel 540. Figure 5E In the example, the drawing panel 540 displays a drawing 545A corresponding to the detected drawing input. In various implementations, the drawing input may be one or more finger inputs and / or one or more stylus inputs.

[0179] Figure 5E Finger contact 581C is shown detected at a location outside of drawing panel 540.

[0180] Fig. 5F User interface 500 is shown, which is displayed in response to detecting finger contact 581C. Fig. 5F , the first portion 551A and the second portion 551B of the first column 521 displaying text are moved relative to each other to stop displaying the non-text area and its drawing panel 540.

[0181] exist Fig. 5F , user interface 500 includes a drawing indicator 547A displayed at a location corresponding to a location between first portion 551A and second portion 551B. Specifically, drawing indicator 547A is displayed in the margin to the left of the location between first portion 551A and second portion 551B. In addition, the selected text (e.g., the word "apple") is highlighted to indicate that a drawing annotation is associated with the text.

[0182] Figure 5G Shows Fig. 5F5 , wherein expand gesture 581D is detected within first column 521A. Expand gesture 581D is detected between first portion 552A of displayed text in first column 521 (which is different from first portion 551A shown previously) and second portion 552B of displayed text in first column 521 (which is different from second portion 551B shown previously).

[0183] Figure 5H User interface 500 is shown, which is displayed in response to detecting the first portion of expand gesture 581D. Specifically, Figure 5H User interface 500 is shown in response to detecting a first finger contact and a second finger contact moving a first distance away from each other. Figure 5H , the first portion 552A moves relative to the second portion 552B to display the non-text area 549 between the first portion 552A and the second portion 552B.

[0184] Fig.5I User interface 500 is shown, which is displayed in response to detecting the second portion of expand gesture 581D. Specifically, Fig.5I User interface 500 is shown in response to detecting a first finger contact and a second finger contact moving a second distance away from each other and liftoff of the first finger contact and the second finger contact.

[0185] exist Fig.5I , the size of the non-text area 549 has increased, and at least a portion of the non-text area 549 has been converted into the drawing panel 540.

[0186] and Figure 5G similar, Figure 5J Shows Fig. 5F 500 of user interface 500, wherein expand gesture 581E is detected within first column 521A. Figure 5J The expansion gesture of 581E and Figure 5G The expand gesture differs from the previous one in that the two finger contacts move a greater distance.

[0187] Figure 5K User interface 500 is shown, which is displayed in response to detecting expand gesture 581E. Figure 5K , the first portion 552A and the second portion 552B are moved relative to each other to display the non-text area, at least a portion of which has been converted into the drawing area 540.

[0188] In comparison Fig.5I and Figure 5K hour, Figure 5K The drawing panel is 540 Fig.5IIn various implementations, the size of the drawing panel 540 is proportional to the size of the expand gesture, for example, proportional to the distance moved by the two finger contacts.

[0189] As described above, when the drawing panel 540 is displayed, the first portion 552A of the text moves relative to the second portion 552B of the text. Specifically, the first portion 552A moves a first distance in a first direction (e.g., upward), and the second portion 552B moves a second distance in a second direction (e.g., downward). In some embodiments, the first distance is the same as the second distance. However, in some embodiments, the first distance is different from the second distance. In various implementations, the first distance and the second distance are selected based on the location of the input that opens the drawing panel 540. In some implementations, the first distance and the second distance are selected so as to maintain at least a subset of the first portion 552A and at least a subset of the second portion 552B on the display. For example, as Figure 5K As shown, the user interface 500 includes a subset of the first portion 552A above the drawing panel 540 and a subset of the second portion 522B below the drawing panel. Specifically, the first distance that the first portion 552A moves is small enough so that the first portion 552A is not removed from the display. For example, Figure 5K As shown, if the boundary between the first portion 552A above the boundary and the second portion 552B below the boundary is closer to the top of the display, the first portion 552A moves less than the second portion 552B. Figure 5D As shown (and similarly, in the following description Figure 5P ), if the boundary between the first portion 551A and the second portion 551B is closer to the bottom of the display, the first portion 551A moves more than the second portion 551B.

[0190] Figure 5L Shows Figure 5K The user interface 500 is displayed in response to detecting a drawing input within the drawing panel 540. Figure 5L , the drawing panel 540 displays a graphic of a drawing 545B corresponding to the detected drawing input.

[0191] Figure 5M Shows Figure 5L 500, wherein a pinch gesture 581F is detected within the first column 521. The pinch gesture 581F includes two finger contacts moving toward each other. In various specific implementations, such as Figure 5M As shown, one or both of the two finger contacts can be detected within the drawing panel 540. In some embodiments, one or both of the two finger contacts are detected outside the drawing panel 540.

[0192] Figure 5N User interface 500 is shown, displayed in response to detecting pinch gesture 581F. In response to pinch gesture 581F, first portion 552A and second portion 552B have moved relative to each other to cease displaying the non-text area and its drawing panel 540.

[0193] exist Figure 5N , user interface 500 includes a drawing indicator 547B displayed at a location corresponding to a location between first portion 552A and second portion 552B. Drawing indicator 547A associated with previously selected text (eg, the word "apple") is also displayed.

[0194] Fig.5O Shows Figure 5N 500, wherein stylus contact 581G is detected somewhere between two segments of text in second column 522. Specifically, stylus contact 581G is typically between first portion 553A of text displayed in second column 522 and second portion 553B of text displayed in second column 522.

[0195] Figure 5P User interface 500 is shown, which is displayed in response to detecting stylus contact 581G. Figure 5P , the first portion 553A and the second portion 533A are moved relative to each other to display the non-text area, which has been converted into the drawing area 540.

[0196] In various implementations, in response to detecting a tap from a stylus, the user interface 500 displays the drawing panel 540, but in response to detecting a tap from a finger, the user interface 500 does not display the drawing panel 540. Conversely, in response to detecting a tap from a finger, the user interface 500 does not change or changes in another manner, such as a navigation operation or displaying the selected text menu 530 described above.

[0197] In various implementations, the size of drawing panel 540 is proportional to the magnitude of a property of stylus contact 581G. In some embodiments, the property is the force, time, or distance of stylus contact 581G.

[0198] Figure 5Q Shows Figure 5P The user interface 500 is displayed in response to detecting a drawing input within the drawing panel 540. Figure 5Q In FIG. 5 , the drawing panel 540 displays a drawing 545C corresponding to the detected drawing input. Figure 5QAs shown, the drawing panel 540 is displayed with a feathered border so that the portion of the drawing 545C near the edge of the drawing panel 540 gradually fades out. Therefore, the user interface 500 displays the drawing 545C by fading the graphics near the edge of the drawing panel 540.

[0199] Figure 5R Shows Figure 5Q User interface 500 of , wherein stylus contact 581H is detected at a location outside of drawing panel 540.

[0200] Figure 5S Shows Figure 5R 5. The user interface 500 is displayed in response to detecting stylus contact 581H. In response to detecting stylus contact 581H, first portion 553A and second portion 553B have moved relative to each other to stop displaying drawing panel 540. Figure 5S , user interface 500 includes a drawing indicator 547C displayed at a location corresponding to a location between first portion 553A and second portion 554B. Drawing indicators 547A-547B associated with other drawings are also displayed.

[0201] Figure 5S Finger contact 5811 is shown detected at the location of drawing indicator 547A associated with previously selected text (eg, the word "apple").

[0202] Figure 5T User interface 500 is shown displayed in response to detecting finger contact 5811. In response to detecting finger contact 5811, first portion 551A and second portion 551B move to display drawing panel 540 including a graphic of previously input drawing 545A.

[0203] Figure 5T Finger contact 581J is shown detected at a location outside of drawing panel 540.

[0204] Figure 5U User interface 500 is shown, displayed in response to detecting finger contact 581J. In response to detecting finger contact 581J, first portion 551A and second portion 551B move to stop displaying drawing panel 540.

[0205] Figure 5U Finger contact 581K detected at the location of directory enable representation 512 is shown.

[0206] Figure 5VUser interface 500 is shown, which is displayed in response to detecting finger contact 581K. In response to detecting finger contact 581K at the location of table of contents affordance 512, user interface 500 displays a table of contents view. Within the table of contents view, application bar 510 changes to display a resume affordance 514 for returning to the reading view shown in the previous figures, replacing table of contents affordance 512. Within the table of contents view, text display area 520 is replaced with metadata display area 527, which displays a table of contents 528 for the electronic book in the table of contents view, which has affordances for displaying corresponding portions of the text of the electronic book.

[0207] The metadata display area 527 includes a view switching indication 529 for switching between a catalog view, a bookmark view including a list of bookmarks of the electronic book, and an annotation view including a list of annotations of the electronic book.

[0208] Figure 5V Finger contact 581L is shown detected at a location of view switch enable indication 529 that corresponds to the notes view.

[0209] Figure 5W User interface 500 is shown, which is displayed in response to detecting finger contact 581L. Figure 5W In the note view, the user interface 500 displays a note view. In the note view, the metadata display area 527 includes an annotation list 528 that lists the annotations of the electronic book. In addition to other annotations (e.g., notes and highlights) added by the user, the annotation list 528 also includes drawing annotations added by the user. Specifically, Figure 5W As shown, annotation list 528 includes the graphic of drawing 545A (and a portion of the graphic of a portion of drawing 545B) and the attached annotations.

[0210] Figure 5W Finger contact 581M detected at the location of restore enable indication 514 is shown.

[0211] Figure 5X User interface 500 is shown, which is displayed in response to detecting finger contact 581M and additional input to navigate to another portion of the electronic book. Figure 5X , the text display area 520 includes a first arrangement of text 525A in two columns. The first arrangement of text 525A includes various portions of the text at various locations. As shown below, reflowable text allows other arrangements of text to include various portions of the text at different locations.

[0212] Figure 5XFinger contact 581N detected at the location of hand-drawn annotation enable representation 590 is shown. In response to detecting finger contact 581N at the location of hand-drawn annotation enable representation 590, user interface 500 enters a hand-drawn annotation mode, where the finger contact is interpreted as a hand-drawn annotation. In some embodiments, the stylus contact is interpreted as a hand-drawn annotation without explicitly entering the hand-drawn annotation mode (e.g., through hand-drawn annotation enable representation 590). In response to detecting a second finger contact at the location of hand-drawn annotation enable representation 590, user interface 500 exits the hand-drawn annotation mode and interprets the finger contact according to the default behavior.

[0213] Figure 5Y A user interface 500 is shown that is displayed in response to receiving multiple groups of one or more annotation inputs, each group corresponding to a separate annotation associated with a portion of text (each portion of the text itself includes multiple portions, as described below). In various specific implementations, a portion of the text includes a character, multiple characters, a word, multiple words, a line, multiple lines, a paragraph, or multiple paragraphs. The user interface 500 displays a representation of each annotation associated with the portion of the text. The shape of the representation corresponds to the location on the display where a group of annotation inputs were detected. Therefore, on a touch-sensitive display, the representation is displayed (at least initially) at the same location where contact was detected. Therefore, the representation is not automatically generated based on a selection of an annotation type, but the representation reflects the input provided by an individual user. This allows a user to enter an unlimited number of different types of annotations.

[0214] User interface 500 includes a first representation of a strikethrough annotation 591A on a portion of text reading content "(as already related)". The strikethrough annotation can be generated in response to detecting a set of one or more annotation inputs, the set of one or more annotation inputs including one or more strikethrough inputs detected at least in part at a set of locations on the text portion associated with the strikethrough annotation. The strikethrough annotation can take various forms, such as a line through the text, across the text (as shown in the first representation of strikethrough annotation 591A), highlighting the text with non-black, or any other annotation on the text portion.

[0215] The user interface includes a first representation of a wraparound annotation 592A surrounding a portion of the text reading "apples." The wraparound annotation can be generated in response to detecting a set of one or more annotation inputs, the set of one or more annotation inputs including one or more wraparound inputs detected at least partially at a set of positions around the portion of text associated with the wraparound annotation. The wraparound annotation can take various forms, such as a circle around the text (as shown in the first representation of the wraparound annotation 592A), a box around the text, or a cloud shape around the text. In addition, the wraparound annotation does not need to completely surround the text, but includes portions on at least two sides of the text portion. Therefore, the wraparound annotation can include a line above the text and a line below the text (sandwiching the text) or enclosing the text with brackets, or any other annotation that surrounds (even if not completely surrounds) the text portion.

[0216] User interface 500 includes a first representation of an underline annotation 593A below a portion of text reading "three of the golden apples". An underline annotation may be generated in response to detecting a set of one or more annotation inputs, the set of one or more annotation inputs including one or more underline inputs detected at least partially at a set of locations below the portion of text associated with the underline annotation. The underline annotation may take various forms, such as a straight line, a wavy line (as shown in the first representation of underline annotation 593A), a dashed line, or any other annotation below the portion of text.

[0217] User interface 500 includes a first representation of a text connection annotation 594A that connects a first portion of text reading content, "golden apples," and a second portion of text reading content, "sacred fruit." The text connection annotation can be generated in response to detecting a set of one or more annotation inputs, the set of one or more annotation inputs including: a first set of one or more annotation inputs detected at a location associated with the first text portion; a second set of one or more annotation inputs detected at a location associated with the second text portion; and a third set of one or more annotation inputs detected at a location between the first text portion and the second text portion. The text connection annotation can take various forms, such as one or more lines between surrounding annotations (as shown in the first representation of text connection annotation 594A) or one or more lines between underline annotations.

[0218] User interface 500 includes a first representation of a drawing connection annotation 595A that connects a portion of the text reading "Pallas-Athene" to a drawing of a question mark. The drawing connection annotation may be generated in response to detecting a set of one or more annotation inputs, the set of one or more annotation inputs including a first set of one or more annotation inputs detected at a location associated with a first portion of text, a second set of one or more annotation inputs detected at a location not associated with a portion of text, and a third set of one or more annotation inputs detected at a location between the first portion of text and the location not associated with a portion of text.

[0219] User interface 500 includes a first representation of an underline annotation 596A below a portion of textual reading content "Realm of Shades". The first representation of underline annotation 596A includes two separate sub-portions, one below the textual reading content sub-portion "Realm of" and the other below the textual reading content sub-portion "Shades".

[0220] Figure 5Z Shows Figure 5Y User interface 500 in which stylus contact 581O is detected at the location where text display option indication 517 is represented.

[0221] Figure 5AA User interface 500 is shown, displayed in response to detecting stylus contact 5810. In response to detecting stylus contact 5810 at the location of text display options affordance 517, user interface 500 includes text display options menu 571. Text display options menu 571 includes a reduction affordance 572A for reducing the font size of displayed text, an increase affordance 572B for increasing the font size of displayed text, and a font selection affordance 573 for selecting a font for displayed text.

[0222] Figure 5AA Stylus contact 581P detected at the location of font selection enable indication 573 is shown.

[0223] Figure 5AB User interface 500 is shown displayed in response to detecting stylus contact 581 P. In response to detecting stylus contact 581 P, user interface 500 includes a font selection menu 574 that includes a plurality of affordances for selecting a particular font for displayed text.

[0224] Figure 5AB Stylus contact 581Q is shown detected at the location of the "Century" selection affordance.

[0225] Figure 5AC User interface 500 is shown, which is displayed in response to detecting stylus contact 581Q. Figure 5AC , text display area 520 includes a second arrangement of text 525B in two columns. The second arrangement of text 525B includes portions of the text that were previously displayed at various locations in the first arrangement of text 525A.

[0226] User interface 500 includes a second representation of a strikethrough annotation 591B on a portion of text reading content “(as already related)”. First representation 591A includes a portion displayed in a first arrangement adjacent to a first portion of text (e.g., “(as already related)”) and another portion displayed in a first arrangement adjacent to a second portion of text (e.g., “related)”), while second representation 591B includes a portion displayed in a second arrangement adjacent to a first portion of text and a portion displayed in a second arrangement adjacent to a second portion of text. Thus, after the text is rearranged, the same portion of the text is marked with a strikethrough.

[0227] However, the second representation 591B is different from the first representation 591A because the second representation is slightly wider due to different fonts. In addition, the second representation 591B is located at a different position from the first representation 591A. However, the second representation 591B has the same shape as the first representation 591A. When one representation is a linear transformation of another representation, it is considered that the two representations (or a portion thereof) have the same shape. Therefore, performing move, resize, reflect, rotate and stretch operations on the representation does not change its shape. In fact, the second representation 591B is a repositioned and resized version of the first representation 591A. Therefore, the look and feel of the annotated representation is maintained by reflowing the text.

[0228] User interface 500 includes a second representation of a wraparound annotation 592B around a portion of the text reading "apples." The first representation 592A includes a portion displayed in a first arrangement adjacent to a first portion of the text (e.g., "app") and another portion displayed in the first arrangement adjacent to a second portion of the text (e.g., "les."), while the second representation 592B includes a portion displayed in a second arrangement adjacent to the first portion of the text and a portion displayed in the second arrangement adjacent to the second portion of the text. Thus, after the text is rearranged, the same portion of the text is wraparound marked.

[0229] However, the second representation 592B is different from the first representation 592A because the second representation is slightly wider due to the different fonts. In addition, the second representation 592B is located at a different position than the first representation 592A. However, the second representation 592B has the same shape as the first representation 592A. In fact, the second representation 592B is a repositioned and resized version of the first representation 592A. Therefore, the look and feel of the annotated representation is maintained by reflowing the text.

[0230] User interface 500 includes a second representation of an underline annotation 593B below the text reading content portion "three of the golden apples". The first representation 593A includes a portion displayed in a first arrangement adjacent to a first portion of the text (e.g., "thee of the") and another portion displayed in the first arrangement adjacent to a second portion of the text (e.g., "golden apples"), while the second representation 593B includes a portion displayed in a second arrangement adjacent to the first portion of the text and a portion displayed in the second arrangement adjacent to the second portion of the text. Therefore, after the text is rearranged, the same portion of the text is underlined.

[0231] Specifically, in the first arrangement, the first portion of the text is adjacent to the second portion of the text in the first arrangement, while in the second arrangement, the first portion of the text is not adjacent to the second portion of the text. Therefore, in the second arrangement, the second representation 593B is divided into two parts, wherein the first part is displayed adjacent to the first portion of the text in the second arrangement, and the second part is displayed adjacent to the second portion of the text in the second arrangement.

[0232] In some embodiments, as described below, the second representation is a copied version of the first representation, rather than a split version of the first representation.

[0233] Thus, although the same portion of the text is underlined after the text is reflowed, the second representation 593B is different from the first representation 592A at least because it has two portions. However, the look and feel of the annotated representation is maintained by reflowing the text.

[0234] User interface 500 includes a second representation of a text connection annotation 594B that connects a first portion of the text reading content, "golden apples," and a second portion of the text reading content, "sacred fruit." First representation 594A includes a portion displayed in a first arrangement adjacent to a first portion of the text (e.g., "golden apples") and another portion displayed in the first arrangement adjacent to a second portion of the text (e.g., "sacred fruit"), while second representation 594B includes a portion displayed in a second arrangement adjacent to the first portion of the text and a portion displayed in a second arrangement adjacent to the second portion of the text. Thus, after rearranging the text, the same portions of the text are connected.

[0235] However, the second representation 594B is different from the first representation 594A because the two portions of the text have been moved. The first representation 594A includes a double slash connector ( / / ) between the different portions of the text, and the second representation 594B includes a double backslash connector (\\) between the different portions of the text. Both representations include connectors between the two portions of the text, and the shapes of the connectors are the same. Therefore, the look and feel of the representation of the annotation is maintained by reflowing the text.

[0236] User interface 500 includes a second representation of a drawing connection annotation 595B that connects the text reading content portion "Pallas-Athene" to the question mark drawing. First representation 595A includes a portion displayed in a first arrangement adjacent to a first portion of text (e.g., "Pallas-") and another portion displayed in the first arrangement adjacent to a second portion of text (e.g., "Athene"), while second representation 595B includes a portion displayed in a second arrangement adjacent to the first portion of text and a portion displayed in the second arrangement adjacent to the second portion of text. Thus, after the text is rearranged, the same portion of the text is identified (and connected to the drawing).

[0237] However, the second representation 595B differs from the first representation 595A because the connector between the text and the drawing has been resized. Thus, the look and feel of the annotated representation is maintained by reflowing the text.

[0238] User interface 500 includes a second representation of an underline annotation 596B below the text reading content portion "Realm of Shades". First representation 596A includes a portion displayed in a first arrangement adjacent to a first portion of the text (e.g., "Realm of") and another portion displayed in the first arrangement adjacent to a second portion of the text (e.g., "Shades"), while second representation 595B includes a portion displayed in a second arrangement adjacent to the first portion of the text and a portion displayed in a second arrangement adjacent to the second portion of the text. Thus, after the text is rearranged, the same portion of the text is underlined.

[0239] However, the second representation 596B is different from the first representation 596A because the two portions of the first representation 596A have been merged. Specifically, in the first arrangement, the first portion of the text (e.g., "Realm of") is not adjacent to the second portion of the text (e.g., "Shades"), while in the second arrangement, the first portion of the text is adjacent to the second portion of the text. Therefore, the second representation 596B includes two separate elements of the first representation 596A merged together. For example, the endpoints of the two separate elements are displayed at the same location in the second representation. Therefore, the look and feel of the annotated representation is maintained by reflowing the text.

[0240] Figure 5AD Shows Figure 5AC User interface 500 in which stylus contact 581R is detected at the location where text display option indication 517 is indicated.

[0241] Figure 5AE User interface 500 is shown displayed in response to detecting stylus contact 581R. In response to detecting stylus contact 581R at the location of text display options affordance 517, user interface 500 includes text display options menu 571.

[0242] Figure 5AE Stylus contact 581S is shown detected at the location of enlarged enable indication 572B.

[0243] Figure 5AF User interface 500 is shown, which is displayed in response to detecting stylus contact 581S. Figure 5AF , text display area 520 includes a third arrangement of text 525C in two columns. The third arrangement of text 525C includes portions of the text that were previously displayed at various locations in the first arrangement of text 525A and the second arrangement of text 525B.

[0244] User interface 500 includes a third representation of a strikethrough annotation 591C on a portion of text reading content “(as already related)”. First representation 591A includes a portion displayed in a first arrangement adjacent to a first portion of text (e.g., “(as already related)”) and another portion displayed in a first arrangement adjacent to a second portion of text (e.g., “related)”), while third representation 591C includes a portion displayed in a second arrangement adjacent to a first portion of text and a portion displayed in a second arrangement adjacent to a second portion of text. Thus, after the text is rearranged, the same portion of the text is marked with a strikethrough.

[0245] However, the third representation 591C differs from the first representation 591C in that it includes two separate portions, each having the same shape as the first representation 591A. Specifically, in the first arrangement, the first portion of text (e.g., "(as already ready")) is adjacent to the second portion of text (e.g., "related)"), while in the second arrangement, the first portion of text is not adjacent to the second portion of text. Thus, the third representation includes a first instance displayed adjacent to the first portion of text and a second instance displayed adjacent to the second portion of text. Each instance has the same shape as the first representation 591A. Thus, the look and feel of the annotated representation is maintained by reflowing the text.

[0246] FIG. 6A to FIG. 6D A flowchart of a method 600 for displaying a drawing panel according to some embodiments is shown. The method 600 is performed on an electronic device (e.g., Figure 1A The portable multifunction device 100, or Figure 3 The method 600 is performed at a device 300 in the embodiment of the present invention. In some embodiments, as described in detail below as an example, the display is a touch screen display, and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 600 are optionally combined, and / or the order of some operations is optionally changed.

[0247] As described below, method 600 provides an intuitive way to add drawing annotations to text. The method reduces the cognitive burden on the user when adding drawing annotations to text, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, enabling users to add drawing annotations to text faster and more efficiently saves power and increases the time between battery charges.

[0248] The device displays (602) a text area on the touch-sensitive display, the text area including a first portion of text and a second portion of text. Figure 5A, the device displays a text display area 520 that includes various portions of text at various locations.

[0249] The device detects (604) input within a text area on the touch-sensitive display. Figure 5B In , the device detects finger contact 581A in text display area 520. In another example, Figure 5G In , the device detects an expansion gesture 581D in the text display area 520. In another example, Fig.5O , the device detects stylus contact 581G within text display area 520. In some embodiments, finger contact is processed in the same manner as stylus contact. In some embodiments, finger contact is processed in a different manner than stylus contact.

[0250] In some embodiments, when detecting input within the text area, the device detects (606) a first input within the text area on the touch-sensitive display. Figure 5B , the device detects finger contact 581A within text display area 520. In some embodiments, in response to detecting the first input, the device displays (608) an annotation menu including a drawing option on the touch-sensitive display. Figure 5C , the device displays a selected text menu 530 including a drawing affordance 535 for adding a drawing annotation to the selected text. In some embodiments, the device detects (610) a second input at the location of the drawing option on the touch-sensitive display. For example, Figure 5D , the device detects finger contact 581B at the location of the drawing indication 535.

[0251] In some embodiments, upon detecting input within the text area, the device detects (612) an expand gesture on the touch-sensitive display that includes a first contact and a second contact that move away from each other. Thus, a user can quickly open a drawing panel to add drawing annotations to text with a single gesture, thereby reducing interaction with the device. Reducing interaction with the device saves power, increases the time between battery charges, and reduces wear and tear on the device. In addition, screen space does not need to be used to display an affordance for adding drawing annotations, resulting in a more efficient human-computer interface (because the user does not need to manipulate the user interface to find an affordance for performing the function). For battery-powered electronic devices, a more efficient human-computer user interface saves power and increases the time between battery charges. In addition, a more efficient human-computer user interface reduces the amount of user interaction with the device and reduces wear and tear on the device. By using less space on the screen, a smaller (and less expensive) screen can provide the same usability. For example, on a Figure 5G In , the device detects an expand gesture 581D, which includes a first contact and a second contact moving away from each other. Figure 5J , the device detects expand gesture 581E, which includes the first contact and the second contact moving away from each other.

[0252] In some embodiments, the size of the drawing panel is proportional to the amount of movement of the first contact and / or the second contact as part of the expand gesture (614). Thus, the user can quickly set the size of the drawing panel with a single gesture, thereby reducing interaction with the device. As described above, reducing interaction with the device saves power, increases the time between battery charges, and reduces wear and tear on the device. For example, in Fig.5I In response to the expand gesture 581D for the first size, the drawing panel 540 is in the first size, and Figure 5K In response to an expand gesture 581E to the second size, the drawing panel 540 is in the second size.

[0253] In some embodiments, while detecting input within a text area, the device detects (616) a contact on the touch-sensitive display that is adjacent to a paragraph divider between a first portion of text and a second portion of text. By using the location of the contact as input to open a drawing panel, the paragraph divider provides information to the user and also acts as an interactive affordance. This allows for more efficient use of screen space, resulting in a more efficient human-computer interface. For battery-powered electronic devices, a more efficient human-computer user interface saves power and increases the time between battery charges. In addition, a more efficient human-computer user interface reduces the amount of user interaction with the device and reduces wear and tear on the device. By using less space on the screen, a smaller (and less expensive) screen can provide the same usability. For example, on Fig.5O , the device detects stylus contact 581G that is adjacent to a paragraph divider between a first portion 553A of text and a second portion 553B of text.

[0254] In some embodiments, based on determining (620) that the contact is a stylus contact, the first portion of the text moves relative to the second portion of the text to display the non-text area, and based on determining that the contact is a finger contact, the first portion of the text does not move relative to the second portion of the text to display the non-text area. By using the input type (finger or stylus) to determine the user interface response, the space on the screen can be used more efficiently and the amount of user interaction with the device is reduced. For example, in Figure 5P In FIG. 5 , based on determining that stylus contact 581G is a stylus contact, first portion 553A of the text moves relative to second portion 553B of the text to display the non-text area that has been converted into drawing panel 540. In contrast, in FIG. Figure 5C In the example, based on determining that finger contact 581A is a finger contact, the drawing panel is not displayed.

[0255] In response to detecting the input within the text area, the device moves (622) the first portion of the text relative to the second portion of the text to display a non-text area between the first portion of the text and the second portion of the text, the non-text area not including any text. Figure 5H , the first portion 552A of the text has been moved relative to the second portion 552B of the text to display a non-text region 549 between the first portion 552A and the second portion 552B of the text, and the non-text region 549 does not include any text. Figure 5D , the first portion 551A of the text has been moved relative to the second portion 551B of the text to show that the non-text area has been converted to the drawing panel 540.

[0256] In some embodiments, the first portion of the text moves (624) relative to the second portion of the text to display the non-text area in response to the first portion of the expand gesture. By associating the movement of the text with the first portion of the gesture and converting the non-text area to a drawing panel with the second portion of the gesture, the partial gesture provides the user with additional information about the availability of the drawing panel feature and increases the likelihood that the user will manipulate the user interface in an efficient manner, thereby saving power, increasing the time between battery charges, and reducing wear and tear on the device. For example, in Figure 5H , first portion 552A of text has moved relative to second portion 552B of text to display non-text area 549 in response to a first portion of expand gesture 581D (eg, detection of two contacts and their movement away from each other a first distance).

[0257] In some embodiments, when moving the first portion of the text relative to the second portion of the text, the device moves (626) the first portion of the text in a first direction by a first distance and moves the second portion of the text in a second direction by a second distance. Thus, both the first portion of the text and the second portion of the text move. For example, in Figure 5D , a first portion 551A of the text moves a first distance in a first direction (eg, upward), and a second portion 552B of the text moves a second distance in a second direction (eg, in the opposite direction, downward).

[0258] In some embodiments, the first distance and the second distance (628) are selected so as to maintain at least a subset of the first portion of the text and at least a subset of the second portion of the text on the display. Thus, additional context is provided to the user when providing drawing input, thereby reducing the likelihood that the user will close the drawing panel to access the context and reopen the drawing panel to provide drawing input. This reduces user interaction with the device, thereby saving power, increasing the time between battery charges, and reducing wear and tear on the device. For example, in Figure 5K, moving the first portion 552A of text and the second portion 552B of text such that at least a subset of the first portion 552A of text and at least a subset of the second portion 552B of text remain on the display.

[0259] In some embodiments, the first distance and the second distance are selected (630) based on the input location. Fig.5I , the first portion of text 552A moves a specific first distance based on the position of expand gesture 581D closer to the top of the display. Figure 5P , first portion 553A of text moves a different first distance based on the location of stylus contact 581G closer to the bottom of the display.

[0260] The device converts (632) at least a portion of the non-text area into a drawing panel that is configured to receive drawing input via the touch-sensitive display and display a corresponding graphic in response to receiving the drawing input. Fig.5I In the embodiment, the device increases the size of the non-text area 549 and converts at least a portion of the non-text area 549 into the drawing panel 540. In another example, Figure 5P At least a portion of the non-text area converted to the drawing panel 540 is shown.

[0261] In some embodiments, the device converts (634) at least a portion of the non-text area into a drawing panel in response to the second portion of the expand gesture. Fig.5I , the device converts at least a portion of the non-text area 549 to the drawing panel 540 in response to the second part of the expand gesture 581D. In some embodiments, the first part of the expand gesture moves the contacts away from each other a first distance, and the second part of the expand gesture moves the contacts away from each other a second distance. Thus, when the contacts begin to move away from each other, the non-text area appears, and once the contacts are a threshold distance away from each other, the non-text area is converted. In some embodiments, the first part of the expand gesture moves the contacts away from each other, and the second part of the expand gesture lifts the contacts off the display. Thus, when the contacts move away from each other, the non-text area appears (and changes size), and once the contacts end, the non-text area is converted.

[0262] In some embodiments, the drawing panel includes (636) one or more drawing affordances that are used to change the color or modify a virtual drawing tool of a corresponding graphic. Figure 5D, the drawing panel 540 includes a plurality of drawing affordances 541-544, which are used to change the color or modify the virtual drawing tools of the corresponding graphics. Specifically, the drawing panel 540 includes a drawing mode affordance 541, which is used to enter a drawing mode (e.g., select a virtual pencil tool) to add lines to the drawing within the drawing panel 540. The drawing panel 540 includes a brush affordance 540, which is used to select a virtual brush for adding strokes to the drawing within the drawing panel 542. The drawing panel 540 includes an eraser affordance 540, which is used to erase some parts of the drawing within the drawing panel 543. The drawing panel 540 includes a color affordance 540, which is used to select a color to be added to the drawing within the drawing panel 544.

[0263] In some embodiments, the device receives (638) drawing input via the touch-sensitive display and displays (640) the corresponding graphics. Figure 5E , the device displays graphic 545A of a golden apple in response to receiving drawing input via the touch-sensitive display.

[0264] In some embodiments, while displaying the corresponding graphic, the device fades (642) the corresponding graphic near the edge of the drawing panel. This provides user feedback about the limitations of the drawing panel, thereby preventing the user from attempting to provide drawing input outside of the drawing panel (and thereby closing the drawing panel and potentially causing the user to reopen the drawing panel). This reduces user interaction with the device, thereby saving power, increasing the time between battery charges, and reducing wear and tear on the device. For example, in Figure 5Q In the example, the device displays a graphic of a strong man 545C that fades in and out near the edge of the drawing panel.

[0265] In some embodiments, the device detects (644) a close input on the touch-sensitive display. Figure 5E In , the device detects finger contact 581C. In another example, Figure 5M In , the device detects a pinch gesture 581F. In another example, in Figure 5R In the process, the device detects that the stylus touches 581H.

[0266] In some embodiments, when detecting the close input, the device detects (646) a contact on the touch-sensitive display at a location outside the drawing panel. Therefore, no screen space is required for the affordance of closing the drawing panel, thereby efficiently utilizing the space on the screen and achieving a more efficient human-computer interface. For example, in Figure 5E In , the device detects finger contact 581C at a position outside the drawing panel 540. For another example, in Figure 5R , the device detects stylus contact 581H at a location outside of drawing panel 540.

[0267] In some embodiments, when detecting the close input, the device detects (648) a pinch gesture on the touch-sensitive display, the pinch gesture comprising a first contact and a second contact moving toward each other. Thus, no screen space is required for the affordance of closing the drawing panel, thereby efficiently utilizing the space on the screen and achieving a more efficient human-computer interface. For example, in Figure 5M , the device detects a pinch gesture 581F that includes a first contact and a second contact moving toward each other.

[0268] In some embodiments, in response to detecting the closing input, the device moves (650) the first portion of the text relative to the second portion of the text to cease displaying the non-text area. Fig. 5F , the device moves the first portion 551A of the text relative to the second portion 551B of the text to stop displaying the non-text area converted into the drawing panel 540.

[0269] In some embodiments, the device displays (652) a drawing indicator on the touch-sensitive display at a location corresponding to a location between the first portion of the text and the second portion of the text. Fig. 5F , the device displays a drawing indicator 547A at a location to the left of (eg, in the margin) a location between a first portion 551A of text and a second portion 551B of text.

[0270] In some embodiments, the device detects (654) an open input at the location of the drawing indicator on the touch-sensitive display. Thus, the drawing indicator provides information to the user (about the presence of the drawing) and also serves as an interactive affordance. This uses the space on the screen more efficiently, resulting in a more efficient human-computer interface (e.g., the user does not need to navigate the user interface to find affordances for performing functions to interact with the image). For battery-powered electronic devices, a more efficient human-computer user interface saves power and increases the time between battery charges. In addition, a more efficient human-computer user interface reduces the amount of user interaction with the device and reduces wear and tear on the device. By using less space on the screen, a smaller (and less expensive) screen can provide the same usability. For example, on a Figure 5S , the device detects finger contact 581I at the location of drawing indicator 547A.

[0271] In some embodiments, in response to detecting the opening input, the device moves the first portion of the text relative to the second portion of the text to display the drawing panel. Figure 5T , the device moves a first portion 551A of the text relative to a second portion 551B of the text to display a drawing panel 540 including a corresponding graphic 545A.

[0272] In some embodiments, the device displays an annotation list on the touch-sensitive display, the annotation list including one or more drawings received via the corresponding drawing panel together with one or more additional annotations. Figure 5W , the device displays annotation list 528, which includes the graphic of drawing 545A (and a portion of the graphic of a portion of drawing 545B) together with the attached annotations.

[0273] It should be understood that FIG. 6A to FIG. 6D The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with reference to other methods described herein (e.g., method 700) are also applicable in a similar manner to the details of the processes described above with reference to FIG. 6A to FIG. 6D For example, the displayed text arrangement, input associated with the displayed text arrangement, affordances, and user interface responses described above with reference to method 600 optionally have one or more of the features described herein with reference to other methods described herein (e.g., method 700): displayed text areas, input associated with text areas, affordances, and user interface responses. For the sake of brevity, these details are not repeated here.

[0274] FIG. 7A to FIG. 7D A flowchart of a method 700 for adding annotations to reflowable text according to some embodiments is shown. The method 700 is performed on an electronic device (e.g., Figure 1A The portable multifunction device 100 or Figure 3 The method 700 is performed at a device 300 in the embodiment of the present invention. In some embodiments (as described below as an example), the display is a touch screen display and the touch-sensitive surface is on the display or integrated with the display. In some embodiments, the display is separate from the touch-sensitive surface. Some operations in method 700 are optionally combined and / or the order of some operations is optionally changed.

[0275] As described below, method 800 provides an intuitive way to annotate reflowable text. The method reduces the cognitive burden on users when annotating reflowable text, thereby creating a more efficient human-computer interface. For battery-powered electronic devices, enabling users to annotate reflowable text faster and more efficiently saves power and increases the time between battery charges.

[0276] The device displays (702) a first arrangement of text on the touch-sensitive display, wherein a first portion of the text has a first position relative to a second portion of the text. Figure 5X, text display area 520 includes a first arrangement of text 525A in which various portions of the text have various positions relative to other portions of the text.

[0277] When the text is displayed in the first arrangement, the device detects (704) a set of one or more annotation inputs via the touch-sensitive display. Figure 5Y , user interface 500 shows the results of multiple groups of one or more annotation inputs.

[0278] In some embodiments, when detecting a set of one or more annotation inputs, the device detects (706) an input via the touch-sensitive display at a location represented by an annotation indicator displayed on the touch-sensitive display. Figure 5X , finger touch 581N is detected at the location of annotation enable representation 590 displayed as part of the user interface. In some embodiments, when detecting a set of one or more annotation inputs, the device detects (708) one or more finger contacts via the touch-sensitive display. In some embodiments, when detecting a set of one or more annotation inputs, the device detects (710) one or more stylus contacts via the touch-sensitive display. By utilizing the input type (finger or stylus) to determine the user interface response, the space on the screen can be used more efficiently and the amount of user interaction with the device is reduced. In some embodiments, finger contacts are processed in the same manner as stylus contacts. In some embodiments, finger contacts are processed differently than stylus contacts.

[0279] In some embodiments, the set of one or more annotation inputs includes (712) one or more underline inputs that are at least partially detected at a set of locations below the first portion of the text and the second portion of the text. Figure 5Y , the device displays a first representation of underline annotation 593A in response to underline input being at least partially detected at a set of locations below a first portion of text (e.g., “three of the”) and a second portion of text (e.g., “golden apples”).

[0280] In some embodiments, the set of one or more annotation inputs includes (714) one or more strikethrough inputs that are at least partially detected at a set of locations on the first portion of the text and the second portion of the text. Figure 5Y , the device displays a first representation of strikethrough annotation 591A in response to at least partially detecting strikethrough input at a set of locations on a first portion of text (e.g., “(as already)” and a second portion of text (e.g., “related)”).

[0281] In some embodiments, the set of one or more annotation inputs includes (716) one or more surrounding inputs that are at least partially detected at a set of locations around the first portion of the text and the second portion of the text. Figure 5Y , the device displays a first representation of surround annotation 592A in response to surround input at least partially detected at a set of locations around a first portion of text (e.g., “app”) and a second portion of text (e.g., “les.”).

[0282] In some embodiments, a set of one or more annotation inputs includes (718): a first set of one or more annotation inputs detected at a location associated with a first portion of text; a second set of one or more annotation inputs detected at a location associated with a second portion of text; and a third set of one or more annotation inputs detected at a location between the first portion of text and the second portion of text. Figure 5Y , the device displays a first representation of a text connection annotation 594A in response to: a first set of one or more annotation inputs detected at a location associated with a first portion of text (e.g., an surround gesture around text reading “golden apples”); a second set of one or more annotation inputs detected at a location associated with a second portion of text (e.g., an surround gesture around text reading “sacred fruit”); and a third set of one or more annotation inputs detected at a location between two pieces of text (e.g., a double slash connector).

[0283] The device displays (720) a first representation of an annotation on the touch-sensitive display that is associated with both the first portion of the text and the second portion of the text. At least a portion of the first representation is displayed (722) adjacent to the first portion of the text in a first arrangement, and at least a portion of the first representation is displayed adjacent to the second portion of the text in the first arrangement. For example, in Figure 5Y In , the device displays a first representation of a strikethrough annotation 591A, which is associated with the first portion of the text reading content "(as already)" and the second portion of the text reading content "related" . For another example, in Figure 5Y In , the device displays a first representation of the surrounding annotation 592A, which is associated with the first part of the text reading content "app" and the second part of the text reading content "les.". For another example, in Figure 5Y In , the device displays a first representation of an underline annotation 593A, which is associated with the first part of the text reading content "three of the" and the second part of the text reading content "golden apples". Figure 5Y, the device displays a first representation of a textual connection annotation 594A, which is associated with a first portion of the textual reading content, "golden apples," and a second portion of the textual reading content, "sacred fruit."

[0284] In some embodiments, at least a portion of the first representation is displayed in a first arrangement below the first portion of the text (724), and at least a second portion of the first representation is displayed in the first arrangement below the second portion of the text. Figure 5Y In the first arrangement 525A of text, at least a portion of a first representation of an underline annotation 593A is displayed below a first portion of the text reading content, "three of the", and at least a portion of the first representation of the underline annotation 593A is displayed below a second portion of the text reading content, "golden apples".

[0285] In some embodiments, at least a portion of the first representation is displayed in a first arrangement over the first portion of the text (726), and at least a portion of the first representation is displayed in a first arrangement over the second portion of the text. Figure 5Y , in a first arrangement 525A of text, at least a portion of a first representation of a strikethrough annotation 591A is displayed above a first portion "(as already") of text reading content, and at least a portion of the first representation of the strikethrough annotation 591A is displayed above a second portion "related)" of the text reading content. By separating the strikethrough annotation 591A into two or more components according to various specific implementations, the strikethrough annotation 591A can track the rearranged text to allow a user to change text reading and / or presentation preferences. In the present example, the text reading content "(as already") and the text reading content "related" can thus be separated to reflow the text to conform to the user's reading and / or presentation preferences, and to provide visual feedback to the user indicating that an annotation to one arrangement of the displayed text will cause the device to associate and adjust the presentation of annotations to other arrangements of the displayed text (e.g., reflow from the first arrangement to the second arrangement). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by encouraging the user to provide input using available features and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0286] In some embodiments, at least a portion of the first representation is displayed in a first arrangement around a first portion of the text (728), and at least a portion of the first representation is displayed in a first arrangement around a second portion of the text. Figure 5Y , in the first arrangement 525A of text, at least a portion of the first representation of the surrounding annotation 592A is displayed around the first portion "app" of the text reading content, and at least a portion of the first representation of the surrounding annotation 592A is displayed around the second portion "les." of the text reading content. Similar to dividing the strikethrough annotation 591A, by dividing the surrounding annotation 592A into two or more components according to various specific implementations, the surrounding annotation 592A can track the rearranged text to allow the user to change the text reading and / or presentation preferences. In this example, the text containing "apples" can therefore be separated to reflow the text to conform to the user's reading and / or presentation preferences, and provide visual feedback to the user, which indicates that the annotation of one arrangement of the displayed text will cause the device to associate and adjust the presentation of the annotation of other arrangements of the displayed text (e.g., reflow from the first arrangement to the second arrangement). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by encouraging the user to provide input using available features and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0287] In some embodiments, the first representation includes (730) a connector displayed in a first arrangement between the first portion of text and the second portion of text. Figure 5Y In the first arrangement 525A of text, a first representation of a text connection annotation 594A includes a double slash connector that is displayed between a first portion of the text reading "golden apples" and a second portion of the text reading "sacred fruit".

[0288] After displaying the first representation of the annotation (which may include continuing to display the first representation of the annotation), the device receives (732) a request to rearrange the text. Figure 5AB , the device detects stylus contact 581Q at a location indicated by the Century selection affordance of font selection menu 574. Thus, the request to rearrange the text is a request to change the font of the text. Figure 5AE , the device detects stylus contact 581S at the location of enlarged affordance 572B of text display options menu 571. Thus, the request to rearrange the text is a request to change the text font size.

[0289] In response to receiving the request to rearrange the text, the device concurrently displays (734) on the touch-sensitive display a second arrangement of the text in which the first portion of the text has a second position relative to the second portion of the text, wherein the second position is different from the first position, and a second representation of the annotation is associated with both the first portion of the text and the second portion of the text, wherein the second representation is different from the first representation. At least a portion of the second representation is displayed adjacent to the first portion of the text in the second arrangement (736), and at least a second portion of the second representation is displayed adjacent to the second portion of the text in the second arrangement.

[0290] Rearranging text can make it easier for users to read text, improve their reading speed and reduce the amount of user-device interaction, thereby saving power, increasing the time between battery charges and reducing the wear of the device. After rearrangement, the annotations associated with the same part of the text are kept to increase the possibility that the user will flow the text back to its reading preference. In other words, the annotation tracks the rearranged text so as to allow the user to change the text reading and / or presentation preferences. According to the change to the text display, the display of the annotation is adjusted to provide visual feedback to the user, and the visual feedback indicates that the annotation of one arrangement of the displayed text will make the device associate and adjust the presentation of the annotation of other arrangements of the displayed text (for example, from the first arrangement to the second arrangement). Providing improved visual feedback to the user enhances the operability of the device, and makes the user device interface more effective (for example, by encouraging the user to provide input using available features and reducing user errors when operating the device / interacting with the device), which in turn reduces power use and extends the battery life of the device by enabling the user to use the device more quickly and effectively.

[0291] For example, in Figure 5AC , the device displays a second arrangement of text 525B, wherein the first portion of the text reading content "(as already)" has a second position relative to the second portion of the text reading content "related)", wherein the second position is different from ( Figure 5Y The device also displays a second representation of strikethrough annotation 591B associated with both the first portion of text and the second portion of text. The second representation of strikethrough annotation 591B is different from strikethrough annotation 591A ( Figure 5Y ), at least because the second representation is slightly wider due to the different font.

[0292] For example, in Figure 5AC, the device displays a second arrangement of text 525B, wherein the first portion of the text reading content "three of the" has a second position relative to the second portion of the text reading content "golden apples", wherein the second position is different from ( Figure 5Y ) first position. For example, in Figure 5Y , the first part of the text is to the left of the second part of the text, and Figure 5AC , the first portion of the text is to the right of the second portion of the text. The device also displays a second representation of underline annotation 593B associated with both the first portion of the text and the second portion of the text. The second representation of underline annotation 593B is different from underline annotation 593A ( Figure 5Y ), at least because it has two parts.

[0293] In some embodiments, at least a portion of the second representation is displayed below the first portion of the text in a second arrangement (738), and at least a second portion of the second representation is displayed below the second portion of the text in a second arrangement. Figure 5AC In the second arrangement 525B of the text, at least a portion of the second representation of the underline annotation 593B is displayed below the first portion of the text reading content "three of the", and at least a portion of the second representation of the underline annotation 593B is displayed below the second portion of the text reading content "golden apples".

[0294] In some embodiments, at least a portion of the second representation is displayed in a second arrangement over the first portion of the text (740), and at least a second portion of the second representation is displayed in a second arrangement over the second portion of the text. Figure 5AC In the second arrangement 525B of the text, at least a portion of the second representation of the strikethrough annotation 591B is displayed above the first portion of the text reading content "(as already"), and at least a portion of the second representation of the strikethrough annotation 591B is displayed above the second portion of the text reading content "related)".

[0295] In some embodiments, at least a portion of the second representation is displayed in a second arrangement around the first portion of the text (742), and at least a second portion of the second representation is displayed in a second arrangement around the second portion of the text. Figure 5AC, in the second arrangement 525B of text, at least a portion of the second representation of the surrounding annotation 592B is displayed around the first portion "app" of the text reading content, and at least a portion of the second representation of the surrounding annotation 592B is displayed around the second portion "les." of the text reading content. In addition, similar to dividing the strikethrough annotation 591A, by dividing the surrounding annotation 592B into two or more components according to various specific implementations, the surrounding annotation 592B can track the rearranged text to allow the user to change the text reading and / or presentation preferences. In this example, the text containing "apples" can therefore be separated to reflow the text to conform to the user's reading and / or presentation preferences, and provide visual feedback to the user, which indicates that the annotation of one arrangement of the displayed text will cause the device to associate and adjust the presentation of the annotation of other arrangements of the displayed text (e.g., reflow from the first arrangement to the second arrangement). Providing improved visual feedback to the user enhances the operability of the device and makes the user-device interface more efficient (e.g., by encouraging the user to provide input using available features and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0296] In some embodiments, the second representation includes (744) a connector displayed in a second arrangement between the first portion of text and the second portion of text. Figure 5AC In the second arrangement 525B of text, the second representation of the text connection annotation 594B includes a double backslash connector, which is displayed between the first part of the text reading content "goldenapples" and the second part of the text reading content "sacred fruit". In some embodiments, the first connector of the first representation has the same shape as the second connector of the second representation (746). For example, in Figure 5AC , the double backslash connector of the second representation of text connection annotation 594B has the same shape as the double backslash connector of the first representation of text connection annotation 594A that has been rotated (or reflected) and possibly resized.

[0297] In some embodiments, the second representation has the same shape as the first representation (748). By maintaining the look and feel of the annotation, the user can enter an unlimited number of different types of annotations, thereby effectively providing a variety of information, meaning, and context. Providing this information efficiently reduces the user's interaction with the device (e.g., a squiggly line indicating that a word is confusing can be entered instead of a detailed explanation of the problem). For example, in Figure 5AC The second representation of the deleted line annotation 591B has the same Figure 5Y The first representation of the deleted line annotation 591A has the same shape. Figure 5AC , the second representation surrounding comment 593B has the same Figure 5Y The same shape as the first representation of the surrounding annotation 593A. Presenting similar annotations using similar shapes and visual markers provides visual feedback to the user that enhances the operability of the device and makes the user-device interface more efficient (e.g., by encouraging the user to provide input using available features and reducing user errors when operating / interacting with the device), which in turn reduces power usage and extends the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0298] In some embodiments, the second representation is a repositioned and resized version of the first representation. Figure 5AC The second representation of the deleted line comment 591B is Figure 5Y The first of the strikethrough annotations 591A represents the repositioned and resized version. Figure 5AC The second representation of the surrounding comment 593B is Figure 5Y The first of the surrounding comments 593A represents the repositioned and resized version.

[0299] In some embodiments, the first portion of the text is adjacent to the second portion of the text in the first arrangement and is not adjacent to the second portion of the text in the second arrangement. Thus, in some embodiments, the first portion of the second representation corresponding to the first portion of the first representation is displayed adjacent to the first portion of the text in the second arrangement (752), and the second portion of the second representation corresponding to the second portion of the first representation is displayed adjacent to the second portion of the text in the second arrangement. For example, in Figure 5AC , with underlined comment 593A ( Figure 5Y )'s first part of the second representation of the underline annotation 593B corresponding to the left part of the first representation of the underline annotation 593A is displayed adjacent to the text reading content part "three of the", and the second part of the second representation of the underline annotation 593B corresponding to the right part of the first representation of the underline annotation 593A is displayed adjacent to the text reading content part "goldenapples".

[0300] As described above, in some embodiments, the first portion of the text is adjacent to the second portion of the text in the first arrangement and is not adjacent to the second portion of the text in the second arrangement. Thus, in some embodiments, the first portion of the second representation corresponding to at least a portion of the first representation is displayed adjacent to the first portion of the text in the second arrangement (754), and the second portion of the second representation corresponding to at least a portion of the first representation is displayed adjacent to the second portion of the text in the second arrangement. For example, in Figure 5AFIn the figure, the first part of the third representation of the strikethrough annotation 591C corresponding to the entire first representation of the strikethrough annotation 591A is displayed adjacent to the text reading content part "(as already"), and the second part of the third representation of the strikethrough annotation 591C corresponding to the same entire first representation of the strikethrough annotation 591A is displayed adjacent to the text reading content part "related)".

[0301] In some embodiments, the first portion of text is not adjacent to the second portion of text in the first arrangement and is adjacent to the second portion of text in the second arrangement. Thus, in some embodiments, the second representation includes (756) two separate portions of the first representation that are merged together. For example, in Figure 5AC , the second representation of underline annotation 596B includes two separate portions of the first representation of underline annotation 596A merged together (eg, the representation portion below the text reading content portion "Realm of" and the representation portion below the text reading content portion "Shades").

[0302] It should be understood that FIG. 7A to FIG. 7D The specific order in which the operations in the description are described is merely exemplary and is not intended to indicate that the order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with reference to other methods described herein (e.g., method 600) are also applicable in a similar manner to the details of the above referenced methods. FIG. 7A to FIG. 7D Method 700 described herein. For example, the displayed text areas, inputs associated with the text areas, affordances, and user interface responses described above with reference to method 700 optionally have one or more of the following features of other methods described herein with reference to the present invention (e.g., method 600): the displayed text arrangements, inputs associated with the displayed text arrangements, affordances, and user interface responses described herein. For the sake of brevity, these details are not repeated here.

[0303] According to some embodiments, Figure 8 800 is a functional block diagram of an electronic device 800 configured according to the principles of the various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Figure 8 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0304] like Figure 8As shown in , the electronic device 800 includes: a display unit 802, which is configured to display a user interface; a touch-sensitive surface unit 804, which is configured to detect contact; and a processing unit 810, which is coupled to the display unit 802 and the touch-sensitive surface unit 804. In some embodiments, the processing unit 810 includes a display control unit 812, an input detection unit 814, and a drawing panel unit 816.

[0305] The processing unit 810 is configured to display (eg, using the display control unit 812 ) a text region on the display unit 802 , the text region including a first portion of text and a second portion of text.

[0306] The processing unit 810 is configured to detect (e.g., using the input detection unit 814) an input within the text area via the touch-sensitive surface unit 804. In some embodiments, the processing unit 810 is configured to detect (e.g., using the input detection unit 814) a first input within the text area via the touch-sensitive surface unit 804, and in response to detecting the first input, display (e.g., using the display control unit 812) an annotation menu including a drawing option on the display unit 802. In some embodiments, the processing unit 810 is configured to detect (e.g., using the input detection unit 814) a second input at the location of the drawing option via the touch-sensitive surface unit 804.

[0307] In some embodiments, the processing unit 810 is configured to detect (e.g., using the input detection unit 814) an expand gesture via the touch-sensitive surface unit 804, the expand gesture comprising the first contact and the second contact moving away from each other. In some embodiments, the size of the drawing panel (e.g., as provided by the drawing panel unit 816 as described below) is proportional to the amount of movement of the first contact and / or the second contact as part of the expand gesture.

[0308] In some embodiments, the processing unit 810 is configured to detect (e.g., using the input detection unit 814) a contact via the touch-sensitive surface unit 804 that is adjacent to a paragraph divider between a first portion of the text and a second portion of the text. In some embodiments, the size of the drawing panel (e.g., as provided by the drawing panel unit 816 as described below) is proportional to the magnitude of the attribute of the contact. In some embodiments, the processing unit 810 is configured to, based on a determination that the contact is a stylus contact, move the first portion of the text relative to the second portion of the text (e.g., using the drawing panel unit 816 as described below) to display a non-text area, and based on a determination that the contact is a finger contact, not move the first portion of the text relative to the second portion of the text to display the non-text area.

[0309] The processing unit 810 is configured to move the first portion of the text relative to the second portion of the text (e.g., using the drawing panel unit 816) to display a non-text area between the first portion of the text and the second portion of the text. In some embodiments, the processing unit 810 moves the first portion of the text relative to the second portion of the text (e.g., using the drawing panel unit 816) to display the non-text area in response to the first portion of the expand gesture. In some embodiments, the processing unit 810 is configured to move the first portion of the text in a first direction (e.g., using the drawing panel unit 816) a first distance and move the second portion of the text in a second direction (e.g., using the drawing panel unit 816) a second distance. In some embodiments, the processing unit 810 is configured to select (e.g., using the drawing panel unit 816) the first distance and the second distance so as to keep at least a subset of the first portion of the text and a subset of the second portion of the text on the display unit 802. In some embodiments, the processing unit 810 is configured to select (e.g., using the drawing panel unit 816) the first distance and the second distance based on the position of the input.

[0310] Processing unit 810 is configured to convert (e.g., using drawing panel unit 816) at least a portion of the non-text area into a drawing panel that is configured to receive drawing input via touch-sensitive surface 804 and display a corresponding graphic in response to receiving the drawing input. In some embodiments, processing unit 810 converts (e.g., using drawing panel unit 816) at least a portion of the non-text area into a drawing panel in response to a second portion of the expand gesture. In some embodiments, the drawing panel includes one or more drawing affordances that are used to change the color or modify a virtual drawing tool of the corresponding graphic.

[0311] In some embodiments, the processing unit 810 is configured to receive (e.g., using the input detection unit 814) drawing input via the touch-sensitive surface 804. In some embodiments, the processing unit 810 is configured to display (e.g., using the display control unit 802) the corresponding graphics. In some embodiments, the processing unit 810 is configured to fade the corresponding graphics near the edge of the drawing panel (e.g., using the display control unit 802).

[0312] In some embodiments, the processing unit 810 is configured to detect (e.g., using the input detection unit 814) a close input via the touch-sensitive surface unit 804. In some embodiments, the processing unit 810 is configured to detect (e.g., using the input detection unit 814) a contact at a location outside the drawing panel via the touch-sensitive surface unit 804. In some embodiments, the processing unit 810 is configured to detect (e.g., using the input detection unit 814) a pinch gesture via the touch-sensitive surface unit 804 that includes a first contact and a second contact that move toward each other.

[0313] In some embodiments, the processing unit 810 is configured to move (e.g., using the drawing panel unit 816) the first portion of the text relative to the second portion of the text to cease displaying the non-text area in response to detecting a closing input.

[0314] In some embodiments, the processing unit 810 is configured to display (e.g., using the display control unit 812) a drawing indicator on the display unit 802 at a position corresponding to a position between the first portion of the text and the second portion of the text.

[0315] In some embodiments, the processing unit 810 is configured to detect (e.g., using the input detection unit 814) via the touch-sensitive surface unit 804 an open input at the location indicated by the drawing.

[0316] In some embodiments, the processing unit 810 is configured to move (e.g., using the drawing panel unit 816) a first portion of the text relative to a second portion of the text to display a drawing panel in response to detecting the opening input.

[0317] In some embodiments, the processing unit 810 is configured to display (e.g., using the display control unit 812) via the display unit 802 an annotation list that includes one or more drawings received via the corresponding drawing panel together with one or more additional annotations.

[0318] The operations in the above information processing method are optionally implemented by running one or more functional modules in an information processing device, such as a general-purpose processor (for example, as described above with respect to Figure 1A and Figure 3 described) or application-specific chips.

[0319] References FIG. 6A to FIG. 6D The operation is optionally performed by Figure 1A to Figure 1B or Figure 8. For example, display operation 602, detection operation 604, move operation 622, and conversion operation 632 are optionally implemented by event classifier 170, event identifier 180, and event handler 190. Event monitor 171 in event classifier 170 detects contact on touch-sensitive display 112, and event distributor module 174 transmits event information to application 136-1. The corresponding event identifier 180 of application 136-1 compares the event information with the corresponding event definition 186, and determines whether the first contact at the first position on the touch-sensitive surface (or whether the rotation of the device) corresponds to a predefined event or sub-event, such as selection of an object on a user interface, or rotation of the device from one orientation to another orientation. When a corresponding predefined event or sub-event is detected, event identifier 180 activates an event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update application internal state 192. In some embodiments, the event handler 190 accesses the corresponding GUI updater 178 to update the content displayed by the application. Figure 1A to Figure 1B How the components depicted in the figure can implement other processes.

[0320] According to some embodiments, Fig. 9 900 is a functional block diagram of an electronic device configured according to the principles of the various described embodiments. The functional blocks of the device are optionally implemented by hardware, software, firmware, or a combination thereof that implements the principles of the various described embodiments. It should be understood by those skilled in the art that Fig. 9 The functional blocks described in the present invention are optionally combined or separated into sub-blocks to implement the principles of various described embodiments. Therefore, the description herein optionally supports any possible combination or separation or further limitation of the functional blocks described herein.

[0321] like Fig. 9 As shown in , the electronic device 900 includes: a display unit 902, which is configured to display a user interface; a touch-sensitive surface unit 904, which is configured to detect contact; and a processing unit 910, which is coupled to the display unit 902 and the touch-sensitive surface unit 904. In some embodiments, the processing unit 910 includes a display control unit 912, an input detection unit 914, and an annotation text presentation unit 916.

[0322] The processing unit 910 is configured to display (eg, using the display control unit 912 ) a first arrangement of text on the display unit 902 , wherein a first portion of the text has a first position relative to a second portion of the text.

[0323] Processing unit 910 is configured to, when displaying text in a first arrangement, detect (e.g., using input detection unit 914) a set of one or more annotation inputs via touch-sensitive surface unit 904. In some embodiments, processing unit 910 is configured to detect (e.g., using input detection unit 914) inputs at locations of annotation affordances displayed on display unit 902 via touch-sensitive surface unit 904, and detect (e.g., using input detection unit) one or more finger contacts via touch-sensitive surface unit 904. In some embodiments, processing unit 910 is configured to detect (e.g., using input detection unit 914) one or more stylus contacts via touch-sensitive surface unit 904.

[0324] In some embodiments, the processing unit 910 is configured to detect (e.g., using the input detection unit 914) via the touch-sensitive surface unit 904 a set of one or more annotation inputs that include one or more underline inputs detected at least partially below a first portion of text and a second portion of text.

[0325] In some embodiments, the processing unit 910 is configured to detect (e.g., using the input detection unit 914) via the touch-sensitive surface unit 904 a set of one or more annotation inputs that include one or more strikethrough inputs detected at least partially at a set of locations above a first portion of text and a second portion of text.

[0326] In some embodiments, the processing unit 910 is configured to detect (e.g., using the input detection unit 914) via the touch-sensitive surface unit 904 a set of one or more annotation inputs that include one or more surround inputs detected at least partially at a set of locations surrounding a first portion of text and a second portion of text.

[0327] In some embodiments, the processing unit 910 is configured to detect (e.g., using the input detection unit 914) via the touch-sensitive surface unit 904 a group of one or more annotation inputs, comprising: a first group of one or more annotation inputs detected at a location associated with a first portion of text; a second group of one or more annotation inputs detected at a location associated with a second portion of text; and a third group of one or more annotation inputs detected at a location between the first portion of text and the second portion of text.

[0328] The processing unit 910 is configured to display (e.g., using the display control unit 912) on the display unit 902 a first representation of an annotation associated with both the first portion of the text and the second portion of the text. The processing unit 910 is configured to display (e.g., using the display control unit 912) on the display unit 902 at least a portion of the first representation adjacent to the first portion of the text in a first arrangement and at least a second portion of the first representation adjacent to the second portion of the text in the first arrangement.

[0329] In some embodiments, such as when a group of one or more annotation inputs includes an underline input, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) at least a portion of the first representation arranged first below a first portion of text and at least a second portion of the first representation arranged first below a second portion of text.

[0330] In some embodiments, such as when a group of one or more annotation inputs includes a strikethrough input, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) at least a portion of the first representation arranged first over a first portion of text and at least a second portion of the first representation arranged first over a second portion of text.

[0331] In some embodiments, such as when a group of one or more annotation inputs includes a surround input, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) at least a portion of the first representation arranged first around a first portion of text and at least a second portion of the first representation arranged first around a second portion of text.

[0332] In some embodiments, the processing unit 910 is configured to display (e.g., using the display control unit 912) on the display unit 902 a first representation having a connector between the first portion of text and the second portion of text.

[0333] Processing unit 910 is configured to receive (eg, via input detection unit 914) a request to rearrange text.

[0334] Processing unit 910 is configured to, in response to receiving a request to rearrange text, concurrently display (e.g., using display control unit 914) on display unit 902 a second arrangement of text and a second representation of the annotation, wherein in the second arrangement of text, the first portion of the text has a second position relative to the second portion of the text, wherein the second position is different from the first position, and the second representation of the annotation is associated with both the first portion of the text and the second portion of the text, wherein the second representation is different from the first representation. Processing unit 910 is configured to display (e.g., using display control unit 912) on display unit 902 at least a portion of the second representation of the first portion of the text adjacent to the second arrangement and at least a second portion of the second representation of the second portion of the text adjacent to the second arrangement.

[0335] In some embodiments, such as when a group of one or more annotation inputs includes an underline input, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) at least a portion of the second representation arranged in a second manner below the first portion of the text and at least a second portion of the second representation arranged in a second manner below the second portion of the text.

[0336] In some embodiments, such as when a group of one or more annotation inputs includes a strikethrough input, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) at least a portion of the second representation arranged in a second manner above the first portion of the text and at least a second portion of the second representation arranged in a second manner above the second portion of the text.

[0337] In some embodiments, such as when a group of one or more annotation inputs includes a surround input, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) at least a portion of a second representation arranged in a second manner around a first portion of text and at least a second portion of the second representation arranged in a second manner around a second portion of text.

[0338] In some embodiments, the processing unit 910 is configured to display (e.g., using the display control unit 912) a second representation having a connector between the first portion of text and the second portion of text on the display unit 902. In some embodiments, the connector has the same shape as the connector of the first representation.

[0339] In some embodiments, the processing unit 910 is configured to display (e.g., using the display control unit 912) a second representation having the same shape as the first representation on the display unit 902. In some embodiments, the processing unit 910 is configured to display (e.g., using the display control unit 912) the second representation on the display unit 902 as a repositioned and resized version of the first representation.

[0340] In some embodiments, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) a first portion of a second representation corresponding to a first portion of a first representation of a first portion of adjacent text in a second arrangement and a second portion of a second representation corresponding to a second portion of the first representation of a second portion of adjacent text in a second arrangement.

[0341] In some embodiments, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) a first portion of a second representation corresponding to at least a portion of a first representation of a first portion of adjacent text in a second arrangement and a second portion of the second representation corresponding to at least a portion of the first representation of a second portion of adjacent text in a second arrangement.

[0342] In some embodiments, the processing unit 910 is configured to display on the display unit 902 (e.g., using the display control unit 912) a second representation that includes two separate portions of the first representation merged together.

[0343] References FIG. 7A to FIG. 7D The operation is optionally performed by Figure 1A to Figure 1B or Fig. 9. For example, display operation 702, detection operation 704, display operation 720, receiving operation 732, and display operation 734 are optionally implemented by event classifier 170, event identifier 180, and event handler 190. Event monitor 171 in event classifier 170 detects contact on touch-sensitive display 112, and event distributor module 174 transmits event information to application 136-1. The corresponding event identifier 180 of application 136-1 compares the event information with the corresponding event definition 186 and determines whether the first contact at the first position on the touch-sensitive surface (or the rotation of the device) corresponds to a predefined event or sub-event, such as selection of an object on a user interface, or rotation of the device from one orientation to another orientation. When a corresponding predefined event or sub-event is detected, event identifier 180 activates event handler 190 associated with the detection of the event or sub-event. Event handler 190 optionally uses or calls data updater 176 or object updater 177 to update application internal state 192. In some embodiments, event handler 190 accesses corresponding GUI updater 178 to update the content displayed by the application. Similarly, those skilled in the art should clearly understand that based on Figure 1A to Figure 1B How the components depicted in the figure can implement other processes.

[0344] For the purpose of explanation, the foregoing description is described by reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the invention to the disclosed precise form. According to the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best illustrate the principles of the present invention and its practical application, so that other technical personnel in the field can best use the present invention and the various described embodiments with various modifications suitable for the specific purpose contemplated.

Claims

1. A method for displaying annotations, include: At a device having one or more processors, non-transitory memory, and a touch-sensitive display: displaying a first arrangement of text on the touch-sensitive display, in which a first portion of the text has a first position relative to a second portion of the text; displaying, on the touch-sensitive display, a first representation of an annotation associated with the first portion of the text and the second portion of the text; receiving a request to rearrange the text; as well as In response to receiving the request to rearrange the text, concurrently displaying on the touch-sensitive display: a second arrangement of the text, wherein the first portion of the text has a second position relative to the second portion of the text, wherein the second position is different from the first position; and A second representation of the annotation is associated with both the first portion of the text and the second portion of the text, wherein the second representation of the annotation is different from the first representation of the annotation. 2 . The method of claim 1 , wherein the second representation of the annotation has the same shape as the first representation of the annotation. 3 . The method of claim 1 , wherein the second representation of the annotation is a repositioned and resized version of the first representation of the annotation.

4. The method according to claim 1, further comprising: include: detecting, via the touch-sensitive display, a set of one or more annotation inputs while displaying the text in the first arrangement; as well as In response to detecting the set of one or more annotation inputs, the first representation of the annotation is displayed.

5. The method according to claim 4, in: The set of one or more annotation inputs includes one or more underline inputs detected at a set of locations at least partially beneath the first portion of the text and the second portion of the text; at least a portion of the first representation of the annotation is displayed below the first portion of the text in the first arrangement, and at least a portion of the first representation of the annotation is displayed below the second portion of the text in the first arrangement; as well as At least a portion of the second representation of the annotation is displayed below the first portion of the text in the second arrangement, and at least a portion of the first representation of the annotation is displayed below the second portion of the text in the second arrangement.

6. The method according to claim 4, in: the set of one or more annotation inputs comprising one or more strikethrough inputs, the one or more strikethrough inputs being detected at least in part at a set of locations on the first portion of the text and the second portion of the text; at least a portion of the first representation of the annotation is displayed on the first portion of the text in the first arrangement, and at least a portion of the first representation of the annotation is displayed on the second portion of the text in the first arrangement; as well as At least a portion of the second representation of the annotation is displayed on the first portion of the text in the second arrangement, and at least a portion of the first representation of the annotation is displayed on the second portion of the text in the second arrangement.

7. The method according to claim 4, in: The set of one or more annotation inputs includes one or more surrounding inputs detected at least in part at a set of locations surrounding the first portion of the text and the second portion of the text; at least a portion of the first representation of the annotation is displayed around the first portion of the text in the first arrangement, and at least a portion of the first representation of the annotation is displayed around the second portion of the text in the first arrangement; as well as At least a portion of the second representation of the annotation is displayed around the first portion of the text in the second arrangement, and at least a portion of the first representation of the annotation is displayed around the second portion of the text in the second arrangement.

8. The method according to claim 4, in: the set of one or more annotation inputs comprising a first set of one or more annotation inputs detected at a location associated with the first portion of the text, a second set of one or more annotation inputs detected at a location associated with the second portion of the text, and a third set of one or more annotation inputs detected at a location between the first portion of the text and the second portion of the text; the first representation of the annotation comprising a first connector displayed between the first portion of the text and the second portion of the text in the first arrangement; and The second representation of the annotation includes a second connector displayed between the first portion of the text and the second portion of the text in the second arrangement. The method of claim 8 , wherein the second connector has the same shape as the first connector.

10. The method according to claim 1, in: the first portion of the text having the first position adjacent to the second portion of the text in the first arrangement; the first portion of the text having the second position not adjacent to the second portion of the text in the second arrangement; a first portion of the second representation of the annotation corresponding to a first portion of the first representation of the annotation is displayed adjacent to the first portion of the text in the second arrangement; as well as A second portion of the second representation of the annotation corresponding to a second portion of the first representation of the annotation is displayed adjacent to the second portion of the text in the second arrangement.

11. The method according to claim 1, in: the first portion of the text having the first position adjacent to the second portion of the text in the first arrangement; the first portion of the text having the second position not adjacent to the second portion of the text in the second arrangement; a first portion of the second representation of the annotation corresponding to at least a portion of the first representation of the annotation is displayed adjacent to the first portion of the text in the second arrangement; as well as A second instance of the second representation of the annotation corresponding to the at least a portion of the first representation of the annotation is displayed adjacent to the second portion of the text in the second arrangement.

12. The method according to claim 1, in: the first portion of the text having the first position not adjacent to the second portion of the text in the first arrangement; the first portion of the text having the second position adjacent to the second portion of the text in the second arrangement; and The second representation of the annotation includes two separate portions of the first representation of the annotation merged together.

13. The method of claim 1, wherein the first representation of the annotation has a positional relationship with the first portion and the second portion of the text, and the second representation of the annotation maintains the positional relationship with the first portion and the second portion of the text.

14. The method of claim 1, wherein the request specifies a particular font option, wherein the second arrangement of the text is based on the particular font option, and wherein the second representation of the annotation is based on the particular font option. The method of claim 14 , wherein the specific font option is font size. The method of claim 14 , wherein the specific font option is a font type.

17. An electronic device, include: A display unit configured to display a user interface; one or more input units configured to receive input; and a processing unit, the processing unit being coupled to the display unit and the one or more input units, the processing unit being configured to: displaying a first arrangement of text on the display unit, in which a first portion of the text has a first position relative to a second portion of the text; displaying on the display unit a first representation of an annotation associated with the first portion of the text and the second portion of the text; receiving a request to rearrange the text; as well as In response to receiving the request to rearrange the text, concurrently displaying on the display unit: a second arrangement of the text, wherein the first portion of the text has a second position relative to the second portion of the text, wherein the second position is different from the first position; and A second representation of the annotation is associated with both the first portion of the text and the second portion of the text, wherein the second representation of the annotation is different from the first representation of the annotation.

18. The electronic device of claim 17, wherein the first representation of the annotation has a positional relationship with the first portion and the second portion of the text, and the second representation of the annotation maintains the positional relationship with the first portion and the second portion of the text.

19. The electronic device of claim 17, wherein the request specifies a particular font option, wherein the second arrangement of the text is based on the particular font option, and wherein the second representation of the annotation is based on the particular font option.

20. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions which, when executed by an electronic device having a display and an input device, cause the electronic device to perform the method according to any one of claims 1 to 16.