Handwriting input on electronic devices

Abstract

To enhance user interaction including handwriting desired text on a touch screen display of an electronic device by using a stylus or another handwriting device.SOLUTION: An electronic device receives a handwriting input in a text input field, converts the handwriting input into font-based text, selects and deletes text on the basis of an input from a stylus, inserts the text into existing text on the basis of the input from the stylus, manages timing for converting the handwriting input into font-based text, presents a handwriting input menu, controls characteristics of the handwriting input on the basis of selection on the handwriting input menu, presents an automatic completion proposal, converts the handwriting input into the font-based text, and displays options in a content input palette.SELECTED DRAWING: None

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 62 / 843,976, filed May 6, 2019; U.S. Provisional Patent Application No. 62 / 859,413, filed Jun. 10, 2019; and U.S. Provisional Patent Application No. 63 / 020,496, filed May 5, 2020, the contents of which are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0002] This specification generally relates to electronic devices that accept handwritten input and user interaction with such devices.

Background Art

[0003] In recent years, user interaction with electronic devices has increased significantly. These devices can be devices such as computers, tablet computers, televisions, multimedia devices, mobile devices, and the like.

[0004] In some situations, a user may desire to enter text on an electronic device or otherwise interact with the electronic device using a stylus. In some situations, a user may desire to use a stylus or other hand - writing device to handwrite desired text on the touch - screen display of an electronic device. Enhancing these interactions improves the user experience with the device and reduces the time of user interaction, which is particularly important when the input device operates on battery power.

[0005] It should be fully understood that the use of personally identifiable information should be in accordance with generally recognized privacy policies and practices that meet or exceed industry or government requirements for maintaining user privacy. In particular, personally identifiable information data should be managed and handled in a manner that minimizes the risk of unintentional or unauthorized access or use, and the nature of authorized use should be clearly indicated to the user. [Overview of the project]

[0006] Some embodiments described in this disclosure aim to receive handwritten input within a text input field and convert the handwritten input to font-based text. Some embodiments described in this disclosure aim to select and delete text using a stylus. Some embodiments of this disclosure aim to insert text into existing text using a stylus. Some embodiments of this disclosure aim to control the timing of converting handwritten input to font-based text. Some embodiments of this disclosure aim to present a handwritten input menu on an electronic device. Some embodiments of this disclosure aim to control the characteristics of handwritten input based on selections on the handwritten input menu. Some embodiments of this disclosure aim to present auto-completion suggestions. Some embodiments of this disclosure aim to convert handwritten input to font-based text. Some embodiments of this disclosure aim to display options within a content input palette. [Brief explanation of the drawing]

[0007] For a better understanding of the various embodiments described, please refer to the following “Modes for Carrying Out the Invention” in conjunction with the following drawings, where similar reference numbers refer to corresponding parts throughout those drawings.

[0008] [Figure 1A]This is a block diagram showing a portable multifunctional device with a touch-sensitive display according to several embodiments.

[0009] [Figure 1B] This is a block diagram showing exemplary components for event handling according to several embodiments.

[0010] [Figure 2] This figure shows a portable multifunctional device having a touchscreen according to several embodiments.

[0011] [Figure 3] This is a block diagram of an exemplary multifunctional device comprising a display and a touch-sensitive surface, according to several embodiments.

[0012] [Figure 4A] This figure shows an exemplary user interface for an application menu on a portable multifunction device, according to several embodiments.

[0013] [Figure 4B] This document illustrates an exemplary user interface for a multifunctional device having a touch-sensitive surface separate from the display, according to several embodiments.

[0014] [Figure 5A] This figure shows a personal electronic device according to several embodiments.

[0015] [Figure 5B] This is a block diagram showing a personal electronic device according to several embodiments.

[0016] [Figure 5C] This figure shows exemplary components of a personal electronic device having a touch-sensitive display and an intensity sensor, according to several embodiments. [Figure 5D] A diagram showing exemplary components of a personal electronic device having a touch sensing display and an intensity sensor, according to some embodiments.

[0017] [Figure 5E] Exemplary components and user interfaces of a personal electronic device, according to some embodiments, are shown. [Figure 5F] Exemplary components and user interfaces of a personal electronic device, according to some embodiments, are shown. [Figure 5G] Exemplary components and user interfaces of a personal electronic device, according to some embodiments, are shown. [Figure 5H] Exemplary components and user interfaces of a personal electronic device, according to some embodiments, are shown.

[0018] [Figure 5I] A block diagram of an exemplary architecture of a device according to some embodiments of the present disclosure is shown.

[0019] [Figure 6A] An exemplary method by which an electronic device converts handwritten input to font-based text, according to some embodiments, is shown. [Figure 6B] An exemplary method by which an electronic device converts handwritten input to font-based text, according to some embodiments, is shown. [Figure 6C] An exemplary method by which an electronic device converts handwritten input to font-based text, according to some embodiments, is shown. [Figure 6D] An exemplary method by which an electronic device converts handwritten input to font-based text, according to some embodiments, is shown. [Figure 6E] An exemplary method by which an electronic device converts handwritten input to font-based text, according to some embodiments, is shown. [Figure 6F]This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6G] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6H] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6I] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6J] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6K] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6L] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6M] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6N] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6O] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6P] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6Q] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6R] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6S] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6T] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6U] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6V] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6W] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6X] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6Y] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6Z] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6AA] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6BB] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6CC] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6DD] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6EE]This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6FF] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6GG] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6HH] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6II] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6JJ] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6KK] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6LL] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6MM] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6NN] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 600] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6PP] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6QQ] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6RR] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6SS] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6TT] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6UU] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6VV] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6WW] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6XX] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 6YY] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text.

[0020] [Figure 7A] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 7B] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 7C] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 7D] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 7E]This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 7F] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 7G] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 7H] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 7I] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text.

[0021] [Figure 8A] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8B] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8C] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8D] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8E] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8F] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8G] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8H]This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8I] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8J] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8K] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8L] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8M] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8N] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8O] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8P] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8Q] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8R] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8S] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8T] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8U] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8V] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8W] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8X] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8Y] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8Z] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8AA] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8BB] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8CC] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8DD] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8EE] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8FF] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8GG]This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8HH] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8II] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8JJ] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8KK] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8LL] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text. [Figure 8MM] This document illustrates, in several embodiments, how an electronic device interprets handwritten input to select or delete text.

[0022] [Figure 9A] This flowchart illustrates a method for interpreting handwritten input to select or delete text, according to several embodiments. [Figure 9B] This flowchart illustrates a method for interpreting handwritten input to select or delete text, according to several embodiments. [Figure 9C] This flowchart illustrates a method for interpreting handwritten input to select or delete text, according to several embodiments. [Figure 9D] This flowchart illustrates a method for interpreting handwritten input to select or delete text, according to several embodiments. [Figure 9E] This flowchart illustrates a method for interpreting handwritten input to select or delete text, according to several embodiments. [Figure 9F]This flowchart illustrates a method for interpreting handwritten input to select or delete text, according to several embodiments. [Figure 9G] This flowchart illustrates a method for interpreting handwritten input to select or delete text, according to several embodiments.

[0023] [Figure 10A] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10B] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10C] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10D] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10E] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10F] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10G] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10H] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10I] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10J] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10K] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10L] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10M] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10N] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10O] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10P] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10Q] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10R] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10S] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10T] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10U] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10V] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10W] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10X] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10Y]This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10Z] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10AA] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10BB] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10CC] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10DD] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10EE] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10FF] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10GG] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10HH] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10II] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10JJ] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10KK] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10LL]This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10MM] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10NN] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 1000] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10PP] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10QQ] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10RR] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10SS] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10TT] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10UU] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10VV] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10WW] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10XX] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10YY]This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10ZZ] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10AAA] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10BBB] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10CCC] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10DDD] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10EEE] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10FFF] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10GGG] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10HHH] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10III] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10JJJ] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10KKK] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10LLL]This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10MMM] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10 NNN] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 1000] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10PPP] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10QQQ] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10RRR] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text. [Figure 10SSS] This document illustrates, through several embodiments, how an electronic device inserts handwritten input into existing text.

[0024] [Figure 11A] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11B] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11C] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11D] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11E] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11F]This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11G] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11H] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11I] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11J] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11K] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11L] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text. [Figure 11M] This flowchart illustrates several embodiments of methods for inserting handwritten input into existing text.

[0025] [Figure 12A] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12B] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12C] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12D] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12E]This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12F] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12G] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12H] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12I] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12J] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12K] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12L] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12M] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12N] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12O] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12P]This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12Q] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12R] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12S] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12T] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12U] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12V] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12W] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12X] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12Y] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12Z] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12AA]This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12BB] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12CC] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12DD] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12EE] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12FF] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12GG] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12HH] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12II] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12JJ] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12KK] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12LL]This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12MM] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12NN] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 1200] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12PP] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12QQ] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12RR] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text. [Figure 12SS] This document illustrates, through several embodiments, how an electronic device can manage the timing of converting handwritten text to font-based text.

[0026] [Figure 13A] This flowchart illustrates a method for controlling the timing of converting handwritten text to font-based text, according to several embodiments. [Figure 13B] This flowchart illustrates a method for controlling the timing of converting handwritten text to font-based text, according to several embodiments. [Figure 13C] This flowchart illustrates a method for controlling the timing of converting handwritten text to font-based text, according to several embodiments. [Figure 13D] This flowchart illustrates a method for controlling the timing of converting handwritten text to font-based text, according to several embodiments. [Figure 13E] This flowchart illustrates a method for controlling the timing of converting handwritten text to font-based text, according to several embodiments. [Figure 13F] This flowchart illustrates a method for controlling the timing of converting handwritten text to font-based text, according to several embodiments. [Figure 13G] This flowchart illustrates a method for controlling the timing of converting handwritten text to font-based text, according to several embodiments.

[0027] [Figure 14A] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14B] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14C] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14D] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14E] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14F] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14G] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14H] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14I]This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14J] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14K] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14L] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14M] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14N] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14O] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14P] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14Q] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14R] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14S] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14T] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14U] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu. [Figure 14V] This document illustrates, according to several embodiments, how an electronic device presents a handwriting input menu.

[0028] [Figure 15A] This flowchart illustrates a method for presenting a handwriting input menu according to several embodiments. [Figure 15B] This flowchart illustrates a method for presenting a handwriting input menu according to several embodiments. [Figure 15C] This flowchart illustrates a method for presenting a handwriting input menu according to several embodiments. [Figure 15D] This flowchart illustrates a method for presenting a handwriting input menu according to several embodiments. [Figure 15E] This flowchart illustrates a method for presenting a handwriting input menu according to several embodiments. [Figure 15F] This flowchart illustrates a method for presenting a handwriting input menu according to several embodiments.

[0029] [Figure 16A] This flowchart illustrates a method for controlling the characteristics of handwriting input based on selections made on a handwriting input menu, according to several embodiments. [Figure 16B] This flowchart illustrates a method for controlling the characteristics of handwriting input based on selections made on a handwriting input menu, according to several embodiments. [Figure 16C] This flowchart illustrates a method for controlling the characteristics of handwriting input based on selections made on a handwriting input menu, according to several embodiments. [Figure 16D] This flowchart illustrates a method for controlling the characteristics of handwriting input based on selections made on a handwriting input menu, according to several embodiments.

[0030] [Figure 17A] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17B] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17C] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17D] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17E] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17F] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17G] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17H] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17I] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17J] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17K] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17L] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17M] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17N] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17O] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17P] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17Q] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17R]This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17S] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17T] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17U] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17V] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments. [Figure 17W] This document describes exemplary methods by which an electronic device presents automatic completion suggestions, based on several embodiments.

[0031] [Figure 18A] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments. [Figure 18B] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments. [Figure 18C] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments. [Figure 18D] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments. [Figure 18E] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments. [Figure 18F] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments. [Figure 18G] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments. [Figure 18H] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments. [Figure 18I] This flowchart illustrates a method for presenting auto-completion suggestions according to several embodiments.

[0032] [Figure 19A] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19B] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19C] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19D] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19E] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19F] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19G] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19H] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19I] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19J] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19K] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19L] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19M]This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19N] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19O] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19P] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19Q] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19R] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19S] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19T] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19U] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19V] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19W] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19X] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19Y] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19Z] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19AA] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text. [Figure 19BB] This document illustrates, through several embodiments, how an electronic device converts handwritten input into font-based text.

[0033] [Figure 20A] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 20B] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 20C] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text. [Figure 20D] This flowchart illustrates several embodiments of methods for converting handwritten input to font-based text.

[0034] [Figure 21A] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21B] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21C] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21D] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21E] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21F] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21G] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21H] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21I] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21J] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21K] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21L] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21M] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21N] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21O] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21P] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21Q] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21R]This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21S] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21T] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21U] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21V] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21W] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21X] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21Y] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21Z] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21AA] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21BB] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21CC] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette. [Figure 21DD] This document illustrates, through several embodiments, how an electronic device displays options in a content input palette.

[0035] [Figure 22A] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22B] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22C] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22D] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22E] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22F] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22G] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22H] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22I] A flowchart showing a method of displaying options on a content input palette according to some embodiments. [Figure 22J] A flowchart showing a method of displaying options on a content input palette according to some embodiments.

Embodiments for Carrying Out the Invention

[0036] The following description describes exemplary methods, parameters, etc. However, it should be recognized that such descriptions are not intended as limitations on the scope of the present disclosure, but rather as descriptions of exemplary embodiments.

[0037] There is a need for an electronic device that provides an efficient method for receiving and interpreting handwritten input (e.g., from a stylus or other handwriting input device). Such technology can reduce the cognitive burden on the user of such device. Furthermore, such technology can reduce the power consumption of the processor and battery that would normally be wasted on redundant user input.

[0038] In the following description, terms such as “first,” “second,” etc., are used to describe various elements, but these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the various embodiments described, the first touch may be called the second touch, and similarly, the second touch may be called the first touch. Both the first touch and the second touch are touches, but they are not the same touch.

[0039] The terminology used in the descriptions of the various embodiments described herein is intended solely to describe specific embodiments and is not intended to be limiting. In the descriptions of the various embodiments and the accompanying claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless otherwise explicitly stated in the context. Furthermore, it should be understood that, as used herein, the term “and / or” refers to and includes any and all possible combinations of one or more of the enumerated items relating to the invention. It will be further understood that, as used herein, the terms “includes,” “comprises,” and / or “comprising,” specify the presence of the described features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0040] The term "if" can be interpreted, at will, depending on the context, as "when" or "upon," or "in response to determining" or "in response to detecting." Similarly, the phrases "if it is determined" or "if [a stated condition or event] is detected" can be interpreted, at will, depending on the context, as "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0041] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communication device, such as a mobile phone, which also includes other functions such as PDA functionality and / or music player functionality. Exemplary embodiments of portable multifunction devices include, but are not limited to, the iPhone®, iPod Touch®, and iPad® devices from Apple Inc., Cupertino, California. Optionally, other portable electronic devices such as laptops or tablet computers having touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads) are also used. It should also be understood that in some embodiments, the device is not a portable communication device but a desktop computer having touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0042] The following discussion describes electronic devices including displays and touch-sensitive surfaces. However, it should be understood that electronic devices optionally include one or more other physical user interface devices such as physical keyboards, mice, and / or joysticks.

[0043] The device typically supports a variety of applications, including drawing applications, presentation applications, word processing applications, website creation applications, disk authoring applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and / or digital video player applications.

[0044] Various applications running on this device optionally utilize at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the device, are optionally adjusted and / or modified on an application-by-application basis and / or within each application. In this way, the device's common physical architecture (such as the touch-sensitive surface) optionally supports a variety of applications with intuitive and transparent user interfaces for the user.

[0045] Here, we turn our attention to embodiments of portable devices having a touch-sensitive display. Figure 1A is a block diagram of a portable multifunction device 100 having a touch-sensitive display system 112 according to several embodiments. The touch-sensitive display 112 may be conveniently referred to as a “touchscreen” and may be known or referred to as a “touch-sensitive display system”. Device 100 includes a memory 102 (optionally including one or more computer-readable storage media), a memory controller 122, one or more processing units (CPUs) 120, a peripheral interface 118, an RF circuit 108, an audio circuit 110, a speaker 111, a microphone 113, an input / output (I / O) subsystem 106, other input control devices 116, and an external port 124. Device 100 optionally includes one or more optical sensors 164. Device 100 optionally includes one or more contact intensity sensors 165 (e.g., touch-sensitive surfaces such as the touch-sensitive display system 112 of Device 100) that detect the intensity of contact on Device 100. Device 100 optionally includes one or more tactile output generators 167 that generate tactile outputs on Device 100 (for example, on touch-sensitive surfaces such as the touch-sensitive display system 112 of Device 100 or the touchpad 355 of Device 300). These components optionally communicate via one or more communication buses or signal lines 103.

[0046] As used herein and in the claims, the term “strength” of contact on a touch-sensitive surface refers to the force or pressure (force per unit area) of contact on the touch-sensitive surface (e.g., finger contact), or a proxy for the force or pressure of contact on the touch-sensitive surface. The strength of contact has a range of values, including at least four distinct values, and more typically, including several hundred (e.g., at least 256) distinct values. The strength of contact is optionally determined (or measured) using various methods and various sensors or combinations of sensors. For example, one or more force sensors below or adjacent to the touch-sensitive surface are optionally used to measure the force at various points on the touch-sensitive surface. In some implementations, force measurements from multiple force sensors are combined (e.g., weighted averaged) to determine an estimated force of contact. Similarly, the pressure-sensitive tip of a stylus is optionally used to determine the pressure of the stylus on the touch-sensitive surface. Alternatively, the size and / or variation of the contact area detected on the touch-sensing surface, the capacitance and / or variation of the touch-sensing surface adjacent to the contact, and / or the resistance and / or variation of the touch-sensing surface adjacent to the contact may optionally be used as a substitute for the force or pressure of the contact on the touch-sensing surface. In some implementations, the substitute measurement of the contact force or pressure is used directly to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is described in units corresponding to the substitute measurement). In some implementations, the substitute measurement of the contact force or pressure is converted into an estimate of the force or pressure, which is then used to determine whether it exceeds an intensity threshold (e.g., the intensity threshold is a pressure threshold measured in units of pressure). By using the intensity of contact as an attribute of user input, it becomes possible for users to access additional device functions that might otherwise be inaccessible (e.g., on a touch-sensitive display) and / or receive user input (e.g., via a touch-sensitive display, touch-sensitive surface, or physical / mechanical control such as a knob or button) on reduced-size devices where the implementation area for displaying affordances is limited.

[0047] As used herein and in the claims, the term “tactile output” refers to the physical displacement of a device relative to its previous position, the physical displacement of a component of a device (e.g., a touch-sensitive surface) relative to another component of the device (e.g., a housing), or the displacement of a component relative to the center of mass of a device, which will be detected by the user through the user’s sense of touch. For example, in a situation where a device or component of a device is in contact with a touch-sensitive user’s surface (e.g., the user’s fingers, palm, or other part of their hand), the tactile output generated by the physical displacement will be interpreted by the user as a tactile sensation corresponding to a perceived change in the physical properties of the device or component of the device. For example, the movement of a touch-sensitive surface (e.g., a touch-sensitive display or trackpad) may be optionally interpreted by the user as a “down-click” or “up-click” of a physical actuator button. In some cases, the user may feel a tactile sensation such as a “down-click” or “up-click” even when there is no movement of a physical actuator button associated with a touch-sensitive surface that has been physically pressed (e.g., displaced) by the user’s action. As another example, movement of a touch-sensitive surface may be interpreted or perceived by the user as "roughness" of that surface, even if there is no change in the smoothness of the touch-sensitive surface. Such user interpretations of touch depend on the user's personal sensory perception, but there are many touch sensory perceptions common to the majority of users. Therefore, when a tactile output is described as corresponding to a user's specific sensory perception (e.g., "up-click," "down-click," "roughness"), unless otherwise stated, the generated tactile output corresponds to the physical displacement of the device or its components that produce the described sensory perception of a typical (or average) user.

[0048] It should be understood that device 100 is merely an example of a portable multifunction device, and that device 100 may optionally have more or fewer components than those shown, may optionally be a combination of two or more components, or may optionally have different configurations or arrangements of those components. The various components shown in Figure 1A are implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application-specific integrated circuits.

[0049] Memory 102 optionally includes high-speed random-access memory and optionally includes non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Memory controller 122 optionally controls access to memory 102 by other components of device 100.

[0050] The peripheral interface 118 can be used to connect the device's input and output peripherals to the CPU 120 and memory 102. One or more processors 120 operate or execute various software programs and / or instruction sets stored in memory 102 to perform various functions for device 100 and process data. In some embodiments, the peripheral interface 118, CPU 120, and memory controller 122 are optionally implemented on a single chip, such as chip 104. In some other embodiments, they are optionally implemented on separate chips.

[0051] The RF (radio frequency) circuit 108 transmits and receives RF signals, also known as electromagnetic signals. The RF circuit 108 converts electrical signals to electromagnetic signals or electromagnetic signals to electrical signals and communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 108 optionally includes well-known circuits for performing these functions, which include, but are not limited to, antenna systems, RF transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, CODEC chipsets, subscriber identity module (SIM) cards, and memory. The RF circuit 108 optionally communicates wirelessly with networks such as the Internet, also known as the World Wide Web (WWW), intranets, and / or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs), as well as with other devices. The RF circuit 108 optionally includes a well-known circuit for detecting a near-field communication (NFC) field using a short-range communication radio. Wireless communication is not limited to this, but optionally includes Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), high-speed downlink packet access (HSDPA), high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO), HSPA, HSPA+, Dual-Cell HSPA (DC-HSPADA), and long-term evolution.Evolution (LTE), Near Field Communication (NFC), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or IEEE 802.11ac), Voice over Internet Protocol (VoIP), Wi-MAX, Email protocols (e.g., Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP)), Instant messaging (e.g., Extensible Messaging and Presence Protocol) Using any of several communication standards, protocols, and technologies, including the XMPP protocol, the Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), the Instant Messaging and Presence Service (IMPS), and / or the Short Message Service (SMS), or any other suitable communication protocol, including a communication protocol not yet developed as of the filing date of this specification.

[0052] The audio circuit 110, speaker 111, and microphone 113 provide an audio interface between the user and the device 100. The audio circuit 110 receives audio data from the peripheral interface 118, converts this audio data into an electrical signal, and transmits this electrical signal to the speaker 111. The speaker 111 converts the electrical signal into human audible sound waves. The audio circuit 110 also receives the electrical signal converted from the sound waves by the microphone 113. The audio circuit 110 converts the electrical signal into audio data and transmits this audio data to the peripheral interface 118 for processing. The audio data is optionally retrieved from and / or transmitted to the memory 102 and / or RF circuit 108 by the peripheral interface 118. In some embodiments, the audio circuit 110 also includes a headset jack (e.g., 212 in Figure 2). The headset jack provides an interface between the audio circuit 110 and detachable audio input / output peripherals such as output-only headphones or headsets that have both output (e.g., headphones for one or both ears) and input (e.g., a microphone).

[0053] The I / O subsystem 106 connects input / output peripherals on device 100, such as the touchscreen 112 and other input control devices 116, to the peripheral interface 118. The I / O subsystem 106 optionally includes a display controller 156, an optical sensor controller 158, an intensity sensor controller 159, a haptic feedback controller 161, and one or more input controllers 160 for other input or control devices. One or more input controllers 160 receive electrical signals from / transmit electrical signals to other input control devices 116. The other input control devices 116 optionally include physical buttons (e.g., push buttons, rocker buttons), dials, slider switches, joysticks, click wheels, etc. In some alternative embodiments, the input controller(s) 160 are optionally connected to (or not connected to) one or more of the following: a keyboard, an infrared port, a USB port, and a pointer device such as a mouse. One or more buttons (e.g., 208 in Figure 2) optionally include up / down buttons for volume control of speaker 111 and / or microphone 113. One or more buttons optionally include push buttons (e.g., 206 in Figure 2).

[0054] As described in U.S. Patent Application No. 11 / 322,549, filed December 23, 2005, "Unlocking a Device by Performing Gestures on an Unlock Image," U.S. Patent No. 7,657,849, which is hereby incorporated by reference in its entirety, a quick press of a push button optionally unlocks the touch screen 112 or, optionally, initiates a process of unlocking the device using gestures on the touch screen. A longer press of a push button (e.g., 206) optionally turns the power to the device 100 on or off. The functionality of one or more of the buttons is optionally customizable by the user. The touch screen 112 is used to implement virtual or soft buttons and one or more soft keyboards.

[0055] The touch-sensitive display 112 provides an input interface and an output interface between the device and the user. The display controller 156 receives electrical signals from and / or transmits electrical signals to the touch screen 112. The touch screen 112 displays a visual output to the user. This visual output optionally includes graphics, text, icons, video, and any combination thereof (collectively referred to as "graphics"). In some embodiments, some or all of the visual output optionally corresponds to user interface objects.

[0056] The touchscreen 112 has a touch-sensing surface, sensor, or set of sensors that accept user input based on touch and / or tactile contact. The touchscreen 112 and the display controller 156 (together with any associated modules and / or instruction sets in memory 102) detect contact (and any movement or interruption of contact) on the touchscreen 112 and translate the detected contact into interaction with user interface objects displayed on the touchscreen 112 (e.g., one or more soft keys, icons, web pages, or images). In an exemplary embodiment, the point of contact between the touchscreen 112 and the user corresponds to the user's finger.

[0057] The touchscreen 112 optionally uses LCD (liquid crystal display) technology, LPD (polymer light-emitting display) technology, or LED (light-emitting diode) technology, but other display technologies may also be used in other embodiments. The touchscreen 112 and the display controller 156 optionally, but not limited to, use any of a number of currently known or future-developed touch sensing technologies, including capacitive, resistive, infrared, and surface acoustic technologies, as well as other proximity sensor arrays or other elements that determine one or more points of contact with the touchscreen 112, to detect contact and any movement or interruption thereof. In exemplary embodiments, projected mutual capacitive sensing technology is used, such as that found in the iPhone® and iPod Touch® from Apple Inc. of Cupertino, California.

[0058] The touch-sensitive displays in some embodiments of the touchscreen 112 are optionally similar to the multi-touch-sensitive touchpads described in U.S. Patent No. 6,323,846 (Westerman et al.), No. 6,570,557 (Westerman et al.), and / or No. 6,677,932 (Westerman), and / or U.S. Patent Publication No. 2002 / 0015024(A1), which are each incorporated herein in whole by reference. However, the touchscreen 112 displays visual output from device 100, whereas the touch-sensitive touchpad does not provide visual output.

[0059] Touch-sensitive displays in some embodiments of the touchscreen 112 are described in the following application. (1) U.S. Patent Application No. 11 / 381,313, filed May 2, 2006, "Multipoint Touch Surface Controller", (2) U.S. Patent Application No. 10 / 840,862, filed May 6, 2004, "Multipoint Touchscreen", (3) U.S. Patent Application No. 10 / 903,964, filed July 30, 2004, "Gestures For Touch Sensitive Input Devices", (4) U.S. Patent Application No. 11 / 048,264, filed January 31, 2005, "Gestures For Touch Sensitive Input Devices", (5) U.S. Patent Application No. 11 / 038,590, filed January 18, 2005, "Mode-Based Graphical User Interfaces For Touch Sensitive Input (6) U.S. Patent Application No. 11 / 228,758, filed September 16, 2005, "Virtual Input Device Placement On A Touch Screen User Interface", (7) U.S. Patent Application No. 11 / 228,700, filed September 16, 2005, "Operation Of A Computer With A Touch Screen Interface", (8) U.S. Patent Application No. 11 / 228,737, filed September 16, 2005, "Activating Virtual Keys Of A Touch-Screen Virtual Keyboard", and (9) U.S. Patent Application No. 11 / 367,749, filed March 3, 2006, "Multi-Functional Hand-Held Device". All of these applications are incorporated herein by reference in their entirety.

[0060] The touchscreen 112 optionally has a video resolution greater than 100 dpi. In some embodiments, the touchscreen has a video resolution of approximately 160 dpi. The user optionally touches the touchscreen 112 using any suitable object or attachment such as a stylus or finger. In some embodiments, the user interface is designed to operate primarily using finger-based touch and gestures, which may be less precise than stylus-based input due to the larger contact area of ​​the finger on the touchscreen. In some embodiments, the device translates coarse finger input into a precise pointer / cursor position or command to perform an action desired by the user.

[0061] In some embodiments, in addition to the touchscreen, the device 100 optionally includes a touchpad (not shown) for activating or deactivating specific functions. In some embodiments, the touchpad is a touch-sensitive area of ​​the device that, unlike the touchscreen, does not display a visual output. The touchpad is optionally a touch-sensitive surface separate from the touchscreen 112 or an extension of the touch-sensitive surface formed by the touchscreen.

[0062] Device 100 also includes a power system 162 that supplies power to various components. The power system 162 optionally includes a power management system, one or more power sources (e.g., a battery, alternating current (AC)), a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator (e.g., a light-emitting diode (LED)), and any other components associated with generating, managing, and distributing power within the portable device.

[0063] The device 100 also optionally includes one or more optical sensors 164. Figure 1A shows optical sensors coupled to an optical sensor controller 158 in the I / O subsystem 106. The optical sensors 164 optionally include a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) phototransistor. The optical sensors 164 receive light from the environment projected through one or more lenses and convert that light into data representing an image. The optical sensors 164 work in conjunction with an imaging module 143 (also called a camera module) to optionally capture still images or video. In some embodiments, the optical sensors are located on the back of the device 100, opposite the touchscreen display 112 which is on the front of the device, so that the touchscreen display can be used as a viewfinder for acquiring still images and / or video. In some embodiments, the optical sensors are located on the front of the device so that the user's image is optionally acquired for video conferencing while the user is viewing other video conference participants on the touchscreen display. In some embodiments, the position of the light sensor 164 can be changed by the user (for example, by rotating the lens and sensor within the device housing), so that a single light sensor 164 can be used for both video conferencing and acquiring still images and / or videos, together with the touchscreen display.

[0064] Device 100 also optionally includes one or more contact intensity sensors 165. Figure 1A shows a contact intensity sensor coupled to an intensity sensor controller 159 in the I / O subsystem 106. The contact intensity sensor 165 optionally includes one or more piezoresistive strain gauges, capacitive force sensors, electric force sensors, pressure-power sensors, optical force sensors, capacitive touch-sensing surfaces, or other intensity sensors (e.g., sensors used to measure the force (or pressure) of contact on a touch-sensing surface). The contact intensity sensor 165 receives contact intensity information (e.g., pressure information, or a proxy for pressure information) from the environment. In some embodiments, at least one contact intensity sensor is positioned juxtaposed with or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112). In some embodiments, at least one contact intensity sensor is positioned on the back of Device 100, opposite the touchscreen display 112 located on the front of Device 100.

[0065] The device 100 also optionally includes one or more proximity sensors 166. Figure 1A shows a proximity sensor 166 coupled to a peripheral interface 118. Alternatively, the proximity sensor 166 is optionally coupled to an input controller 160 in an I / O subsystem 106. The proximity sensor 166 optionally functions as described in U.S. Patent Applications 11 / 241,839, “Proximity Detector In Handheld Device”, 11 / 240,788, “Proximity Detector In Handheld Device”, 11 / 620,702, “Using Ambient Light Sensor To Augment Proximity Sensor Output”, 11 / 586,862, “Automated Response To And Sensing Of User Activity In Portable Devices”, and 11 / 638,251, “Methods And Systems For Automatic Configuration Of Peripherals”, which are all incorporated herein by reference. In some embodiments, if the multifunction device is placed near the user's ear (for example, when the user is making a phone call), the proximity sensor turns off and disables the touchscreen 112.

[0066] Device 100 also optionally includes one or more tactile output generators 167. Figure 1A shows a tactile output generator coupled to a tactile feedback controller 161 in the I / O subsystem 106. The tactile output generator 167 optionally includes one or more electroacoustic devices such as a speaker or other audio component, and / or electromechanical devices that convert energy into linear motion, such as a motor, solenoid, electroactive polymer, piezoelectric actuator, electrostatic actuator, or other tactile output generating component (e.g., a component that converts an electrical signal into a tactile output on the device). The contact intensity sensor 165 receives a tactile feedback generation command from the tactile feedback module 133 and generates a tactile output on device 100 that can be sensed by the user of device 100. In some embodiments, at least one tactile output generator is positioned alongside or adjacent to a touch-sensing surface (e.g., a touch-sensing display system 112) and optionally generates a tactile output by moving the touch-sensing surface vertically (e.g., inward / outward from the surface of device 100) or horizontally (e.g., forward / backward in the same plane as the surface of device 100). In some embodiments, at least one tactile output generator sensor is positioned on the back of device 100, opposite the touchscreen display 112 which is positioned on the front of device 100.

[0067] The device 100 also optionally includes one or more accelerometers 168. Figure 1A shows an accelerometer 168 coupled to a peripheral interface 118. Alternatively, the accelerometer 168 is optionally coupled to an input controller 160 in the I / O subsystem 106. The accelerometer 168 optionally functions as described in U.S. Patent Publication No. 20050190059, “Acceleration-based Theft Detection System for Portable Electronic Devices,” and U.S. Patent Publication No. 20060017692, “Methods And Apparatuses For Operating A Portable Device Based On An Accelerometer,” both of which are incorporated herein by reference in their entirety. In some embodiments, information is displayed on a touchscreen display in portrait or landscape orientation based on an analysis of data received from one or more accelerometers. Device 100 optionally includes, in addition to one or more accelerometers 168, a magnetometer (not shown), and a GPS (or GLONASS or other global navigation system) receiver (not shown) for acquiring information regarding the position and orientation of Device 100 (e.g., portrait or landscape).

[0068] In some embodiments, the software components stored in memory 102 include an operating system 126, a communications module (or instruction set) 128, a contact / motion module (or instruction set) 130, a graphics module (or instruction set) 132, a text input module (or instruction set) 134, a Global Positioning System (GPS) module (or instruction set) 135, and an application (or instruction set) 136. Furthermore, in some embodiments, memory 102 (Figure 1A) or 370 (Figure 3) stores a device / global internal state 157, as shown in Figures 1A and 3. The device / global internal state 157 includes one or more of the following: an active application state indicating which application is active, if there is an application currently active; a display state indicating which applications, views, or other information occupy different areas of the touchscreen display 112; a sensor state including information obtained from various sensors and input control devices 116 of the device; and location information relating to the device's position and / or orientation.

[0069] An operating system 126 (for example, an embedded operating system such as Darwin, RTXC, LINUX, UNIX, OS X, iOS, WINDOWS, or VxWorks) includes various software components and / or drivers that control and manage general system tasks (e.g., memory management, storage control, power management, etc.) and facilitate communication between various hardware components and software components.

[0070] The communication module 128 facilitates communication with other devices via one or more external ports 124 and also includes various software components for processing data received by the RF circuit 108 and / or external ports 124. The external ports 124 (e.g., Universal Serial Bus (USB), FireWire, etc.) are adapted to connect to other devices directly or indirectly via a network (e.g., the Internet, Wi-Fi, etc.). In some embodiments, the external ports are multi-pin (e.g., 30-pin) connectors that are the same as and / or compatible with the 30-pin connector used on iPod® (a trademark of Apple Inc.) devices.

[0071] The contact / motion module 130 optionally detects contact with the touchscreen 112 and other touch-sensitive devices (e.g., a touchpad or physical click wheel) (in cooperation with the display controller 156). The contact / motion module 130 includes various software components for performing various contact detection operations, such as determining whether contact has occurred (e.g., detecting a finger down event), determining the intensity of the contact (e.g., the force or pressure of the contact, or a substitute for the force or pressure of the contact), determining whether there is movement of contact and tracking movement across the touch-sensitive surface (e.g., detecting one or more events of a finger dragging), and determining whether contact has stopped (e.g., detecting a finger up event or interruption of contact). The contact / motion module 130 receives contact data from the touch-sensitive surface. Determining the movement of the contact point, represented by a series of contact data, optionally includes determining the speed (magnitude), velocity (magnitude and direction), and / or acceleration (change in magnitude and / or direction) of the contact point. These actions can be optionally applied to a single contact (e.g., a single finger contact) or multiple simultaneous contacts (e.g., "multi-touch" / multiple finger contacts). In some embodiments, the contact / motion module 130 and the display controller 156 detect contact on the touchpad.

[0072] In some embodiments, the contact / motion module 130 uses a set of one or more intensity thresholds to determine whether an action has been performed by a user (for example, to determine whether a user has "clicked" on an icon). In some embodiments, at least one subset of the intensity thresholds is determined according to a software parameter (for example, the intensity thresholds can be adjusted without modifying the physical hardware of device 100, rather than being determined by the activation threshold of a particular physical actuator). For example, the mouse "click" threshold for a trackpad or touchscreen display can be set to one of a range of default thresholds without modifying the trackpad or touchscreen display hardware. In addition, in some implementations, the user of the device is provided with software settings to adjust one or more of the set of intensity thresholds (for example, by adjusting individual intensity thresholds and / or by adjusting multiple intensity thresholds at once using a system-level click "intensity" parameter).

[0073] The contact / motion module 130 optionally detects gesture input from the user. Different gestures on the touch-sensitive surface have different contact patterns (e.g., different motion, timing, and / or intensity of the detected contact). Therefore, gestures are optionally detected by detecting a specific contact pattern. For example, detecting a finger tap gesture involves detecting a finger down event, followed by a finger up (lift-off) event at the same location (or substantially the same location) as the finger down event (e.g., the location of an icon). As another example, detecting a finger swipe gesture on the touch-sensitive surface involves detecting a finger down event, followed by one or more finger drag events, and then a finger up (lift-off) event.

[0074] The graphics module 132 includes various known software components for rendering and displaying graphics on the touchscreen 112 or other display, including components for changing the visual effects of the displayed graphics (e.g., brightness, transparency, saturation, contrast, or other visual properties). In this specification, the term “graphics” includes, but is not limited to, any object that can be displayed to the user, including text, web pages, icons (such as user interface objects including soft keys), digital images, videos, animations, etc.

[0075] In some embodiments, the graphics module 132 stores data representing the graphics to be used. Each graphic is optionally assigned a corresponding code. The graphics module 132 receives one or more codes from an application or the like, as needed, along with coordinate data and other graphic characteristic data, specifying the graphics to be displayed, and then generates screen image data to be output to the display controller 156.

[0076] The haptic feedback module 133 includes various software components for generating commands used by the haptic output generator 167, which generates haptic outputs at one or more locations on the device 100 in response to the user's interaction with the device 100.

[0077] The text input module 134 is optionally a component of the graphics module 132 and provides a soft keyboard for entering text in various applications (e.g., contacts 137, email 140, IM 141, browser 147, and any other applications that require text input).

[0078] The GPS module 135 determines the location of the device and provides this information for use in various applications (for example, to the telephone 138 for use in location-based dialing, to the camera 143 as metadata for photos / videos, and to applications that provide location-based services such as weather widgets, local occupational phonebook widgets, and map / navigation widgets).

[0079] Application 136 optionally includes the following modules (or instruction sets) or subsets or supersets thereof: ● Contact module 137 (sometimes called the address book or contact list), ● Telephone module 138, ● Video conferencing module 139, ● Email client module 140, ● Instant messaging (IM) module 141, ● Training support module 142, ● Camera module 143 for still images and / or video, ● Image management module 144, ● Video player module, ● Music player module, ● Browser module 147, ● Calendar module 148, ● A widget module 149 that optionally includes one or more of the following: weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, dictionary widget 149-5, and other widgets obtained by the user, as well as user-created widgets 149-6. ● Widget creator module 150 for creating user-created widget 149-6, ● Search module 151, ● A video and music player module 152 that integrates a video player module and a music player module. ● Memo Module 153, ● Map module 154, and / or, ● Online video module 155.

[0080] Examples of other applications 136 that may be optionally stored in memory 102 include other word processing applications, other image editing applications, drawing applications, presentation applications, Java-enabled applications, encryption, digital rights management, speech recognition, and speech duplication.

[0081] The contact module 137 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to optionally manage an address book or contact list (for example, by storing it in the application internal state 192 of the contact module 137 in memory 102 or memory 370). Management by the contact module 137 includes: This includes adding names to an address book, removing names (single or multiple) from an address book, associating names with telephone numbers (single or multiple), email addresses (single or multiple), physical addresses (single or multiple), or other information, associating images with names, categorizing and sorting names, and providing telephone numbers or email addresses to initiate and / or facilitate communication by telephone 138, video conferencing module 139, email 140, or IM 141.

[0082] The telephone module 138 works in conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to optionally input character sequences corresponding to telephone numbers, access one or more telephone numbers in the contact module 137, modify entered telephone numbers, dial each telephone number, make a call, and disconnect and terminate a call at the end of the call. As previously mentioned, wireless communication may optionally use any of several communication standards, protocols, and technologies.

[0083] The video conferencing module 139 works in conjunction with the RF circuit 108, audio circuit 110, speaker 111, microphone 113, touchscreen 112, display controller 156, light sensor 164, light sensor controller 158, contact / motion module 130, graphics module 132, text input module 134, contact module 137, and telephone module 138 to include executable commands for starting, running, and ending video conferences between the user and one or more other participants in accordance with the user's commands.

[0084] The email client module 140, in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, includes executable commands for creating, sending, receiving, and managing emails in response to user commands. In conjunction with the image management module 144, the email client module 140 makes it extremely easy to create and send emails containing still or video images captured by the camera module 143.

[0085] The instant messaging module 141, in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, includes executable commands for inputting character sequences corresponding to instant messages, modifying previously entered characters, sending each instant message (e.g., using the Short Message Service (SMS) or Multimedia Message Service (MMS) protocol for telephone-based instant messaging, or XMPP, SIMPLE, or IMPS for internet-based instant messaging), receiving instant messages, and viewing received instant messages. In some embodiments, the transmitted and / or received instant messages optionally include graphics, photographs, audio files, video files, and / or other attachments, such as those supported by MMS and / or Enhanced Messaging Service (EMS). In this specification, “instant messaging” refers to both telephone-based messaging (e.g., messages sent using SMS or MMS) and internet-based messaging (e.g., messages sent using XMPP, SIMPLE, or IMPS).

[0086] The training support module 142 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, map module 154, and music player module to create training (e.g., with time, distance, and / or calorie burn goals), communicate with training sensors (sports devices), receive training sensor data, calibrate sensors used to monitor training, select and play music for training, and includes executable commands for displaying, storing, and transmitting training data.

[0087] The camera module 143 works in conjunction with the touchscreen 112, display controller 156, light sensor 164, light sensor controller 158, contact / motion module 130, graphics module 132, and image management module 144 to include executable commands for capturing still images or videos (including video streams) and storing them in memory 102, modifying the characteristics of still images or videos, or deleting still images or videos from memory 102.

[0088] The image management module 144 works in conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, text input module 134, and camera module 143 to include executable commands for arranging, modifying (e.g., editing), or otherwise manipulating still images and / or videos, labeling, deleting, presenting (e.g., in a digital slideshow or album), and storing them.

[0089] The browser module 147, in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134, includes executable commands for browsing the internet according to user commands, including searching, linking, receiving, and displaying web pages or portions thereof, as well as attachments and other files linked to web pages.

[0090] The calendar module 148 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, email client module 140, and browser module 147 to include executable commands for creating, displaying, modifying, and storing a calendar and data associated with the calendar (e.g., calendar items, to-do lists, etc.) in accordance with user commands.

[0091] The widget module 149 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147 to optionally download and use mini-applications (e.g., weather widget 149-1, stock widget 149-2, calculator widget 149-3, alarm clock widget 149-4, and dictionary widget 149-5) or mini-applications created by the user (e.g., user-created widget 149-6). In some embodiments, the widget includes an HTML (Hypertext Markup Language) file, a CSS (Cascading Style Sheets) file, and a JavaScript file. In some embodiments, the widget includes an XML (Extensible Markup Language) file and a JavaScript file (e.g., Yahoo! widget).

[0092] The widget creator module 150 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, and browser module 147 to be used by the user to optionally create widgets (for example, to turn a user-specified portion of a web page into a widget).

[0093] The search module 151 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to include executable commands for searching for text, music, sounds, images, videos, and / or other files in memory 102 that match one or more search criteria (e.g., one or more user-specified search terms) according to user commands.

[0094] The video and music player module 152 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, and browser module 147 to include executable commands that allow the user to download and play recorded music and other sound files stored in one or more file formats such as MP3 or AAC files, as well as executable commands for displaying, presenting, or otherwise playing videos (for example, on the touchscreen 112 or on an external display connected via the external port 124). In some embodiments, the device 100 optionally includes the functionality of an MP3 player such as an iPod (a trademark of Apple Inc.).

[0095] The memo module 153 works in conjunction with the touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, and text input module 134 to include executable commands for creating and managing memos, to-do lists, etc., according to user commands.

[0096] The map module 154 works in conjunction with the RF circuit 108, touchscreen 112, display controller 156, contact / motion module 130, graphics module 132, text input module 134, GPS module 135, and browser module 147 to optionally receive, display, modify, and store maps and map-related data (e.g., driving directions, data on shops and other points of interest in or near a specific location, and other location-based data) in accordance with user commands.

[0097] The online video module 155, in conjunction with the touchscreen 112, display controller 156, touch / motion module 130, graphics module 132, audio circuit 110, speaker 111, RF circuit 108, text input module 134, email client module 140, and browser module 147, includes instructions that enable the user to access a specific online video, browse a specific online video, receive it (e.g., by streaming and / or downloading), play it (e.g., on the touchscreen or on an external display connected via external port 124), send an email with a link to a specific online video, and perform other management of online videos in one or more file formats such as H.264. In some embodiments, an instant messaging module 141 is used instead of the email client module 140 to send a link to a specific online video. For further information regarding online video applications, please refer to U.S. Provisional Patent Application No. 60 / 936,562, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed June 20, 2007, and U.S. Patent Application No. 11 / 968,067, “Portable Multifunction Device, Method, and Graphical User Interface for Playing Online Videos,” filed December 31, 2007, the entire contents of which are incorporated herein by reference.

[0098] Each of the modules and applications identified above corresponds to a set of executable instructions that perform one or more of the functions described above and the methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (e.g., instruction sets) do not need to be implemented as separate software programs, procedures, or modules; therefore, in various embodiments, various subsets of these modules can be optionally combined or otherwise reconfigured. For example, a video player module can optionally be combined with a music player module to form a single module (e.g., the video and music player module 152 in Figure 1A). In some embodiments, memory 102 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 102 optionally stores additional modules and data structures not described above.

[0099] In some embodiments, device 100 is a device in which the operation of a default set of functions in the device is performed solely via a touchscreen and / or touchpad. By using a touchscreen and / or touchpad as the primary input control device for device 100 to operate, the number of physical input control devices (push buttons, dials, etc.) on device 100 is optionally reduced.

[0100] A default set of functions, which are performed only through the touchscreen and / or touchpad, optionally includes navigation between user interfaces. In some embodiments, the touchpad, when touched by the user, navigates the device 100 from any user interface displayed on the device 100 to the main menu, home menu, or root menu. In such embodiments, a “menu button” is implemented using the touchpad. In some other embodiments, the menu button is a physical push button or other physical input control device, rather than a touchpad.

[0101] Figure 1B is a block diagram showing exemplary components for event processing according to several embodiments. In some embodiments, memory 102 (Figure 1A) or 370 (Figure 3) includes an event sorter 170 (e.g., within the operating system 126) and each application 136-1 (e.g., any of the aforementioned applications 137-151, 155, 380-390).

[0102] The event sorter 170 receives event information and determines the application 136-1 that distributes the event information, and the application view 191 of application 136-1. The event sorter 170 includes an event monitor 171 and an event dispatcher module 174. In some embodiments, application 136-1 includes an application internal state 192 that indicates the current application view displayed on the touch-sensitive display 112 when the application is active or running. In some embodiments, a device / global internal state 157 is used by the event sorter 170 to determine which application(s) are currently active, and the application internal state 192 is used by the event sorter 170 to determine the application view 191 from which the event information is distributed.

[0103] In some embodiments, the application internal state 192 includes additional information such as resume information to be used when the application 136-1 resumes execution, user interface state information that indicates or is ready to display information displayed by the application 136-1, a state queue that allows the user to return to a previous state or view of the application 136-1, and one or more redo / undo queues of previous actions performed by the user.

[0104] The event monitor 171 receives event information from the peripheral interface 118. The event information includes information about sub-events (for example, user touch as part of a multi-touch gesture on the touch-sensitive display 112). The peripheral interface 118 transmits information received from the I / O subsystem 106, or from sensors such as the proximity sensor 166, one or more accelerometers 168, and / or the microphone 113 (via the audio circuit 110). The information received by the peripheral interface 118 from the I / O subsystem 106 includes information from the touch-sensitive display 112 or the touch-sensitive surface.

[0105] In some embodiments, the event monitor 171 sends requests to the peripheral interface 118 at predetermined intervals. In response, the peripheral interface 118 transmits event information. In other embodiments, the peripheral interface 118 transmits event information only when there is a significant event (e.g., reception of input exceeding a predetermined noise threshold and / or exceeding a predetermined duration).

[0106] In some embodiments, the event sorter 170 also includes a hit view determination module 172 and / or an active event recognition determination module 173.

[0107] The hit view determination module 172 provides a software procedure for determining where in one or more views a sub-event occurred when the touch-sensitive display 112 is displaying two or more views. A view consists of control devices and other elements that the user can see on the display.

[0108] Another aspect of the user interface associated with an application is a set of views, sometimes referred to herein as application views or user interface windows, in which information is displayed and touch-based gestures occur. The application view (of each application) in which a touch is detected optionally corresponds to the program level within the application's program hierarchy or view hierarchy. For example, the lowest-level view in which a touch is detected optionally refers to the hit view, and the set of events recognized as appropriate input is optionally determined at least in part based on the hit view of the initial touch that initiates the touch gesture.

[0109] The hit view determination module 172 receives information related to sub-events of touch-based gestures. When an application has multiple views arranged in a hierarchy, the hit view determination module 172 identifies the hit view as the lowest-level view in the hierarchy from which the sub-events should be processed. In most situations, the hit view is the lowest-level view from which the initiating sub-event (e.g., the first sub-event in a sub-event sequence that forms an event or potential event) occurs. Once a hit view is identified by the hit view determination module 172, the hit view typically receives all sub-events related to the same touch or input source that identified it as the hit view.

[0110] The active event recognition determination module 173 determines which view(s) in the view hierarchy should receive a particular sequence of sub-events. In some embodiments, the active event recognition determination module 173 determines that only the hit view should receive a particular sequence of sub-events. In other embodiments, the active event recognition determination module 173 determines that all views, including the physical location of the sub-event, are actively involved views, and therefore all actively involved views should receive a particular sequence of sub-events. In other embodiments, even if the touch sub-event is entirely confined to an area associated with one particular view, higher-level views in the hierarchy still remain actively involved views.

[0111] The event dispatcher module 174 dispatches event information to an event recognition unit (e.g., an event recognition unit 180). In embodiments including an active event recognition unit determination module 173, the event dispatcher module 174 distributes the event information to the event recognition unit determined by the active event recognition unit determination module 173. In some embodiments, the event dispatcher module 174 stores the event information acquired by each event receiving unit 182 in an event queue.

[0112] In some embodiments, the operating system 126 includes an event sorter 170. Alternatively, application 136-1 includes an event sorter 170. In yet another embodiment, the event sorter 170 is a standalone module or part of another module stored in memory 102, such as a contact / motion module 130.

[0113] In some embodiments, application 136-1 includes a plurality of event processing units 190 and one or more application views 191, each containing instructions for handling touch events occurring within each view of the application's user interface. Each application view 191 of application 136-1 includes one or more event recognition units 180. Typically, each application view 191 includes a plurality of event recognition units 180. In other embodiments, one or more of the event recognition units 180 are part of a separate module, such as a user interface kit (not shown) or a higher-level object from which application 136-1 inherits methods and other characteristics. In some embodiments, each event processing unit 190 includes one or more of the following: a data update unit 176, an object update unit 177, a GUI update unit 178, and / or event data 179 received from an event sorter 170. The event processing unit 190 optionally uses or calls the data update unit 176, the object update unit 177, or the GUI update unit 178 to update the application's internal state 192. Alternatively, one or more of the application views 191 include one or more event processing units 190. In some embodiments, one or more of the data update unit 176, object update unit 177, and GUI update unit 178 are included in each application view 191.

[0114] Each event recognition unit 180 receives event information (e.g., event data 179) from the event sorter 170 and identifies an event from the event information. The event recognition unit 180 includes an event receiving unit 182 and an event comparison unit 184. In some embodiments, the event recognition unit 180 also includes at least a subset of metadata 183 and event distribution commands 188 (optionally including sub-event distribution commands).

[0115] The event receiving unit 182 receives event information from the event sorter 170. The event information includes information about sub-events, such as touches or the movement of touches. Depending on the sub-event, the event information also includes additional information, such as the position of the sub-event. When the sub-event involves the movement of a touch, the event information also optionally includes the speed and direction of the sub-event. In some embodiments, an event includes the rotation of the device from one orientation to another (e.g., from portrait to landscape, or vice versa), and the event information includes corresponding information about the current orientation of the device (also called the device's orientation).

[0116] The event comparison unit 184 compares event information with a predefined definition of an event or sub-event, and based on the comparison, determines an event or sub-event, or determines or updates the state of an event or sub-event. In some embodiments, the event comparison unit 184 includes an event definition 186. The event definition 186 includes definitions of events (e.g., a sequence of default sub-events), such as event 1 (187-1) and event 2 (187-2). In some embodiments, sub-events within event (187) include, for example, touch start, touch end, touch movement, touch cancel, and multiple touches. In one embodiment, the definition for event 1 (187-1) is a double tap on a displayed object. A double tap includes, for example, a first touch on the displayed object for a predetermined stage (touch start), a first lift-off for the predetermined stage (touch end), a second touch on the displayed object for the predetermined stage (touch start), and a second lift-off for the predetermined stage (touch end). In another embodiment, event 2(187-2) is defined as a drag on a displayed object. The drag includes, for example, a touch (or contact) on the displayed object to a predetermined stage, movement of the touch across the touch-sensitive display 112, and lift-off of the touch (end of touch). In some embodiments, the event also includes information about one or more associated event processing units 190.

[0117] In some embodiments, the event definition 187 includes an event definition for each user interface object. In some embodiments, the event comparison unit 184 performs a hit test to determine which user interface object is associated with a sub-event. For example, in an application view where three user interface objects are displayed on the touch-sensitive display 112, when a touch is detected on the touch-sensitive display 112, the event comparison unit 184 performs a hit test to determine which of the three user interface objects is associated with the touch (sub-event). If each displayed object is associated with its respective event processing unit 190, the event comparison unit uses the results of the hit test to determine which event processing unit 190 should be activated. For example, the event comparison unit 184 selects the sub-event and the event processing unit associated with the object that triggers the hit test.

[0118] In some embodiments, the definition of each event 187 also includes a delay action that delays the transmission of event information until it is determined whether the sequence of sub-events corresponds to the event type of the event recognition unit.

[0119] If each event recognition unit 180 determines that a series of sub-events does not match any of the events in the event definition 186, each event recognition unit 180 enters an event impossible, event failed, or event terminated state and thereafter ignores subsequent sub-events of the touch-based gesture. In this situation, if there are other event recognition units that remain active for the hit view, those event recognition units continue to track and process the sub-events of the ongoing touch-based gesture.

[0120] In some embodiments, each event recognition unit 180 includes metadata 183 having configurable properties, flags, and / or lists that indicate to the actively involved event recognition unit how the event distribution system should perform sub-event distribution. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how the event recognition units interact with each other, or how they can interact with each other. In some embodiments, the metadata 183 includes configurable properties, flags, and / or lists that indicate how sub-events are distributed to various levels in the view hierarchy or program hierarchy.

[0121] In some embodiments, each event recognition unit 180 activates an event processing unit 190 associated with an event when one or more specific sub-events of an event are recognized. In some embodiments, each event recognition unit 180 delivers event information associated with the event to the event processing unit 190. Activating the event processing unit 190 is separate from sending (and delaying the sending of) sub-events to the respective hit view. In some embodiments, the event recognition unit 180 sets a flag associated with the recognized event, and the event processing unit 190 associated with that flag captures the flag and executes a default process.

[0122] In some embodiments, the event distribution command 188 includes a sub-event distribution command that distributes event information about a sub-event without activating an event processing unit. Instead, the sub-event distribution command distributes event information to an event processing unit associated with a set of sub-events, or to a view that is actively involved. The event processing unit associated with the set of sub-events or the view that is actively involved receives the event information and performs predetermined processing.

[0123] In some embodiments, the data update unit 176 creates and updates data used in application 136-1. For example, the data update unit 176 updates telephone numbers used in contact module 137 or stores video files used in video player module. In some embodiments, the object update unit 177 creates and updates objects used in application 136-1. For example, the object update unit 177 creates new user interface objects or updates the position of user interface objects. The GUI update unit 178 updates the GUI. For example, the GUI update unit 178 prepares display information and sends it to graphics module 132 for display on touch-sensitive display.

[0124] In some embodiments, the event processing unit(s) 190 includes or has access to a data update unit 176, an object update unit 177, and a GUI update unit 178. In some embodiments, the data update unit 176, the object update unit 177, and the GUI update unit 178 are included in a single module of their respective applications 136-1 or application view 191. In other embodiments, they are included in two or more software modules.

[0125] The foregoing description regarding the handling of user touch events on a touch-sensitive display also applies to other forms of user input for operating the multifunction device 100 using input devices, but it should be understood that not all of these begin on the touchscreen. For example, mouse movement and mouse button presses, touch movements such as taps, drags, and scrolls on a touchpad, pen stylus input, device movement, verbal commands, detected eye movements, biometric input, and / or any combination thereof may be optionally used as inputs corresponding to sub-events that define the events to be recognized.

[0126] Figure 2 shows a portable multifunction device 100 having a touchscreen 112 according to several embodiments. The touchscreen optionally displays one or more graphics within a user interface (UI) 200. In this embodiment, and in other embodiments described below, the user can select one or more of the graphics by performing gestures on the graphics using, for example, one or more fingers 202 (not shown in the figure to an exact scale) or one or more styluses 203 (not shown in the figure to an exact scale). In some embodiments, the selection of one or more graphics is performed when the user interrupts contact with that one or more graphics. In some embodiments, the gesture optionally includes one or more taps, one or more swipes (from left to right, right to left, upward and / or downward) and / or rolling (from right to left, left to right, upward and / or downward) with a finger in contact with the device 100. In some implementations or situations, accidental contact with a graphic does not constitute a selection of that graphic. For example, if the gesture corresponding to selection is a tap, a swipe gesture sweeping over an application icon does not arbitrarily select the corresponding application.

[0127] In some embodiments, the stylus 203 is an active device and includes one or more electronic circuits. For example, the stylus 203 includes one or more sensors and one or more communication circuits (such as communication module 128 and / or RF circuit 108). In some embodiments, the stylus 203 includes one or more processors and power systems (such as power system 162). In some embodiments, the stylus 203 includes an accelerometer (such as accelerometer 168), a magnetometer, and / or a gyroscope that can determine the position, angle, location, and / or other physical characteristics of the stylus 203 (e.g., whether the stylus is positioned downwards, tilted toward or away from the device, and / or positioned near or far from the device). In some embodiments, the stylus 203 communicates with electronic devices (e.g., via communication circuits, via wireless communication protocols such as Bluetooth) and transmits sensor data to the electronic devices. In some embodiments, the stylus 203 can determine whether a user is holding the device (e.g., via an accelerometer or other sensor). In some embodiments, the stylus 203 can accept tap inputs (e.g., single taps or double taps) on the stylus 203 from a user (e.g., received by an accelerometer or other sensor), and the inputs can be interpreted as commands or requests to perform a function or to change to a different input mode.

[0128] Device 100 also optionally includes one or more physical buttons, such as a "Home" button or a menu button 204. As previously mentioned, the menu button 204 is optionally used to navigate to any application 136 within a set of applications running on device 100. Alternatively, in some embodiments, the menu button is implemented as a soft key in a GUI displayed on the touchscreen 112.

[0129] In some embodiments, device 100 includes a touchscreen 112, a menu button 204, a push button 206 for turning the device on / off and locking the device, volume control buttons 208, a subscriber identification module (SIM) card slot 210, a headset jack 212, and an external port 124 for docking / charging. The push button 206 is optionally used to turn the device on / off by pressing down and holding the button down for a predetermined period of time, to lock the device by pressing down and releasing the button before a predetermined period of time has elapsed, and / or to unlock the device or initiate an unlocking process. In alternative embodiments, device 100 also accepts verbal input via a microphone 113 to activate or deactivate certain functions. Device 100 also optionally includes one or more contact intensity sensors 165 for detecting the intensity of contact on the touchscreen 112, and / or one or more tactile output generators 167 for generating tactile output to the user of device 100.

[0130] Figure 3 is a block diagram of an exemplary multifunctional device having a display and a touch-sensitive surface according to several embodiments. The device 300 does not need to be portable. In some embodiments, the device 300 is a laptop computer, a desktop computer, a tablet computer, a multimedia player device, a navigation device, an educational device (such as a children's learning toy), a game system, or a control device (e.g., a home or commercial controller). The device 300 typically includes one or more processing units (CPUs) 310, one or more network or other communication interfaces 360, memory 370, and one or more communication buses 320 that interconnect these components. The communication buses 320 optionally include circuitry (sometimes called a chipset) that interconnects and controls communication between system components. The device 300 includes an input / output (I / O) interface 330 including a display 340, the display 340 is typically a touchscreen display. The I / O interface 330 also optionally includes a keyboard and / or mouse (or other pointing device) 350 and a touchpad 355, a tactile output generator 357 that generates tactile output on device 300 (for example, similar to the tactile output generator 167 described above with reference to Figure 1A), and a sensor 359 (for example, light, acceleration, proximity, touch sensing, and / or a contact intensity sensor similar to the contact intensity sensor 165 described above with reference to Figure 1A). The memory 370 includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and optionally includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 370 optionally includes one or more storage devices located remotely from the CPU(s) 310.In some embodiments, memory 370 stores programs, modules, and data structures similar to, or subsets thereof, that are stored in memory 102 of the portable multifunction device 100 (Figure 1A). Furthermore, memory 370 optionally stores additional programs, modules, and data structures that are not present in memory 102 of the portable multifunction device 100. For example, memory 370 of device 300 optionally stores a drawing module 380, a presentation module 382, ​​a word processing module 384, a website creation module 386, a disk authoring module 388, and / or a spreadsheet module 390, whereas memory 102 of the portable multifunction device 100 (Figure 1A) optionally does not store these modules.

[0131] Each of the elements identified above in Figure 3 is optionally stored in one or more of the memory devices described above. Each of the modules identified above corresponds to an instruction set that performs the function described above. The modules or programs (e.g., instruction sets) identified above do not need to be implemented as separate software programs, procedures, or modules, and therefore in various embodiments, various subsets of these modules are optionally combined or otherwise reconfigured. In some embodiments, memory 370 optionally stores a subset of the modules and data structures identified above. Furthermore, memory 370 optionally stores additional modules and data structures not described above.

[0132] Next, we optionally turn our attention to an embodiment of a user interface implemented in, for example, a portable multi-functional device 100.

[0133] Figure 4A shows an exemplary user interface for an application menu on a portable multifunction device 100 according to several embodiments. A similar user interface is optionally implemented on device 300. In some embodiments, the user interface 400 includes the following elements, or subsets or supersets thereof. ● Signal strength indicators (single or multiple) for wireless communication (single or multiple) such as cellular signals and Wi-Fi signals 402, ● Time 404, ● Bluetooth indicator 405, ● Battery status indicator 406, ● Tray 408 containing icons for frequently used applications, as shown below. ○ An icon 416 of the telephone module 138, labeled "Telephone," optionally including an indicator 414 for the number of missed calls or voicemail messages. ○ An icon 418 of the email client module 140, labeled "Mail," optionally including an indicator 410 of the number of unread emails. ○ Icon 420 of browser module 147, labeled "Browser", and ○ Icon 422 for the video and music player module 152, also known as the iPod (trademark of Apple Inc.) module 152, which is labeled "iPod", and ● Icons of other applications, such as the following: ○ Icon 424 of IM module 141, labeled "Message", ○ Icon 426 of calendar module 148, labeled "Calendar" ○ Icon 428 of image management module 144, labeled "Photo" ○ Icon 430 of camera module 143, labeled "Camera" ○ Icon 432 of online video module 155, labeled "online video" ○ Icon 434 of stock widget 149-2, labeled "Stock Price" ○ Icon 436 of map module 154, labeled "Map" ○ Icon 438 of weather widget 149-1, labeled "Weather" ○ Icon 440 of the alarm clock widget 149-4, labeled "Clock", ○ Icon 442 of training support module 142, labeled "Training Support" ○ Icon 444 of memo module 153, labeled as "Memo", and ○ An icon 446 labeled "Settings," which provides access to the settings of device 100 and its various applications 136, for a settings application or module.

[0134] Please note that the icon labels shown in Figure 4A are for illustrative purposes only. For example, the icon 422 for the video and music player module 152 is labeled "Music" or "Music Player," and other labels are optionally used for various application icons. In some embodiments, the label for each application icon includes the name of the application to which that application icon corresponds. In some embodiments, the label for a particular application icon is different from the name of the application to which that particular application icon corresponds.

[0135] Figure 4B shows an exemplary user interface on a device (e.g., device 300 in Figure 3) having a touch-sensitive surface 451 (e.g., tablet or touchpad 355 in Figure 3) separate from the display 450 (e.g., touchscreen display 112). Device 300 also optionally includes one or more contact intensity sensors (e.g., one or more of sensors 359) for detecting the intensity of contact on the touch-sensitive surface 451, and / or one or more tactile output generators 357 for generating tactile output to the user of device 300.

[0136] Some of the following examples are given by referring to input on a touchscreen display 112 (a combination of a touch-sensing surface and a display), but in some embodiments, the device detects input on a touch-sensing surface separate from the display, as shown in Figure 4B. In some embodiments, the touch-sensing surface (e.g., 451 in Figure 4B) has a primary axis (e.g., 452 in Figure 4B) corresponding to a primary axis (e.g., 453 in Figure 4B) on the display (e.g., 450). According to these embodiments, the device detects contact with the touch-sensing surface 451 (e.g., 460 and 462 in Figure 4B) at positions corresponding to each of the positions on the display (e.g., 460 corresponds to 468 and 462 corresponds to 470 in Figure 4B). In this way, user input (e.g., touches 460 and 462, and their movement) detected by the device on a touch-sensitive surface (e.g., 451 in Figure 4B) is used by the device to operate the user interface on the display of the multifunction device (e.g., 450 in Figure 4B) when the touch-sensitive surface is separate from the display. It should be understood that a similar method may be optionally used for other user interfaces described herein.

[0137] In addition, while the following examples are given primarily with reference to finger input (e.g., finger touch, finger tap gesture, finger swipe gesture), it should be understood that in some embodiments, one or more of the finger inputs may be replaced by input from another input device (e.g., mouse-based input or stylus input). For example, a swipe gesture may optionally be replaced by a mouse click (e.g., instead of touch), followed by a mouse click with cursor movement along the swipe path (e.g., instead of touch movement). As another example, a tap gesture may optionally be replaced by a mouse click (e.g., instead of touch detection and subsequent cessation of touch detection) while the cursor is located over the tap gesture position. Similarly, it should be understood that when multiple user inputs are detected simultaneously, multiple computer mice may optionally be used simultaneously, or mouse and finger touch may optionally be used simultaneously.

[0138] Figure 5A shows an exemplary personal electronic device 500. Device 500 includes a body 502. In some embodiments, device 500 may include some or all of the features described with respect to devices 100 and 300 (e.g., Figures 1A to 4B). In some embodiments, device 500 has a touch-sensitive display screen 504, hereafter referred to as touchscreen 504. Alternatively, in addition to touchscreen 504, device 500 may have a display and a touch-sensitive surface. Similar to devices 100 and 300, in some embodiments, touchscreen 504 (or touch-sensitive surface) optionally includes one or more intensity sensors that detect the intensity of the applied contact (e.g., touch). One or more intensity sensors on touchscreen 504 (or touch-sensitive surface) may provide output data representing the intensity of the touch. The user interface of device 500 may respond to touches based on their intensity, meaning that touches of different intensity may invoke different user interface behaviors on device 500.

[0139] For example, see, for instance, International Patent Application PCT / US2013 / 040061, “Device, Method, and Graphical User Interface for Displaying User Interface Objects Corresponding to an Application,” filed 8 May 2013, published as International Patent WO / 2013 / 169849, and International Patent Application PCT / US2013 / 069483, “Device, Method, and Graphical User Interface for Transitioning Between Touch Input to Display Output Relationships,” filed 11 November 2013, published as International Patent WO / 2014 / 105276.

[0140] In some embodiments, the device 500 has one or more input mechanisms 506 and 508. The input mechanisms 506 and 508 may be physical, if included. Examples of physical input mechanisms include push buttons and rotatable mechanisms. In some embodiments, the device 500 has one or more attachment mechanisms. Such attachment mechanisms, if included, can allow the device 500 to be attached to, for example, hats, eyeglasses, earrings, necklaces, shirts, jackets, bracelets, watch bands, chains, trousers, belts, shoes, wallets, backpacks, etc. These attachment mechanisms allow the user to wear the device 500.

[0141] Figure 5B shows an exemplary personal electronic device 500. In some embodiments, the device 500 may include some or all of the components described with respect to Figures 1A, 1B, and 3. The device 500 has a bus 512 that operably connects an I / O section 514 to one or more computer processors 516 and memory 518. The I / O section 514 may be connected to a display 504, which may have a touch-sensing component 522 and optionally an intensity sensor 524 (e.g., a contact intensity sensor). In addition, the I / O section 514 may be connected to a communication unit 530 that receives application and operating system data using Wi-Fi, Bluetooth, near-field communication (NFC), cellular, and / or other wireless communication technologies. The device 500 may include input mechanisms 506 and / or 508. The input mechanism 506 may optionally be, for example, a rotatable input device or a pressable and rotatable input device. In some embodiments, the input mechanism 508 may optionally be a button.

[0142] In some embodiments, the input mechanism 508 is optionally a microphone. The personal electronic device 500 optionally includes various sensors such as a GPS sensor 532, an accelerometer 534, a direction sensor 540 (e.g., a compass), a gyroscope 536, a motion sensor 538, and / or a combination thereof, all of which can be operably connected to the I / O section 514.

[0143] The memory 518 of the personal electronic device 500 may include one or more non-temporary computer-readable storage media for storing computer-executable instructions, which, when executed by one or more computer processors 516, can cause the computer processors to execute techniques described below, including processes 700, 900, 1100, 1300, 1500, 1600, 1800, 2000, and 2200 (Figures 7, 9, 11, 13, 15, 16, 18, 20, and 22). The computer-readable storage media may be any medium capable of tangibly containing or storing computer-executable instructions used by or in connection with an instruction execution system, apparatus, or device. In some embodiments, the storage medium is a temporary computer-readable storage medium. In some embodiments, the storage medium is a non-temporary computer-readable storage medium. Non-temporary computer-readable storage media may include, but are not limited to, magnetic, optical, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, CDs, DVDs, or optical disks based on Blu-ray technology, as well as resident solid-state memory such as flash and solid-state drives. The personal electronic device 500 may include, but is not limited to, the components and configurations shown in Figure 5B, and may include other or additional components in multiple configurations.

[0144] As used herein, the term “affordance” optionally refers to user-interactive graphical user interface objects displayed on the display screens of devices 100, 300, and / or 500 (Figures 1A, 3, and 5A-5B). For example, images (e.g., icons), buttons, and text (e.g., hyperlinks) each optionally constitute an affordance.

[0145] As used herein, the term “focus selector” refers to an input element that indicates the current part of the user interface that the user is interacting with. In some implementations, including a cursor or other position marker, the cursor acts as a “focus selector,” and therefore, when an input (e.g., a press input) is detected on a touch-sensitive surface (e.g., touchpad 355 in Figure 3 or touch-sensitive surface 451 in Figure 4B) while the cursor is positioned over a particular user interface element (e.g., a button, window, slider, or other user interface element), the particular user interface element is adjusted according to the detected input. In some implementations, including a touchscreen display that enables direct interaction with user interface elements on the touchscreen display (e.g., touch-sensitive display system 112 in Figure 1A or touchscreen 112 in Figure 4A), a detected contact on the touchscreen acts as a “focus selector,” and therefore, when an input (e.g., a press input by touch) is detected at the location of a particular user interface element (e.g., a button, window, slider, or other user interface element) on the touchscreen display, the particular user interface element is adjusted according to the detected input. In some implementations, focus is moved from one area of ​​the user interface to another without corresponding cursor movement or touch movement on the touchscreen display (for example, by using the tab key or arrow keys to move focus from one button to another), and in these implementations, the focus selector moves in accordance with the movement of focus between different areas of the user interface. Regardless of the specific form the focus selector takes, the focus selector is generally a user interface element (or touch on the touchscreen display) controlled by the user to communicate about the user's intended interaction with the user interface (for example, by pointing to the device the user interface element through which the user intends to interact).For example, the position of a focus selector (e.g., cursor, touch, or selection box) over a corresponding button while pressure input is detected on a touch-sensitive surface (e.g., a touchpad or touchscreen) indicates that the user intends to activate that corresponding button (rather than other user interface elements displayed on the device's display).

[0146] As used herein and in the claims, the term “characteristic intensity” of a contact refers to the characteristics of that contact based on one or more intensities of the contact. In some embodiments, the characteristic intensity is based on multiple intensity samples. The characteristic intensity is optionally based on a set of intensity samples collected over a predetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10 seconds) associated with a predetermined event (e.g., after detection of contact, before detection of lift-off of contact, before or after detection of the start of movement of contact, before detection of the end of contact, before or after detection of an increase in contact intensity, and / or before or after detection of a decrease in contact intensity). The characteristic intensity of a contact is optionally based on one or more of the following: the maximum value of the contact intensity, the mean value of the contact intensity, the average value of the contact intensity, the top 10 percentile value of the contact intensity, the maximum half value of the contact intensity, the maximum 90 percent value of the contact intensity, and so on. In some embodiments, the duration of contact is used when determining characteristic intensity (for example, when characteristic intensity is the average intensity of contact over time). In some embodiments, characteristic intensity is compared to a set of one or more intensity thresholds to determine whether an action has been performed by the user. For example, the set of one or more intensity thresholds optionally includes a first intensity threshold and a second intensity threshold. In this example, contact with a characteristic intensity not exceeding the first threshold results in a first action, contact with a characteristic intensity above the first intensity threshold but not exceeding the second intensity threshold results in a second action, and contact with a characteristic intensity above the second threshold results in a third action. In some embodiments, the comparison between characteristic intensity and one or more thresholds is not used to determine whether a first action should be performed or a second action should be performed, but rather to determine whether one or more actions should be performed at all (for example, whether each action should be performed or whether each action should be postponed).

[0147] Figure 5C shows that multiple intensity sensors 524A-524D detect multiple contacts 552A-552E on a touch-sensitive display screen 504. Figure 5C also includes an intensity diagram showing the current intensity measurements of intensity sensors 524A-524D relative to intensity units. In this example, the intensity measurements of intensity sensors 524A and 524D are 9 intensity units each, and the intensity measurements of intensity sensors 524B and 524C are 7 intensity units each. In some implementations, the aggregate intensity is the sum of the intensity measurements of the multiple intensity sensors 524A-524D, which in this example is 32 intensity units. In some embodiments, each contact is assigned a corresponding intensity, which is a portion of the aggregate intensity. Figure 5D shows that the aggregate intensity is assigned to contacts 552A-552E based on their distance from the center of force 554. In this example, contacts 552A, 552B, and 552E are each assigned a contact intensity of 8 intensity units of the aggregate intensity, and contacts 552C and 552D are each assigned a contact intensity of 4 intensity units of the aggregate intensity. More generally, in some implementations, each contact j is assigned an intensity Ij, which is a portion of the aggregate intensity A, according to a predetermined mathematical function Ij = A·(Dj / ΣDi), where Dj is the distance from the center of force to each contact j, and ΣDi is the sum of the distances from the center of force to each of the contacts (e.g., from i=1 to the end). The operations described with reference to Figures 5C-5D can be performed using electronic devices similar to or identical to devices 100, 300, or 500. In some embodiments, the characteristic intensity of a contact is based on one or more intensities of the contact. In some embodiments, an intensity sensor is used to determine a single characteristic intensity (e.g., a single characteristic intensity of a single contact). Please note that the intensity diagrams are not part of the display user interface, but are included in Figures 5C-5D to assist the reader.

[0148] In some embodiments, a portion of the gesture is identified for the purpose of determining characteristic intensity. For example, the touch-sensitive surface optionally receives a series of swipe contacts that transition from a starting position to an ending position, where the intensity of contact increases. In this example, the characteristic intensity of the contact at the ending position is optionally based only on a portion of the series of swipe contacts (e.g., only the portion of the swipe contact at the ending position) rather than the entire swipe contact. In some embodiments, optionally, a smoothing algorithm is applied to the intensity of the swipe contact before determining the characteristic intensity of the contact. For example, the smoothing algorithm optionally includes one or more of the following: an unweighted moving average smoothing algorithm, a triangular smoothing algorithm, a median filter smoothing algorithm, and / or an exponential smoothing algorithm. In some situations, these smoothing algorithms eliminate narrow spikes or drops in the swipe contact intensity for the purpose of determining characteristic intensity.

[0149] The intensity of contact on a touch-sensitive surface is optionally characterized to one or more intensity thresholds, such as a contact detection intensity threshold, a light press intensity threshold, a deep press intensity threshold, and / or one or more other intensity thresholds. In some embodiments, the light press intensity threshold corresponds to the intensity at which the device performs an action typically associated with clicking a physical mouse button or trackpad. In some embodiments, the deep press intensity threshold corresponds to the intensity at which the device performs an action different from the action typically associated with clicking a physical mouse button or trackpad. In some embodiments, when a contact with a characteristic intensity below the light press intensity threshold (for example, above a nominal contact detection intensity threshold below which contact is no longer detected) is detected, the device moves the focus selector in accordance with the movement of the contact on the touch-sensitive surface without performing an action associated with the light press intensity threshold or the deep press intensity threshold. Generally, unless otherwise specified, these intensity thresholds are consistent across various sets of user interface values.

[0150] An increase in the characteristic intensity of contact from an intensity below a light press intensity threshold to an intensity between the light press intensity threshold and the deep press intensity threshold is sometimes called a "light press" input. An increase in the characteristic intensity of contact from an intensity below a deep press intensity threshold to an intensity above the deep press intensity threshold is sometimes called a "deep press" input. An increase in the characteristic intensity of contact from an intensity below a contact detection intensity threshold to an intensity between the contact detection intensity threshold and the light press intensity threshold is sometimes called detection of contact on the touch surface. A decrease in the characteristic intensity of contact from an intensity above a contact detection intensity threshold to an intensity below a contact detection intensity threshold is sometimes called detection of contact lift-off from the touch surface. In some embodiments, the contact detection intensity threshold is zero. In some embodiments, the contact detection intensity threshold is greater than zero.

[0151] In some embodiments described herein, one or more actions are performed in response to the detection of a gesture including each press input, or in response to the detection of each press input performed by each (or more) contact, and each press input is detected at least in part on the detection of an increase in the intensity of the contact (or more) above a press input intensity threshold. In some embodiments, each action is performed in response to the detection of an increase in the intensity of each contact above a press input intensity threshold (e.g., a "downstroke" of each press input). In some embodiments, a press input includes an increase in the intensity of each contact above a press input intensity threshold, followed by a decrease in the intensity of the contact below the press input intensity threshold, and each action is performed in response to the detection of a decrease in the intensity of each contact below the press input threshold (e.g., an "upstroke" of each press input).

[0152] Figures 5E-5H show the light pressure threshold in Figure 5E (for example, "IT L From an intensity below ) to a deep pressing force threshold (for example, "IT" in Figure 5H) DThe device indicates the detection of a gesture, including a press input, corresponding to an increase in the intensity of contact 562 to an intensity exceeding a deep press intensity threshold (e.g., "IT"). The gesture performed by contact 562 is detected on the touch-sensitive surface 560, and on the display user interface 570, which includes application icons 572A-572D displayed within a predetermined area 574, a cursor 576 is displayed over the application icon 572B corresponding to app 2. In some embodiments, the gesture is detected on the touch-sensitive display 504. An intensity sensor detects the intensity of contact on the touch-sensitive surface 560. The device indicates that the intensity of contact 562 exceeds a deep press intensity threshold (e.g., "IT"). D It is determined that the touch has reached its peak by exceeding the "IT" threshold. Contact 562 is maintained on the touch sensing surface 560. In response to the detection of the gesture, a deep press intensity threshold (e.g., "IT") is set during the gesture. D According to a contact 562 having an intensity exceeding '', a scaled representation 578A-578C (e.g., thumbnail) of the recently opened document is displayed for app 2, as shown in Figures 5F-5I. In some embodiments, this intensity, compared to one or more intensity thresholds, is the characteristic intensity of the contact. Note that the intensity diagram for contact 562 is not part of the display user interface but is included in Figures 5E-5H to assist the reader.

[0153] In some embodiments, the display of representations 578A to 578C includes animation. For example, as shown in Figure 5F, representation 578A is initially displayed close to the application icon 572B. As the animation progresses, as shown in Figure 5G, representation 578A moves upward and representation 578B is displayed close to the application icon 572B. Then, as shown in Figure 5H, representation 578A moves upward and representation 578B moves upward toward representation 578A and representation 578C is displayed close to the application icon 572B. Representations 578A to 578C form an array above the icon 572B. In some embodiments, as shown in Figures 5F to 5G, the animation progresses according to the intensity of the contact 562, where the intensity of the contact 562 reaches a deep press intensity threshold (e.g., "IT").D As it increases toward (), expressions 578A-578C appear and move upward. In some embodiments, the intensity on which the animation progresses is based is the characteristic intensity of the contact. The actions described with reference to Figures 5E-5H can be performed using electronic devices similar to or identical to devices 100, 300, or 500.

[0154] Figure 5I shows a block diagram of an exemplary architecture of device 580 according to some embodiments of the present disclosure. In the embodiment of Figure 5I, media or other content is optionally received by device 580 via a network interface 582, which is optionally wireless or wired. One or more processors 584 optionally execute any number of programs stored in memory 586 or storage. Memory 586 or storage optionally contains instructions that execute one or more methods and / or processes described herein (e.g., methods 700, 900, 1100, 1300, 1500, 1600, 1800, 2000, and 2200).

[0155] In some embodiments, a display controller 588 causes various user interfaces of the Disclosure to be displayed on a display 594. Furthermore, input to device 580 is optionally provided by a remote control 590 via a remote interface 592. The remote interface 592 is optionally wireless or wired. In some embodiments, as will be described in more detail below, input to device 580 is provided by a multifunction device 591 (e.g., a smartphone), on which a remote control application constituting the multifunction device 591 is executed to simulate remote control functions. In some embodiments, the multifunction device 591 corresponds to one or more of the devices 100 in Figures 1A and 2, the device 300 in Figure 3, and the device 500 in Figure 5A. The embodiment in Figure 5I is not intended to limit the features of the devices of the Disclosure, and it is understood that other components to facilitate other features described in the Disclosure are also optionally included in the architecture of Figure 5I. In some embodiments, device 580 optionally corresponds to one or more of the multifunction device 100 in Figures 1A and 2, device 300 in Figure 3, and device 500 in Figure 5A. Network interface 582 optionally corresponds to one or more of the RF circuit 108, external port 124, and peripheral device interface 118 in Figures 1A and 2, and network communication interface 360 ​​in Figure 3. Processor 584 optionally corresponds to one or more of the processor(s) 120 in Figure 1A and CPU(s) 310 in Figure 3. Display controller 588 optionally corresponds to one or more of the display controller 156 in Figure 1A and I / O interface 330 in Figure 3. Memory 586 optionally corresponds to one or more of the memory 102 in Figure 1A and memory 370 in Figure 3. The remote interface 592 optionally corresponds to one or more of the peripheral device interface 118 and I / O subsystem 106 (and / or its components) in Figure 1A, and the I / O interface 330 in Figure 3.The remote 590 optionally corresponds to and / or includes one or more of the following: speaker 111, touch-sensitive display system 112, microphone 113, optical sensor(s) 164, contact intensity sensor(s) 165, tactile output generator(s) 167, other input control device 116, accelerometer(s) 168, proximity sensor 166, and I / O subsystem 106, keyboard / mouse 350, touchpad 355, tactile output generator(s) 357, and contact intensity sensor(s) 359 in Figure 3, and touch-sensitive surface 451 in Figure 4; and the display 594 optionally corresponds to one or more of the touch-sensitive display systems 112 in Figures 1A and 2, and the display 340 in Figure 3.

[0156] In some embodiments, the device employs intensity hysteresis to avoid accidental inputs, which may be referred to as “jitter,” and the device defines or selects a hysteresis intensity threshold that has a predetermined relationship with a press input intensity threshold (for example, the hysteresis intensity threshold is X intensity units lower than the press input intensity threshold, or the hysteresis intensity threshold is 75%, 90%, or some reasonable percentage of the press input intensity threshold). Thus, in some embodiments, a press input includes an increase in the intensity of each contact above the press input intensity threshold, followed by a decrease in the intensity of the contact below the hysteresis intensity threshold corresponding to the press input intensity threshold, and each action is performed in response to the detection of a subsequent decrease in the intensity of each contact below the hysteresis intensity threshold (for example, an “upstroke” of each press input). Similarly, in some embodiments, a press input is detected only when the device detects an increase in contact intensity from below a hysteresis intensity threshold to above a press input intensity threshold, and optionally a decrease in contact intensity to below the hysteresis intensity, and each action is performed in response to the detection of a press input (e.g., depending on the situation, an increase in contact intensity or a decrease in contact intensity).

[0157] For the sake of clarity, the description of an action performed in response to a press input associated with a press input intensity threshold, or a gesture including a press input, is optionally triggered in response to the detection of any of the following: an increase in contact intensity above the press input intensity threshold, an increase in contact intensity from below the hysteresis intensity threshold to above the press input intensity threshold, a decrease in contact intensity below the press input intensity threshold, and / or a decrease in contact intensity below the hysteresis intensity threshold corresponding to the press input intensity threshold. Furthermore, in examples where an action is described to be performed in response to the detection of a decrease in contact intensity below the press input intensity threshold, the action is optionally performed in response to the detection of a decrease in contact intensity below a hysteresis intensity threshold corresponding to and lower than the press input intensity threshold.

[0158] In this specification, “installed application” refers to a software application that has been downloaded onto an electronic device (e.g., device 100, 300, and / or 500) and is ready to be launched on the device (e.g., opened). In some embodiments, a downloaded application becomes an installed application by an installation program that extracts the program portion from the downloaded package and integrates the extracted portion with the operating system of the computer system.

[0159] In this specification, the terms “open application” or “running application” refer to a software application that has retained state information (e.g., as part of the device / global internal state 157 and / or application internal state 192). An open or running application is optionally one of the following types of applications: ● The active application currently displayed on the display screen of the device on which the application is being used. ● Background applications (or background processes) that are not currently displayed but whose processes are handled by one or more processors, as well as ● An application that is not running but has state information stored in memory (volatile and non-volatile, respectively) that can be used to resume the execution of the application, either suspended or suspended.

[0160] In this specification, the term “closed application” refers to a software application that does not retain state information (for example, state information for a closed application is not stored in the device’s memory). Therefore, closing an application involves stopping and / or removing the application process for the application and removing the state information for the application from the device’s memory. Generally, opening a second application while a first application is running does not close the first application. When the second application is displayed and the first application is closed, the first application becomes a background application.

[0161] Next, we will focus on embodiments of user interfaces ("UI") and related processes implemented on electronic devices such as portable multifunction device 100, device 300, or device 500. User interface and related processes Convert handwritten input to text

[0162] Users interact with electronic devices in many different ways, including by typing text into them. In some embodiments, electronic devices provide a virtual keyboard (e.g., a soft keyboard) that mimics the layout of a physical keyboard and allows the user to select the characters to type. The embodiments described below provide a method by which electronic devices accept handwritten input from a handwriting input device (e.g., a stylus) and convert the handwritten input into font-based text (e.g., computer text, digital text, etc.). By enhancing bidirectional interaction with the device, the amount of time required by the user to perform an action is reduced, and therefore the power consumption of the device is reduced, increasing battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally called a user of that device.

[0163] Figures 6A to 6YY illustrate exemplary methods by which an electronic device converts handwritten input into font-based text. The embodiments in these figures are used to illustrate the processes described below, including those described with reference to Figures 7A to 7I.

[0164] Figures 6A to 6YY illustrate the operation of an electronic device 500 that converts handwritten input to font-based text. Figure 6A shows an exemplary device 500 including a touchscreen 504. As shown in Figure 6A, the electronic device 500 presents a user interface 600. In some embodiments, the user interface 600 is any user interface that includes one or more text input fields (e.g., text input areas). In some embodiments, a text input field (e.g., text input area) is a user interface element that allows a user to input text (e.g., letters, characters, words, etc.). For example, a text input field could be a text field on a form, a URL input element on a browser, a login field, etc. In other words, any user interface element that allows a user to input text and perform edit, delete, copy, cut, or any other text-based operations. It is understood that a text input field (e.g., text input area) is not limited to a user interface element that accepts only text, but can also accept and display audio and / or visual media.

[0165] In some embodiments, as shown in Figure 6A, the user interface 600 is that of an internet browser application displaying (e.g., navigating) a passenger information input user interface (e.g., for purchasing an airline ticket). It should be understood that the embodiments shown in Figures 6A to 6YY are illustrative and should not be considered as limiting to the illustrative user interfaces and / or applications only. In some embodiments, the user interface 600 includes text input fields 602-1 to 602-9, where the user can enter text to fill in the respective text input fields (e.g., information about two passengers).

[0166] In Figure 6B, user input is received (e.g., detected) on the touchscreen 504 from the stylus 203. As shown in Figure 6B, the stylus 203 is touching down on the touchscreen 504. In some embodiments, the stylus 203 touches down on the touchscreen 504 to provide handwriting input 604-1. For example, as shown in Figure 6B, the handwriting input 604-1 is for the character "12". In some embodiments, if the handwriting input is performed entirely within a text input field, the handwriting input is interpreted as a request to enter text within each text input field. In some embodiments, if the handwriting is performed in a threshold area near the boundary of a text input field, the handwriting input is still interpreted as a request to enter text within each text input field. In some embodiments, a text input field has a margin of error or tolerance that allows handwriting inputs that are slightly outside the literal boundaries of the text input field (e.g., 1mm, 2mm, 3mm, 5mm, 3 points, 6 points, 12 points, etc.) to still be considered a request to enter text within each respective text input field. In some embodiments, handwriting inputs that begin outside the boundaries of the text input field but enter within the boundaries of the text input field are considered a request to enter text within each respective text input field. In some embodiments, handwriting inputs that have the majority of the stroke within a text input field are considered a request to enter text within each respective text input field. In some embodiments, handwriting inputs that begin within a text input field but extend outside the text input field and optionally into another text input field are still considered a request to enter text within each respective text input field (e.g., not into another text input field).In some embodiments, by providing an error or tolerance margin near the boundary of the text input field, the system can accept handwritten input that is not entirely within the text input field (e.g., larger than the text input field, "missing" the text input field, or unintentionally extending beyond the boundary of the text input field).

[0167] As shown in Figure 6B, the handwriting input 604-1 is directed towards the text input field 602-3. In some embodiments, the handwriting input 604-1 starts slightly outside the text input field 602-3 (e.g., within the error or tolerance margin of the text input field 602-3) and / or optionally has the majority of the stroke within the boundaries of the text input field 602-3. Thus, in some embodiments, the handwriting input 602-1 is interpreted as a request to input the character "12" into the text input field 602-3.

[0168] In Figure 6C, the user continues handwriting input 604-1 and writes "1234" into text input field 602-3. In some embodiments, the user further provides handwriting input 604-2 corresponding to "E". In some embodiments, handwriting input 604-2 begins outside the boundary of text input field 602-3, but the majority of handwriting input 604-2 is within the boundary of 602-3 so that handwriting input 604-2 is considered a request to enter text into text input field 602-3. In some embodiments, whether a handwriting input is considered a request to enter text into a specific text input field is determined by an analysis of each character (e.g., whether each character is considered directed towards its respective text input field), each word (e.g., whether each word as a whole is considered directed towards its respective text input field), or the entire sequence of handwriting input (e.g., whether the entire sequence from the initial touchdown until the handwriting input pauses or ends within a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds) is considered directed towards its respective text input field).

[0169] In Figure 6D, the user continues handwriting 604-2 and writes "Elm" in the text input field 602-3. In some embodiments, this occurs after a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds) has elapsed since the user entered handwriting input 604-1 (e.g., "1234"). In some embodiments, after the threshold amount of time, device 600 determines that handwriting input 604-1 corresponds to the characters "1234". In other words, device 600 analyzes handwriting input 604-1 and recognizes the user's writing as the characters "1234". In some embodiments, handwriting input 604-1 changes color and / or opacity to indicate that handwriting input 604-1 has been recognized by device 500 and / or that handwriting input 604-1 is being converted to font-based text (e.g., computer text, digital text). For example, handwriting input 604-1 becomes gray when handwriting input 604-1 is being converted to font-based text. In some embodiments, changes in color and / or opacity are part of an animation that converts the handwritten input 604-1 to font-based text (for example, the handwritten input becomes gray for a short time, such as 0.2 seconds, 0.3 seconds, 0.5 seconds, 1 second, etc., during the animation that converts the handwritten input to font-based text). In some embodiments, when the handwritten input is received, an animation is displayed that changes the color and / or opacity (e.g., an ink-drying effect) similar to the ink-drying animation described later with respect to Method 2000 (e.g., described with respect to Figures 19B-19I). In some embodiments, an ink-drying effect animation is performed while the handwritten input is being received (e.g., optionally, before the device starts the process of converting the handwritten input to font-based text). In some embodiments, an ink-drying effect animation is performed when the handwritten input is converted to font-based text (e.g., as part of the animation that converts the handwritten input to font-based text).

[0170] In Figure 6E, the user enters a handwriting input 604-3 corresponding to the word "Streat". In some embodiments, the handwriting input 604-3 begins within the boundaries of text input field 602-3, ends outside the boundaries of text input field 602-3, and enters within the boundaries of text input field 602-4. In some embodiments, even if the handwriting input 604-3 leaves the boundaries of text input field 602-3 and enters within the boundaries of text input field 602-4, the handwriting input 604-3 is considered a request to enter text into text input field 602-3 (for example, directed to text input field 602-3).

[0171] In some embodiments, the handwritten input 604-1 is converted to font-based text. In some embodiments, the font-based text is text entered using a conventional text input system such as a physical keyboard or a soft keyboard. In some embodiments, the text is formatted using a specific font style. For example, the font-based text may be 12-point Times New Roman or 10-point Arial. In some embodiments, the handwritten input 604-3 is converted after a threshold delay (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds). In some embodiments, the handwritten input 604-3 is converted after its visual characteristics have been altered to indicate that it is about to be converted (e.g., as shown in Figure 6D). In some embodiments, the visual characteristics of the handwritten input 604-3 do not change before conversion.

[0172] In some embodiments, the size of the converted handwritten input is the default font size of the text input field. In some embodiments, the size of the handwritten input changes before it is converted to font-based text. In some embodiments, the size of the font-based text matches the size of the handwritten input, in which case the size of the font-based text is changed to match the default size of the text input field (for example, the size is changed after the animation that converts the handwritten input to font-based text). In some embodiments, the size changes during the animation from handwritten input to font-based text. In some embodiments, the animation that converts handwritten input to font-based text includes morphing the handwritten input to font-based text. In some embodiments, the handwritten input is broken down (for example, into fragments or inflected words) and reassembled as font-based text (for example, as described below with respect to Method 2000). In some embodiments, the handwritten input dissolves or fades out and dissolves or fades in to font-based text. In some embodiments, the handwritten input moves toward the final position of the font-based text while dissolving (e.g., aligning itself with the text input area or any existing text), and the font-based text appears simultaneously while moving toward the final position. Thus, in some embodiments, the handwritten input and the font-based text can be displayed on the display simultaneously for at least a portion of the animation (e.g., to reduce animation time).

[0173] In Figure 6F, the user enters a handwriting input 604-4 corresponding to the character "Apt.". In some embodiments, the handwriting input 604-4 is entirely outside of any text input field (e.g., both text input fields 604-3 and 602-4). In some embodiments, the handwriting input 604-4 is executed quickly and consecutively after the handwriting input 604-3, so that it is considered to be in the same sequence of handwriting inputs as the handwriting input 604-3 (e.g., 0.25 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds after the writing of the handwriting input 604-3). In some embodiments, since the handwriting input 604-4 is considered to be within the same sequence of inputs as the handwriting input 604-3, the handwriting input 604-4 is also considered to be a request to enter text into the text input field 602-3 (e.g., directed to the text input field 602-3).

[0174] Figure 6G shows the user lifting off the stylus 203 from touchscreen 504 after completing the writing of handwritten inputs 602-4 to 604-4. In some embodiments, in response to the lift-off of the stylus 203 from touchscreen 504 within a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds), device 500 analyzes, interprets, and converts the handwritten inputs into font-based text, as shown in Figure 6H. As shown in Figure 6H, each of the converted handwritten inputs 604-2 to 604-4 is entered into text input field 602-3 and visually aligned with text input field 602-3 and, optionally, with the converted handwritten input 604-1.

[0175] In Figure 6I, after a threshold time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds), and after lifting the stylus 203 from the touchscreen 504, the user continues to input handwriting inputs 604-5. However, because the user pauses handwriting input, any further handwriting inputs are no longer considered to be in the same sequence as handwriting inputs 604-3 and 604-4. Therefore, in the example shown in Figure 6I, further handwriting inputs such as handwriting input 604-5 are analyzed separately to determine which text input field the handwriting input is directed to (e.g., in this case, text input field 602-4).

[0176] In Figure 6J, in some embodiments, when a user enters handwritten input 604-5 near or at the end of the text input field 602-4 (e.g., within 1 mm, 2 mm, 3 mm, etc.), the text input field 602-4 expands horizontally to accommodate further handwritten input. For example, after the user writes the character "1", the text input field 602-4 optionally expands to provide space for the user to write the character "2", etc. Alternatively, in some embodiments, after the user writes the character "1", the text input field 602-4 does not expand. However, after the user writes the character "2" outside the text input field 602-4, the text input field 602-4 expands to include the character "2".

[0177] In Figure 6K, the user continues to input "1234" into the handwriting input field 604-5. In some embodiments, the handwriting input field 604-5 reaches the end of the touchscreen 504 to such an extent that the text input field 602-4 cannot be further expanded horizontally. Therefore, as shown in Figure 6K, the text input field 602-4 expands vertically to provide the user with an additional line to continue inputting the handwriting.

[0178] In some embodiments, after the user lifts the stylus 203 off the touchscreen 504 for a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds), the device 500 analyzes, interprets, and converts the handwriting input (e.g., handwriting input 604-5) into font-based text. In some embodiments, as described above, because the user paused the handwriting input for a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds) so that the handwriting input 604-5 is no longer considered a continuation of handwriting input 604-3 or handwriting input 604-5 (e.g., so that the handwriting input can optionally be entered into text input field 602-3), the handwriting input 604-5 is entered into text input field 602-4 instead of text input field 602-3. In some embodiments, simultaneously with or after the handwriting input 604-5 is converted into font-based text, the text input field 602-4 returns to its original size.

[0179] Figures 6M to 6O illustrate alternative methods by which device 500 provides additional space for continued handwriting input when the handwriting input approaches or reaches the end of the text input field. In Figure 6M, the user provides handwriting input 604-5 at or near the end of the text input field 602-4. In some embodiments, as shown in Figure 6N, the handwriting input 604-5 is shifted to the left from the end of the text input field 602-4 to provide the user with room to continue handwriting input. In some embodiments, the handwriting input 604-5 is shifted to the left after the user has finished writing the character (e.g., after a short lift-off of 0.2 seconds, 0.4 seconds, 0.6 seconds, 1 second, 2 seconds, etc.). In some embodiments, the leftward shift of the handwriting input is performed simultaneously with the expansion of the text input field. In some embodiments, after the user lifts the stylus 203 off the touchscreen 504 for a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds), the device 500 converts the handwritten input 604-5 into font-based text, as shown in Figure 6O.

[0180] In Figure 6P, handwritten input 604-6 is detected (e.g., received) on the touchscreen 504 in the text input fields 602-5. In some embodiments, the handwritten input 604-6 is difficult to recognize. For example, the confidence level of device 500 in the written characters in handwritten input 604-6 falls below a threshold confidence level (e.g., 25% confidence, 50% confidence, 75% confidence, etc.). In some embodiments, if the confidence level of the characters written by the user falls below a threshold confidence level, a popup is displayed to the user in proposed font-based text, as shown in Figure 6Q.

[0181] In some embodiments, the popup 606 appears above the handwriting input 604-6 or otherwise within its vicinity (e.g., within 5mm, 1cm, 1.5cm, 3cm, etc.). In some embodiments, the word or character associated with the popup 606 is highlighted. In some embodiments, the popup 606 includes the highest confidence interpretation of the handwriting input 604-6 (e.g., "Salem"). In some embodiments, the popup 606 includes two or more potential interpretations of the handwriting input 604-6 (e.g., corresponding to one or more selectable options). In some embodiments, the popup 606 is selectable to trigger a conversion of the handwriting input 604-6 to a selected interpretation (as opposed to conversion after a threshold time delay or other time-based heuristics). In some embodiments, the popup 606 appears after the user lifts the stylus 203 off the touchscreen 504, and the device 600 has the opportunity to analyze and interpret the entire handwriting sequence (e.g., an entire word, an entire sentence, a string, etc.). In some embodiments, the popup 606 appears at any point while the user is performing handwriting input and is updated when the user writes additional characters recognized by the device 500. For example, the popup 606 may optionally first appear after the user writes "Sa" and display "Sa". In such embodiments, when the user writes "l", the popup 606 is updated to display "Sal". In some embodiments, after the user writes "em", the popup 606 is updated to display "Salem" (for example, in some embodiments, the popup is updated with a new character after each character, or after several characters).In some embodiments, the popup 606 is displayed regardless of the confidence level of the interpretation of the handwriting input (for example, the popup 606 is optionally always displayed, providing the user with a way to “accept” the proposed font-based text and trigger the conversion of the handwriting input to the proposed font-based text, independently of the timer used to determine when to convert the handwriting input to the font-based text). In some embodiments, the popup 606 includes selectable options to reject the proposal or, otherwise, to dismiss the popup 606. In some embodiments, dismissing the popup or rejecting the proposal does not prevent the handwriting input 604-6 from ever being converted. In some embodiments, dismissing the popup or rejecting the proposal prevents the handwriting input 604-6 from being converted at that point, but the handwriting input 604-6 may still be optionally converted at a later point based on other heuristics, such as a timer-based conversion heuristic.

[0182] As shown in Figure 6R, device 500 detects a tap on the touchscreen 504 from a stylus 203 to select a popup 606. In some embodiments, in response to user input to select a popup 606 (e.g., to select a selectable option corresponding to the proposed font-based text "Salem"), device 500 replaces the handwritten input 604-6 with font-based text, as shown in Figure 6S. In some embodiments, as described above, replacing (e.g., converting) the handwritten input with font-based text optionally includes changing the size and / or shape of the handwritten input, optionally including performing an animation to convert the handwritten input to font-based text, aligning the font-based text with a text input field (e.g., text input fields 602-5), or optionally aligning the font-based text with any existing text in a text input field (optionally, in a manner similar to the process described below with respect to method 2000).

[0183] In some embodiments, the converted font-based text is displayed in different positions within the text input field based on the confidence level of device 500 in the handwritten characters in the handwritten input 604-6. For example, if the confidence level of device 500 is below a threshold level (e.g., 25% confidence, 50% confidence, 75% confidence, etc.), the converted font-based text is not aligned with any existing text or text input field. Instead, in some embodiments, the converted font-based text is left in the same position as the original handwritten input, indicating to the user that device 500 is not confident in the conversion. In some embodiments, if the confidence level is above a threshold level, the converted font-based text is aligned with any existing text in the text input field, or aligned to the left of the text input field (e.g., if no existing text exists).

[0184] Figures 6T to 6W illustrate embodiments in which a text input field expands its boundaries to provide a more comfortable or natural writing position based on the location of the text input field on the display. In Figure 6T, user input is detected from a stylus 203 touching down on the touchscreen 504 with text input fields 602-8 (e.g., tap input, long press input (e.g., tap and hold)). In some embodiments, the text input fields 602-8 are located at or near the bottom of the touchscreen 504 (e.g., the bottom third, bottom half, bottom quarter, etc.). In some embodiments, writing on the bottom of the touchscreen 504 with the stylus 203 is cumbersome because the user has little to no surface to rest their palm on while writing. Therefore, in some embodiments, the device 500 determines that the text input field should be expanded upwards so that it can provide handwriting input in a less uncomfortable location for the user, based on the location of the text input field the user is interacting with. Therefore, as shown in Figure 6U, in response to receiving input by tapping or selecting on the text input field 602-8, the boundary of the text input field 602-8 expands vertically upward. In some embodiments, the text input field 602-8 expands to the midpoint of the screen, the two-thirds point of the screen, and so on. In some embodiments, the text input field 602-8 expands horizontally and vertically.

[0185] In Figure 6V, user input 604-7 is received from the stylus 203, providing handwriting input 604-7 to write the word "Bob" into the enlarged text input field 602-8. In some embodiments, the determination of whether the handwriting input is directed to or corresponds to a request to enter text into the enlarged text input field 602-8 is the same as the determination for entering text into the non-enlarged text input field. In some embodiments, after the writing of the handwriting input 604-7 is completed (e.g., stylus 203 lift-off and / or detection of further handwriting input after a threshold amount of time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds)), the handwriting input 604-7 is converted to font-based text, and the text input field 602-8 returns to its original size and shape (e.g., simultaneously with, after, or before conversion), as shown in Figure 6W.

[0186] In Figure 6X, user input from the stylus 203 is detected on the touchscreen 504 outside the boundary of an arbitrary text input field. In some embodiments, user input is not considered handwritten text input if it does not meet any of the criteria for determining whether user input is directed to a text input field or a request to enter text into a text input field. In some embodiments, if user input is not handwritten text input, the gesture performed by the user input is not displayed on the screen. In some embodiments, when the user is performing handwritten text input, the user's handwriting of characters and words appears on the screen at the location and time the input is received. In contrast, in some embodiments, when the user is not performing handwritten text input, the user's gesture does not appear on the screen. Similarly, in some embodiments, user input is interpreted as a non-text input command or a non-text input gesture based on the element the user is interacting with and the characteristics of the input. For example, in Figure 6X, device 500 detects that the user has initiated an upward scroll input (e.g., touching down on the touchscreen 504 with the stylus 203 and moving upward while continuously touching the touchscreen 504). In some embodiments, in response to the upward scroll input from the stylus 203, the user interface 600 scrolls upward in accordance with the movement of the scroll input, as shown in Figure 6Y. As shown in Figures 6X to 6Y, the user's upward gesture while touching down on the touchscreen 504 is not displayed on the touchscreen 504 (in contrast to, for example, when the user is performing text input using the stylus 203).

[0187] Figures 6Z to 6MM illustrate exemplary methods for receiving handwritten input within a multi-line text input field. In Figure 6Z, device 500 displays a user interface 610 including text input fields 612-1 and 612-2. In some embodiments, text input field 612-2 is a multi-line text input field capable of receiving and displaying multiple lines of text. In Figure 6AA, text 616-1 is added to text input field 612-1, and text input field 612-2 is receiving handwritten input 616-2. In some embodiments, when handwritten input 616-2 reaches or begins to reach the horizontal end of text input field 612-2, a popup 618 appears, presenting a selectable option to create a new line of text for input. In some embodiments, creating a new line of text includes vertically increasing the size of the text input field (e.g., optionally, based on the size of the handwritten input) to accept further handwritten input. For example, as shown in Figure 6BB, user input is detected in which the stylus 203 selects a popup 618 to create (e.g., insert) a new line of text. In some embodiments, as a result of the user input, as shown in Figure 6CC, the text input field 612-2 expands its lower boundary downward to create a line of text for the user to provide further handwriting input.

[0188] In Figure 6DD, a further handwritten input 616-3 is received from the stylus 203 into the newly created space within the text input field 612-2. In some embodiments, as shown in Figure 6EE, the device 500 receives a handwritten input 616-4. In some embodiments, the handwritten input 616-4 is received in the text input field 612-2 at a vertical position lower than the handwritten input 616-3. However, in some embodiments, since the handwritten input 616-4 is not at a threshold distance below the handwritten input 616-3 (e.g., it at least partially overlaps with the vertical space of the handwritten input 616-3, 1 mm below the handwritten input 616-3, 2 mm below the handwritten input 616-3, etc.), the handwritten input 616-4 is not considered to be written on a different line from the handwritten input 616-3 and is not considered a request to insert a new line of text.

[0189] In Figure 6FF, the handwritten input 616-5 is received beyond a threshold distance of handwritten input 616-3 (e.g., 1mm, 2mm, 3mm below handwritten input 616-3). In some embodiments, even if the handwritten input 616-5 is received outside the boundary of the text input field 612-2 (e.g., slightly overlapping the area of ​​the text input field 612-2), it is generally considered that the handwritten input 616-5 was entered immediately after the handwritten input 616-4 with little delay, and / or that there are no further text input fields below the text input field 612-2, so the handwritten input 616-5 is considered a request to enter text into a new line within the text input field 612-2. In some embodiments, depending on whether the handwritten input 616-5 is received below the threshold distance of handwritten input 616-3, the text input field 612-2 creates a new text line to contain the handwritten input 616-5, as shown in Figure 6GG.

[0190] In Figure 6HH, user input from the stylus 203 is received by tapping on the space in the text input field 612-2 below the handwriting input 616-5, in response to a request to add a new line of text. In some embodiments, upon receiving a tap input (e.g., a long press input), the text input field 612-2 expands further to create space for the new line of text, as shown in Figure 6II. In Figures 6JJ to 6KK, the handwriting input 616-6 is received within the space for the new line of text. In Figure 6LL, after the user has completed the handwriting input and lifted the stylus 203 off the touchscreen 504, the device 500 optionally converts the handwriting input to font-based text. In some embodiments, after the handwriting input has been converted to font-based text, the text input field 612-2 returns to its original size and shape, as shown in Figure 6MM. In some embodiments, if the text in the text input field 612-2 overflows the size of the text input field 612-2, a scroll bar or navigation element (not shown) is provided to allow the user to view the overflowed text.

[0191] Figures 6NN to 6RR illustrate exemplary criteria for converting handwritten input to font-based text. In Figure 6NN, device 500 displays a user interface 620 corresponding to a notepad application. In some embodiments, the user interface 620 includes a text input area 622 where the user can enter multiple lines of text. In Figure 6OO, handwritten input 624-1 is received within the text input area 622. In some embodiments, the handwritten input 624-1 includes punctuation after one or more characters or words (e.g., a comma in Figure 6OO). In some embodiments, upon detection of punctuation, the handwritten input before and including the punctuation is analyzed and converted to font-based text, as shown in Figure 6PP. In some embodiments, the conversion is performed after a short time delay (e.g., according to method 1300).

[0192] In Figure 6PP, a further handwritten input 624-2 is received within the text input area 622. In some embodiments, as shown in Figure 6QQ, the handwritten input 624-2 is converted after a certain time delay following the user's completion of writing the handwritten input 624-2. In some embodiments, the device 500 recognizes the handwritten input 624-2 as a word that the user has completed writing, and at that time, the handwritten input 624-2 is converted. In some embodiments, the handwritten input 624-2 is converted after the device 500 detects that the user has started writing on a different line than the handwritten input 624-2 (e.g., handwritten input 624-3). In Figure 6QQ, the handwritten input 624-3 is received within the text input area 622. In some embodiments, the handwritten input 624-3 includes a word to which no additional characters can be added (e.g., "o'clock"). In some embodiments, when device 500 detects that it is not possible to add additional characters to a recently written word, the handwritten input up to the word to which additional characters cannot be added is analyzed and converted to font-based text, as shown in Figure 6RR. In some embodiments, a word to which additional characters cannot be added is a word to which, based on the device's default dictionary, no further characters can be added to generate a valid word. In other words, any additional characters to that word will generate a non-existent word (for example, no combination of additional characters will generate a valid word). In some embodiments, or because the user has written a threshold number of words (e.g., 3 words, 5 words, 6 words, etc.), the handwritten input 624-3 is converted to font-based text.

[0193] Figures 6SS to 6YY illustrate an exemplary method of transmitting font-based text from a first electronic device to a second electronic device. In Figure 6SS, device 500 is communicating with device 631. In some embodiments, device 631 is a set-top box or other electronic device (e.g., device 580) that communicates with a display 632. In some embodiments, device 500 communicates with device 631 wirelessly via a wireless communication protocol (e.g., WiFi, WiFi Direct, NFC, IR, RF, etc.). In some embodiments, device 631 communicates with other electronic devices that can remotely control device 631, such as device 590 and / or device 591. In some embodiments, as shown in Figure 6SS, device 631 displays a user interface 634 that includes a text input field 636. Thus, in some embodiments, device 631 anticipates user input, where text is entered into the text input field 636. In some embodiments, device 500 displays a user interface 630 that corresponds to a remote control application that remotely controls device 631. In some embodiments, the user interface 630 includes a text input area that can accept handwritten input. For example, in Figure 6TT, handwritten input 638 is detected within the text input area of ​​the user interface 630. In some embodiments, after receiving (or while receiving) the handwritten input 638, the handwritten input 638 is converted to font-based text, as shown in Figure 6UU. In some embodiments, in response to (or simultaneously with) the conversion of the handwritten input 638 to font-based text, the text is sent to the device 631 and optionally entered and displayed in the text input field 636.

[0194] Figures 6VV to 6YY illustrate alternative exemplary methods for transmitting font-based text from a first electronic device to a second electronic device. In some embodiments, as shown in Figure 6VV, device 631 displays one or more text input fields (e.g., text input fields 644-1 to 644-4) on a user interface 642. In some embodiments, device 631 transmits data relating to one or more text input fields to device 500 (or device 500 otherwise receives data relating to one or more text input fields). In some embodiments, device 500 displays one or more text input fields on a user interface 640. In some embodiments, one or more text input fields mimic the position and arrangement of corresponding text input fields on display 632. In some embodiments, device 500 does not mimic the position and arrangement of text input fields.

[0195] In Figure 6WW, the handwritten input 648 is received in a text input field 646-1 on the user interface 640 of device 500. In some embodiments, after the user completes the handwritten input 648 and lifts off the stylus 203 (for example, as shown in Figure 6XX), device 500 converts the handwritten input 648 into font-based text, as shown in Figure 6YY. In some embodiments, after or simultaneously with the conversion of the handwritten input 648 to font-based text, device 500 sends the text to device 631. In some embodiments, upon receiving the text, device 631 inputs and displays the received text in a text input field 644-1 (for example, corresponding to text input field 646-1).

[0196] Figures 7A–7I are flowcharts illustrating a method 700 for converting handwritten input to font-based text. Method 700 is optionally performed on electronic devices such as devices 100, 300, 500, 501, 510, and 591, as described above with reference to Figures 1A–1B, 2–3, 4A–4B, and 5A–5I. Some operations in Method 700 are optionally combined, and / or the order of some operations is optionally changed.

[0197] As described below, Method 700 provides a method for converting handwritten input into font-based text. This method reduces the cognitive burden on the user when interacting with the user interface of the device of this disclosure, thereby creating a more efficient human-machine interface. With respect to battery-operated electronic devices, power is saved and the interval between battery charging is increased by improving the efficiency of user interaction with the user interface.

[0198] In some embodiments, an electronic device communicating with a touch-sensitive display (e.g., an electronic device such as device 100, device 300, device 500, device 501, or device 591, a mobile device including a touchscreen (e.g., a tablet, smartphone, media player, or wearable device), or a computer including a touchscreen) displays a user interface including a first text input area on the touch-sensitive display, as shown in Figure 6A (702). For example, the user interface may have multiple text fields (or text input areas), and the selection of a particular text field (e.g., by touch) optionally displays a soft keyboard for entering text into the text field. In some embodiments, a physical keyboard may be optionally used to enter text into each text field.

[0199] In some embodiments, while displaying a user interface, the electronic device receives user input via a touch-sensitive display, including handwritten input directed towards a first text input area as shown in Figure 6B (704) (e.g., receiving handwritten input on or near a text field (or text input area)). In some embodiments, the user input is received from a stylus or other writing device. In some embodiments, the user input is received from a finger. In some embodiments, the handwritten input is directed towards the first text input field when the handwritten input is received on or near a text field (or text input area). In some embodiments, handwritten input indicating a request to enter text into a text input field (or text input area) is considered directed towards the first text input field. For example, handwritten input starting from a text field (or text input area) optionally indicates that the entire sequence of handwritten input is intended to be entered within the text field (or text input area), even if part (e.g., part or all) of the handwritten input extends outside the text field (or text input area). In some embodiments, user input that starts outside the text field (or text input area) but results in a substantial amount of handwritten input within the text field (or text input area) is optionally considered to be an intention to input text within the text field (or text input area) (e.g., 30%, 50%, etc., within the text field or text input area). In some embodiments, the text input field (or text input area) includes a predetermined margin of error within which handwritten input within a certain distance from the text input field (or text input area) is considered to be handwritten input within the text input field (or text input area).In some embodiments, user input that lies entirely outside the text field (or text input area) is considered to be an intention to enter text into the text field (or text input area) if the timing of the input indicates that the input is a continuation of handwritten input that should be entered into the text field (for example, the user continues writing without a pause or with a short pause, and the writing extends beyond the text field).

[0200] In some embodiments, while receiving user input, the electronic device displays a representation of the handwritten input within the user interface at a location corresponding to a text input area, as shown in Figure 6B (706) (for example, displaying the handwritten input trajectory on the display at the location where the handwritten input was received when the input is received). In some embodiments, when a user “draws” on a touch-sensitive display, the display indicates the user’s handwritten input at the location where the input was received. In some embodiments, if the handwritten input is received within a text field, the handwritten input trajectory is shown within the text field. More generally, in some embodiments, the handwritten input trajectory is shown regardless of where the handwritten input is received on the touch-sensitive display. In some embodiments, the display of the handwritten input occurs after the reception of each character, each word, or each sentence. In some embodiments, user input that is not determined to be handwritten input (e.g., input not directed at a text input field or area) using an input device (e.g., stylus, finger, etc.) does not trigger a simultaneous display of the input trajectory.

[0201] In some embodiments, as shown in Figure 6E (for example, after handwriting input has finished, or after handwriting input has started, while the user is still inputting further handwriting input), after displaying a representation of the handwriting input in the user interface (708), the electronic device, as shown in Figure 6E, stops displaying at least a portion of the representation of the handwriting input and displays font-based text corresponding to at least a portion of the representation of the handwriting input in a text input area (710) (for example, removing at least a portion of the handwriting input on the display and displaying computerized text (e.g., font-based text) corresponding to the removed portion of the handwriting input in the text input field).

[0202] In some embodiments, the substitution occurs while input is being received (for example, the first part of a handwritten input is substituted while the user is still writing the second part of the handwritten input). In some embodiments, the substitution occurs after the input is finished (for example, after a threshold amount of time without receiving handwritten input, after the user has finished writing a word or sentence, or after some other input termination criterion has been met). In some embodiments, the substitution occurs after the suggested text is displayed to the user and input is received to select or confirm the suggested text.

[0203] In some embodiments, the system detects characters and / or words written by the user via handwriting input and converts them into computerized text. For example, the handwriting input is optionally replaced with text in a 12-point Times New Roman font (e.g., or another preferred font). In some embodiments, the font-based text is 10-point, 12-point, etc., and optionally in fonts such as Arial, Calibri, or Times New Roman. In some embodiments, the computerized text (e.g., font-based text) replaces the handwriting input. In some embodiments, the font-based text is displayed before or after a portion of the handwriting input is removed from display (e.g., 0.5 seconds before or after, 1 second before or after, 3 seconds before or after, etc.). In some embodiments, an animation is shown that converts the handwriting input to computerized text or removes the handwriting input in another way and displays the computerized text. In some embodiments, the position of the computerized text overlaps with the position where the handwriting input was before conversion. In some embodiments, the computerized text is smaller in size than the handwriting input (e.g., the font size is smaller than the handwriting input). In some embodiments, the handwritten input is converted to font-based text having the same size as the handwritten input (e.g., the font-based text size matches the handwritten input), and then the font-based text is updated to its final size (e.g., the default size of the font-based text or the default size of the text input area). In some embodiments, the size of the handwritten input is changed to the final size of the font-based text (e.g., the default size of the font-based text or the default size of the text input area), and then the handwritten input is converted to font-based text (e.g., at its final size, matching the final size of the handwritten input). In some embodiments, the size of the handwritten input is not changed, and the font-based text already appears at its final size without changing from its initial size to its final size, without matching the size of the handwritten input.Similarly, in some embodiments, the text position is optionally updated before or after conversion. In some embodiments, the handwritten input is moved to its final position before conversion, and the font-based text appears at the position of the handwritten input before it moved to its final position (e.g., when it is converted), or the font-based text appears at its final position (e.g., when it is converted) without an animation moving the font-based text from its initial position to its final position. In some embodiments, the animation includes any combination of changing size and / or position of the handwritten input or font-based text (e.g., and in any order), resulting in the handwritten input going from its initial position and size to its final position and size. In some embodiments, regardless of the size of the user's writing, the representation of the handwritten text is displayed at the final size of the font-based text (e.g., the default size of the font-based text or the default size of the text input area). In some embodiments, as a result of the conversion operation, the font-based text is provided to the text input or text input area as text input. In some embodiments, the animation of the conversion from handwritten text to font-based text is similar to, or shares similar features with, the conversion from handwritten input to font-based text described later with respect to Method 2000. In some embodiments, when handwritten input is converted to font-based text, an animation is displayed in which the handwritten input dissolves into inflected words and moves to the position where the font-based position appears, similar to the animations described later with respect to Method 2000 (for example, described later with respect to and / or Figures 19I-19N and / or Figures 19O-19V).

[0204] In some embodiments, after displaying a representation of handwritten input on the user interface (708) (for example, after handwritten input has finished or started, while the user is still entering further handwritten input), as shown in Figure 6C, the electronic device maintains the display of the handwritten input representation without displaying font-based text within the text input area (712) (for example, if the criteria for converting text are not met, the handwritten input is not converted to font-based text). In some embodiments, after the criteria are met (for example, if the criteria are timing-related, or if further input is required to meet the criteria for converting text), the handwritten input is converted at a later time. In some embodiments, the handwritten input is unrecognizable and is not converted to computer text. In some embodiments, unrecognized handwritten input is ignored or interpreted as a command. In some embodiments, the trajectory of the handwritten input remains on the display and is not removed. For example, the handwritten input is interpreted as a drawing instead of handwritten input, and therefore the drawing remains displayed within the text input area.

[0205] By converting handwritten input to text (for example, by receiving the input in or near a text input field and replacing the handwritten input with text if certain criteria are met), electronic devices can provide users with the ability to directly write and input text on the user interface (for example, by accepting handwritten input, automatically determining the text corresponding to the handwritten input, and entering the text into the respective text input fields). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by allowing the user to handwrite text directly on the touchscreen display without requiring the user to select each text field and then use a keyboard (e.g., a physical or virtual keyboard) to enter text into the text field). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently while reducing errors in device use.

[0206] In some embodiments, displaying font-based text corresponding to at least a portion of the handwritten input representation in the text input area occurs while the reception of handwritten input continues (e.g., displaying font-based text while still receiving handwritten input) (714), as shown in Figure 6B. In some embodiments, the handwritten input is converted "live" as the input is received. In some embodiments, the conversion occurs after each word (or optionally, after every two, three, four words, etc.). In some embodiments, the conversion occurs after a certain time delay. In some embodiments, the conversion occurs after some trigger event. In some embodiments, if the conversion is "live", the handwritten input is converted to font-based text while the user is still writing further words or characters.

[0207] The aforementioned method of converting handwritten input to text (for example, by displaying font-based text while continuing to receive handwritten input) allows electronic devices to provide users with the ability to receive immediate feedback on the text they are writing (for example, by accepting handwritten input and converting it to text while the user continues to provide handwritten input). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by allowing the user to verify that the conversion is correct without having to wait for all input to be converted at once or until a separate input is made to trigger the conversion). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently while reducing errors in device use.

[0208] In some embodiments, displaying font-based text corresponding to at least a portion of the handwritten input representation in the text input area is done in response to detecting a pause in the handwritten input that is longer than a time threshold (e.g., 0.5, 1, 2, 3, 5 seconds), as shown in Figure 6H (716) (for example, performing a conversion from handwritten input to font-based text after the user pauses handwritten input for a certain time threshold). For example, if the user writes a particular phrase and pauses writing for a time of a threshold amount, the system converts the phrase to font-based text. In some embodiments, after the pause provides a balance between improved text recognition and reduced delay in the conversion of handwritten text, text recognition is improved by converting the handwritten text taking into account strings of words.

[0209] The above method of converting handwritten input to text (for example, by displaying font-based text after pausing in handwriting input) enables electronic devices to convert handwritten text without unnecessarily confusing the user (for example, by converting handwritten text after the user pauses handwriting input). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (by improving the balance between conversion accuracy and providing the user with feedback on their handwriting input, while reducing opportunities to confuse the user by allowing the user to complete their current input before performing the conversion). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently while reducing errors in device use.

[0210] In some embodiments, after displaying a representation of handwritten input within the user interface, the electronic device simultaneously displays on a touch-sensitive display, as in Figure 6Q, at least a portion of the representation of the handwritten input (720) and selectable options (724) corresponding to font-based text corresponding to at least a portion of the representation of the handwritten input (718), as in Figure 6Q (e.g., displaying a popup or other type of dialog box with one or more selectable options that, if selected, allow the system to convert a portion of the representation of the handwritten input into font-based text). In some embodiments, the selectable options are suggestions for font-based text after the conversion of the portion of the handwritten input. In some embodiments, the popup is displayed when the confidence in the recognition of the handwritten input falls below a certain threshold. For example, if the system does not guarantee what the user's handwritten input is, the popup may provide the user with one or more choices of what to convert the handwritten input into. In some embodiments, if the user continues to handwrite while the popup is displayed, the suggested text in the popup continues to update based on the continued handwritten input. For example, the handwritten input continues to be interpreted and evaluated, and the suggestions continue to update to reflect any new characters or words added to the handwritten input. In some embodiments, a pop-up is displayed for each word. In some embodiments, a pop-up is displayed for the entire handwritten input. In some embodiments, a pop-up is displayed for a subset of the handwritten words (e.g., 2 words, 3 words, 4 words, etc.).

[0211] In some embodiments, the display of at least a portion of the handwritten input representation is stopped and font-based text corresponding to at least a portion of the handwritten input representation in the text input area is displayed in response to the detection of a selection of a selectable option (726) (e.g., conversion is performed in response to the user selecting a selectable option), as shown in Figure 6S. In some embodiments, if the user does not select a selectable option, no conversion is performed. In some embodiments, conversion is performed at a later time (e.g., when another selectable option is presented to the user or when other conversion criteria are met). In some embodiments, if multiple suggestions of font-based text are presented to the user, the option selected by the user is the option that is displayed.

[0212] By presenting handwriting conversion options to the user (for example, by displaying selectable options for converting handwritten text) as described above, electronic devices can present the user with options on whether to convert handwritten text and what to convert it to (for example, by converting the handwritten text when the user selects a selectable option and confirms the conversion). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by requiring the user to visually verify the conversion and confirm it after the conversion, and then approving and / or confirming the conversion without making any necessary edits if the conversion is incorrect). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently while reducing errors in device use.

[0213] In some embodiments, as shown in Figure 6G, the text input area includes text input fields (728) (for example, font-based text is entered into the text field to which the user's handwriting input is directed). In some embodiments, the determination of which text field the user's handwriting input is directed to is based on the characteristics of the handwriting input. In some embodiments, if the handwriting input is biased within a given text field, the font-based text is entered into the given text field. In some embodiments, if the handwriting input begins in a given text field, the font-based text is entered into the given text field. In some embodiments, if the handwriting input overlaps with two or more text input fields, the font-based text is entered into the text input field to which the majority of the handwriting input overlaps. In some embodiments, if the handwriting input is entirely outside a text input field, but part of the determined sequence of words is entered into a given text input field, the entirely outside handwriting input is entered into the given text field.

[0214] The above method of inputting font-based text (for example, by converting font-based text and inputting it into a text input field) enables electronic devices to input user handwritten input into appropriate text fields (for example, by converting handwritten text and displaying the font-based text in a text input field that accepts font-based text). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by inputting the converted text into the appropriate text field without requiring the user to accurately provide handwritten input into the desired text input field, and without requiring the user to separately move the converted text into the text input field after conversion). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0215] In some embodiments, as shown in Figure 6G, at least a portion of the handwriting input includes handwriting input detected within the boundaries of a text input area and handwriting input detected outside the boundaries of a text input area (730) (for example, handwriting text that partially overlaps a given text input area but extends outside a given text input area is optionally entered within the given text input area). In some embodiments, if the handwriting input begins in a given text field, the font-based text is entered in the given text field. In some embodiments, if the handwriting input ends within a given text field, the font-based text is entered in the given text field. In some embodiments, if the handwriting input overlaps with two or more text input fields, the font-based text is entered in the text input field over which the majority of the handwriting input overlaps.

[0216] By accepting handwriting input in the manner described above (for example, by recognizing handwriting input both inside and outside the text input area as directed towards the text input area), electronic devices can provide users with compatibility with natural handwriting characteristics (for example, by accepting handwriting text that potentially extends outside the text input area and is not entirely within it). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by accepting natural handwriting input that may be large and extend outside a given text input area without requiring the user to write entirely within a given text input area for the handwriting input to be accepted). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently.

[0217] In some embodiments, as shown in Figure 6B, handwritten input detected within the margin of an error area surrounding the text input area is eligible to be converted to font-based text within the text input area, while handwritten input detected outside the margin of the error area is not eligible to be converted to font-based text within the text input area (732) (for example, the area where handwriting is accepted as directed towards a given text input area is a predetermined size that is larger than the text input area (e.g., 10%, 20%, 30% larger)). In some embodiments, if a portion of the user's handwritten input extends beyond a given text input area but remains within the margin of the error area of ​​the text input area, the entire handwritten input is recognized as directed towards the given text input area. In some embodiments, if handwritten input extends beyond the margin of the error area, the handwritten input is not considered directed towards the given text input area. In some embodiments, if handwritten input extends beyond the margin of the error area, the portion of the handwritten input within the margin of the error area is processed and optionally converted, while the portion of the handwritten input outside the margin is not processed and optionally converted (optionally, a portion of the handwritten input is retained on the display).

[0218] By accepting handwriting input in the manner described above (for example, by providing an error area margin around a text input area where the handwriting input is eligible to be converted to font-based text), electronic devices can provide users with compatibility with natural handwriting characteristics (for example, by accepting handwriting that potentially extends outside the text input area and is not entirely within the text input area). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by accepting natural handwriting that may be large and extend outside a given text input area without requiring the user to write entirely within a given text input area for the handwriting input to be accepted). This further reduces power consumption and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently.

[0219] In some embodiments, as shown in Figure 6E, the electronic device receives a second user input via a touch-sensitive display, including a handwritten input directed to a second text input area in the user interface (e.g., receiving a continuation of a handwritten input). In some embodiments, the second user input is an input in a sequence of one or more handwritten inputs. In some embodiments, the second user input follows quickly and consecutively after the first user input. In some embodiments, the second user input is not directed to the first text input area. In some embodiments, the second user input is directed to the second text input area, or even to a non-text input area (e.g., a space on the user interface not associated with a text input area, such as a space between two text fields).

[0220] In some embodiments, after receiving a second user input (736), the electronic device displays font-based text corresponding to the second user input in the text input area, as shown in Figure 6H, according to a determination that the second user input satisfies one or more second criteria, including criteria that are satisfied when the second user input is detected within a user input time threshold (738) (for example, if the second user input is received (e.g., within a previous handwriting input time threshold) and the system determines that it is associated with a sequence of handwriting inputs directed to the text input area, the converted text will be entered in the text input area and not in the second input area, even if the second user input is directed to the second text input area). In some embodiments, the time threshold is 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds, etc.

[0221] In some embodiments, after receiving a second user input (736), and in accordance with the determination that the second user input does not satisfy one or more second criteria, the electronic device displays font-based text corresponding to the second user input in the second text input area, as shown in Figure 6L (740) (for example, if the second user input is received after a delay of a threshold amount, the second user input is not considered to be associated with a sequence of handwritten inputs directed to the text input area). In some embodiments, the second user input is then interpreted as being directed to the second text input area, and the converted text is entered into the second text input area instead of the text input area.

[0222] The above method of converting handwriting input (for example, by inputting subsequent handwriting input into a given text input area, even if the subsequent handwriting input is directed to a different text input area) allows the electronic device to provide the user with compatibility with natural handwriting characteristics (for example, by accepting continuous handwriting text that is entirely outside a given text input area and potentially directed to another text input area, as long as the continuous handwriting text is within a certain time threshold from the previous handwriting text directed to the given text input area). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by accepting natural handwriting input without requiring the user to pause their handwriting input and reposition it to the desired text input area, or to separately move the converted text from the second text input area to the text input area after conversion). This further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0223] In some embodiments, one or more second criteria are met when the majority of the second user input is directed to the text input area rather than the second text input area, as shown in Figure 6G, and are not met when the majority of the second user input is directed to the second text input area rather than the second text input area, as shown in Figure 6K (742) (for example, if the second (e.g., continuous) handwriting input is performed more in the text input area (e.g., with or without error margins) than in the second text input area (e.g., with or without considering the space between the text input area and the second text input area), the second criterion is met and the converted text of the second user input is entered into the text input area rather than the second text input area). In some embodiments, if the majority of the second user input is located in the second input area (e.g., with or without error margins), the second criterion is not met and the converted text is optionally entered into the second user input.

[0224] The above method of converting handwriting input (for example, by inputting subsequent handwriting input into a given text input area when the majority of subsequent handwriting input is directed to that given text input area rather than another text input area) enables the electronic device to provide the user with compatibility with natural handwriting characteristics (for example, by accepting continuous handwriting text extending outside a given text input area when the majority of the continuous handwriting text is within that given text input area). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by allowing natural handwriting input to continue without requiring the user to pause their handwriting input and reposition it to a desired text input area, or to separately move the converted text from a second text input area to a text input area after conversion). This further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0225] In some embodiments, displaying font-based text corresponding to at least a portion of the handwritten input representation within a text input area (744), as shown in Figures 6D to 6E, includes displaying the font-based text with a first value for a visual characteristic (746) (e.g., updating one or more visual characteristics of the handwritten input to indicate that the handwritten input has been detected as text that can be interpreted and converted to font-based text), as shown in Figure 6D, but before committing the font-based text to the text input area; and displaying the font-based text with a second value for a visual characteristic different from the first value (748) (e.g., updating one or more visual characteristics of the font-based text to indicate that the font-based text has now been committed (e.g., entered) to the text input area), as shown in Figure 6E. In some embodiments, updating the handwritten input includes changing the color and / or opacity of the handwritten input. In some embodiments, alternatively or additionally, font-based text is displayed with a specific visual characteristic (e.g., gray) to indicate that it is a provisionally proposed font-based text (e.g., after converting handwritten input) and committed (e.g., formally entered into the text input area) after a specific time delay (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds). In some embodiments, the font-based text is updated to black, or otherwise to the default color and / or size of the text input area.

[0226] By displaying font-based text in the manner described above (for example, by displaying font-based text with a first visual characteristic before committing text to a text input field, and by displaying font-based text with a second visual characteristic after committing text to a text input field), the electronic device can provide the user with feedback on the progress of converting the user's handwritten text (for example, by displaying font-based text with a first visual characteristic before committing font-based text to a text input area, and with a second visual characteristic after committing). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing the user with visual feedback on the progress of converting handwritten input to font-based text). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0227] In some embodiments, displaying font-based text corresponding to at least a portion of the handwritten input representation within a text input area (750), as shown in Figures 6D and 6H, includes displaying the font-based text with a first value for each visual characteristic according to the determination that the detection of the font-based text has a first confidence level (752) (e.g., displaying the font-based text in a specific color or opacity based on the confidence level of the interpretation of the handwritten input), as shown in Figure 6D, and displaying the font-based text with a second value for each visual characteristic different from the first value according to the determination that the detection of the font-based text has a second confidence level different from the first confidence level (754) (e.g., if the system does not have high confidence in converting the handwritten input (e.g., if the handwritten input is crude or otherwise difficult to interpret), displaying the font-based text with a different visual characteristic than when the system has high confidence in interpretation). For example, if the system has high confidence in converting the handwritten input to a given font-based text, the font-based text is displayed in black. For example, if the system has low reliability, font-based text will be displayed in gray or red.

[0228] By providing visual feedback in the manner described above (for example, by displaying font-based text with a first visual characteristic when the reliability in interpretation and conversion is at a first level, and by displaying font-based text with a second visual characteristic when the reliability in interpretation and conversion is at a second level), the electronic device can provide the user with visual feedback on the reliability and / or accuracy of the conversion. This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing the user with a visual cue of the reliability level of the conversion of the user's handwritten user input, and thus providing the user with instructions on whether to confirm that the conversion is accurate). Furthermore, it reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently while reducing errors in using the device.

[0229] In some embodiments, displaying font-based text corresponding to at least a portion of the handwritten input representation within a text input area (756), as shown in Figure 6S, includes displaying the font-based text at a first position within the text input area (758), as shown in Figure 6S, according to a determination that the detection of the font-based text has a first confidence level (e.g., the font-based text is displayed at a different position within the text input area based on the confidence level of the conversion), and displaying the font-based text in the text input area at a second position different from the first position (760), as shown in Figure 6S, according to a determination that the detection of the font-based text has a second confidence level different from the first confidence level (e.g., the font-based text is optionally left at the same position as the original handwritten input), as shown in Figure 6S (e.g., if the confidence level of the conversion is low). For example, if the confidence level of the conversion is high, the font-based text is moved to align to the left within the text input area (e.g., if the text input area is empty), or otherwise aligned with other text within the text input area. In some embodiments, if the confidence level of the conversion is low, the handwritten input is converted and left in the same position to allow the user to verify whether the conversion is accurate before aligning the text with other text in the text input area (for example, or aligning the text to the left if the text input area is empty). In some embodiments, separate user input is required to verify or otherwise accept font-based text with low confidence.

[0230] The above method of displaying font-based text (for example, by displaying the font-based text at a position based on the confidence level of the conversion from handwritten input) enables the electronic device to provide the user with visual feedback on the reliability and / or accuracy of the conversion. This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing the user with a visual cue on the confidence level of the conversion of the user's handwritten input by not moving the font-based text to its final position, and thus providing the user with an indication of whether to confirm that the conversion is accurate). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently while reducing errors in device use.

[0231] In some embodiments, as shown in Figures 6A to 6RR, one or more first criteria are: timing characteristics of handwriting input (e.g., converting text after handwriting input has paused for a predetermined time), context associated with handwriting input (e.g., converting a word written by the user to font-based text if no further characters can be added to that word), punctuation in handwriting input (e.g., converting text written up to the punctuation mark if the user writes a period or other punctuation mark), distance of the stylus from the touch-sensitive display (e.g., converting handwriting input up to the point the user puts the stylus down or moves the stylus away from the device by a threshold distance (e.g., 6 inches, 12 inches, 2 feet, etc.)), input directed to a second text input area in the user interface (e.g., converting handwritten text entered in the first text input area if the user starts typing text in another text input area), and the user interface The system includes one or more criteria that are satisfied based on scrolling input (e.g., converting previously entered handwriting input when the user interacts with the user interface and scrolls or otherwise navigates around the user interface), stylus angle (e.g., converting previously entered handwriting input when the user points the stylus away from the device), distance of handwriting input from the edge of the text input area (e.g., converting text faster when the user reaches the end of the text input area to make space for the user to make more handwriting input), gestures detected on the stylus (e.g., converting previously entered handwriting input by detecting user input tapped on the stylus), or input from a finger detected on a touch-sensitive display (e.g., converting handwritten text entered by the stylus before user input from the finger if user input is received from a finger instead of the stylus) (762).

[0232] The above method of converting handwritten input (for example, by converting handwritten text based on several different factors) allows electronic devices to select the most appropriate time to convert handwritten text based on context (for example, by converting text based on input, context, punctuation, stylus distance and angle, input interacting with other elements, etc.). This simplifies the interaction between the user and the electronic device (for example, by converting text at a time that is as uninterruptible to the user as possible while balancing the speed of text conversion), enhances the usability of the electronic device, and makes the user-device interface more efficient. Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0233] In some embodiments, while receiving user input, according to a determination that one or more second criteria are met, the electronic device moves at least a portion of the representation of the handwritten input in the user interface to make space for receiving additional handwritten input in the user interface, as shown in Figure 6N (764) (for example, while receiving handwritten user input, the handwritten user input is moved to provide room for the user to continue providing further handwritten input in the text input area). For example, when handwritten user input is received, the previously provided handwritten input is scrolled to the left. In some embodiments, as a result of the scrolling, the user can continue writing at the same position or the user's writing is simply shifted slightly to the right. In some embodiments, the text scrolled to the left scrolls beyond the boundary of the text input area, in which case the text is displayed above the text input area (for example, scrolls beyond the text input area and is not hidden from view) or behind the text input area (for example, scrolls beyond the text input area, but any text that crosses the boundary of the text input area is displayed so as to be hidden by the boundary of the text input area).

[0234] The aforementioned method of receiving handwritten input (for example, by moving previous handwritten inputs when handwritten input is received to provide room for more handwritten input) allows the electronic device to provide space for handwritten input to the user (for example, by spatially moving previously entered handwritten inputs to provide room for receiving further handwritten input). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by allowing the user to continue providing handwritten input without requiring the user's input position to be reset to ensure that the user's input remains within the text input area). Furthermore, this reduces power consumption and improves the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0235] In some embodiments, while receiving user input, according to a determination that one or more third criteria are met, the electronic device expands the boundaries of the text input area to create space within the text input area to receive additional handwriting input, as shown in Figure 6J (766) (for example, when the user reaches the boundary of the text input area, the text input area is expanded horizontally and / or vertically to provide space for the user to continue entering handwriting input). In some embodiments, the text input area expands into the area of ​​another text input area, in which case the text input area covers the other text input area or is displayed on top of it. In some embodiments, after the user has completed the handwriting input and / or after the handwriting input has been converted to font-based text, the text input area shrinks back to its original size.

[0236] The above method of receiving handwriting input (for example, by expanding the size of the text input area) allows the electronic device to provide space for handwriting input to the user (for example, by expanding the text input area horizontally and / or vertically to provide room for receiving further handwriting input when the user begins to reach the boundaries of the text input area). This simplifies the interaction between the user and the electronic device (for example, by allowing the user to continue providing handwriting input within the text input area), enhances the usability of the electronic device, and makes the user-device interface more efficient. This further reduces power consumption and improves the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0237] In some embodiments, expanding the boundary of the text input area (768), as shown in Figure 6J, includes expanding the first boundary of the text input area (770), as shown in Figure 6J, according to the determination that the text input area is in a first position within the user interface (for example, expanding the text input area vertically upward if the text input area is in a specific default position on the touchscreen, such as the bottom third of the touchscreen). In some embodiments, expanding the text input area vertically upward allows the user to provide handwriting input in a more comfortable or natural handwriting position. For example, writing in the bottom third of the touchscreen is potentially awkward or uncomfortable, and by expanding the text input area vertically upward, the user can avoid awkward or uncomfortable handwriting positions.

[0238] In some embodiments, expanding the boundary of the text input area (768), as shown in Figure 6K, includes expanding the second boundary of the text input area (772) without expanding the first boundary of the text input area, as shown in Figure 6K, according to the determination that the text input area is in a second position different from a first position in the user interface (for example, not expanding the text input area vertically upward if the text input area is not in a default position on the touchscreen, such as the bottom third of the touchscreen). In some embodiments, the text input area expands vertically downward and / or horizontally to the right to provide a natural expansion of space for handwriting (for example, since the natural progression of handwriting is from left to right and from top to bottom, the natural expansion of the text input area is horizontally to the right and vertically downward, as opposed to expanding vertically upward when the text input area is in the bottom third of the touchscreen).

[0239] By receiving handwriting input in the manner described above (for example, by expanding the boundaries of the text input area based on its position on the screen), the electronic device can provide the user with space to perform handwriting input (for example, by moving the respective boundaries of the text input area based on its position to provide the most natural place to perform handwriting input). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing the user with space to perform handwriting input comfortably and naturally without requiring the user to write in awkward positions). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0240] In some embodiments, displaying a representation of handwritten input within the user interface while receiving user input includes displaying an animation of one or more visual properties of the representation of handwritten input that change as a function of elapsed time after the corresponding handwritten input has been received (774) (e.g., displaying an animation of the handwritten input when the handwritten input is received). For example, the handwritten input is displayed as if it were ink writing, and the animation appears as if the ink writing is drying over time. In some embodiments, the color and / or opacity of the handwritten input changes to reach a final color and / or opacity level. In some embodiments, the animation of a visual property (e.g., ink drying) is similar to, or shares similar features with, the conversion of the handwritten input to font-based text (e.g., the handwritten input changing to gray) as described later with respect to Method 2000.

[0241] By displaying handwriting input (for example, by changing the visual characteristics of the handwriting input over time) as described above, the electronic device can provide the user with a visual cue of how long it has been since the handwriting input was received and how long the handwriting input has been processed (for example, by displaying an animation of the handwriting input that changes its visual characteristics based on how long it has been since the handwriting input was received). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing the user with a visual indicator of the elapsed time since the handwriting input was received). Furthermore, this further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0242] In some embodiments, stopping the display of at least a portion of the handwritten input representation and displaying font-based text corresponding to at least a portion of the handwritten input representation in the text input area includes displaying an animation in which the handwritten input representation morphs into font-based text, as shown in Figure 6D (776) (e.g., animating the conversion of handwritten input to font-based text). In some embodiments, the handwritten input changes shape and size to produce font-based text. In some embodiments, the animation includes changing the size, shape, color, and / or opacity of the handwritten input. In some embodiments, the handwritten input appears to be broken down and reassembled into font-based text (e.g., broken down and reassembled into large fragments, small fragments, inflected words, sprays, or any combination thereof, as described later with respect to Method 2000). In some embodiments, the handwritten input fades out and the font-based text fades in. In some embodiments, during animation, font-based text is displayed on the display simultaneously with the handwritten input (for example, the font-based text is displayed on the display as the handwritten input is removed from the display so that both the font-based text and the handwritten input are displayed on the display at the same time at some point). In some embodiments, the animation in which the handwritten input morphs into font-based text is similar to, or shares similar features with, the conversion from handwritten input to font-based text described later with respect to Method 2000 (for example, the handwritten input dissolves into an invariable and moves toward the position where the font-based text appears).

[0243] The above-described method of displaying handwritten input (for example, by displaying an animation in which the handwritten input morphs into font-based text) allows the electronic device to provide the user with a visual cue that the handwritten input is being converted into font-based text (for example, by displaying an animation in which the handwritten input morphs into font-based text). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing the user with a visual indication that it is the user's handwritten input that is being processed, interpreted, and converted into font-based text). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0244] In some embodiments, as shown in Figure 6H, at least a portion of the handwritten input corresponds to font-based text containing errors, and displaying font-based text corresponding to at least a portion of the handwritten input representation in the text input area includes displaying font-based text with the errors corrected (778) (for example, in some embodiments, if the handwritten input contains errors that allow the system to determine that the input is correct, the process of converting the handwritten text to font-based text also automatically corrects the errors). In some embodiments, automatic correction of the conversion occurs when the confidence that what is correct input exceeds a certain threshold confidence level (e.g., a high confidence level).

[0245] The above method of converting handwritten input (for example, by removing errors when converting handwritten input to font-based text) enables electronic devices to automatically provide the user with error-free font-based text (for example, by automatically removing errors when converting handwritten input to font-based text). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by automatically removing errors for the user without requiring the user to separately determine whether errors exist and perform additional input to edit the font-based text and remove the errors). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0246] In some embodiments, after displaying a handwritten input representation within the user interface (780), and in accordance with the determination that the user input satisfies one or more first criteria (782), the electronic device transmits font-based text corresponding to at least a portion of the handwritten input representation to a second electronic device separate from itself (784), as shown in Figure 6UU (for example, if the device controls the second electronic device (e.g., wirelessly or wired) and the second electronic device requests text input, the handwritten input is converted to font-based text and then the text is transferred to the second electronic device to satisfy the text input request). For example, if the second electronic device is a set-top box and the user requests a search user interface on the second electronic device, the user can use the electronic device to remotely transmit text into a search field on the search user interface of the second electronic device.

[0247] By transmitting text to a second electronic device (for example, by receiving handwritten input on an electronic device, converting it to font-based text, and sending the font-based text to a second electronic device) as described above, the electronic device can provide the user with a handwritten input method for entering text on the second electronic device (for example, by receiving handwritten input from the user, converting the handwritten input to font-based text, and sending the text to the second electronic device). This simplifies the interaction between the user and the electronic device (by accepting the user's handwritten input and sending the font-based text to the second electronic device without requiring the user to use a virtual keyboard or conventional remote control to enter text on the second electronic device), enhances the usability of the electronic device, and makes the user-device interface more efficient. Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0248] In some embodiments, as shown in Figure 6SS, the second electronic device displays a user interface that includes one or more text input areas, each corresponding to a text input area displayed by the electronic device (786) (for example, the second electronic device displays one or more text input areas).

[0249] In some embodiments, as shown in Figure 6VV, the electronic device detects on the electronic device one or more text input areas that are displayed by the second electronic device (788). In some embodiments, in response to detecting one or more text input areas that are displayed by the second electronic device, as shown in Figure 6VV, the electronic device displays one or more text input areas within the user interface that correspond to one or more text input areas, including the text input areas (790) (for example, extracting text input areas from the user interface of the second electronic device and displaying them on the electronic device). In some embodiments, the electronic device mirrors the user interface of the second electronic device, including any labels, text, graphics, etc., so that the electronic device displays the same user interface as the second electronic device. In some embodiments, the electronic device does not mirror the user interface of the second electronic device, but rather displays only a portion of the elements of the user interface of the second electronic device (for example, displaying text fields and text field labels from the user interface of the second electronic device, and not displaying other elements of the user interface of the second electronic device).

[0250] In some embodiments, transmitting font-based text corresponding to at least a portion of the handwritten input representation to a second electronic device includes transmitting font-based text to each text input area on the second electronic device, as shown in Figure 6YY (792) (for example, the electronic device receives handwritten input directed to each text input area, the handwritten input is converted to font-based text, and the font-based text is transmitted to the second electronic device and entered into the corresponding text input area on the user interface of the second electronic device).

[0251] By transmitting text to a second electronic device (for example, by displaying on the electronic device the same text input area that is displayed on the second electronic device), the electronic device can provide the user with an intuitive interface for transmitting text to the second electronic device (for example, by mirroring the user interface of the second electronic device to the electronic device and transmitting text from the electronic device to the appropriate text input area on the second electronic device). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing on the electronic device the same user interface that is displayed on the first electronic device, so that the user can easily and intuitively select which text input area to input text into without requiring the user to perform additional input or use conventional remote control to select which text input area to input text into). This further reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0252] In some embodiments, as shown in Figure AA, the text input area is a multi-line text input area, and font-based text corresponding to at least a portion of the handwritten input representation is displayed on the first line of the multi-line text input area (794) (for example, the text input area supports multi-line text).

[0253] In some embodiments, as shown in Figure 6DD, while displaying font-based text corresponding to at least a portion of the representation of handwritten input in a first line of a multi-line text input area, the electronic device receives a second user input via a touch-sensitive display, including handwritten input directed towards the first text input area (796) (for example, after detecting handwritten input directed towards the text input area, receiving a second input directed towards the text input area). In some embodiments, the second input corresponds to a request to insert a second line below the previous handwritten input. In some embodiments, the request to insert a second line includes tapping below the previous handwritten input. In some embodiments, the request includes receiving further handwritten input below the previous handwritten input. In some embodiments, the request includes creating a second line by selecting selectable options. In some embodiments, creating a second line includes vertically expanding the size of the text input area.

[0254] In some embodiments, after receiving a second user input (798), and in accordance with the determination that one or more second criteria are met, the electronic device displays font-based text corresponding to the second user input in a second line different from the first line of a multi-line text input area (798-2), as shown in Figure 6LL (e.g., converting the handwritten input of the second user input and entering the converted text into the second line of the text input area (e.g., the line below the previous line of handwritten text)). In some embodiments, one or more second criteria are met if the second user input includes a tap in the space below the previous line of handwritten text, includes the selection of a selectable option to create a new line, and / or includes handwritten input at a threshold distance below the previous line of handwritten text (e.g., 6 points, 12 points, 18 points, 24 points, etc.).

[0255] In some embodiments, after receiving a second user input (798), according to a determination that one or more second criteria are met, or according to a determination that one or more second criteria are not met, the electronic device displays font-based text corresponding to the second user input in the first line of the multi-line text input area (798-4), as shown in Figure 6EE (for example, if the second user input does not reflect input that would input text into the second line, the font-based text is entered into the same line as the previous line of the previous handwritten text). For example, if the user continues handwriting input just below but not too far below the previous line so that the second user input becomes input into the previous line (for example, if it appears that the user intended to continue writing on the previous line), the converted text continues to be entered into the previous line.

[0256] The above method of inputting handwritten text (for example, by entering text into a second line of a text input area that supports multiline text when user input indicates a request to enter text into a second line) allows electronic devices to provide users with an intuitive way to input multiline text (for example, by entering text into a second line of a text input area when certain criteria for handwritten input are met). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by determining whether a new line should be created and entering text into the new line, without requiring the user to perform additional user input to manually edit font-based text to insert line breaks at desired positions or to wait for handwritten text to be converted). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently.

[0257] In some embodiments, as shown in Figures 6EE to 6FF, one or more second criteria are met when the second user input is detected beyond a threshold distance (e.g., 6 points, 12 points, 18 points, 20 points, 24 points, etc.) below the user input, and one or more second criteria are not met when the second user input is detected below a threshold distance (798-6) (for example, if the second user input exceeds the threshold distance below the previous handwritten text, the second user input indicates a request to insert text into the second line (e.g., below the previous line of handwritten text)). In some embodiments, if the second user input does not exceed the threshold distance below the previous handwritten text, the second user input indicates a request to continue inserting text into the previous line of text.

[0258] The above method of inputting multi-line handwritten text (for example, by entering text in the second line of a text input area when user input exceeding a threshold distance below the previous text line is received, indicating a request to enter text in the second line) makes it possible for electronic devices to provide users with an intuitive way to input multi-line text (for example, by interpreting the input as a request to accept handwritten text below the previous text line and enter handwritten text in the line below the previous text line). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by entering text in a new line when handwritten text is received at a threshold distance below the previous text line, without requiring the user to perform additional user input to manually edit font-based text and insert line breaks at the desired positions or wait for the handwritten text to be converted). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently.

[0259] In some embodiments, one or more second criteria are satisfied if the second user input includes a stylus input detected on the second line in a multi-line text input area, and one or more second criteria are not satisfied if the second user input does not include a stylus input detected on the second line in a multi-line text input area, as shown in Figure 6FF (798-8) (for example, if the second user input includes a tap, long press, or input exceeding a certain force threshold at a position below the previous text line, the second user input is interpreted as including a request to insert a second line of text below the previous text line).

[0260] The above method of inputting multi-line handwritten text (for example, by entering text in the second line and receiving a tap on the second line indicating a request to insert text into the second line of the text input area) allows electronic devices to provide users with an intuitive way to input multi-line text (for example, by accepting gesture input below the previous text line and interpreting the input as a request to enter handwritten text on the line below the previous text line). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by entering text on a new line when a tap is received below the previous text line, without requiring the user to perform additional user input to manually edit font-based text and insert line breaks at the desired positions or wait for handwritten text to be converted). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently.

[0261] In some embodiments, as shown in Figure 6BB, a selectable option for moving to a second line is displayed simultaneously with font-based text corresponding to at least a portion of the handwritten input representation, and one or more second criteria are met when the selectable option is selected and one or more second criteria are not met when the selectable option is not selected (798-10) (e.g., receiving user input to select a selectable option for inserting a new line of text). In some embodiments, the selectable option is displayed or presented in response to receiving tap input or other instruction requesting to insert a new line of text. In some embodiments, in response to receiving user input to select a selectable option for inserting a new line of text, the font-based text is inserted into a new line of text below the previous line of text.

[0262] The above method of inputting multi-line handwritten text (for example, by receiving a selection of a selectable option to insert a new text line below the previous text line) allows the electronic device to provide the user with an easy way to input multi-line text (for example, by providing a selectable option to insert handwritten text on a line below the previous text line). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing a selectable option to input a new text line in response to receiving a selection of a selectable option, without requiring the user to manually edit the font-based text after the handwritten text has been converted to font-based text to insert line breaks at the desired positions, and then inputting text into the new line). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0263] In some embodiments, as shown in Figure 6B, the electronic device receives a second user input via a touch-sensitive display (798-12). In some embodiments, upon receiving the second user input (798-14), and upon determining that the second user input is detected within the area of ​​the user interface corresponding to the respective text input area, the electronic device performs a handwriting input operation within the respective text input area based on the second user input (798-16), as shown in Figure 6C (for example, if the user input is directed to a text input area, the user input is interpreted as a request to handwrite or otherwise enter text into the text input area). In some embodiments, upon receiving user input directed to a text input area, the input is accepted as handwriting input.

[0264] In some embodiments, upon receiving a second user input (798-14), and in accordance with the determination that the second user input has been detected in an area of ​​the user interface that does not correspond to a text input area, the electronic device performs a scroll operation within the user interface based on the second user input (798-18), as shown in Figure 6Y (for example, if the user input is not directed to a text input area, the user input is not interpreted as a request to insert text). For example, if the user interacts with another user element that is not a text input area, the handwriting conversion process is not performed. In some embodiments, for example, if the user performs a scroll or other type of navigation gesture, navigation is performed according to the user input instead of inserting font-based text based on the handwriting input.

[0265] By interpreting user input in the manner described above (for example, by interpreting the input as handwritten text if it is received in a text input area, and not interpreting it as handwritten text if it is not received in a text input area), the electronic device can provide the user with an easy way to input text (for example, by allowing the user to interact with the device in a non-textual way if the input does not indicate a request to input text, but by accepting handwritten input if the input indicates a request to input text). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by automatically determining whether the user is requesting to input text or otherwise interact with a user interface without requiring the user to perform additional input to switch to text input mode or to interact with a separate user interface or use a separate device to input text). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently.

[0266] In some embodiments, the animation of a handwritten input representation morphing into font-based text (798-20) includes animating the handwritten input representation to morph into font-based text (e.g., directly) at its final position within the text input area and at the final size in which the font-based text will be displayed, according to the determination that the text input area does not yet contain font-based text (798-22) (for example, if the text input area does not have any font-based text (or the line to which the handwritten input is directed does not have any text), the animation simultaneously changes the size and shape of the handwritten text to font-based text and moves to the final position of the font-based text (e.g., aligned to the left within the text input area)). Thus, in some embodiments, the animation is performed in one step. In some embodiments, the animation of a handwritten input morphing into font-based text is similar to, or shares similar features with, the conversion of handwritten input to font-based text described later with respect to Method 2000. In some embodiments, if the text input area has font-based text, the animation changes the shape of the handwritten text to font-based text and then resizes it to match the size of the existing font-based text.

[0267] By converting handwritten input to text using the methods described above (for example, by displaying an animation in which the handwritten input simultaneously changes to the final size of the font-based text and moves to the final position), electronic devices can provide the user with a visual cue that the handwritten input is being converted to font-based text (for example, by displaying an animation in which the handwritten input morphs into font-based text in one step). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by providing the user with a visual indication that it is the user's handwritten input that is being processed, interpreted, and converted to font-based text). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently while reducing errors in device use.

[0268] In some embodiments, the animation of a handwritten input representation morphing into font-based text (798-24) includes, as shown in Figure 6E, animating the handwritten input representation to morph into intermediate-sized font-based text based on the size of the handwritten input representation, according to the determination that the text input area does not yet contain font-based text, and then animating the intermediate-sized font-based text to morph into font-based text at the final position within the text input area and at the final size where the font-based text will be displayed, unlike the intermediate size (798-26) (for example, if the text input area does not have any font-based text (or the line to which the handwritten input is directed does not have any text), the animation first reshapes the handwritten text into font-based text and resizes it to a size between the final size and the original handwritten size (for example, and optionally remains in the same position as the original handwritten input)). In some embodiments, after reshaping into font-based text, the animation continues to resize the text to the final size and move the text to the final position of the font-based text (for example, aligned to the left within the text input area). Therefore, in some embodiments, the animation is performed in two steps. In some embodiments, the animation of morphing handwritten input into font-based text is similar to, or shares similar features with, the conversion of handwritten input to font-based text described later with respect to Method 2000. For example, in some embodiments, a first animation similar to the animation described in Method 2000 is performed to convert the handwritten input into font-based text of the same size as the handwritten input, and after the first animation, a second animation (for example, optionally similar to the animation described in Method 2000) is performed to morph the size of the resulting font-based text to the final size of the font-based text (e.g., from 36 font size to 12 font size, from 24 font size to 12 font size, etc.).

[0269] By converting handwritten input to text using the method described above (for example, by displaying an animation in which the handwritten input is first converted to font-based text with an intermediate size (between the final size and the size of the handwritten input), and then converted from the intermediate size to the final size while moving to the final position), electronic devices can provide users with a visual cue that the handwritten input is being converted to font-based text. This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient (for example, by displaying an animation in which the handwritten input morphs into font-based text in two steps to emphasize that the process does both convert the handwritten input to font-based text and resize and move the font-based text to the appropriate size and position). Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently while reducing errors in using the device.

[0270] In some embodiments, the animation of a handwritten input representation morphing into font-based text (798-28) is animated to morph into an intermediate-sized font-based text based on the size of the handwritten input representation, according to the determination that the text input area does not contain previously entered font-based text (e.g., font-based text displayed in the text input area before the handwritten input is converted to font-based text (e.g., font-based text corresponding to the handwritten input is added to existing font-based text in the text input area)), as shown in Figure 6H, after which the intermediate-sized font-based text is displayed in the text input area. The handwriting input is animated to morph into font-based text at its final position and at its final size, where the font-based text will be displayed, unlike intermediate sizes, and the final size of the font-based text corresponding to the handwriting input is the same as the size of the previously entered font-based text (798-30) (for example, if the text input area has existing font-based text (or the line to which the handwriting input is directed has existing text), the animation first reshapes the handwriting text into font-based text and resizes it to a size between the size of the existing text and the original handwriting size (for example, and optionally remains in the same position as the original handwriting input)). In some embodiments, after reshaping into font-based text, the animation continues to resize the text to its final size (for example, the same size as the existing text) and move the text to the final position of the font-based text (for example, aligning it to the left with the existing text). Thus, in some embodiments, the animation is performed in two steps to match the font format of the existing text. In some embodiments, the animation of the handwriting input morphing into font-based text is similar to, or shares similar features with respect to, the conversion of handwriting input to font-based text described later with respect to Method 2000.For example, in some embodiments, a first animation similar to the animation described in Method 2000 is performed to convert the handwritten input into intermediate-sized font-based text, and after the first animation, a second animation (optionally similar to the animation described in Method 2000) is performed to morph the size of the resulting font-based text from intermediate size to the final size of the font-based text (for example, from an effective font size of 36 for the handwritten input to 24 font size for the font-based text, and then down to 12 font size).

[0271] By converting handwritten input to text using the method described above (for example, by displaying an animation in which the handwritten input is first converted to font-based text having an intermediate size (between the final size and the size of the handwritten input), and then converted from the intermediate size to the same size as any existing text while moving to the final position (for example, to align with existing text)), the electronic device can provide the user with a visual cue that the handwritten input is being converted to font-based text (for example, by displaying an animation in which the handwritten input morphs into font-based text in two steps to emphasize that the process does both convert the handwritten input to font-based text and resize and move the font-based text to the appropriate size and position). This simplifies the interaction between the user and the electronic device, enhances the usability of the electronic device, and makes the user-device interface more efficient. Furthermore, this reduces power consumption and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently while reducing errors in using the device.

[0272] It should be understood that the specific order of operations described in Figures 7A to 7I is merely illustrative and not intended to indicate that the described order is the only possible order in which the operations can be performed. Those skilled in the art will recognize various methods for rearranging the operations described herein. In addition, it should be noted that details of other processes described herein with respect to other methods described herein (e.g., methods 900, 1100, 1300, 1500, 1600, 1800, 2000, and 2200) are also applicable in a similar manner to method 700 described above with respect to Figures 7A to 7I. For example, the operation of an electronic device that converts handwritten input to font-based text, as described above with reference to Method 700, optionally has one or more of the following characteristics, as described herein with reference to other methods described herein (e.g., Methods 900, 1100, 1300, 1500, 1600, 1800, 2000, and 2200): selecting and deleting text, inserting handwritten input into existing text, controlling the timing of converting handwritten text to font-based text, presenting a handwritten input menu, controlling the characteristics of handwritten input, presenting auto-completion suggestions, converting handwritten input to font-based text, and displaying options within a content input palette. For brevity, those details will not be repeated here.

[0273] The operations in the above-described information processing method are optionally performed by executing one or more functional modules within an information processing device such as a general-purpose processor (for example, as described in Figures 1A to 1B, 3, and 5A to 5I) or an application-specific chip. Furthermore, the operations described above are optionally performed by the components shown in Figures 1A to 1B with reference to Figures 7A to 7I. For example, the display operations 702, 706, 710, 712, 714, 716, 718, 738, 740, 744, 746, 748, 750, 752, 754, 756, 758, 760, 774, 776, 778, 790, 798-2, and 798-4, as well as the receiving operations 704, 734, 796, and 798-12, are optionally performed by the event sorter 170, event recognition unit 180, and event processing unit 190. When a corresponding default event or sub-event is detected, the event recognition unit 180 activates the event processing unit 190 associated with the detection of that event or sub-event. The event handler 190 optionally uses or calls the data update unit 176 or the object update unit 177 to update the application's internal state 192. In some embodiments, the event processing unit 190 accesses the corresponding GUI update unit 178 to update what is displayed by the application. Similarly, it will be obvious to those skilled in the art how other processes can be carried out based on the components shown in Figures 1A-1B. Selecting and deleting text using a stylus

[0274] Users interact with electronic devices in many different ways, including by typing text into them. In some embodiments, the electronic device displays text within a text field or text area. The embodiments described below provide a method by which the electronic device selects and / or deletes text using a handwriting input device (e.g., a stylus). By enhancing bidirectional interaction with the device, the amount of time required for the user to perform an action is reduced, and therefore the power consumption of the device is reduced, increasing battery life for battery-powered devices. It is understood that people use devices. When a person uses a device, that person is optionally called a user of that device.

[0275] Figures 8A to 8II illustrate exemplary methods by which an electronic device interprets handwritten input for selecting or deleting text. Embodiments in these figures are used to illustrate the processes described below, including the processes described with reference to Figures 9A to 9G.

[0276] Figure 8A shows an exemplary device 500 including a touchscreen 504. In Figure 8A, the device 500 displays a user interface 800 corresponding to a notepad application. In some embodiments, the user interface 800 includes a text input area 802 on which the user can enter multiple lines of text. In some embodiments, the text input area 802 includes one or more pre-existing texts 804. In some embodiments, the pre-existing texts 804 were previously entered as handwritten input and converted to font-based text. In some embodiments, the pre-existing texts 804 were entered using a soft keyboard (for example, by the user or another user on this device or another device).

[0277] In Figure 8B, user input is received from the stylus 203. In some embodiments, as shown in Figure 8B, the user input is a gesture on the touchscreen 504 that passes through a portion of the existing text 804. In some embodiments, in response to the user input, a trajectory 806 of the handwriting input is displayed on the display. In some embodiments, the trajectory 806 is a visual indication on the display that corresponds to the handwritten user input at the location of the handwriting input. In other words, the trajectory 806 is a representation of the user's handwriting input. In some embodiments, as shown in Figure 8B, the handwriting input passes horizontally through the letter "ck" of the word "clock". In some embodiments, the trajectory 806 provides a visual indication that the user performed a horizontal gesture through the letter "ck" of the word "clock". In Figure 8C, user input continues to be received from the stylus 203 (e.g., without lift-off) as it crosses out the entire word "clock". In some embodiments, a horizontal gesture (e.g., or substantially horizontal gesture) is considered a request to select (e.g., highlight) each portion of the existing text 804.

[0278] In Figure 8D, the handwritten user input is completed (e.g., the stylus 203 lifts off the touchscreen 504). In some embodiments, in response to the lift-off of the stylus 203, the existing text 804 corresponding to the word "clock" is selected. In some embodiments, selecting a word includes highlighting the word (e.g., as indicated by highlight 808), displaying one or two selection adjustment elements 810-1 and 810-2, and / or displaying a pop-up menu 812. In some embodiments, the selection adjustment elements 810-1 and 810-2 are selectable to move the selection to include more or fewer characters or words (e.g., the user can drag the selection adjustment elements 810-1 and 810-2 to include more or fewer characters). In some embodiments, the pop-up menu 812 includes one or more selectable options for performing actions on the highlighted text. In some embodiments, the pop-up menu 812 includes selectable options for cutting selected text (e.g., copying selected text to the clipboard and deleting selected text simultaneously), selectable options for copying text (e.g., copying selected text to the clipboard), selectable options for changing the font of selected text (e.g., changing the font, size, whether it is bold, underlined, italicized, etc.), and / or selectable options for sharing selected text (e.g., with another user and / or another electronic device).

[0279] Figures 8E–8H illustrate alternative exemplary embodiments for selecting text based on handwriting input. In Figure 8E, device 500 displays a user interface 800 corresponding to a notepad application. In some embodiments, the user interface 800 includes a text input area 802 where the user can enter multiple lines of text. In some embodiments, the text input area 802 includes one or more existing texts 804. In some embodiments, the existing texts 804 were previously entered as handwriting input and converted to font-based text. In some embodiments, the existing texts 804 were entered using a soft keyboard (for example, by the user or another user on this device or another device).

[0280] In Figure 8F, user input is received from the stylus 203. In some embodiments, the user input is a gesture on the touchscreen 504 that passes through a portion of the existing text 804, as shown in Figure 8F. In some embodiments, in response to the user input, a trajectory 806 of the handwriting input is displayed on the display. In some embodiments, the trajectory 806 is a visual indication on the display that corresponds to the handwritten user input at the location of the handwriting input. In some embodiments, as shown in Figure 8F, the handwriting input passes through the letter "ck" of the word "clock". In some embodiments, the trajectory 806 provides a visual indication that the user has performed a horizontal gesture through the letter "ck" of the word "clock". In some embodiments, after the handwriting input is recognized as a selection gesture, the previously selected letters are highlighted, and the highlighting is updated "live" (e.g., moving along with the handwriting input). Thus, as shown in Figure 8F, the highlighting 808 is now highlighting the letter "ck".

[0281] In Figure 8G, user input continues to be received from the stylus 203 crossing out the entire word "clock" (e.g., without lift-off). In some embodiments, the highlight 808 is updated to highlight the additional characters selected by the user input when the user selects additional characters (e.g., now highlighting the entire word "clock").

[0282] In some embodiments, as shown in Figures 8F to 8G, handwriting input does not need to be perfectly straight or perfectly horizontal in order to be interpreted as a request to select a letter or word. In some embodiments, substantially straight and / or substantially horizontal handwriting input is interpreted as a request to select a letter or word. In some embodiments, any handwriting input that passes through at least part of a letter or word and is not interpreted as a delete command (as will be discussed in more detail below) is interpreted as a request ...

Claims

1. In an electronic device that communicates with a touch-sensitive display, A text input user interface including the first character sequence, which includes a first part of the first character sequence and a second part of the first character sequence, is displayed on the touch-sensitive display. While the text input user interface is displayed, user input within the text input user interface is received via the touch-sensitive display between the first part of the first character sequence and the second part of the first character sequence. In response to receiving the aforementioned user input, The text input user interface is updated by creating a space between the first part of the first character sequence and the second part of the first character sequence, in accordance with the determination that the user input corresponds to a request to input separate font-based text between the first part of the first character sequence and the second part of the first character sequence using handwriting input, wherein the space between the first part and the second part is configured to receive the handwriting input for inserting the separate font-based text between the first part and the second part of the first character sequence. After updating the text input user interface by creating the space between the first part of the first character sequence and the second part of the first character sequence, the handwritten input is received via the touch-sensitive display within the space between the first part and the second part of the first character sequence. Upon receiving the aforementioned handwritten input, Displaying the handwritten input representation within the space between the first and second parts of the first character sequence, In accordance with the determination that one or more new line criteria are met by the handwriting input, the text input user interface is updated to create the new line configured to receive the additional handwriting input in the new line between the first and second parts of the first character sequence before the handwriting input and additional handwriting input are converted to font-based text, Methods that include...

2. After updating the text input user interface by creating the space between the first part of the first character sequence and the second part of the first character sequence, the handwritten input is received via the touch-sensitive display within the space between the first part and the second part of the first character sequence. After receiving the handwritten input, the handwritten input is converted into font-based text between the first and second parts of the first character sequence. The method according to claim 1, further comprising:

3. The method according to claim 2, wherein the handwriting input is detected after the user input is detected between the first and second parts of the first character sequence without detecting a lift-off from the touch-sensitive display.

4. The method according to claim 1, wherein when the user input includes a touchdown of a stylus on the touch-sensitive display between the first and second parts of the first character sequence, the user input responds to the request to input separate text between the first and second parts of the first character sequence using the handwriting input, and the text input user interface is updated by creating the space between the first and second parts of the first character sequence, in accordance with the detection of the touchdown of the stylus before detecting further input from the stylus.

5. The method according to claim 4, wherein the touchdown of the stylus is between two words of the first character sequence.

6. The method according to claim 1, wherein when the user input includes a touchdown of a stylus on the touch-sensitive display for a period longer than a time threshold, the user input responds to the request to input separate text between the first and second parts of the first character sequence using handwriting input, and the text input user interface is updated by creating the space between the first and second parts of the first character sequence, in response to the detection of the touchdown of the stylus on the touch-sensitive display for a period longer than the time threshold.

7. The method according to claim 1, wherein when the user input includes a separate gesture, the user input responds to the request to input separate text between the first and second parts of the first character sequence using handwriting input, and the text input user interface is updated by creating the space between the first and second parts of the first character sequence, in response to the detection of the separate gesture.

8. The user input includes touching down a stylus on the touch-sensitive display, and the method is In response to detecting the stylus touchdown between the first and second parts of the first character sequence on the touch-sensitive display, selectable options for creating the space between the first and second parts of the first character sequence are displayed on the touch-sensitive display. While displaying the selectable options for creating the space between the first and second parts of the first character sequence, the system receives a selection of the selectable options via the touch-sensitive display. It further includes, The method according to claim 1, wherein updating the text input user interface by creating the space between the first portion of the first character sequence and the second portion of the first character sequence is performed in response to detecting the selection of the selectable option.

9. After updating the text input user interface by creating the space between the first part of the first character sequence and the second part of the first character sequence, the handwritten input is received via the touch-sensitive display within the space between the first part and the second part of the first character sequence. Upon receiving the aforementioned handwritten input, Displaying the handwritten input representation within the space between the first and second parts of the first character sequence, In accordance with the determination that the handwritten input satisfies one or more criteria, the space between the first and second parts of the first character sequence is expanded, The method according to claim 1, further comprising:

10. The method according to claim 9, wherein the handwriting input satisfies one or more criteria when the handwriting input includes a first individual gesture, and does not satisfy one or more criteria when the handwriting input includes a second individual gesture different from the first individual gesture.

11. The method according to claim 1, wherein the one or more new line criteria include criteria that are met when the handwritten input reaches the end of the current line in the text input user interface.

12. The method according to any one of claims 1 to 11, wherein the one or more new line criteria include criteria that are met when the additional handwriting input is detected under existing font-based text in the text input user interface.

13. The method according to any one of claims 1 to 12, wherein the one or more new line criteria include criteria that are met when tap input is detected under existing font-based text in the text input user interface.

14. Upon receiving the aforementioned handwritten input, The method further includes, upon determining that the handwritten input is within a threshold distance of the end of the current line in the text input user interface, displaying a selectable option within the text input user interface for creating a new line. The method according to any one of claims 1 to 13, wherein the one or more new row criteria include criteria that are met when a selection of the selectable option for creating the new row is detected within the text input user interface.

15. While the new line configured to receive the additional handwriting input is included in the text input user interface, individual user input is received via the touch-sensitive display, In response to receiving the individual user inputs, In accordance with the determination that the individual user input includes a tap input detected at the end of the last word of the previous line preceding the new line in the text input user interface, or a tap input detected at the beginning of the first word in the new line in the text input user interface, a selectable option to remove the new line from the text input user interface is displayed within the text input user interface. The method according to any one of claims 1 to 14, further comprising:

16. The new line configured to receive the additional handwriting input is included in the text input user interface, and while the new line includes a separate character sequence, the touch-sensitive display receives separate inputs including a touch-down of the stylus on the separate character sequence and a movement of the stylus to a separate line in the text input user interface that is different from the new line. In response to receiving the aforementioned individual inputs, Moving the aforementioned individual character sequences to the aforementioned individual lines in the text input user interface, To remove the new line from the text input user interface, The method according to any one of claims 1 to 15, further comprising:

17. After updating the text input user interface by creating the space between the first part of the first character sequence and the second part of the first character sequence, the handwritten input is received via the touch-sensitive display within the space between the first part and the second part of the first character sequence. Upon receiving the aforementioned handwritten input, The text input user interface displays the handwritten input representation within the space between the first and second parts of the first character sequence. In accordance with the determination that the handwritten input has not reached the end of the current line in the text input user interface, the display of the representation of the handwritten input is stopped after a first elapsed time from the time the handwritten input is received. In accordance with the determination that the handwritten input has reached the end of the current line in the text input user interface, the display of the representation of the handwritten input is stopped after a second elapsed time shorter than the first elapsed time since the handwritten input was received. The method according to claim 1, further comprising:

18. After updating the text input user interface by creating the space between the first part of the first character sequence and the second part of the first character sequence, the handwritten input is received via the touch-sensitive display within the space between the first part and the second part of the first character sequence. After receiving the aforementioned handwritten input, In accordance with the determination that no additional handwriting input is received during the time threshold after the completion of the aforementioned handwriting input, the size of the space between the first and second parts of the first character sequence is reduced to remove the space not consumed by the handwriting input within the text input user interface. The method according to claim 1, further comprising:

19. After updating the text input user interface by creating the space between the first part of the first character sequence and the second part of the first character sequence, the handwritten input is received via the touch-sensitive display within the space between the first part and the second part of the first character sequence. After receiving the aforementioned handwritten input, In accordance with the determination that no additional handwriting input is received during the time threshold after the completion of the handwriting input, the handwriting input is converted into font-based text in the space between the first and second parts of the first character sequence. The method according to claim 1, further comprising:

20. The text input user interface displays the second character sequence, which includes the first part of the second character sequence and the second part of the second character sequence. While the text input user interface is displayed, a second user input is received within the text input user interface between the first part of the second character sequence and the second part of the second character sequence via the touch-sensitive display. Upon receiving the second user input, According to the determination that the second user input corresponds to a request to input a second separate font-based text between the first part of the second character sequence and the second part of the second character sequence using handwriting input, Displaying a handwriting input user interface element within the text input user interface configured to receive handwriting input for inserting the second separate font-based text between the first and second parts of the second character sequence, The method according to claim 1, further comprising:

21. While the aforementioned handwriting input user interface element is being displayed, a second handwriting input is received within the handwriting input user interface element via the touch-sensitive display. In response to receiving the second handwriting input within the aforementioned handwriting input user interface element, Inserting font-based text corresponding to the second handwriting input into the text input user interface, While the handwriting input user interface element remains stationary on the touch-sensitive display, the text input user interface is scrolled according to the movement of the current text insertion point within the text input user interface. The method according to claim 20, further comprising:

22. While the aforementioned handwriting input user interface element is being displayed, a second handwriting input is received within the handwriting input user interface element via the touch-sensitive display. In response to receiving the second handwriting input within the aforementioned handwriting input user interface element, Displaying the second handwriting input representation within the aforementioned handwriting input user interface element, In accordance with the determination that the second handwriting input has not reached the end of the handwriting input user interface element, the display of the representation of the second handwriting input is stopped after a first elapsed time has elapsed since the reception of the second handwriting input. In accordance with the determination that the second handwriting input has reached the end of the handwriting input user interface element, the display of the representation of the second handwriting input is stopped after a second elapsed time shorter than the first elapsed time since the reception of the second handwriting input. The method according to claim 20, further comprising:

23. While displaying the text input user interface including the first character sequence, the system receives individual user inputs via the touch-sensitive display, including movement across individual parts of the first character sequence, while maintaining contact with the touch-sensitive display at a position between the first character and the second character in the first character sequence. In response to receiving the individual user inputs, The text input user interface is updated by adding a whitespace character between the first character and the second character in the first character sequence, in accordance with the determination that there is no character separating the first character and the second character in the first character sequence. The text input user interface is updated by removing the whitespace character between the first character and the second character in the first character sequence, based on the determination that only whitespace characters separate the first character and the second character in the first character sequence. The method according to claim 1, further comprising:

24. After updating the text input user interface to create the new line that receives the additional handwriting input, a second handwriting input is received within the new line via the touch-sensitive display. Upon receiving the second handwritten input, the touch-sensitive display displays a representation of the second handwritten input within the new line, The method according to any one of claims 1 to 23, further comprising:

25. One or more processors, Memory and One or more programs, An electronic device comprising, wherein one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 1 to 24.

26. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 24.

Images